posted 07-05-2002 10:08 PM           

quote:

---

Originally posted by jstrazzere:
I don't think it's an either-or situation.
In my shops automated testing compliments manual testing.

For example, smoke tests are almost always automated.
Here, I'm trying to quickly find out if the new build is "broken" or not.

It's not an issue of using deep, heuristic testing. It's a matter of efficient use of resources. My automated smoke test typically runs automatically - immediately after our daily build. It often runs at night, unattended, and we get the results in the morning when we arrive.

Now, I could pay a tester to stay overnight and perform the smoke test, but that wouldn't make much sense. Yes, a good human tester could perform a deeper test, but that's not what I'm looking for here.

Similarly complete automated regression test suites can give me confidence that my system continues to work well - even when the operating system is changed underneath, or a major component is rewritten.

SO when deciding the "goodness" of automated testin, it's not a matter of comparing automated tests to heuristic processes involved in manual testing. It's more a matter of knowing when its appropriate to apply which method.

---

I agree with most of what you say.

What I don't agree with is that you seem to be using it to make a case against having guidelines.

Now to take your last sentence.

It's more a matter of knowing when its appropriate to apply which method.

This is very true but it's also a recipe for anarchy. Not everybody will come to the same conclusion about timing and appropriateness.

At work I have been participating in a workshop on Gui Automation Tool rollout - coming up with a methodology for advising clients on tool implementations. It is headed up by a methodologist, and a recurring theme from him is "I am yet to hear anything that makes this any different from the implementation of any other kind of software".

So it is here.

The issues arising are

1. Smoke Testing is an important class of test and we should have guidelines outlining how we would normally (thats my get out for exceptions) approach it. The guidelines should cover things like the degree of coverage required in a smoke test, the level
of robustness expected of a script, documentation requirements, etc etc.

2. Other classes of test also need guidelines. What is an important consideration for one class may not be for another class. There also should be overall guidelines. Now the degree to which these overall guidelines are influenced by the smoke testing considerations will depend upon the relative importance of smoke testing in the projects/team/enterprise testing strategy.

Looking at the flip side, my testing strategy may call for exploratory testing and it may well be deemed that this testing is to be script free. You would not use that to argue against having general guidelines which may stipulate that as a rule you will document test cases for testing you may carry out.

It's a matter of efficient use of resources.

I tell you what would be an efficient use of resources. Taking all them proprietary checks out of your smoke test so that the it could be handed over to the development team. Then they could run it, check it and fix anything that they found wrong before they handed it over to the test team.

posted 07-05-2002 10:19 PM           

quote:

---

Originally posted by jstrazzere:
Again, I don't see any general rule that would favor deferred checking of test results.

---

Several have come up in the thread. Here's a few

a) Not knowing what repository an outcome is written to because of load balancing.
b) Making verification of testing results the province of the few who have access to specialist testing tool skills and licenses. Do you not like the idea of developers taking responsibility for running smoke tests.
c) Your automation script breaks when the repository you are checking changes. Are we so resistant to the benefits of implementation hiding.

quote:

---

Nor, do I find that, in general, deferred checking of GUI-intensive applications would be probitive based on storage requirements.

---

My bad use of terminology.
By graphical apps I am referring to applications mainly dependent on bitmap checks.

posted 07-06-2002 05:20 AM         

quote:

---

Originally posted by wooks:

..... I am really interested in doing is coercing out refutations so that I can validate my hypotheses. A hypotheses that will guide my work in the future.
......

---

The answer to a question such as whether you should used "deferred checking" or "instant checking" is as usual, it depends.
I wouldn't try to chase that one and only answer, since as with most everything when it comes to testing, the answer will depend on the circumstances (i.e. task at hand, testing strategy, technology used, schedules, budgets, resources, preferences, etc).

quote:

---

Originally posted by wooks:

Looking back on the thread there have been 3 concrete objections to the universal application of deffered checking.
......

---

Again, I don't think there ever will be a "universal" application of deferred checking. You could use it as a general guideline for your testing approach, but it never will be a "universal" application for everyone to follow.

By the way, as the moderator to this forum, I think we should now close this discussion – the last 20 or so posts are between Wooks and Joe (Maybe you can take if off line, if you want to keep debating this topic).

To summarize, it seems the answer to Wooks’ question posted in the subject heading “...Instant Gratification” should be “it depends on the circumstances.”

Elfriede

posted 07-06-2002 05:23 AM         

quote:

---

Originally posted by wooks:
{Again, I don't see any general rule that would favor deferred checking of test results.}

Several have come up in the thread. Here's a few

a) Not knowing what repository an outcome is written to because of load balancing.
b) Making verification of testing results the province of the few who have access to specialist testing tool skills and licenses. Do you not like the idea of developers taking responsibility for running smoke tests.
c) Your automation script breaks when the repository you are checking changes. Are we so resistant to the benefits of implementation hiding.

---

a) In most of the shops in which I have worked, load balancing is more the exception than the rule. And, doesn't incomplete knowledge of the repository location favor immediate checking over deferred checking?
By the way, in the cases where I have tested a load balanced implementation, it's not difficult to check both sides of a balanced pair.

b) Personally, I don't like making developers responsible for running smoke tests. I make smoke test scripts available to the devlopers, who often use them as part of their unit testing. But, in most of the shops where I have worked, the integration of their units occurs first in the QA Lab, after the nightly build. Thus, the developers would only be testing their individual units, not the integration of all the units. The nightly smoke test does that for us.

c) I'm not sure I understand your point. I understand implementation hiding. I don't see how that applies here or favors deferred checking.
[/B][/QUOTE]

posted 07-06-2002 05:27 AM         

quote:

---

Originally posted by wooks:
I agree with most of what you say.

What I don't agree with is that you seem to be using it to make a case against having guidelines.

---

I'm not sure how you come to that conclusion.
I just don't see any need to set a guideline for immediate versus deferred checking. I teach my QA teams to use whatever methods are most appropriate for the situation at hand.

Do you really propose to set a guideline like "Thou shalt not check immediately, thou must only check after-the-fact (unless of course you are a maverick, in which case do whatever you think is proper) "?

posted 07-06-2002 06:01 AM           

I am getting value out of the exchange and would love to hear more views so look in comp.software testing for my rejoinder.

One last thing.

It is oxymoronic for the word MUST to appear in a guideline.

posted 07-08-2002 04:05 AM           

a) only just referred to the book. I have long since disposed of my personal copy and have had to purloin this one from the office library.
b) extracted comments relevant to both sides.

From Chapter 2.

There are a number of aspect to automating comparison that must be designed into the automated testing regime. Whether and when to do dynamic comparison and post-execution comparison is one example. It is not a case of one alternative being better than another ; each alternative has disadvantages and advantages.....Usually a combination of the two should be used to get the best results.

Many of the choices come down to trading off the resilience of the tests to changes in the software, with the ease of finding defects when there is a discrepancy.

....

If dynamic comparison is embedded in the script ... this can make the script more complex....
The more information the script contains the more susceptible it will be to changes (either in the software or to test cases themeselves).
This can increase the cost of maintaining scripts considerably.

From Chapter 4

Dynamic comparison appears at first to provide compelling reasons for using it as much as possible, but there is a downside (..goes on to repeat issues of script complexity and maintenance cost).

....

Unlike dynamic comparison, where outputs have to be compared as they occur, with post-execution comparison it is possible to be more selective over the order and the extent of comparisons performed. .... For example, general results may be verified first, since if they have failed, there is no point in spending any time on detailed verification of those results.

....

However when a large number of automated test cases are in place or there are a large number of time consuming complex comparisons, you may run out of time to run and verify all of the tests.

Attention PeterNairn - psuedo deferred checks (Graham calls this Active post-execution comparison)

The active approach to post-execution comparison offers only subtle benefits over dynamic comparison that may be valuable in some situations but not in others. The benefits are:

1. ..A more detailed record of the outcome of the test cases will be retained given potentially useful information ...and perhaps to archive for future audit trails.

2. Comparison may be done offline (may take considerable time and be processor intensive).

3. Different comparators may be used. The sophistication of dynamic comparison is usually limited ot the capabilities of the test tool....

The final excerpt is somewhat self contradictory and thus could be said to be reflective of the whole discussion. Having warned of the disadvantages of dynamic comparison she then goes on to say this

Verifying outputs as they occur during a test case is generally superior to verifying them at the end of a test case since the test execution tool can be instructed to act differently depending on the results.

So there we have it ....in the meantime I think I had better retrieve my old copy and put it back on my bookshelf.