posted 07-04-2002 01:31 PM         

quote:

---

Originally posted by wooks:
So my conclusion. All things being equal (and they are more often than not) - deferred is the way to go.

---

I guess I have to conclude that things are never equal.
In general, I use whatever works best for the situation at hand. I use "instant gratification" when it makes sense. I use "deferred gratification" when it makes sense.
I see no compelling reason to limit myself to one or the other.

posted 07-04-2002 10:10 PM           

quote:

---

Originally posted by jstrazzere:
I guess if all you cared about were the attributes of GUI objects that would work. Not much of a test, though

---

Automated tests never are much of a test. The most powerful tests are heuristic manual ones and the way you conduct your most powerful tests should drive your testing strategy.
Verifying that A = B is not rocket science, does not need to be proprietary and it serves no useful purpose to the enterprise if it becomes the province of a few people with access to specialist tools and skills.

posted 07-05-2002 04:05 AM           

Looking back on the thread there have been 3 concrete objections to the universal application of deffered checking.

1. With deferred checks you cannot direct the flow of your test execution.

Thats ok. The immediate check that was made for the purpose of checking the outcome of a test is now being used for a 2nd purpose - to drive the path through test execution.
Good Software engineering principles discourage us from doing that sort of thing (using an attribute for more than one purpose).
So I can put things into my test suite that will check an outcome and drive my test execution. I just won't call it an immediate check - or rather I just won't equate it to an immediate check since it is serving a different purpose.

2. Deferred checking may hide certain errors. This is the point Peter raised, typical of this problem is the equal offset type error (C should = A + B, C is 40 and A and B are both 20 you think it's ok. In actual fact A and B (both intermediate steps) should have been 10 and 30 respectively. You don't see this because you are only looking at the end result.

Again I'm ok with the psuedo deferred (or psuedo immediate if you favour Peters semantics) solution to this.

3. Graphical apps would lead to prohibitive storage requirements for deferred checking.

True. Thats why my advocation of deferred checking is as as a guideline not a standard.

posted 07-05-2002 04:09 AM         

quote:

---

Originally posted by wooks:
Automated tests never are much of a test.

---

That's not true of my automated tests.
Your mileage may vary.

posted 07-05-2002 04:09 AM         

quote:

---

Originally posted by wooks:
and whats wrong with looking at GUI attributes.
aren't these the very same things that you look at with an immediate check anyway?

---

No, I look at lots of things - the GUI, the database, the registry, etc, etc.
Your mileage may vary.

posted 07-05-2002 04:12 AM         

quote:

---

Originally posted by wooks:
3. Graphical apps would lead to prohibitive storage requirements for deferred checking.

True. Thats why my advocation of deferred checking is as as a guideline not a standard.

---

Not necessarily true.
In testing a graphical CASE tool, the result was a "model file" which could be extracted into text format and used for checking correctness after-the-fact.

posted 07-05-2002 04:16 AM         

quote:

---

Originally posted by wooks:

This started out as a hunch. It still is not dogma, thus much as it may look like I am trying to coerce others into my point of view, what I am really interested in doing is coercing out refutations so that I can validate my hypotheses. A hypotheses that will guide my work in the future.

... Thats why my advocation of deferred checking is as as a guideline not a standard.

---

I guess I was confused by your earlier statement:

quote:

---

So my conclusion. All things being equal (and they are more often than not) - deferred is the way to go.

---

posted 07-05-2002 04:35 AM         

quote:

---

Originally posted by wooks:
Automated tests never are much of a test. .....

---

I agree with Joe.
It depends on how the automated tests are developed. One cannot make a general statement, such as "Automated tests never are much of a test..."
The testing of some of our core functionality is automated. Much of our regression testing is automated. Our automated testing however is also supplemented with manual testing.
Automated testing is an important part in our testing efforts.

posted 07-05-2002 04:58 AM           

quote:

---

Originally posted by Elfriede Dustin:
I agree with Joe.
It depends on how the automated tests are developed. One cannot make a general statement, such as "Automated tests never are much of a test..."
The testing of some of our core functionality is automated. Much of our regression testing is automated. Our automated testing however is also supplemented with manual testing.
Automated testing is an important part in our testing efforts.

Elfriede

---

Ok it was poorly phrased let me try again.

Automated tests are never much of a test compared to the heuristic processes involved in manual testing.

All the metrics I have ever seen for the percentage of bugs found by automated testing range from 15% to 30%.

Now Joe will correct me if I am wrong but I don't think he was referring to the amount of testing done with automation. Rather I believe he was making a reference to how powerful (or should I say weak) a test that looked at a dump of GUI attributes would be.

From what I know of immediate checks they too are derived from a dump of GUI attributes so the only difference they can make to the power of a test is by influencing what you decide to test next. There is nothing stopping you from putting such decision points in a deferred test only you would call it what it really was rather than an immediate check.

posted 07-05-2002 05:06 AM           

quote:

---

Originally posted by jstrazzere:
No, I look at lots of things - the GUI, the database, the registry, etc, etc.
Your mileage may vary.

---

Already been covered earlier in the thread.
In particular the investment banking example.

Re your other responses.

You can find an exception to everything in life. So you have to decide whether the fact that you could generate a text file from a graphical file is the general case for graphical apps or not.
Then you decide what to base your guidelines on if you want to have any guidelines at all.

posted 07-05-2002 05:26 AM           

The main situation I can think of that necessitates immediate checks, is where you do not have a controlled test environment and this is not an uncommon phenomenon.

That being the case, though other factors come into play. How feasible is your test automation or more to the point the verification of your test automation outcomes in an uncontrolled environment.
Is such a situation to be treated as an exception or is it regarded as a norm fit to influence or define the basis of architectural guidelines.

Your mileage may vary

posted 07-05-2002 07:14 PM         

quote:

---

Originally posted by wooks:
Automated tests are never much of a test compared to the heuristic processes involved in manual testing.

---

I don't think it's an either-or situation.
In my shops automated testing compliments manual testing.

For example, smoke tests are almost always automated.
Here, I'm trying to quickly find out if the new build is "broken" or not.

It's not an issue of using deep, heuristic testing. It's a matter of efficient use of resources. My automated smoke test typically runs automatically - immediately after our daily build. It often runs at night, unattended, and we get the results in the morning when we arrive.

Now, I could pay a tester to stay overnight and perform the smoke test, but that wouldn't make much sense. Yes, a good human tester could perform a deeper test, but that's not what I'm looking for here.

Similarly complete automated regression test suites can give me confidence that my system continues to work well - even when the operating system is changed underneath, or a major component is rewritten.

SO when deciding the "goodness" of automated testin, it's not a matter of comparing automated tests to heuristic processes involved in manual testing. It's more a matter of knowing when its appropriate to apply which method.

posted 07-05-2002 07:17 PM         

quote:

---

Originally posted by wooks:
You can find an exception to everything in life. So you have to decide whether the fact that you could generate a text file from a graphical file is the general case for graphical apps or not.
Then you decide what to base your guidelines on if you want to have any guidelines at all.

---

Again, I don't see any general rule that would favor deferred checking of test results.
Nor, do I find that, in general, deferred checking of GUI-intensive applications would be probitive based on storage requirements.

posted 07-05-2002 07:19 PM         

quote:

---

Originally posted by wooks:
For the most part I still think all things being equal there is nothing you can check now that you cannot check later.

---

Perhaps. But I guess the converse would be true as well. All thing being equal (if they ever are), there are few things you can check later that you cannot check now.