posted 06-29-2002 11:49 PM           

There are 2 options - immediate verification which involves interrogating of a database (typically a db check) via a tool, or delayed verification where you keep track of updated records and check them at the end of the run.

By the way how do you handle an immediate verification where the repository is not a database - does ODBC (which most of these tools use for db verification) support a text file driver. What other repositories are there to consider and how do they get checked.

posted 06-30-2002 05:03 AM         

quote:

---

Originally posted by wooks:
This is to do with the timing of the checking of the backend while doing a gui test.

There are 2 options - immediate verification which involves interrogating of a database (typically a db check) via a tool, or delayed verification where you keep track of updated records and check them at the end of the run.

By the way how do you handle an immediate verification where the repository is not a database - does ODBC (which most of these tools use for db verification) support a text file driver. What other repositories are there to consider and how do they get checked.

---

Sometimes I use immediate checks, sometimes delayed database checks.

When I write an automated test for a specific bug, I usually do immediate checks.

When I write a comprehensive regression it usually begins with an empty database, builds it through a series of steps, and checks it all at the end.

And, yes - ODBC does support a text file driver. So sometimes I use ODBC, sometimes I read and parse the text file myself.

Other "repositories" might include a non-standard file type. I once tested a CASE tool whose end result was a "model" file. We built an export tool that could take a model, strip out the timestamps, and export it to a .csv file.

posted 06-30-2002 06:30 AM           

Unless you are doing ad-hoc testing, I think that

a) the timing of the check should be an architectural decision, mandated by the testing strategy (as opposed to determined by the tester).

b) that decision should be that the checks should be deferred. In the cases where timing is a crucial element of the test, specific tests can address that, but the generality of the checks should be deferred.

b stems from a. Some tests may be verified immediately because a tool may make it expedient to do so (typically db checks), but what if the backend check is to see that an email has been sent for instance.

posted 07-01-2002 04:48 AM         

I would totally agree that the decision should be an architectural one and that may be to defer checking.

My concerns with deferring checking are with:

- the potential difficulty of tracking which test caused the bug, it may not be the most obvious one
- the possibility of two wrongs making a right, where two bugs cause the right result
- a test's incorrect result being overwritten by a test's correct result.

posted 07-01-2002 05:09 AM         

quote:

---

Originally posted by peternairn:

- the possibility of two wrongs making a right

---

my mother said 2 wrongs never make a right

posted 07-01-2002 05:29 AM           

quote:

---

Originally posted by peternairn:
Wooks,
I am interested in why you think that checks should be deferred - you haven't said.

I would totally agree that the decision should be an architectural one and that may be to defer checking.

My concerns with deferring checking are with:

- the potential difficulty of tracking which test caused the bug, it may not be the most obvious one
- the possibility of two wrongs making a right, where two bugs cause the right result
- a test's incorrect result being overwritten by a test's correct result.

---

Well I've hinted at my reasons and again I will stress it is more hunch than dogma. Nevertheless this is very much the kind of response I was hoping for because it brings to the fore real issues.

My stance is very much a conceptual one.

1. If you mandate immediate verification then people will tend to check the things that are easy to check immediately at the expense of things were that is not the case.

2. I don't like a hodgepodge approach where some checks are done immediately because there may be tool support for them and others are not. I reckon if consistency is an architectural goal (whether it should be is of course debatable) , more often than not deferred checking is the only option.

3. Deferred checking implies some sort of audit trail being built which is a good thing.

4. Tests where timing is an integral part of the test should be distinguishable from tests where that is not the case. The former by necessity must be immediate, and should be distinguishable from the latter.

5. Deferred checking forces you to plan your test more carefully and is by magnitude much more efficient.

6. If you have a commandline and a gui interface to your system the same deferred checks can be used to test it.

posted 07-01-2002 06:31 AM         

quote:

---

1. If you mandate immediate verification then people will tend to check the things that are easy to check immediately at the expense of things were that is not the case.

---

quote:

---

2. I don't like a hodgepodge approach where some checks are done immediately because there may be tool support for them and others are not. I reckon if consistency is an architectural goal (whether it should be is of course debatable) , more often than not deferred checking is the only option.

---

quote:

---

3. Deferred checking implies some sort of audit trail being built which is a good thing.

---

quote:

---

4. Tests where timing is an integral part of the test should be distinguishable from tests where that is not the case. The former by necessity must be immediate, and should be distinguishable from the latter.

---

quote:

---

5. Deferred checking forces you to plan your test more carefully and is by magnitude much more efficient.

---

quote:

---

6. If you have a commandline and a gui interface to your system the same deferred checks can be used to test it.

---

Wooks, I think you are hinting at your dislike for GUI Automation tools again here. I think you are stretching your point a bit, trying to tie the benefit of "deferred checks" into it.

Most GUI Automation tools are quite capable of automating deferred checks well.

posted 07-01-2002 12:16 PM           

Re 2. See 4

Re 3.
Deferred checking relies on their being evidence that something has happened, and I have taking the precaution of checking out Websters for their own definition

a record of a sequence of events (as actions performed by a computer) from which a history may be reconstructed

Interrogations of logs, database journals, the existence of files with file creation/modification timestamps or even date/time stamps on database records qualify as audit trails.
Without an audit trail I am yet to be convinced that deferred checking is possible and I still believe that one does indeed imply the other.

Re 4.
I think it is beneficial to be able to distinguish tests where the timing of the event under test is significant to the test outcome, from tests where that is not the case. If data being collected on line doesn't need to be processed until the overnight batch run start then it shouldn't be verified immediately. Supposing a resilience measure (over which you have no control) was in effect that queued transactions if a server was down. Whither your immediate check then. It doesn't happen now. Whose to say it won't tommorow. Supposing a load balancing measure is implemented as a performance enhancement , the consequence of which is that you no longer know which database server the record is going to get written to. That is why it is worth distinguishing between tests where timing of the event is critical and where it is not, and more importantly that is why timing of checks should be an architectural consideration. Another reason - immediate checks unnecessarily done make your script less robust - vulnerable to changes in whatever repository you are immediately checking.
PeterNairn raised some very good points and real problems with deferred checking but my gut feel is that they are soluble with effective audit trails and are less intractable than the problems immediate checks pose.

Re 5.
I take it you are doubting the "plan test more carefully" aspect.
When you do an immediate check do you
a) Consider what else should be checked immediately apart from - I'll revert to the typical example - the database.
b) Do you always check for the date timestamp or are you merely satisfied if a record appears with the right value. Supposing the date time stamp has a bearing on the timing of another activity being triggered.
c) What other updates do you check as well. Supposing there is an explicit audit trail or system log. Suppose the trigger in (b) is based on that (might be a performance optimisation) rather than the presence database record you checked (which may not be the final destination by the way as was mentioned before). How likely are you to do an immediate check on the audit trail or system log if it isn't convenient to do so.

This post is about an important and often overlooked test architectural consideration. In the absence of well defined architectural guidelines testers will do what they or their tool if they are using one find easiest, and will do so with varying degrees of thoroughness. After all who is supposed to be in charge?

Another architectural consideration is what I would call an psuedo deferred check. This is where all the data to be checked is captured immediately but the actual checking of it is deferred (as opposed to a deferred check that may for instance just capture the key to the data to be checked).

I believe this would address the problems Peter raised. It can apply to GUI attributes as well and has benefits, because the correct configuration of the expected gui attributes is exposed outside the script code or whatever tool mechanism is used and thus is more visible. This means there would not be a need to maintain a "golden" version of what those attributes should be within the tool, and would makes such testing more accessible to non-tool specialists as a consequence.

So while it is not the focal point of the post there is some sort of tool advocacy going on, but I hope it has provided enough food for thought to be elevated above mere dislike (not that that is a crime).

posted 07-01-2002 03:01 PM           

For instance you have made reference to "the correct sequence of events" in doing a test.
But the sequence of events may be immaterial as long as all the events happen.

Further if the sequence of events is critical
the only way you can really verify that they happened in the correct order is by having the events time-stamped and checking them. That to me indicates a deferred check, we could be talking message protocols, audit trails, job submissions, NT event log, emails as well as databases amongst the events to be checked for. Some of those things simply don't lend themselves to immediate verification.

As ever in life a bit of both is probably the ideal mix, but also as often in life the option which seems to be harder work has it's advantages. Not least being the insulation of the GUI automation scripts (I've just destroyed my anti-automation credentials) from changes in the backend repository.

posted 07-01-2002 11:05 PM           

This doesn't rule out deferring checks and still doing them with a GUI automation tool if you want to.

A good parallel can be taken from the development of GUI applications. Most GUI development tools make it easy to develop screens by binding GUI controls directly to columns in database tables. Most developers eschew this convenience and use unbound controls because it offers them more power and flexibility.

posted 07-02-2002 03:16 AM         

Taking each of your points in turn

quote:

---

1. If you mandate immediate verification then people will tend to check the things that are easy to check immediately at the expense of things were that is not the case.

---

Two things on this.

Mandating immediate or deferred is a bad thing to do. I wouldn’t mandate either, I would ask what is the best method for this test.

I don’t agree that people tend to only check things that are easy. A good tester will look for bugs. The easy checks may be done first, but not at the expense of more difficult checks.

quote:

---

2. I don't like a hodgepodge approach where some checks are done immediately because there may be tool support for them and others are not. I reckon if consistency is an architectural goal (whether it should be is of course debatable) , more often than not deferred checking is the only option.

---

Personally, I am not interested in consistency for consistency sake – give me a hodgepodge any day if it is the right thing to do. Where I would agree with you is where the timing of a check is done solely because it can be, rather than when it should be, that is bad practice.

quote:

---

3. Deferred checking implies some sort of audit trail being built which is a good thing.

---

Never seen a bad/buggy audit trail? Never seen one where the information is useless or worse than useless? Never seen one where the information it supplies actually replicates the bug so that it looks correct but isn’t? Don’t get me wrong, audit trails can be useful, but like all software they can have problems.

Now, of course, you need some record of what happened, as in your list of logs, journals, etc. But you have those available for immediate checking as well, so where is the difference?

quote:

---

4. Tests where timing is an integral part of the test should be distinguishable from tests where that is not the case. The former by necessity must be immediate, and should be distinguishable from the latter.

---

Agreed. That is one of the reasons why you cannot mandate the timing of the checks.

quote:

---

5. Deferred checking forces you to plan your test more carefully and is by magnitude much more efficient.

---

I don’t follow the logic here. Any/all checking requires you to plan your tests carefully. Deferred checking is not necessarily more efficient, it may be that you have to write more test code to do deferred checking than you would have to with immediate checking, so maybe less efficient.

quote:

---

6. If you have a commandline and a gui interface to your system the same deferred checks can be used to test it.

---

Going back to my original post:

quote:

---

My concerns with deferring checking are with:

- the potential difficulty of tracking which test caused the bug, it may not be the most obvious one
- the possibility of two wrongs making a right, where two bugs cause the right result
- a test's incorrect result being overwritten by a test's correct result.

---

If I am right in understanding what you said here....

quote:

---

Another architectural consideration is what I would call an psuedo deferred check. This is where all the data to be checked is captured immediately but the actual checking of it is deferred (as opposed to a deferred check that may for instance just capture the key to the data to be checked).

I believe this would address the problems Peter raised. It can apply to GUI attributes as well and has benefits, because the correct configuration of the expected gui attributes is exposed outside the script code or whatever tool mechanism is used and thus is more visible. This means there would not be a need to maintain a "golden" version of what those attributes should be within the tool, and would makes such testing more accessible to non-tool specialists as a consequence.

---

.....I would agree with you, that this solves the problems, but, therefore, I think we differ on the definition of deferred and immediate checking.

You see, I would call your “pseudo deferred check” an immediate check. The fact that a human being hasn’t yet looked at it is irrelevant. I don’t see the argument that this is deferred checking, unless I am totally missing your point.

If I am running a set of tests overnight I will capture all the results I need as I go along and look at the results in the morning. I might write automatic checks in there that send the tests down different paths depending on results (as something like Winrunner will let you do). Maybe it is just me, but that is standard practice for this type of automated test. That is not deferred checking, that is immediate checking. I have to plan that checking, the same way as I would have to if I was manually executing the same tests, the results are still immediate. What I am missing, of course, is the immediacy of unexpected result investigation, which can be more of a time consuming exercise as you might need to do more to get back to the starting position to investigate (restore db, reset timers, reset dates, etc.) which is where you are trading off time to execute and overall coverage versus time to investigate – but we do that trade off all the time with automation anyway.

So, if you accept the point that pseudo deferred checking is the same as immediate checking (let’s assume you do accept that for a moment), my points still stand as a potential problem for “real” deferred checking.

Deferred checking has its place, however. Performance testing is a good example where I don’t want to check as I go along because that might affect my test results. End to end tests may require deferred checking, I am currently involved in some testing where deferred checking is the only option I have given the nature of the system (you might call it bad design, but I couldn’t possibly comment ). You just have to be aware that there is a risk associated with doing it.

posted 07-02-2002 03:20 AM         

quote:

---

Originally posted by wooks:
The thing I am trying to emphasise is that
the timing of your checks is important and shouldn't be unduly influenced how convenient the check is to do or how easy the tool makes it.

---

I think this is the key point in all of this - and it goes back to planning your tests and developing them as a production piece of software, not as something to create and throw away.

------------------