posted 07-02-2002 04:41 AM           

You see, I would call your “pseudo deferred check” an immediate check. The fact that a human being hasn’t yet looked at it is irrelevant. I don’t see the argument that this is deferred checking, unless I am totally missing your point.
I don't think you are missing my point. No. But what is really happening here . In a deferred check we will capture information at execution time that enables us to check the outcome of the test later (unless you have some other mechanism that obviates the need for the real-time capture eg you start from). So in the general case you will capture the key of whatever entity is of interest and use that key to look up the outcome relating to it post run. All pseudo deffered checking (and I admit it is a self - invented nomenclature) is doing is increasing the amount of entity data that we are capturing. So instead of just the key we now capture the key, the transaction amount, the colour attribute of the customer field, and the value of the amount fields enabled attribute (I guess it's more analagous to a screen capture) . These attribute values are evaluated for correctness in the same way as for a deferred check except that they may not have to be looked up from a repository.
I might write automatic checks in there that send the tests down different paths depending on results (as something like Winrunner will let you do). Maybe it is just me, but that is standard practice for this type of automated test.
I don't doubt that alot of people may test this way but there are sources varying from the WinRunner CPS program to Dorothy Graham that say you should strive for atomisticity in your test automation.
That is not deferred checking, that is immediate checking. I have to plan that checking, the same way as I would have to if I was manually executing the same tests, the results are still immediate.
If anything I would say that perhaps the difference between deferred and psuedo deferred checking is spurious as it is only distinguishable by the amount of data captured at test execution time.
So, if you accept the point that pseudo deferred checking is the same as immediate checking (let’s assume you do accept that for a moment), my points still stand as a potential problem for “real” deferred checking.
Well I can see the argument for saying that psuedo deferred is similar to immediate. Personally I think the conceptual semblance to deferred checking is stronger. More importantly it is a solution to the problem of errors of commission and it is one that I think offers more power and flexibility than what I would now call traditional immediate checking.

posted 07-02-2002 04:43 AM           

First up this is great stuff Peter.
Mandating immediate or deferred is a bad thing to do. I wouldn’t mandate either, I would ask what is the best method for this test.
Yes it was poorly expressed wasn't it. What I should have said is if your architecture does not stipulate guidelines for timing of checks...... The assumption then was that there would be a predominance of immediate checks that are easier to conceptualise and execute.
Your architecture guidelines will rarely be granular enough to afford you the opportunity to consider what the best method for THIS test. At best they will provide guidance for a class of test.
I don’t agree that people tend to only check things that are easy. A good tester will look for bugs. The easy checks may be done first, but not at the expense of more difficult checks.
OK, but then the architecture should provide clear guidance for the not so good less experienced tester.
Personally, I am not interested in consistency for consistency sake – give me a hodgepodge any day if it is the right thing to do. Where I would agree with you is where the timing of a check is done solely because it can be, rather than when it should be, that is bad practice.
My gut feel is that if a hodge podge is the right thing to do, then we probably haven't thought about the problem as hard as we could have. Whether we have thought about it enough then depends on the degree of formality and criticality attached to the work we are doing. If either of those things are significant factors, hodge podge will probably hurt you. Straight up a hodge podge gives you a variety of repositories for your test results. Some inside a tool perhaps, some in a spreadsheet or database. There is much to be said I think for being able to go to a single repository for the purpose of interrogating, updating and managing data relating to test outcomes.
Never seen a bad/buggy audit trail? Never seen one where the information is useless or worse than useless? Never seen one where the information it supplies actually replicates the bug so that it looks correct but isn’t? Don’t get me wrong, audit trails can be useful, but like all software they can have problems.
Point taken.
Now, of course, you need some record of what happened, as in your list of logs, journals, etc. But you have those available for immediate checking as well, so where is the difference?
Not necessarily. If your application is load balanced you may not know what server they are written to. The writing of them might be batched for performance reasons. Even if they are immediately available they may not be held in a structured format conducive to immediate verification (maybe you need to parse the log with a regular expression beyond the capabilities of your tool).
I don’t follow the logic here. Any/all checking requires you to plan your tests carefully. Deferred checking is not necessarily more efficient, it may be that you have to write more test code to do deferred checking than you would have to with immediate checking, so maybe less efficient.
Deferred checking does not impact on test execution throughput. It's the batch v online argument. As for it requiring you to plan more carefully, I can relate from experience. I had a situation where I had to verify that the correct validation had taken place. The correct validation was dependent on input and had been identified as mission-critical by the client. The test manager told me to use an immediate gui check and that if I didn't get an error message then it had obviously passed. I told him that the test would also pass if it didn't apply any validation at all and might pass if the wrong validation was applied (so long as the wrong validation didn't produce an error). The fundamental problem was that the GUI did not give any indication that any validation had taken place and unless there was an error gave no indication of what validation had been applied (remember the essence of this project was to make sure that the correct validation was always applied). Immediate checks as mandated by the Test Manager would have glossed over this problem. Deferred checking forces you to think about it and the need for an audit trail of exactly what validation was being applied and to what data.

You see, I would call your “pseudo deferred check” an immediate check. The fact that a human being hasn’t yet looked at it is irrelevant. I don’t see the argument that this is deferred checking, unless I am totally missing your point.
I don't think you are missing my point. No. But what is really happening here . In a deferred check we will capture information at execution time that enables us to check the outcome of the test later (unless you have some other mechanism that obviates the need for the real-time capture eg you start from). So in the general case you will capture the key of whatever entity is of interest and use that key to look up the outcome relating to it post run. All pseudo deffered checking (and I admit it is a self - invented nomenclature) is doing is increasing the amount of entity data that we are capturing. So instead of just the key we now capture the key, the transaction amount, the colour attribute of the customer field, and the value of the amount fields enabled attribute (I guess it's more analagous to a screen capture) . These attribute values are evaluated for correctness in the same way as for a deferred check except that they may not have to be looked up from a repository.
I might write automatic checks in there that send the tests down different paths depending on results (as something like Winrunner will let you do). Maybe it is just me, but that is standard practice for this type of automated test.
I don't doubt that alot of people may test this way but there are sources varying from the WinRunner CPS program to Dorothy Graham that say you should strive for atomisticity in your test automation.
That is not deferred checking, that is immediate checking. I have to plan that checking, the same way as I would have to if I was manually executing the same tests, the results are still immediate.
If anything I would say that perhaps the difference between deferred and psuedo deferred checking is spurious as it is only distinguishable by the amount of data captured at test execution time.
So, if you accept the point that pseudo deferred checking is the same as immediate checking (let’s assume you do accept that for a moment), my points still stand as a potential problem for “real” deferred checking.
Well I can see the argument for saying that psuedo deferred is similar to immediate. Personally I think the conceptual semblance to deferred checking is stronger. More importantly it is a solution to the problem of errors of commission and it is one that doesn't have the drawbacks of what I will now call traditional immediate checking.

posted 07-02-2002 05:08 AM         

quote:

---

Further if the sequence of events is critical
the only way you can really verify that they happened in the correct order is by having the events time-stamped and checking them. That to me indicates a deferred check, we could be talking message protocols, audit trails, job submissions, NT event log, emails as well as databases amongst the events to be checked for. Some of those things simply don't lend themselves to immediate verification.

---

I'd question 'the only way to validate a sequence being deferring checks until the end and checking time stamps - you can validate at the end of each step in the sequence, and ensure that the data is correct then just as easily. In some cases, this is a better solution. If I know that Step 7 of the sequence is going to fail if Step 2 is incorrect, why bother continuing through if step 2 fails? In my own opinion and experience, it's much more efficient, and easier to track testing metrics, if I stop the test at the point that it fails, and analyze from there, rather than continue to run a sequence I know is going to fail if the first step was not recorded properly.

For that matter, consider a system where multiple sequences are dependant upon one another. Mandating deferred checks would mean that I could run many sequences, all of which fail due to an error back at step 2 on the first script. Doesn't make much sense to me.

posted 07-02-2002 05:34 AM           

And remember, you cannot stop an analyse an automated test that is running unattended.

posted 07-02-2002 05:44 AM         

quote:

---

Originally posted by wooks:
The fact that you call it step 7 and it is the 7th thing you check doesn't mean it was the seventh thing done in the scenario.
If it is important to the test that it was the 7th thing then how else are you going to verify it without looking at timing of the other activities in the scenario.

And remember, you cannot stop an analyse an automated test that is running unattended.

---

Hm. In a test where the sequence of events IS critical to the test, and part of it, whether I call it 7 or I call it a flying saucer doesn't matter - it will need to be the seventh thing completed. Or more importantly, it will need to be the thing completed BEFORE the thing it must follow and AFTER the thing it must precede. There are transactions in systems that operate this way - or there are in all systems I have ever tested.

You're right, I can't analyze it while it runs unattended, but then again, I can't analyze anything else while it's running unattended either - that's the point of running them unattended. Coming in the next day to view results. And I can do that much more quickly and effectively if I have targeted the failure.

If that horrible flying saucer test failed, I simply close that particular test or sequence of tests, and move on to the next script that was not dependant on it.

I'm not saying it is the best way for all tests - my point was to highlight a particular scenario that 'forcing delayed validation' would not work for. If you've seen no systems such as this in your own experiences, you'll disagree. Such is the joy of varied experiences.

posted 07-02-2002 06:01 AM           

quote:

---

Originally posted by QALady:
Or more importantly, it will need to be the thing completed BEFORE the thing it must follow and AFTER the thing it must precede.

---

So you need to look at the time those things were done.

If you don't, then you know 7 (or however many) things happened but are making an assumption that they happened in the right sequence and you will miss timing errors. Things may not have occured in the order in which you decided to check them, but may still have a semblance of looking correct.

I don't understand the talk about flying saucers so I'll leave it at that.

posted 07-02-2002 06:13 AM         

quote:

---

Originally posted by wooks:

If you don't, then you know 7 (or however many) things happened but are making an assumption that they happened in the right sequence and you will miss timing errors. Things may not have occured in the order in which you decided to check them, but may still have a semblance of looking correct.

---

What gives you the impression I am making an assumption they happen in the right sequence? It is my script, so I program them to happen in that sequence, and if I validate after the preceding step, after the current, and after the final, I've validated that they occurred in the proper order, and were correctly recorded or changed in the database as necessary. That's the whole point of checking them as they occur when the sequence is a necessary part of the test.

posted 07-02-2002 06:40 AM           

If your script executes a financial transaction in a banking application in your script, your script does not control, the sequence in which the respective customer accounts are updated, whether an interbank transaction is generated after or before the accounts are updated, the timing of updates to the GL and or P & L account, the timing of notification to the customer or any internal dept in relation to the transaction etc.

Now it may be important to the test that no interbank activity takes place until all "in-house" transactions arising have been completed so that if any internal update gets rolled back there are no external ramifications to worry about.

Similarly perhaps the bank don't want the GL and P & L accounts updated until the customer account updates have gone through and notifications to the customer should only go out after the interbank transactions have been completed.

So one thing in your script has generated several outcomes and the sequence is material.

If you don't look at the timing of these activities then you have made an assumption that they have occured in the right sequence.

posted 07-02-2002 07:00 AM         

You keep saying that we have to make assumptions for this to work, but again, you are missing the fact that you just stated several sequential requirements - and that is what I would be testing against. If I _know_ that is the sequence they want, then I test to ensure that it happens correctly. I also test to be sure that one cannot occur if the other doesn't - ie, making sure that a particular type of transaction is NOT processed if the other type has not yet completed - all of which need to be done at runtime.

Say transaction type A should not be completed unless transaction type C has already been updated in the system. I would have one execution sequence where Transaction type C IS updated first (and confirm it as such), and ensure that transaction A is updated accordingly when it is executed.

I would have a second execution sequence, using the same tests, where Transaction A is first, I validate that it has NOT been updated in the system, I execute transaction type C, and then validate after C that A has also been updated in the database.

It seems to me that you're overcomplicating things in an attempt to support your theory, but perhaps my own experiences and test approach vary so vastly from your own that I am missing something important.

posted 07-02-2002 07:17 AM           

If there is a one to one correspondence between user action and outcome then what you are describing is feasible.

posted 07-02-2002 07:43 AM           

I would have thought that any financial system would record a transaction in more than one place. Isn't that the whole premise on which accounting is based.

Is a system that has multiple outcomes from a single operation that atypical? The point I made is equally valid even if there are only 2 outcomes.

The example I quoted is quite typical in investment banking, where you may have interbank electronic funds transfers, nostro and client account updates, updates to security accounts, and accrued interest considerations arising out of a single transaction, in fact it doesn't take into account other updates that may occur to the back office and front office systems.

So is it indeed that I am complicating things or is this indicative of immediate checks having a tendency to result in other outcomes not being investigated as thoroughly.

posted 07-02-2002 08:08 AM         

quote:

---

Your architecture guidelines will rarely be granular enough to afford you the opportunity to consider what the best method for THIS test. At best they will provide guidance for a class of test.

OK, but then the architecture should provide clear guidance for the not so good less experienced tester.

---

quote:

---

My gut feel is that if a hodge podge is the right thing to do, then we probably haven't thought about the problem as hard as we could have. Whether we have thought about it enough then depends on the degree of formality and criticality attached to the work we are doing. If either of those things are significant factors, hodge podge will probably hurt you. Straight up a hodge podge gives you a variety of repositories for your test results. Some inside a tool perhaps, some in a spreadsheet or database. There is much to be said I think for being able to go to a single repository for the purpose of interrogating, updating and managing data relating to test outcomes.

---

quote:

---

Now, of course, you need some record of what happened, as in your list of logs, journals, etc. But you have those available for immediate checking as well, so where is the difference?

Not necessarily. If your application is load balanced you may not know what server they are written to. The writing of them might be batched for performance reasons. Even if they are immediately available they may not be held in a structured format conducive to immediate verification (maybe you need to parse the log with a regular expression beyond the capabilities of your tool).

---

Yes, these would be good reasons for deferred checking.

quote:

---

I don't doubt that alot of people may test this way but there are sources varying from the WinRunner CPS program to Dorothy Graham that say you should strive for atomisticity in your test automation.

---

quote:

---

If anything I would say that perhaps the difference between deferred and psuedo deferred checking is spurious as it is only distinguishable by the amount of data captured at test execution time.

---

quote:

---

Well I can see the argument for saying that psuedo deferred is similar to immediate. Personally I think the conceptual semblance to deferred checking is stronger. More importantly it is a solution to the problem of errors of commission and it is one that doesn't have the drawbacks of what I will now call traditional immediate checking.

---

posted 07-02-2002 08:12 AM         

quote:

---

Originally posted by wooks:
Re complicating things.

I would have thought that any financial system would record a transaction in more than one place. Isn't that the whole premise on which accounting is based.

Is a system that has multiple outcomes from a single operation that atypical? The point I made is equally valid even if there are only 2 outcomes.

The example I quoted is quite typical in investment banking, where you may have interbank electronic funds transfers, nostro and client account updates, updates to security accounts, and accrued interest considerations arising out of a single transaction, in fact it doesn't take into account other updates that may occur to the back office and front office systems.

So is it indeed that I am complicating things or is this indicative of immediate checks having a tendency to result in other outcomes not being investigated as thoroughly.

---

You aren't working on the same system as I am, are you? (I'm sure not). But these are just the situations I am dealing with every day - yes, one transaction has multiple effects, some you won't see until the overnight batch job has run.

posted 07-02-2002 08:25 AM         

quote:

---

Originally posted by peternairn:
You aren't working on the same system as I am, are you? (I'm sure not). But these are just the situations I am dealing with every day - yes, one transaction has multiple effects, some you won't see until the overnight batch job has run.

---

And for those types of processes or transactions, I agree that deferred checks are necessary - where I disagree is a blanket statement that they are best for all situations, because I actually feel that immediate checks are better in many.

posted 07-02-2002 09:04 AM           

I agree with you, the ideal is a single repository. However, I would rather have different repositories and good tests than a single one and restricted tests. Yes, maybe there isn’t always enough thought about the problem, but if the solution is more time consuming to think about and implement than dealing with multiple repositories for your results, it becomes the pragmatic option. I am not really arguing with you here, just pointing out that the ideal may not be attainable for more reasons than not enough thought.
But I agree with you hence I said "haven't thought about the problem as much as we could have ..." rather than should have. Should we. As ever the answer is it depends - on the sort of things you mentioned.

Did you mean “errors of omission” rather than “errors of commission”? If the latter, I am confused, so I will assume the former.
Actually I meant neither having consulted a glossary of accounting terms. I was referring to the 2 wrongs making a right type of error.