posted 03-13-2002 02:25 PM         

quote:

---

Originally posted by Semantec:
Does any one has an experience with test oracles used for checking complex computations...

---

Yes, we are testing tax management systems. Calculations include (among others) Income Tax Planning, Corporate Tax Planning, Depreciation for fixed asset management systems, while using tax laws from the last 70 years, as they apply for fixed asset depreciation, for example.
Yes, we've developed a test harness that tests these complex calculations.
What type of calculations are you testing?

posted 03-14-2002 05:26 AM           

quote:

---

Originally posted by Semantec:
Dear Elfriede
Of course automated oracle is preferrable but the development of such one may be as complicated as the development of the application itself.

---

Every time I have written an oracle I have run into this paradox. I'd be keen to hear any approaches that guard against this.

posted 03-14-2002 06:52 AM         

quote:

---

Originally posted by wooks:
Every time I have written an oracle I have run into this paradox. I'd be keen to hear any approaches that guard against this.

---

It's always a problem.

One approach is a test harness which stores the "answer" and verifies the program's output against it.
This "answer" is provided by a person, rather than being derived automatically.
This is often used with data-driven approaches.

So, the test harness automates the application to the point where the previously-stored answer can be checked.

Not an automated oracle, but at least no manual intervention is required once the answers are stored.

posted 03-14-2002 07:08 AM         

quote:

---

Originally posted by jstrazzere:

Not an automated oracle, but at least no manual intervention is required once the answers are stored.

---

I was thinking to implement exactly this...to insist that the sponsor team currently responsible for the creation of the invoices should create manually at least several invoices using the algorithm underlying our application so this way we are going to posses some real world results. And later to compare the human generated results against the system generated response of one and the same inputs...

posted 03-14-2002 08:30 AM         

------------------
Charles Reace

posted 03-15-2002 02:24 PM         

quote:

---

Originally posted by Semantec:
Dear Elfriede
First I want to say thank you for the great book "Automated software testing"..Im amazed how valuable it is...
Plamen

---

Thanks, Plamen. Glad you find the book useful.

quote:

---

The question is how to develop such kind of oracles -- whether to go for automated oracles or to compute the procedure manually

---

Building a test harness can range from writing a simple batch file to a complex C++ application. The appropriate language with which to build the tool really depends on the circumstances, and the target of the test. For example, if the target of the test is to thoroughly exercise a complex C++ calculation DLL using some or all of its possible inputs, a suitable solution may be another C++ program that calls directly to the DLL, supplying the necessary combinations of test values, outputting and examining the results.

quote:

---

Of course automated oracle is preferable but the development of such one may be as complicated as the development of the application itself. On other side is great to have an automated oracle procedure because later in the regression testing I can always rerun the oracle in order to check the implementation of the algorithm again.

---

Agreed. The test harness payoff is achieved most often during regression testing. For example, our calculation engine changes from version to version of the application.
The test harness can be played back against the changed calculation engine, produce results, compare the results to the old version, and output the differences.

quote:

---

On other the team is small (only 10 developers and 2 testers) so I dont have the needed resources for the automated oracle...

---

Developing a test harness – like all test automation efforts – is a development effort in itself. Time and dedicated resources have to be made available in order to effectively implement it.

HTH,
Elfriede

posted 03-16-2002 02:30 PM         

If I were you, I'd use either Perl's DBI or Java's JDBC. The database interface is cheap, and should be able to do almost everything you can. For perl, there is one limination and that's passing blobs as objects. As for JDBC, I don't recall any major problems. One major difference is JDBC tends to be slow. Personally for ease, I'd stick with perl.

For data validation, you can try two ways.
1. Scrit/data file based.
Simply, design the tool to be so dumb that it does, 'x', and looks for value 'y' as specified in the tool.

2. Write validation code.
This is pain, but will give you more test cases in the long run. If you can simplify calculation in such a way that you feel confident that your tool can do it, then you have a easy solution as 'call procs with 'x', 'y', and 'z'' -> Get id back -> 'select x from y where xid'. Calculate, then validate values.

If validating the complexity of calculation is the key, your team's choice of tool should be based on development and your team members, programming strength in your test language.

I've done this successfully in the past, putting together an entire automation framework to validate all the stored procedures in Oracle. We also used the same engine for data popluation, and using it to test complex scenarios.

You can opt to write the code in C++ as well, but that tends to be rather too complex. Perl and Java is so easy to do, and you can integrate it into a web cgi/servlet afterwards to get a web driven test tool.

posted 03-19-2002 06:57 AM         

quote:

---

Originally posted by turbotester:
who's gonna test the oracle? And aren't you often just proving that one person scripts better than another? What if both the app and the oracle are created where they both come up with the wrong answer - and they are the same wrong answer? How do you find out that it's wrong?

---

All test procedures, whether manual or automated (oracle or not), should ideally undergo a review/test process of some sort. And of course, one could argue that you then need to test the test of the test, etc., but at some point you reach the point of diminishing returns. In most cases, if someone wrote an independent test oracle, it was reviewed by a 3rd party, and the oracle's results agreed with the AUT's results, I'd feel a lot better about the AUT's quality than if some tester sat down with a pencil and paper and ran a couple manual, ad hoc tests and then said, "It looks OK to me."

------------------
Charles Reace

posted 03-20-2002 07:40 AM         

[/B][/QUOTE]

I'll express a slight disagreement with Charles in this case. OK I wouldn’t trust too the results of the oracle if those results are derived trough a method like this --"some tester sat down with a pencil and paper and ran a couple manual, ad hoc tests and then said, "It looks OK to me.". But in my case there are people with extensive knowledge of the business environment of the application. Those folks have generated thousand of invoices using the semi-automated methods and those invoices seem to be correct. What I'll do in the case is to rely on the human expertise. I am aware that there is a human error but if we are able to generate statistically enough data using the human method we can minimize the risk of error. The other alternative is to develop an fully automated oracle but who is responsible for the oracle testing..Thats not no say that I am against the automated oracles...I just want to say that the automation isnt always the one and only proper track

All the best
Plamen

posted 06-07-2002 11:13 AM           

Or to put it another way. Is it unreasonable to opine that without such in place (i.e no means of tool verification of expected results, that the enterprise is simply not ready to automate that application.

posted 06-07-2002 01:24 PM         

quote:

---

Originally posted by wooks:
Is it worth an enterprise embarking on a test automation program without having in place some sort of test oracle and some means of automatically verifying test results against it.

---

Yes, sometimes it is worth it.

Even if the output must be manually verified, it may be worthwhile to automate the application enough to create that output.

For example, if the setup and execution of an application takes a lot of time or is difficult to perform accurately, then even just automating that part can save some time. Sometimes the time saved is enough to justify the effort of creating the automation.

Sometimes, the time saved can be used toward expanding the automation (and perhaps adding the verification-using-the-oracle steps).