The online community for software testing & quality assurance professionals
   
Active Topics Today's Topics
Sponsors:
Lost Password?

Home
BetaSoft
Jobs
Training
News
Links
Downloads

News Group:
software.testing


Testing
Automation
Performance
Engineering
Miscellaneous
Statistics
Poll
  QA Forums
  Performance & Load Testing
  Loadrunner vs. QA Load (Recommend)

Post New Topic  Post A Reply
profile | register | preferences | faq | search

UBBFriend: Email This Page to Someone! next newest topic | next oldest topic
Author Topic:   Loadrunner vs. QA Load (Recommend)
mike2001
New Member

Posts: 1
Registered: May 2001

posted 05-30-2001 10:11 AM     Click Here to See the Profile for mike2001   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by mike2001
From reading through these forums - I have an idea that Loadrunner is quite a good tool, with appropriate infrastructure monitors but has quite a big foot-print per vuser. I also know it is more expensive than QA Load from Compuware.

I've been looking for comments in the QA Load forum -but information is lacking. Does anyone have anyhing good to say about QA Load ? Does anyone use it ?


IP Logged

abatardi
Member

Posts: 7
Registered: May 2001

posted 06-05-2001 12:50 PM     Click Here to See the Profile for abatardi   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by abatardi Visit abatardi's Homepage! UIN: 21875073
QALoad is a very good tool, and Compuware has recently released version 4.8. One benefit of QALoad is that it is based on straight C. No proprietary scripting language (Mercury TSL). Since it is C, it can be compiled into machine code (DLL) and greatly reduce overhead and increase scalability (instead of being an interpreted script like TSL). It doesn't have as nice an interface as Mercury, and may be a bit harder to learn unless you are fluent in C, but I believe it is potentially much more powerful than LoadRunner in the right hands. All dynamic parameterization is also handled automatically for you, and QALoad can support recording several middlewares at once, unlike Mercury.

------------------

IP Logged

ElSeeker
Member

Posts: 24
Registered: Jun 2001

posted 06-05-2001 10:53 PM     Click Here to See the Profile for ElSeeker   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by ElSeeker
I also prefer QALoad over the other products available on the market because it is very powerfull and has a small footprint in terms of memory usage - I believe it is around 100k per simulated user!! And since it is not interpreted it makes it VERY quick so the timing results you get back are accurate.

------------------
Have a good day!
El Seeker
elseeker@hotmail.com

IP Logged

Nuflex
Member

Posts: 7
Registered: May 2001

posted 06-06-2001 01:55 AM     Click Here to See the Profile for Nuflex   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by Nuflex
Hi Mike,

I wouldn't bother with any of those two. I've gone through extensive evaluations myself, you should definitely have a look at Segue's SilkPerformer if you are serious about your project ... I think they are based in Reading (UK).

Let me know if you need any more help!


quote:
Originally posted by mike2001:
From reading through these forums - I have an idea that Loadrunner is quite a good tool, with appropriate infrastructure monitors but has quite a big foot-print per vuser. I also know it is more expensive than QA Load from Compuware.

I've been looking for comments in the QA Load forum -but information is lacking. Does anyone have anyhing good to say about QA Load ? Does anyone use it ?



IP Logged

Nuflex
Member

Posts: 7
Registered: May 2001

posted 06-06-2001 02:03 AM     Click Here to See the Profile for Nuflex   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by Nuflex
Also, I forgot to mention, if you check out the Voting Poll you can see that SilkPerformer comes out top ...

IP Logged

abatardi
Member

Posts: 7
Registered: May 2001

posted 06-06-2001 06:39 AM     Click Here to See the Profile for abatardi   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by abatardi Visit abatardi's Homepage! UIN: 21875073
Yes, Segue is a good tool, but the problem with Segue is that since certain upper management has stepped down and been replaced, the company has been in a bit of a financial situation. They are actually based out of Lexington, Mass I believe. A lot of customers don't even want to deal with them because with stock now dwindling at $3/share, they are afraid of long term support from the company, and a lot believe that Mercury is just waiting for the right time to buy them out anyways.

I have been to a few test tool "bake offs" in the Boston area where Segue was one of the presenters. When most people showed off the uniqueness of their product and what made them better, Seque basically said "yeah, we can do that too...." and an actual quote from one of the shows "So, when you're ready to purchase a [load testing] tool, remember us...remember Segue." It sounded like a cry for help. Needless to say they got few visitors at their table after the presentation.

I'm not saying in any way that their tool is inferior, but their presentation and marketing of it is, which will cost them a lot of sales and potentially the company in the long run.

But, that's just my feelings. Considering most people in QA want to step away from instability, they aren't willing to buy from a company that are themselves, instable.

- abatardi

------------------

IP Logged

JerrySchwartz
Advanced

Posts: 157
Registered: Dec 2000

posted 06-06-2001 06:51 AM     Click Here to See the Profile for JerrySchwartz   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by JerrySchwartz Visit JerrySchwartz's Homepage!
As a point of correction - Mercury's LoadRunner code is NOT interpreted TSL. It is straight (ANSI) C - compiled and full-strength!

TSL and interpretation is in the WinRunner GUI-based tool.

------------------

IP Logged

abatardi
Member

Posts: 7
Registered: May 2001

posted 06-07-2001 08:52 AM     Click Here to See the Profile for abatardi   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by abatardi Visit abatardi's Homepage! UIN: 21875073
Well, it's straight ANSI C (weeellll....maybe)... But since when is it compiled? Controller just interprets the .c file...

- abatardi

------------------

IP Logged

rkat1
unregistered
posted 06-09-2001 10:43 AM         Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by rkat1
>Well, it's straight ANSI C (weeellll....maybe)... But since when is it compiled? >Controller just interprets the .c file...

Is this true?
(WinRunner is an interpreter, but LoadRunner claims to really compile the code)

How can this be verified?
There doesn't have to be an .exe file, probably the executable code is stored in memory only.

Some thoughts to check what really happens: what code is sent to agents, what time does it take to compile (or not) a very large/short script, what (temporary) files are created? (Ask Mercury?)

rkat

------------------

IP Logged

ElSeeker
Member

Posts: 24
Registered: Jun 2001

posted 06-10-2001 03:38 PM     Click Here to See the Profile for ElSeeker   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by ElSeeker
I am not sure what Mercury does but I do know that QALoad is C code that does get compiled because they require MS VC++ in order to compile them. It ends up creating a .DLL that contains everything that a user will do (ie HTTP requests, etc.) You can use any C functions that you want and also any Windows API calls that you want (keeping in mind that this will not work on a Unix machine). The speed and custom-coding that you can achieve with QALoad is very good since there is no interpreter or "slow" scripting language involved!

Cya.

------------------
Have a good day!
El Seeker
elseeker@hotmail.com

IP Logged

ilia
Moderator

Posts: 358
Registered: Jul 1999

posted 06-14-2001 03:21 AM     Click Here to See the Profile for ilia   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by ilia
Well, it is ANSI C since the first version
of LoadRunner. And the code is COMPILED.

Do not just 'beleive' what is more
powerful, just to check it yourself.

Using ANSI C lets to have a UNIX based
load generators (Linux, Solaris, AIX, HP).

The ANSI C code is being compiled on
remote host. No MSVC installation is needed.

It is also possible to load any win32
compiled DLL and use its functions in
LoadRunner script. In such a case, the Unix
based load generator cannot be used of course.

However, for Java environments like RMI and
CORBA the script is pure java code, which
is COMPILED by a standard Sun's JDK.

Regards,
--Ilia

P.S. The ANSI C interpreter is used by
virtual user generator only. It is used for
script debugging only. During the loadtest
the script is compiled.

[This message has been edited by ilia (edited 06-14-2001).]

IP Logged

Phil Hollows
Advanced

Posts: 131
Registered: Jun 2000

posted 06-14-2001 06:35 AM     Click Here to See the Profile for Phil Hollows   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by Phil Hollows
Interpreted / compiled -- this could become a real holy war, which would be utterly pointless because IT DOESN'T MATTER.

The gating factor on any load tool's performance is going to be the application under test -- it will be spending the vast majority of its time waiting for sockets to fill, servers to respond etc. These times are orders of magnitude larger than the few microseconds difference an interpreted vs compiled script is going to make. And even if timings are "more accurate" (and I'd like to see that proved, please), again -- is anyone that interested in the fifth or sixth significant figure for a time?

[Rant off]

Phil

------------------


[This message has been edited by Phil Hollows (edited 06-14-2001).]

[This message has been edited by Phil Hollows (edited 06-14-2001).]

IP Logged

ElSeeker
Member

Posts: 24
Registered: Jun 2001

posted 06-14-2001 10:19 AM     Click Here to See the Profile for ElSeeker   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by ElSeeker
I agree that a "Holy War" is not going to help anyone out. The things to keep in mind however when deciding if any interpreted language will fill your need is speed. Thinking that a interpreted language is only marginally slower then compiled is a bad misconception - just look at VB versus VC!!! True it may not be very slow in terms of the actual Socket communications but when you start doing string searches and replaces, linking into Windows API calls or external DLL's, etc. you WILL notice a slowdown and increase in CPU usage on the load testing machine. If you look at one system running 100 users of interpreted code you will notice a larger hit in CPU and memory usage then if they were not interpreted. For every user there will need to be one more thread or process that is actually doing the interpreting - unless there is a central process that does then then that process will be the slowdown because it is getting overwhelmed with requests!!!

Overall anyone that is looking at load testing tools should consider what they need, what resources they have (programmers, computers, etc.) and what they may need in the future.

------------------
Have a good day!
El Seeker
elseeker@hotmail.com

IP Logged

Phil Hollows
Advanced

Posts: 131
Registered: Jun 2000

posted 06-14-2001 11:49 AM     Click Here to See the Profile for Phil Hollows   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by Phil Hollows
On a like for like basis, yes: an interpreted solution will be slower in general than a compiled one. BUT I will always contend that the performance of the script (and, more importantly, the scripting engine underneath it) is not typically significant for any load test.

This is because the scripting technology employed is only part of the performance overhead of the tool. For example, if under the covers you link into WinINet then the tool will not scale well simply because WinINet is not designed to scale -- you could have an infinitely fast scripting technology and it won't make any difference to the scalability of the solution. Hence casting the scripting language technology as the defining factor for load tool scalability is very misleading. It isn't.

The difference in overhead between compiled and scripted languages is immaterial compared to other technology, implementation and runtime differences from each vendor. One simple example readily provable in the field is this: LoadRunner is compiled C, yet its virtual user footprint is some 2 to 5 times greater than WebLoad, running JavaScript. WebLoad also consistently supports more users, higher throughput and lower CPU utilization then LoadRunner on like for like tests. There are many reasons why this is so, but none of them -- not one -- is about the nature of the scripting language.

If purchasers want to include whether the scripting technology is compiler or interpreter-based as one of their decision-making criteria that's up to them -- but I would strongly argue it's much less important than the ease of learning the scripting technology in the first place, the testing capabilities of the tool, their reporting needs etc.

Phil

[This message has been edited by Phil Hollows (edited 06-14-2001).]

IP Logged

ElSeeker
Member

Posts: 24
Registered: Jun 2001

posted 06-19-2001 08:40 PM     Click Here to See the Profile for ElSeeker   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by ElSeeker
Hmmm... I would mostly agree with what you say except for one thing.. When you look at a tool you should keep in mind, as you stated, the memory footprint (aka how much hardware will we need in order to simulate x number of users.) If you have a footprint of 2-4 MB per user then you are looking at 3-4 highend systems, with a large amount of memory, in order to simulate a few thousand users!!

The speed of the scripting (interpreted or compiled C) should be considered when choosing a tool.. Yes, you want to make sure it is easy to use BUT with the dynamic nature of web sites today you will need to make sure you can code around things that the tool does not account for - ie. What if you web site needs to make direct calls via Winsock?? Can the tool give you the ability to easily code or add that into the script?

Overall I think we are all on the same page though... "Choose the tool that fits the job!" If you are an avid C programmer then it would do you better to go with a non-interpreted language.. if your C is not that great, then a scripting language that is easier to use would fit you better..

I think we have taken a long road to a center local with this post!! ;-)

------------------
Have a good day!
El Seeker
elseeker@hotmail.com

IP Logged

All times are PT (US)

next newest topic | next oldest topic

Administrative Options: Close Topic | Archive/Move | Delete Topic | Top
Post New Topic  Post A Reply
Hop to:

Contact Us | BetaSoft Inc. | Privacy Statement

Copyright 1997-2003 BetaSoft Inc.


Ultimate Bulletin Board 5.45c