Real-world virtualization benchmarking: the best server CPUs compared

vApus: Virtual Stress Testing

Testing uses real-world databases and websites. To make this as realistic as possible we use vApus. vApus or Virtual Application Unique Stresstest is a stress test developed by Dieter Vandroemme, lead developer of the Sizing Server Lab at the University College of West-Flanders. The basic setup works as follows. Each application is logged during business use in the peak hours. These logs are analyzed by the vApus application and queries/URLs requested are grouped into user actions. vApus will simulate the actual actions that people have done by performing these actions again on the application running on a server in our lab. A well-tuned threading mechanism developed in-house launches one thread per user.

The cool thing is that vApus allows us to perform several completely different stress tests on several servers simultaneously. This is ideal for virtual server testing. Let's discuss this a bit more.

Above you can see the connection to -- in this case -- the Sun XFire 4450 server (which is no longer in our lab). Several performance monitors can be started on each tested server. The CRWebHTTPBenchmark is the first benchmark that will be performed, in our case on virtual servers 2 and 3. The CR means "Continuous Rate": each user will perform an action every second (this is configurable of course). The CRDBBenchmark (the third item under the "SunxFire" connection) is the Continuous Rate benchmark for the Decision Support Database.

Above you see what the results look like, in this case on one of our slowest servers. Concurrency indicates how many users are hitting the website, and Item/s is the throughput. We constantly monitor the client CPU to make sure that the client machine is never the bottleneck.

The vApus "master" process launches several separate processes called "slaves", which will stress test a separate server or VM. In our case, there are three slaves: two web tests and one database test will all run in parallel. As vApus must be cheap to use, we wanted to avoid situations where you need a massive battery of clients. Besides the fact that every Slave has an extremely tuned threading system to simulate users, each Slave can have affinity set to one or more CPU(s) of the client PC it runs on. For this test, we used two client machines: a quad-core Core 2 Quad Q6600 at 2.4GHz and our old dual Opteron 2.6GHz test system.

As you can see above, we carefully monitor CPU usage so that we are sure that our clients are never the bottleneck. The beauty of this system is that we were able to simulate between…

• 600-800 Database users on VM1
  
• Two times 80-110 web users on VM2 and VM3

…while running only two clients. Our quad-core client should have been sufficient, but we wanted to play it safe and give each slave at least two CPUs. The two clients and one master are directly connected via a gigabit switch to the test servers.