The dangers of extrapolation

As a performance tester I’m often asked to predict the behaviour of a client application based on test results. This is often difficult and I was reminded of this when I saw this xkcd cartoon recently.

Extrapolation

Often I’m asked to predict how many users a website will handle based on test results. For example, I may run a test on a cut-down version of a production system, the test system may have 2 webservers and the larger-scale production system may have 5 webservers.

If our 2-server system can handle 200 users, why isn’t it safe to assume that the 5-server system will handle 500 users?

Test Environment
Test Environment

Using extrapolation to predict the scalability or performance of a system is rarely possible in performance testing. The diagram below illustrates some of the reasons why.

 

Production environment
Production environment

 

The “real” network is invariably more complex than the test environment, the network topology or application architecture may be different. As well as this the patterns of user behaviour may not be as predicted, meaning that performance tests were unrealistic.

A few key differences:

  • Network connection speeds:
    Some users are connecting via slower mobile networks, this may hold connections open longer and affect overall system performance. Mobile users may use a disproportionate number of connections due to the higher network latencies.
    Other, local users, may connect at faster “LAN” speeds. I once worked in an office where a network upgrade brought the Exchange mail servers down. Until the network upgrade, the mail servers had their traffic “nicely throttled” by a slow network. Once this bottleneck was removed the servers couldn’t handle the higher throughputs.
  • Network load balancing: ideally this should be the same in production and test environments. In my experience it rarely is!
  • Sources of load: In our example test environment, we may only simulate user load on the webserver and database server. But what about other load on these systems? In a world increasingly built on SOA principles, what else is communicating with your database servers, contributing to network traffic or accessing your shared storage?

You may feel that in this article, I’m arguing against performance testing. Far from it, performance testing is vital, I’m only highlighting some of the pitfalls. Testing will help you to find and fix problems before your “real users” do, this is exactly what you want.

As well as more robust testing you need to do the following:

  • System monitoring, check real against predicted performance.
  • User monitoring, check real against predicted behaviour.
  • Repeat tests once you have this new data.
  • Test early and test often. Repeat tests over and over again throughout the SDLC.
  • Test your application from “all angles”, consider the use of stubs, test harnesses or service virtualisation technology to supplement your performance test tool.

 

Office window time-lapse

As followers of this blog will know, I’ve been experimenting with a Raspberry Pi (RPi) and one of the uses that I found for it is as a webcam server. I have plans to put a permanent webcam in the TrustIV office window in Manchester and I’ll be using a RPi to do this.

My installation is pretty straightforward:

  • Raspberry Pi
  • Logitech C270 webcam
  • Shared windows folder for storage of images

I installed the standard Raspbian operating system and connected the Pi to the Office network. Once I identified it by IP address, I connected to it via SSH. I left the RPi in “headless mode” without a keyboard, monitor or mouse attached.

I installed fswebcam software. Fswebcam is a simple webcam app. It captures images from a V4L1/V4L2 compatible device or file, averages them to reduce noise and outputs the image to PNG or JPEG formats. The resulting image can then be saved to a file, like the image below.

Webcam time-lapse

Click on the image above to see the timelapse, alternatively read on for the geeky bit…

I mounted an external shared windows folder (called nas) using these commands:

cd /mnt
sudo mkdir nas
sudo mount -t cifs -o username=<username>,password=<password> //PC/nas /mnt/nas
(instructions from the Raspberry Pi adventures blog.

Then I wrote a short script called webcam.sh  that created a folder on the NAS and saved images to that share at five-second intervals

mkdir /mnt/nas/$(date +”%Y%m%d”)
sudo fswebcam -r 640×480 -S 15 –jpeg 95 –save /mnt/nas/$(date +”%Y%m%d”)/webcam.jpg

while true
     do
     sleep5
     sudo mv /mnt/nas/$(date +”%Y%m%d”)/webcam.jpg /mnt/nas/$(date
+”%Y%m%d”)/webcam_$(date +”%m%d%Y%H%M%S”).jpg
     sudo fswebcam -r 640×480 -S 15 –jpeg 95 –save /mnt/nas/20130121/webcam.jpg
done

run using
sudo ./webcam.sh

To “stitch” the video together, I used Windows Live Movie Maker to convert the folder of images to a time-lapse video. Images were replaced every 0.05 seconds, meaning that the “action” was sped up 160x faster than real life.

If you read this far, well done. The video is now on the TrustIV blog page, click the image below to see the timelapse footage, and keep checking back there for the webcam. (When I finally get round to installing it) 🙂

Webcam time-lapse

 

 

A very happy Christmas and best wishes for 2013

This year we’ve decided to send Christmas greetings by email once more. We’re donating our card and stamp money to Porch Boxes. We hope that you like James and Katie’s Christmas pictures.

James' picture
James’ picture

 

Katie's picture

 

As well as the children’s pictures, we’ve done our usual “Elf yourself” videos.

Elf Yourself Pic - 2012
Elf Yourself Pic – 2012

Charleston style”, “80s style”, “Soul style”, “Classic”, ……..sadly no “Gangnam Style” 😉

A very happy Christmas to you all.
Love from Richard, Sarah, James and Katie xxxx