How much detail to include in a performance test report?

I’m a member of a LinkedIn group called “Performance Testing“. This morning a member of the group from BlazeMeter posted a link to their blog article about how much detail to put into a performance test report.

http://blazemeter.com/blog/what-include-load-test-report-technical-vs-management-reports

 

Historically I’ve always put large amounts of detail into performance test reports, but over the last 12 months or so I’ve started to reduce the amount of content. This allows me to produce what the customer generally wants in a shorter time frame.

In most performance tests customers tend to want to know the answer to one question:
Will my application perform well under expected user load?

This can generally be answered with “yes” or “no”, although occasionally the answer is “maybe”. (I tend to use traffic lights to show this at a high level.)

Often much more information than this is simply wasting time. In the last year or so I’ve started to produce reports  in Powerpoint that can be easily referred to in conference calls or webinars, be presented to clients at their site and can be re-used internally by my client’s project managers when they want to pass on information to their own internal customers.

I have found that by including less high-level detail and including embedded spreadsheets, charts or other documents allowing technical readers to “drill down” to the detail; I can keep all the potential readers of my reports happy.

I’ve attached a PDF “mock up” of a performance test report based on a test that I ran for a client earlier this year. I’d be interested to hear any comments from other testers about what works for them.

pdf-icon
Sample Test Report

DMA2200 Media Center problem – resolved

DMA2200For the last few weeks, I’ve been having problems with my Linksys Media Extender. This has been causing me a great deal of grief because this is the only method currently of watching TV in bed (I have a Windows 7 Media Center PC in the front room and serve recorded TV as well as the satellite feeds up to the extender in the bedroom).

I’ve noticed that occasionally the Media Center Extender freezes up and becomes unresponsive to the remote control. I replaced the batteries in the remote (twice) but the problem remained. The Extender would happily continue displaying the channel that I had started watching but wouldn’t allow me to change. Occasionally I’d come back to the Extender to find that the screen was blank and the extender was unresponsive.

Restarting the Extender has always temporarily resolved this issue, but in the last few days, the problem has become more acute and the extender has generally only remained stable for a few minutes before crashing again. Since nobody is manufacturing extenders any more (until Ceton releases their new media center later this year) , this could have meant that I needed to drop an aerial into the bedroom and lose the ability to watch recorded TV.

Trawling the Internet for clues to the problem I came across this article, which although it was 2 years old and written in the US, gave me a nugget of information which fixed my problem.
http://experts.windows.com/frms/windows_entertainment_and_connected_home/f/116/p/95097/496301.aspx (Sadly not there anymore….)

It appears that the extender tries to connect to a Cisco or Linksys server to obtain an update. Since support for this product has now been withdrawn it is likely that the server is now offline. My theory is that the problem hit US users a few years ago and perhaps only recently has the same problem surfaced in my region when the UK/European regional servers were decommissioned.

My connection was wired, but I presume that the same steps could be used to fix a problem with a wired DMA2200 (or DMA2100) extender.

Steps taken:

  • Set a static IP address for the media center extender.
  • Enter 0.0.0.0 for both primary and secondary DNS servers.DMA2200

This prevents the extender from accessing the internet and then crashing when it fails to find the relevant update servers.

Finding the MAC address of a device

I always for get how to do this, so I’m putting it on my website as a kind of aide memoire for myself.

Basics:
Every hardware device that connects to a network has a unique hardware address known as a Media Access Control (MAC) address. This MAC address is used by network hardware such as routers and switches to send traffic to the correct device on the network.

Part of the TCP/IP network stack is the ARP service (Address Resolution Protocol) which resolves and tracks the TCP/IP and MAC address of the remote devices that you’re communicating with.

To find out the MAC address of a remote device:

  1. Open the MS-DOS prompt.
  2. Ping a remote device that you want to find the MAC address. e.g. “PING 192.168.1.253”

    PING

  3. Type “ARP -A”, and press Enter.
    (Returns all the entries in your ARP address translation table).
  4. Alternatively, type “ARP -A | FIND “192.168.1.253” and press Enter.
    (Returns just the entry that you’re looking for).

    ARP

  5. The MAC address is in the format xx-xx-xx-xx-xx-xx