Unzipping files in Powershell scripts

I’ve been working for some time on a project which is deploying a complex application to a client’s servers. This project relies on Powershell scripts to push zip files to servers, unzip those files on the servers and then install the MSI files contained within them. The zip files are frequently large (up to 900MB) and the time taken to unzip the files is causing problems with our automated installation software (Tivoli) due to timeouts.

The scripts are currently unzipped using the Copyhere method.

Simple tests on a Windows 8 PC with 8GB RAM and an 8 core processor although a single SATA hard drive show that this method is “disk intensive” and disk utilisation as viewed in Task Manager “flatlines” at 100% during the extraction.

I spent some time looking at alternatives to the “Copyhere” method to unzip files to reduce the time taken for deployments and reduce the risk of Tivoli timeouts which were affecting the project.

Method

A series of test files were produced using a test utility (FSTFIL.EXE), FSTFIL creates test files made up of random data. These files are difficult to compress due to the fact that they contain little or no “whitespace” or repeating characters, similar to the already compressed MSI files which make up our deployment packages.

Files were created that were 100MB, 200MB, 300MB, 400MB and 500MB. Each of these files were zipped into similar sized ZIP files. As well as this a single large ZIP files containing each of the test files was also created.

Tests were performed to establish the time taken to decompress increasingly large ZIP files.

Test were performed to establish whether alternative decompression (unzip) techniques were faster.

Observations

The effect of filesize on CopyHere unzips

Despite initial observations, after averaging out the time taken to decompress different sized files using the CopyHere method the time taken to decompress increasingly larger files was found to be linear.

CopyHere

The difference between CopyHere and ExtractToDirectory unzips

To do this comparison, two PowerShell scripts were written. Each script unzipped the same file (a 1.5GB ZIP file containing each of the 100MB, 200MB, 300MB, 400MB and 500MB test files described earlier). Each script calculated the elapsed time for each extract, this was recorded for analysis.

Unzips took place alternately using one of the two techniques to ensure that resource utilisation on the test PC was comparable for each test.

ExtractToHere

No detailed performance monitoring was carried out during the first tests, but both CPU and disk utilisation was observed to be higher (seen in Task Manager) when using the CopyHere method.

Conclusion


The ExtractToDirectory method introduced in .Net Framework 4.5 is considerably more efficient when UNZIPPING packages. Assuming that this method is not available, alternative techniques to unzip the packages, possibly including the use of “self extracting .exe” files, the use of RAM disks  or memory-mapped files to remove disk bottlenecks or more modern decompression techniques may reduce the risk of Tivoli timeouts and increase the likelihood of successful deployments.
Powershell scripts used

Scripts

 

 

 

What’s new in LoadRunner 11.52 ?

On 12th June at HP Discover, I attended one of lecture sessions led by Shane Evans and Silvia Siquiera. During the presentation, they introduced the new features of LoadRunner 11.52. They started the presentation by introducing the new features that came in LoadRunner 11.50 (last year) and then highlighted the features that had been added since then.

LoadRunner 11.52 session HP Discover

 

Key new features:

  • Improved application performance (particularly VuGen startup and Analysis).
  • Integration with HP Live network. (HP Live network connects partners customers etc.)
  • Replay summary report displays detailed statistics.
  • Improved run time settings interface.
  • Can now configure round robin ramp up across load generators.
  • Integration with Visual studio and eclipse for agile CI/CD.
  • Support for Junit, nUnit and Selenium in LR scenarios.
  • LR scenarios as part of unit testing.
  • Can add performance tests as build steps in Jenkins.
  • VTS 2 is a big feature.
  • Mobile traffic recording apps for Android.*
  • New, improved proxy recording feature.
  • Shunra network virtualisation enhancements, a single load generator can now simulate different network types simultaneously e.g. some users on 3G, others on DSL etc. this removes the requirement to have a load generator for each “network type”.
  • Additional support for Linux load generators (different Linux versions).
    (This alone could be a reason to migrate to 11.52, Linux load generators can significantly reduce license costs for your test equipment).

*Whilst the new mobile recording engine is useful, I can’t help but feel that HP has scored an “own goal” in marketing terms. A number of their competitors are making lots of noise about their ability to test mobile applications and this announcement implies that HP are only just getting their act together on mobile testing. HP has been able to test mobile (and many other) applications via a proxy recorder for as long as I can remember and I’ve been using LoadRunner since 1999. This is new proxy recording engine undoubtedly makes scripting easier but it isn’t their first foray into mobile testing and their competitors are disingenuous if they try to make that claim.

Future trends
64-bit VuGen
One customer asked about whether HP had any plans for 64-bit VuGen or Load Generators. Shane Evans said that they didn’t perceive particular demand for this especially since load generators could each spawn multiple 2GB processes for load generation but that they would consider this if there was sufficient customer demand.

This question prompted a few others and one customer asked whether dcripting languages other than C were being considered. Shane said that JavaScript was being considered as a possible scripting language. This could pave the way towards using NoJS as a test engine.

Network simulation
The discussion then moved to network simulation and Shane asked the room whether people were using network simulation tools like Shunra or WANEM as part of their tests. Only 2 people in an audience of approximately 90 were using network simulation tools.

 

Diagnostics tools
Only one customer was using HP diagnostics and two or three others were using Dynatrace or other diagnostics/profiling tools like Wily Introscope. I would be very surprised if network simulation and application diagnostics weren’t significant growth areas for specialist testers in the coming years. These products add significant value to testing engagements and are becoming more important as users become more mobile and applications are increasingly hosted on virtual platforms where performance optimisation is more important.

 

VTS2 – Virtual Table Server
Shane then did a small demonstration of VTS2. I was a big advocate of the original VTS (virtual table server) and since it’s demise I’ve looked at various ways of managing test data in scripts (such as using MySQL) . VTS was always an unsupported add-on (originally written by Mercury engineers) to allow parameters to be passed from script to script. This worked well but had a number of limitations. As well as being “technically” unsupported, the database had to be loaded with parameters from a CSV file before tests and you had to remember to save the “database” to a file each time that you shut it down.

VTS2 doesn’t have these limitations. It is based on the Mongo NoSQL database and can handle up to 2000 txns/sec easily (I can’t wait to try this properly). Media for VTS2 comes with LoadRunner 11.52 or is downloadable from HP SSO.

LoadRunner 12.0 ?
Silvia mentioned the fact that there were so many new features, some people within HP felt that this could merit a new major version number rather than simply adding 0.02 to the last release. This reminded me of Stuart Moncrieff’s summary of 11.50 last year. Having seen the new features, I think that she’s probably right, LoadRunner 12.0 and Performance Center 12.0 are probably already here 🙂

Google Latitude contacts disappeared…now fixed

I’ve used Google Latitude for about four years now and have found the application really useful. For those of you who don’t know, Latitude is an app that you install on your smart phone (it supports Android, Blackberry or iOS) and it keeps track of your location. You can then use this information how you wish. You can share it with friends or family, so that they know where you are in real time, or just use it to maintain a history of where you’ve been. (I find it useful for expense claims when I can’t remember how many miles I’ve done or exactly which day I visited a particular client).

typicalday

About 2 months ago my wife asked why I wasn’t showing up on her list of contacts on her iPhone. I had a look at my own iPhone and saw that I had “no contacts” either. I hadn’t made any changes to my privacy settings and thought it was curious that it affected both me and my wife on multiple devices. (Windows 7 and Windows 8 PCs, multiple different browsers including Chrome) as well as on my iPad and iPhone. As a result of this, I suspected that the problem was with Google.

I tried deleting and adding my wife as a “Latitude contact”. Despite clicking on the invitations that we sent each other and accepting the request to share location information we never appeared in each others contact lists.

Searching for answers I found plenty of discussions on internet forums. Several people had reported that this was due to new privacy settings which coincided with a new release of Google Maps. The only solution seemed to be to log onto an android device and enable location sharing again.

But I  don’t have an Android tablet…..

As an application tester, I’m reasonably au fait with virtualisation, so I had a quick look around the internet and I found this website which contains OVA files with images of Android Tablet devices. These can be used to emulate an Android tablet. http://androvm.org/blog/download/

Android tablet in VMWare Player
Android tablet in VMWare Player

These are the steps that I took to fix my problem

  • I already had VMWare Player on my home PC.
    (Download and install this if you don’t already have a copy)
  • Downloaded a suitable OVA file
  • Open it in VMWare Player, created a new virtual machine
  • Start the VM and “boot up” your Android tablet
  • Choose a country location and go through the basic setup
    (including typing in your google account credentials.)
  • Open the Google Play Store
  • Install Google Maps
  • Once Google Maps is installed, open maps and enable the “Latitude layer”
  • Enable location sharing

Google Maps

Google Maps”Hey presto” almost immediately all your devices will repopulate your friends lists and location sharing will work again.

Latitude screenshot
Latitude screenshot

It looks like the new version of Google Maps for Android has the functionality to enable sharing and accept invitations but this doesn’t work for existing, non-Android users, including those of us using iPhones.

If you’ve deleted friends whilst trying to fix the problem get your friends to send an invitation whilst you’re logged into your “new tablet” and you can accept the invitation and enable location sharing from within the Maps application.