Unzipping files in Powershell scripts

I’ve been working for some time on a project which is deploying a complex application to a client’s servers. This project relies on Powershell scripts to push zip files to servers, unzip those files on the servers and then install the MSI files contained within them. The zip files are frequently large (up to 900MB) and the time taken to unzip the files is causing problems with our automated installation software (Tivoli) due to timeouts.

The scripts are currently unzipped using the Copyhere method.

Simple tests on a Windows 8 PC with 8GB RAM and an 8 core processor although a single SATA hard drive show that this method is “disk intensive” and disk utilisation as viewed in Task Manager “flatlines” at 100% during the extraction.

I spent some time looking at alternatives to the “Copyhere” method to unzip files to reduce the time taken for deployments and reduce the risk of Tivoli timeouts which were affecting the project.

Method

A series of test files were produced using a test utility (FSTFIL.EXE), FSTFIL creates test files made up of random data. These files are difficult to compress due to the fact that they contain little or no “whitespace” or repeating characters, similar to the already compressed MSI files which make up our deployment packages.

Files were created that were 100MB, 200MB, 300MB, 400MB and 500MB. Each of these files were zipped into similar sized ZIP files. As well as this a single large ZIP files containing each of the test files was also created.

Tests were performed to establish the time taken to decompress increasingly large ZIP files.

Test were performed to establish whether alternative decompression (unzip) techniques were faster.

Observations

The effect of filesize on CopyHere unzips

Despite initial observations, after averaging out the time taken to decompress different sized files using the CopyHere method the time taken to decompress increasingly larger files was found to be linear.

CopyHere

The difference between CopyHere and ExtractToDirectory unzips

To do this comparison, two PowerShell scripts were written. Each script unzipped the same file (a 1.5GB ZIP file containing each of the 100MB, 200MB, 300MB, 400MB and 500MB test files described earlier). Each script calculated the elapsed time for each extract, this was recorded for analysis.

Unzips took place alternately using one of the two techniques to ensure that resource utilisation on the test PC was comparable for each test.

ExtractToHere

No detailed performance monitoring was carried out during the first tests, but both CPU and disk utilisation was observed to be higher (seen in Task Manager) when using the CopyHere method.

Conclusion


The ExtractToDirectory method introduced in .Net Framework 4.5 is considerably more efficient when UNZIPPING packages. Assuming that this method is not available, alternative techniques to unzip the packages, possibly including the use of “self extracting .exe” files, the use of RAM disks  or memory-mapped files to remove disk bottlenecks or more modern decompression techniques may reduce the risk of Tivoli timeouts and increase the likelihood of successful deployments.
Powershell scripts used

Scripts

 

 

 

Google Latitude contacts disappeared…now fixed

I’ve used Google Latitude for about four years now and have found the application really useful. For those of you who don’t know, Latitude is an app that you install on your smart phone (it supports Android, Blackberry or iOS) and it keeps track of your location. You can then use this information how you wish. You can share it with friends or family, so that they know where you are in real time, or just use it to maintain a history of where you’ve been. (I find it useful for expense claims when I can’t remember how many miles I’ve done or exactly which day I visited a particular client).

typicalday

About 2 months ago my wife asked why I wasn’t showing up on her list of contacts on her iPhone. I had a look at my own iPhone and saw that I had “no contacts” either. I hadn’t made any changes to my privacy settings and thought it was curious that it affected both me and my wife on multiple devices. (Windows 7 and Windows 8 PCs, multiple different browsers including Chrome) as well as on my iPad and iPhone. As a result of this, I suspected that the problem was with Google.

I tried deleting and adding my wife as a “Latitude contact”. Despite clicking on the invitations that we sent each other and accepting the request to share location information we never appeared in each others contact lists.

Searching for answers I found plenty of discussions on internet forums. Several people had reported that this was due to new privacy settings which coincided with a new release of Google Maps. The only solution seemed to be to log onto an android device and enable location sharing again.

But I  don’t have an Android tablet…..

As an application tester, I’m reasonably au fait with virtualisation, so I had a quick look around the internet and I found this website which contains OVA files with images of Android Tablet devices. These can be used to emulate an Android tablet. http://androvm.org/blog/download/

Android tablet in VMWare Player
Android tablet in VMWare Player

These are the steps that I took to fix my problem

  • I already had VMWare Player on my home PC.
    (Download and install this if you don’t already have a copy)
  • Downloaded a suitable OVA file
  • Open it in VMWare Player, created a new virtual machine
  • Start the VM and “boot up” your Android tablet
  • Choose a country location and go through the basic setup
    (including typing in your google account credentials.)
  • Open the Google Play Store
  • Install Google Maps
  • Once Google Maps is installed, open maps and enable the “Latitude layer”
  • Enable location sharing

Google Maps

Google Maps”Hey presto” almost immediately all your devices will repopulate your friends lists and location sharing will work again.

Latitude screenshot
Latitude screenshot

It looks like the new version of Google Maps for Android has the functionality to enable sharing and accept invitations but this doesn’t work for existing, non-Android users, including those of us using iPhones.

If you’ve deleted friends whilst trying to fix the problem get your friends to send an invitation whilst you’re logged into your “new tablet” and you can accept the invitation and enable location sharing from within the Maps application.

Raspberry Pi as a WOL server

My electricity supplier in the UK recently sent me a electricity consumption meter which tracks energy use in real time and displays in on a small LCD panel. This has proved to be a real eye opener for me and has helped me to see how much power is used by a variety of devices in the house. I’ve found myself checking the meter at regular intervals and I’m getting good at working out which things are powered on just by checking the meter!

Electricity Consumption Meter
Electricity Consumption Meter

I’m one of those geeky types that likes to leave my home PC on all the time. This means that when I’m away from home, I can connect to the PC remotely and pick up files or refer to old email archives that I don’t keep on my laptop. I haven’t given this much thought until now but I checked the power consumption of my PC and saw that it draws 250W – 300W whenever it’s powered on. I did some rough calculations to see what this was costing me and I was surprised.

Daily power use for my PC = 0.275 KW x 24 = 6.6 KWh
Annual power use for my PC = 6.6 KWh x 365 = 2409 KWh
Average cost in the UK for 1 KWh = £0.14

Annual cost for running my PC all the time = £0.14 x 2409 KWh = £337.26

This made me think about turning my PC off more (as my wife has been suggesting for years) 😉  But I don’t want to lose the ability to turn it on remotely and get access to files….. hence my use of WOL (Wake on Lan).

I recently bought a Raspberry Pi and I’ve been tinkering with it for a while, I’ve used it for time lapse photography and general experimentation. I’ve also put one in the Trust IV office in Manchester which operates a webcam. I thought that I could use the low-powered Raspberry Pi to wake up my high-powered PC when I need it and I’d enable power management on the PC to put the PC to sleep when it isn’t in use.

Here’s how I did it.

  • Installed a base build of Raspbian on my Pi
  • Installed “wakeonlan” on the Pi, using the command [sudo apt-get install wakeonlan]
  • Installed Apache on the Pi, using the command [sudo apt-get install apache2]
  • Installed PHP5 on the Pi using the command [sudo apt-get install php5]
  • Wrote a small PHP script that sends the wakeonlan command to my PC.
  • Changed the default port for the webserver
  • Enabled port forwarding on my router to forward HTTP requests from outside to the Pi.
RPi screenshot
RPi screenshot

The PHP script (above) is pretty straightforward and uses the “exec” command to simply execute the wakeonlan command. I may improve this to give myself some visual feedback to show that the magic packet has been sent to the PC. For now I’m just happy that I can reduce my energy consumption and still access my PC remotely whenever I want to.

Assuming that  this gives me a 50% energy saving, this should reduce the electricity bill by more than £150 per annum.