Using Octane 2.0 to Benchmark WebPagetest Agents

WebPagetest is a fantastic tool for measuring web page performance. But sometimes you have to be careful when using the public test agents. In particular the agents are often run on small machines such as the Amazon EC2 VMs. so if the page has a lot of JavaScript code running this might give a distorted picture of the actual performance.

The best way to assess the overall JavaScript performance of a browser is to run Google’s Octane 2.0 benchmark suite. It consists of 17 individual benchmarks, which are run and then the geometric average is calculated as the Octane score.

Instead of scripting the browser to click on the Start link, the easiest way to automatically start the test is to use a query string parameter like this:

http://chromium.github.io/octane/?auto=1

The only problem is that WebPagetest will stop the test before it has completed, but you can work around that by setting the “Mininum test duration” under the Advanced tab. A value of 80 seconds seems suitable for most agents, but if you get incomplete Octane results for some test locations you might have to increase this.

Here are some example measurements for European locations, collected by making 9 test runs (first+repeat view) and collecting the lowest and the highest scoring runs:

As you can see, the Prague location is actually the best performing one, followed by the IISpeed agent in Amsterdam.