Website Performance Test

This test uses a full headless web browser to accurately capture the user experience of browsing a website from within a user’s network. The measurements are based on what is observable on the screen, the goal is to highlight how much waiting time is needed for a user to be able to start accessing content on the page.

Real customer experience 

The purpose of the SamKnows Website Performance Test is to capture the user experience of a website by testing from inside the user's own network. Since the browser is a graphical program that renders the page step by step as objects are received from the network, the results must reflect what can be seen on the screen.

The goal of the test is to highlight how long someone has to wait before they can start accessing the contents of the page. However, not all objects for a web page have visual relevance. Some objects are invisible, such as an advertising beacon, while others have less impact on the visuals overall. Capturing a “visual loading progress” is a useful metric that allows us to track how much of a page is visible to the user throughout the few seconds that a page is loading.

This is useful to a degree - they tell you how fast your website is and whether it is accessible at a very basic level. But it is also completely unrepresentative of real user experience. Real users do not sit inside data centres on 1Gbps connections when browsing the internet! Instead, they sit at home on their cable or DSL connection, or use their mobile phone.

There are 7 phases to the test sequence

Relay phase

The first phase of the test is to determine which objects the test client needs to measure. This relay phase involves no measurement itself - it simply works out what objects the client needs to fetch and measure later on.

Because many websites dynamically generate content both on the client side (JavaScript) as well as on the server side, a browser needs to load the complete page. This phase loads the page by communicating to web servers, and HTTP requests are relayed via the client.

Transfer measurement phase

When the web page is fully loaded a complete list of URLs are available. The list of URLs, together with saved HTTP request headers, are sent to the client side so that the transfer measurements can be carried out. The client will imitate a browser by carrying out the transfers in parallel. Detailed measurements about TCP connection time, SSL negotiation time, waiting time and transfer time are saved on a per-object basis.

Replay phase

When the client has finished recording its measurements, it asks the offload server to replay the web site using the DNS lookup timing and transfer measurements. The server starts a new web browser session. This time, while in replay mode, it does not fetch anything from the internet or make any external communications. The HTTP layer is wrapped and for every object that is requested the data is taken from the data cache.

Whilst the page is being loaded in the browser during the replay phase, periodic screenshots of the browser window are taken every 200ms and stored in memory. When the page is fully loaded the screenshots are encoded in PNG format and stored on the file system.

Image and video generation phase

Screenshots are analysed to determine page load progress. Elapsed time for 'first initial', 'visually useful', and 'visually complete' are identified as well as the overall page load progress expressed as a percentage.

With the available information, a video is generated where screenshots are mapped to a high frame-rate video timeline. An on-screen display of ticking timestamp, page load progress, and measurement timestamps are also included. The video gives a visual feel of how fast the page is loaded.

HAR file generation phase

By combining all of the available measurements, an HTTP Archive (HAR) file is generated. The network characteristics from the user’s network, the HTTP request, and response header/body pair from the data cache, the replay report, as well as conclusions from the image analysis, are all used to produce the HAR file. It allows detailed inspection of the timing and complete content of the page load operation using third-party tools.

File upload phase

The video file and HAR file are too large to be stored in a database. They are also likely to become less relevant over time and my be purged after a period of time. The files are uploaded using unique test session prefix to a file storage repository. Each resource can be accessed via a public URL.

CSV result generation phase

The final output of the test is the CSV data on the client side, which consists of overall test results as well as per-object statistics which are later stored in a columnar database for data analysis via SamKnows One. 

Measure everything 

Top-level measurements for for data analysis:

  • Timestamp of the test
  • URL fetched
  • Total number of DNS requests made
  • Total number of objects transferred
  • Total download and upload volume
  • Number of successful objects
  • Number of failed objects
  • Peak number of concurrent HTTP requests
  • Name resolution service (DNS or DoH)
  • Average DNS lookup time
  • Distribution of DNS resolution time
  • Average TCP connection time
  • Distribution of TCP connection time
  • Average SSL connection time
  • Distribution of SSL connection time
  • Average object transfer time
  • Distribution of object transfer time
  • Time to first element being visible
  • Time to browser engine reported it was finished
  • Time to the page being visually complete
  • Distribution of the page loading time
  • Screen resolution
  • User agent

Per-object information captured:

  • Timestamp
  • Method (e.g. GET, POST)
  • URL
  • Transfer size
  • Transfer speed
  • HTTP version
  • HTTP response code
  • Server IP
  • Request start time
  • Request end time
  • DNS lookup time
  • TCP connect time
  • SSL connect time
  • Time to first byte

Video of the browser’s rendering of the page

Lastly, the SamKnows Website Performance Test also records a video of the browser’s rendering of the page (with a statistics bar overlaid onto it) and a HTTP Archive file (HAR), which can be analysed using common open source tools.

Analysing Website Performance Test data

As you can see, the SamKnows Website Test is the most advanced way to measure your website’s performance! Its ability to test from inside a consumer’s home network allows you to generate a truly representative view of their experience, factoring in the ISP, CDN relationships, and dozens of other factors. 

Key benefits

The SamKnows Website Test can also run on extremely low power hardware, which means that you can run tests on a significant scale too. Aggregating results across many homes provides a representative picture of page loading performance. And the videos and HTTP Archive files significantly aide troubleshooting and shortens the time it takes to resolve issues dramatically.

The SamKnows Website Performance Test also captures an HTTP Archive which highlights how individual objects are loaded. This is relevant when identifying bottlenecks or isolating an error.

Web browsing lite

Measures the time taken to fetch the HTML and referenced resources from a page of a popular website. This test does not test against centralised testing nodes; instead it tests against real websites, allowing for content distribution networks and other performance enhancing factors to be considered.

The test measures the following:

The time taken to download the page and its associated resources, the number of bytes transferred and the calculated rate per second will be recorded. The primary measure for this test is the total time taken to download the HTML page and all associated images, JavaScript and stylesheet resources. Optionally, the test can capture metrics for each object that is fetched too.

The results include the time taken for DNS resolution. The test can be optionally configured to use a DNS-over-HTTPS resolver for its DNS requests.