In this first of a three-part series about our new Website Performance Test, we’ll explain the problems with conventional testing that we’ve set out to fix, and why we believe it’s important to do so.
In the next two posts, we’ll detail the technical methodology and then we’ll show you how we’ve already identified a widespread, unseen fault which was impacting the experience of millions of Sky customers. Now fixed by Sky.
Identifying unseen faults
At SamKnows, we’ve been hard at work developing a new website performance test, which we’ve imaginatively named the SamKnows Website Performance Test. For this, we use the engine of a real web browser to measure how long a web page takes to render from a real person’s home.
While the concept of instrumenting a real web browser to run performance and availability measurements for websites isn’t a new one - there are many commercial services that offer this already - these existing services all run tests from high-capacity data centres.
A typical website performance test
This is useful to a degree - they tell you how fast your website is and whether it is accessible at a very basic level. But it is also completely unrepresentative of real user experience. Real users do not sit inside data centres on 1Gbps connections when browsing the internet! Instead, they sit at home on their cable or DSL connection, or use their mobile phone.
The SamKnows Website Performance Test carries out its measurements from a SamKnows enabled device (such as a Whitebox or a CPE with the SamKnows Router Agent embedded) inside the user’s home.
The SamKnows Website Performance Test
This approach means that we can accurately simulate a consumer’s actual web browsing experience, including dozens of factors that tests using a data centre cannot.
For example, the SamKnows Website Performance Test considers:
- The user’s internet connection performance
- Their ISP’s core network and routing
- Their ISP’s DNS infrastructure
- Any CDN relationships that their ISP and/or the content provider has
- How their ISP connects with the content providers
Our original test for measuring web page loading time, first produced in 2011, fetched an HTML page and all referenced objects. This was fine for the time, but is now insufficient. Crucially, it does not execute JavaScript (which may lead to further requests), use a real browser engine, or actually render the page - so it has no concept of what objects within the page make a difference to the user’s visual experience. The SamKnows Website Performance Test addresses all of these issues.
Real customer experience
The purpose of the SamKnows Website Performance Test is to capture the user experience of a website by testing from inside the user's own network. Since the browser is a graphical program that renders the page step by step as objects are received from the network, the results must reflect what can be seen on the screen.
The goal of the test is to highlight how long someone has to wait before they can start accessing the contents of the page. However, not all objects for a web page have visual relevance. Some objects are invisible, such as an advertising beacon, while others have less impact on the visuals overall. Capturing a “visual loading progress” is a useful metric that allows us to track how much of a page is visible to the user throughout the few seconds that a page is loading.
In addition to recording the overall web page loading time, we can also measure the performance of specific objects, which can then reveal why the page is slow to load, or even broken (e.g. a slow-to-download JavaScript file may slow down the entire page load, but a slow-to-download GIF may not).
Today, many websites require access to multiple web servers hosted with different providers in different physical locations. A problem with just one of these providers could significantly increase page loading times. Some objects (e.g. HTML, CSS, or JavaScript) have other objects as dependencies, and if they are the bottleneck, can further impact the page load time. By identifying where the slowdown occurs, you can improve the decisions you make for enhancing the network or increasing server performance.
Massive data processing capacity
In an ideal world we would be able to instrument real user’s web browsers to run measurements on-demand or according to a schedule for us. Of course, because of technical limitations and privacy concerns (we take privacy very seriously) we cannot do that, and even if we could, we would still be subject to the differing performance of users’ computers, poor in-home Wi-Fi, and many other confounding factors.
What we do have are many millions of small embedded Linux devices (Whiteboxes and home routers with our Router Agent integrated). These are all directly connected to the internet and run the same code, with similar processing power. The relatively low-power nature of these devices mean that we cannot run a full-blown web browser on them though. To resolve this, we introduce the concept of the SamKnows Offload Server Farm.
The server runs the actual browser while the device in the user’s home only performs time measurements between its network and the web servers. These measurements include DNS resolution time, TCP connection time, SSL/TLS handshake time, object transfer time, and more. The offload server makes sure that the web page with all its objects are loaded exactly according to the measured timings from the user's network.
Since web servers may produce content differently depending on the geographical location of the user, the offload server never talks directly to any web server (e.g. to decide which objects to measure). Instead, a tunnel is established between the user's router and the offload server so that all traffic is originated from the user's own network.
The offload server uses an instrumented web browser that fetches its content from an artificially delayed local cache (delayed according to the timings obtained from the device in the user’s home). The result is a web browsing measurement that gives results that depict exactly what a real user would see if they loaded the same web page on their broadband connection. When scaling up the test to cover thousands of user networks, a good overall picture of browsing experience over a large geographical area, or an ISP as a whole, can be achieved.
By scaling the test to cover thousands of home networks, you gain valuable insight into how people experience your website according to their location and ISP. The test can potentially be run in any broadband router, which is far more than what would be possible when testing with a selected few real user devices. On top of this, by periodically testing a large number of user networks, you can see how performance varies over time. This is far more than what would be possible when testing with a small number of consumers’ computers.
Measure everything
There are three different types of test output from the SamKnows Website Performance Test. The main type is results that are stored in a database to enable queries and visualisation in SamKnows One. It includes top-level information as well as per-object statistics. This is useful to see the big picture among thousands of test results.
The table below shows the metrics used for top-level results that are relevant for data analysis:
- Timestamp of the test
- URL fetched
- Total number of DNS requests made
- Total number of objects transferred
- Total download and upload volume
- Number of successful objects
- Number of failed objects
- Peak number of concurrent HTTP requests
- Name resolution service (DNS or DoH)
- Average DNS lookup time
- Distribution of DNS resolution time
- Average TCP connection time
- Distribution of TCP connection time
- Average SSL connection time
- Distribution of SSL connection time
- Average object transfer time
- Distribution of object transfer time
- Time to first element being visible
- Time to browser engine reported it was finished
- Time to the page being visually complete
- Distribution of the page loading time
- Screen resolution
- User agent
Secondly, the SamKnows Website Performance Test also captures information on a per-object basis during a test session. Given that a web page can often reference dozens or even hundreds of objects, this means that we collect a large amount of data! The following information is captured on a per-object basis:
- Timestamp
- Method (e.g. GET, POST)
- URL
- Transfer size
- Transfer speed
- HTTP version
- HTTP response code
- Server IP
- Request start time
- Request end time
- DNS lookup time
- TCP connect time
- SSL connect time
- Time to first byte
Lastly, the SamKnows Website Performance Test also records a video of the browser’s rendering of the page (with a statistics bar overlaid onto it) and a HTTP Archive file (HAR), which can be analysed using common open source tools. We will show you an example of this further down.
A first look at some results
We have been trialling the SamKnows Website Performance Test on one of our R&D panels of broadband connections in the UK, comparing page loading time on a number of popular websites. The graphs in this section are based upon results collected in February and March 2019, from tests run against skyscanner.net (a popular flights comparison website).
Comparing website loading experiences for different ISPs
Source SamKnows One
Comparing website loading experiences for different ISP products
Source: SamKnows One
Comparing Page Load Time for popular sites
Source: SamKnows One
This graph shows the time it takes for a website to display its first visual compared to other popular websites - an indication of how the website compares to the expectations of most users.
As mentioned earlier, the SamKnows Website Performance Test also captures a video of the page loading in the web browser. This shows graphically how different objects in the page are loaded, with a bar overlaid at the top highlighting key page loading statistics. Instead of just providing a number the video can be used to evaluate how a human would experience loading the website.
Visual progress is calculated against the finished page, and for each new update intermediate progress is shown as a percentage.
Note that all the SamKnows Website Performance Test measurements load everything afresh which provides for comparability. Using a browser on a laptop that recently loaded the page a local cache may be used.
The SamKnows Website Performance Test also captures an HTTP Archive which highlights how individual objects are loaded. This is relevant when identifying bottlenecks or isolating an error. It is normally presented as a waterfall diagram using Firefox or Chrome.
Proof of concept
As you can see, the SamKnows Website Test is the most advanced way to measure your website’s performance! Its ability to test from inside a consumer’s home network allows you to generate a truly representative view of their experience, factoring in the ISP, CDN relationships, and dozens of other factors.
The SamKnows Website Test can also run on extremely low power hardware, which means that you can run tests on a significant scale too. Aggregating results across many homes provides a representative picture of page loading performance. And the videos and HTTP Archive files significantly aide troubleshooting and shortens the time it takes to resolve issues dramatically.
This is the first in a series of blog posts about the SamKnows Website Performance Test. The upcoming posts will cover a more in-depth view on how it works and how test result data can be used for troubleshooting. If you would like to speak to a member of our team to find out more, please contact us here.