SamKnows web test for super-fast speeds

A behind-the-scenes look at how we develop new measurements. Here's our response to traditional web tests struggling to keep up with faster internet speeds. Read all about how we ensured the new SamKnows Web Test consistently provides accurate results for any speed tier on any browser - laying the groundwork for speeds greater than 1Gbps.

At SamKnows, we’re forever striving for a faster and more reliable internet. People from all over the world come to us to test their innovations to improve future internet, and we love helping world-leading ISPs trial and develop their new products. Excitingly, we’re testing faster and faster connections. But importantly, we’re finding that traditional web tests struggle to handle such speeds, and their performance can vary depending on the browser or computer its running on. So, six months ago, we decided to create a SamKnows Web Test, making it sensationally accurate and fully capable of handling these faster speeds.

In doing so, we carried out a huge amount of research and testing to ensure that the new web test always provides the most accurate results for any speed tier on any browser.

SamKnows Lab

First, we wrote four different speed tests. We then ran them under the same conditions in every browser, so that we could work out which test results were the most reliable. We decided that four download and three upload test results looked viable.

Next, we ran our favoured browser test on three different internet speeds on seven different browsers. Figure 1 shows the results for the download speed test:

Figure 1: Source SamKnows

As you can see, Fetch streaming performed best in all browsers, with the exception of Edge, where websockets performed best. Upon this realisation, we decided that we were so determined to provide the most accurate results in every browser that we would write two download speed tests. The web-based test automatically detects the browser that it’s opened with, and then runs the relevant version of the web test for that browser. Thankfully with the upload test, we found that websockets were the best across the board, so we only had to write one of them!

Finally, it was time to put our new web test through our quality assurance checks. We tested it against nine different speed profiles, in seven different browsers, in three different pages. We then repeated this pattern three separate times, using our SamKnows QA lab, which includes equipment with extremely fast connectivity and traffic-shaping abilities. This meant that we could be sure of what internet speeds the web test should be recording. 

After a few solid weeks of testing, we were confident that we had achieved our goal and were ready to deploy the web test to production.

“Field Testing”

On the 9th November, BBC’s Watchdog ran a story about internet performance. Starting at 8pm, we received data from over 15,000 speed tests. In fact, Figure 2 shows us that speed tests were run from the moment that download speeds were discussed on the show, giving us a wealth of data to explore.

Figure 2: Source SamKnows One

Important: Before we begin exploring some of the results, it is important to note that all web browser based tests are subject to a number of limitations, which may make the data unreliable for certain use cases. Firstly, web browsers do not provide a way for us to determine whether the user’s device is connected via Wi-Fi or ethernet (although the Network Information API is slowly being adopted), and often Wi-Fi can be the limiting factor at higher speeds. Secondly, the age and performance of a user’s device (along with their web browser) can have a significant bearing on the results. Lastly, whilst it is possible to accurately determine the user’s ISP (via their public IP address), we cannot automatically know what product or access technology they are using. Many ISPs use a blend of access technologies (e.g. ADSL and FTTC/VDSL), and these results are all conflated together with web based tests due to this lack of product information. Our web based test is subject to all of these limitations, as is every other web based test on the market. In our view, web based tests are useful for individual users troubleshooting their own performance. They are also useful for extremely coarse statistics (e.g. country level averages) when you have a very large sample. But they are far less well suited to providing granular comparisons (e.g. by ISP, product or region) when statistical robustness is an objective.

SamKnows One, our colourful cloud-based performance platform, is designed to carry out such a data exploration, and can automatically identify the ISP that the speed test measures. In Figure 3 we can see a table summarising the results of some of the largest ISPs in the UK. The mean is a simple average over the entire dataset while the median and the 80th percentile shows the speed that 50% and 80% of the tests achieved respectively.

Figure 3: Source SamKnows One

If we take a look at Figure 4, we can see that all the ISPs’ upload speeds performed similarly. However, the Post Office only reached humble heights. Interestingly, Hyperoptic, for which we had one outlier result and so not included in Figure 4, was recorded as having a 1Gbps upload! In fact, another Hyperoptic home recorded a high of 940.47Mbps, using Microsoft Edge on Windows 10. We were thrilled to see that gigabit speeds were being successfully measured from a browser.

Figure 4: Source SamKnows One

However, when it came to download, we didn’t hit any gigabit speeds. But we did later on. In fact,we mostly saw speeds between 13Mbps and 60Mbps, matching the most common UK ISP broadband packages. The box plots shown in Figure 5 demonstrate how ISPs, excluding FTTP and Cable providers, are providing pretty similar speeds. The highest download speed recorded was 779.26Mbps by someone using Chrome on Windows 10 and Vodafone Broadband. And since then we’ve even seen download speeds exceed 900Mbps, with a maximum of 924.34Mbps, with almost all of those top 900+ measurements in Chrome.

Figure 5: Source SamKnows One

Figure 6: Source SamKnows One

Finally, our latency and jitter test also saw a range of results, with some as low as 3.3ms and others as high as 165ms, with 60% of the results between 14ms and 30ms, which is pretty standard and not cause of concern.

Figure 7: Source SamKnows One

The extensive development and testing work we’ve carried out on our updated web based test has laid the groundwork for us to be able to measure performance beyond 1Gbps from a web browser. Of course, this brings many other performance challenges (both inside and outside of the home). But we are fortunate enough to be working in a variety of countries where services over 1Gbps are already available to consumer’s homes (some countries can get 10Gbps FTTH!) and have already been carrying out such tests. Stay tuned to hear more about the challenges we’ve faced, how we’ve overcome them and the results that we’re seeing!

And if you’re interested — give the web-test a go: