Helping care agents resolve customer queries faster
Helping consumers analyse their internet service over time
Helping consumers measure real time in-home performance
Helping network engineers monitor network performance in real time
Helping product managers monitor network performance over time
Helping regulators provide a handy speed testing app for their citizens
Overview of all the available SamKnows tests
We offer a variety of different mechanisms to measure speed
Measures the round trip time between the router and a test server
Measure the performance to real-world gaming servers and stores
Measure the performance of major social media services
Measure the performance of video conferencing services
Measure the performance of video streaming providers
Measure the performance of Content Delivery Networks
Measuring web browsing performance to real homes
Measure the performance of Voice over IP calls
Measuring the time taken to resolve a DNS query
Discover the path that traffic takes between a source and a destination
Passively measure the amount of data used on a broadband connection
We explore how communication technology is changing the world
Find out about our latest products and feature releases.
Listen to our latest podcasts
How our customers are using SamKnows’ technology
Find out about our company culture
Some handy resources if you are writing about SamKnows
Our company vision, our mission and the values we hold dear
Discover how we became the global measurement standard
Find out more about working at SamKnows
Find out what job opportunities we currently have at SamKnows
Get access to a demo version of SamKnows One
Help and advice for users of SamKnows One
Ever wondered how SamKnows makes a test? Jamie Mason, SamKnows’ head of measurement operations explains the meticulous way in which SamKnows tests are designed, engineered and executed to give you an insight into the detailed work that goes into creating each test, how these tests capture real-world performance, how they’re executed with zero impact on a consumer’s connection, and what happens to that test data once it’s collected.
Facebook, Zoom, WhatsApp, Fortnite, Netflix, Twitter, YouTube, FIFA... the list of services that SamKnows tests is as long as your arm. There are 23 different games alone.
What’s amazing about all these tests is that they don’t need the user to be running any of the software involved. Whether it’s the SamKnows Agent inside your router, or a Whitebox connected to it, SamKnows can accurately test your connection’s performance on, say, League of Legends – even if you’ve never heard of the game. And even if you’ve never owned a games console in your life.
This is all down to the meticulous way in which SamKnows tests are designed, engineered and executed. Here, we’re going to map the journey of just one of those tests – created for Call of Duty: Modern Warfare II – to give you an insight into the detailed work that goes into creating each test, how these tests capture real-world performance, how they’re executed with zero impact on a consumer’s connection, and what happens to that test data once it’s collected.
Jamie Mason
Our tour guide will be Jamie Mason, SamKnows’ head of measurement operations, who will take us along every step of the journey. This is the most detailed insight yet into how SamKnows delivers the real-world performance tests that are trusted by broadband companies and regulators all over the world.
This is down to the meticulous way that SamKnows tests are designed, engineered and executed
Before SamKnows reaches the point of designing a test, it first has to identify what it’s going to test in the first place. There are thousands of different games launched every year, for example. How does SamKnows decide which should earn a place in its wide portfolio of games tests?
“It’s very much a case of what is most appropriate in terms of popularity, in terms of the infrastructure that these games use, and also just how impacted they are by latency and packet loss,” said Jamie. “Is this a game where users will genuinely notice the difference if they have high latency?” Naturally, this means SamKnows primarily focuses on games with a significant online element. Games such as Fortnite, FIFA, and the one we’re focusing on for this Spotlight – Call of Duty: Modern Warfare II and its multiplayer online spin-off Warzone.
Just some of the services SamKnows tests
In any of these hugely popular games played by millions of people worldwide, any slowdown in server performance would dramatically degrade the online gaming experience, making them perfect candidates for SamKnows’ performance tests.
Once SamKnows has decided to add a game to its test roster, the next step is to begin dissecting the game, finding out exactly how its online component works. The task of discovering how each game works and, ultimately, of developing the test falls to the people responsible for the SamKnows agent, the Tests & SDK team. “Luckily for us, what this involves is buying, installing and playing the game,” said Jamie.
“It’s a question of trying to work out what’s going on under the hood,” said Jamie. “Is there anything special happening in terms of server discovery? Is the game client doing anything clever?”
Some games will publish a list of their online servers, making discovery easier for the team. This definitely isn’t the case with Call of Duty, however. “Unfortunately, for some of the most popular games it takes a lot of work to try to identify what the underlying server and infrastructure looks like. Not every game uses the same methods.”
Anti-cheat systems must be avoided
Call of Duty required “an absolute ton of work” from the SamKnows team to ensure the tests were accurate and not snagged by the various anti-cheating systems built into the game. The game was installed on a Windows PC and that was connected to another Linux PC, which captured the network traffic flowing from the game client out to the internet. “We had to do this because of the anti-cheat system that’s in place. A lot of these games will prevent us from being able to check the network traffic in the way that we need to, and so we had to come up with a clever workaround,” said Jamie.
Call of Duty required “an absolute ton of work” from the SamKnows team
That Linux machine was also used to make VPN connections. “We were trying to appear as if we were connecting from 94 different countries,” Jamie added. “Every time that we connected to the VPN and connected to a country, and then logged in on the game client, the game would contact a minimum of 15 or 16 different data centres in order to carry out tasks such as authentication. There’s lots of background communication going on, and that’s even before you actually get into the game itself.”
Consequently, the SamKnows team had to capture all that network data and figure out what was happening. “We’re able to build a picture of available servers and available IP addresses for us to use as test targets, and we then use a whole bunch of different methods to determine the location, the host and the provider for each server,” said Jamie.
“We might use geographic IP lookup systems to determine whether a particular IP address is in a particular country or a particular city, and we’ll do tracerouting, for example, from targets all over the world to correlate that information. We’re not just relying on a single source.”
The job doesn’t end once the team has established a list of target servers. Now they have to work out a way of pinging them, because SamKnows can’t install and run a full copy of the game on customers’ routers or Whiteboxes.
Finding target servers is a skilled task
Sometimes, that means creating their own data packets to match a particular protocol. For example, Fortnite doesn’t use the ICMP protocol that’s normally used to determine latency and availability of online hosts. “We discovered using our packing captures and looking at the way that the game communicates that there is actually a special protocol that’s been created by Epic Games, which makes Fortnite, and that’s used as an alternative to ICMP in order to communicate latency between client and servers. So, we actually have to craft our own packets that match this protocol in order to come up with a test that will work.”
To uncover target servers for a particular game, SamKnows has to install the game on a Windows PC and play as any normal player would. Network traffic is captured by a connected Linux PC and analysed. VPN servers are used to identify international servers.
On the journey so far, SamKnows has identified the game it wants to test (in our case, Call of Duty), scrutinised the game’s network behaviour to identify test servers, and written the necessary test code to accurately measure the performance of the game. What’s the next stop?
Next, all the server information is uploaded to SamKnows’ back-end server, so that when a test is run from a device in the consumers’ home, it can query that back-end and pull down an appropriate list of target servers to use for that test based on the consumer’s geographical location. That means, for example, a Call of Duty test run in Houston, Texas, will be pinging a completely different set of servers to a Call of Duty test being run from Riyadh in Saudi Arabia. That, according to Jamie, “enables us to say with a great deal of confidence that this is a really accurate portrayal of end-user experience”.
The tests are run to a schedule, but with millions of devices in homes all around the world, SamKnows takes care to ensure that the live game servers aren’t being hit by a flood of tests all being performed at the same time. “Generally speaking, when we want tests to run every hour, it’s the device itself that uses a particular algorithm to determine what time that actually translates to,” said Jamie.
Agents use the closest game server for tests
“So, my device might run a test at 16 minutes and 32 seconds past the hour, and then the device for my neighbour might run at 42 minutes and 12 seconds past the hour. There’s a bit more randomness, but that is essentially how it happens: a Whitebox or an embedded Agent has a test schedule, which is a list of tasks that it has to run to a particular frequency.”
We could have our device in an empty home and still be able to run all of our tests
However, before a test is performed, another crucial check happens. “When the time comes to actually run this test, we don’t run it straight away. We’ll check it’s appropriate for this test to be running, and by that I mean are we about to interfere with someone’s internet connection? Alternatively, is the use of the internet connection about to interfere with our test? It works both ways.” This is called cross-traffic detection, and it ensures the volume of traffic being carried over the home network won’t disrupt the tests, or the consumer’s experience.
“All we’re doing here is checking to see how busy the network is,” said Jamie. “We don’t care what the traffic actually is. Sometimes, when we’re explaining how this works to clients or consumers, they ask ‘how do you know when I’m running Netflix so that your Netflix test picks this up?’ We have to explain that, actually, it doesn’t matter.
We can run our Netflix test and it has nothing to do with whether you’re watching Netflix or whether you even have a Netflix account. We could have our device in an empty home with no users who don’t have accounts for anything, and we’d still be able to run all of our tests.”
Once a set of tests is completed, SamKnows is naturally keen to get those results into its data pipeline as soon as possible.
One of the strengths of the SamKnows system is that it can discover faults in a broadband provider’s network before the provider’s own systems even report a problem; but, again, consideration is given to the amount of traffic being run over consumers’ networks. “When a test runs, the output is stored on the local device and, if possible, sent back to us as quickly as it can be,” said Jamie. “Sometimes, we’ll store the results for a short while, especially if we’re running a lot of tests. On a particularly poor line, where the device can’t actually connect to our back- end systems, it will store the test results for longer and we’ll keep trying to send that data back to us.”
Similarly, SamKnows has to be careful not to effectively create a denial-of-service attack on its own network! “We have a huge number of Agents, tens of millions of devices, and they’ve been designed to not overwhelm us with communication, but they do need to be sending us test results as soon as they realistically can, because we want to provide a near real- time view of what’s going on.”
Test results travel back to base fast
The results are sent back to SamKnows’ edge servers in a specifically crafted format, where they enter the data pipeline. “Our data pipeline in 2023 is a very interesting and involved thing,” said Jamie. “There are a lot of APIs that are now part of this process. We make heavy use of tools such as Kafka and Flume in order to manage and manipulate the data in certain ways, to ensure that we’re getting all the metadata we need and that everything is formatted correctly. Then, ultimately, it all ends up in our big data storage and we use Google’s BigQuery for this. This is essentially the final resting place for this data. The data isn’t going to change anymore.”
Test results are sent back quickly “to provide a near real-time view of what’s going on
The flowchart below shows how data moves from the Agents performing the tests on consumers’ devices, through multiple steps in SamKnows cloud infrastructure, before it’s finally accessible to customers via the SamKnows One product or via the Data API.
The results data may have reached the end of its journey, but that’s not the end of the story. In fact, this is where things start to get interesting for SamKnows’ clients, because they get to see what all these billions of test results show.
First, there’s the SamKnows One analytics platform, which “allows us to produce a huge variety of graphs and other methods of using statistical analysis on the data that we store,” according to Jamie.
“This is used really heavily by us internally, and also by many of our clients, and by our end users since it allows you to see what your connection looks like over time. It allows you to drill down to individual results. It allows you to compare, say, your FIFA latency to your Call of Duty latency and also to your standard SamKnows latency metrics. It’s a great tool because it’s really easy to use.”
SamKnows also has its own analysts, who prepare data on behalf of clients such as telecoms regulators – data that eventually helps consumers make better choices about their broadband suppliers. The analysts aggregate the data and perform “very heavy-duty number crunching, querying the raw data directly and going through an awful lot of statistical analysis that’s beyond my ability. I have great respect for it, but at some point my brain starts shutting down!” said Jamie.
Customers can see results in SamKnows One
That doesn’t prevent Jamie from wanting to add more and more tests to the portfolio. “I know we’ve talked a lot about speed tests in the past, but if you’re like me, then I’m not downloading that often,” he said. “I’m watching a lot of Netflix, I’m watching a lot of YouTube, I’m playing a lot of video games, and it just so happens that every single one of those things is something that we have a test for. We’re able to produce accurate data that looks a lot like what I’m actually experiencing. There is a direct correlation between my performance when I’m trying to browse Twitter, or watch YouTube or play Call of Duty, and what my SamKnows device is actually telling me is going on as well.
“So it’s very much about representing real-world performance and representing what a consumer is genuinely doing, or what one of our volunteers is experiencing when they’re using the internet.”
Jamie Mason, our Head of measurement operations gives us a detailed description of how SamKnows’ performance tests are designed, executed and analysed.
