The journey of a test
Facebook, Zoom, WhatsApp, Fortnite, Netflix, Twitter, YouTube, FIFA... the list of services that SamKnows tests is as long as your arm. There are 23 different games alone.
What’s amazing about all these tests is that they don’t need the user to be running any of the software involved. Whether it’s the SamKnows Agent inside your router, or a Whitebox connected to it, SamKnows can accurately test your connection’s performance on, say, League of Legends – even if you’ve never heard of the game. And even if you’ve never owned a games console in your life.
This is all down to the meticulous way in which SamKnows tests are designed, engineered and executed. Here, we’re going to map the journey of just one of those tests – created for Call of Duty: Modern Warfare II – to give you an insight into the detailed work that goes into creating each test, how these tests capture real-world performance, how they’re executed with zero impact on a consumer’s connection, and what happens to that test data once it’s collected.
Our tour guide will be Jamie Mason, SamKnows’ head of measurement operations, who will take us along every step of the journey. This is the most detailed insight yet into how SamKnows delivers the real-world performance tests that are trusted by broadband companies and regulators all over the world.
This is down to the meticulous way that SamKnows tests are designed, engineered and executed
Deciding what to test
Before SamKnows reaches the point of designing a test, it first has to identify what it’s going to test in the first place. There are thousands of different games launched every year, for example. How does SamKnows decide which should earn a place in its wide portfolio of games tests?
“It’s very much a case of what is most appropriate in terms of popularity, in terms of the infrastructure that these games use, and also just how impacted they are by latency and packet loss,” said Jamie. “Is this a game where users will genuinely notice the difference if they have high latency?” Naturally, this means SamKnows primarily focuses on games with a significant online element. Games such as Fortnite, FIFA, and the one we’re focusing on for this Spotlight – Call of Duty: Modern Warfare II and its multiplayer online spin-off Warzone.
In any of these hugely popular games played by millions of people worldwide, any slowdown in server performance would dramatically degrade the online gaming experience, making them perfect candidates for SamKnows’ performance tests.
Dissecting the game
Once SamKnows has decided to add a game to its test roster, the next step is to begin dissecting the game, finding out exactly how its online component works. The task of discovering how each game works and, ultimately, of developing the test falls to the people responsible for the SamKnows agent, the Tests & SDK team. “Luckily for us, what this involves is buying, installing and playing the game,” said Jamie.
“It’s a question of trying to work out what’s going on under the hood,” said Jamie. “Is there anything special happening in terms of server discovery? Is the game client doing anything clever?”
Some games will publish a list of their online servers, making discovery easier for the team. This definitely isn’t the case with Call of Duty, however. “Unfortunately, for some of the most popular games it takes a lot of work to try to identify what the underlying server and infrastructure looks like. Not every game uses the same methods.”
Call of Duty required “an absolute ton of work” from the SamKnows team to ensure the tests were accurate and not snagged by the various anti-cheating systems built into the game. The game was installed on a Windows PC and that was connected to another Linux PC, which captured the network traffic flowing from the game client out to the internet. “We had to do this because of the anti-cheat system that’s in place. A lot of these games will prevent us from being able to check the network traffic in the way that we need to, and so we had to come up with a clever workaround,” said Jamie.
Call of Duty required “an absolute ton of work” from the SamKnows team
That Linux machine was also used to make VPN connections. “We were trying to appear as if we were connecting from 94 different countries,” Jamie added. “Every time that we connected to the VPN and connected to a country, and then logged in on the game client, the game would contact a minimum of 15 or 16 different data centres in order to carry out tasks such as authentication. There’s lots of background communication going on, and that’s even before you actually get into the game itself.”
Consequently, the SamKnows team had to capture all that network data and figure out what was happening. “We’re able to build a picture of available servers and available IP addresses for us to use as test targets, and we then use a whole bunch of different methods to determine the location, the host and the provider for each server,” said Jamie.
“We might use geographic IP lookup systems to determine whether a particular IP address is in a particular country or a particular city, and we’ll do tracerouting, for example, from targets all over the world to correlate that information. We’re not just relying on a single source.”
The job doesn’t end once the team has established a list of target servers. Now they have to work out a way of pinging them, because SamKnows can’t install and run a full copy of the game on customers’ routers or Whiteboxes.
Sometimes, that means creating their own data packets to match a particular protocol. For example, Fortnite doesn’t use the ICMP protocol that’s normally used to determine latency and availability of online hosts. “We discovered using our packing captures and looking at the way that the game communicates that there is actually a special protocol that’s been created by Epic Games, which makes Fortnite, and that’s used as an alternative to ICMP in order to communicate latency between client and servers. So, we actually have to craft our own packets that match this protocol in order to come up with a test that will work.”
How SamKnows discovers test servers
To uncover target servers for a particular game, SamKnows has to install the game on a Windows PC and play as any normal player would. Network traffic is captured by a connected Linux PC and analysed. VPN servers are used to identify international servers.
Running the tests
On the journey so far, SamKnows has identified the game it wants to test (in our case, Call of Duty), scrutinised the game’s network behaviour to identify test servers, and written the necessary test code to accurately measure the performance of the game. What’s the next stop?
Next, all the server information is uploaded to SamKnows’ back-end server, so that when a test is run from a device in the consumers’ home, it can query that back-end and pull down an appropriate list of target servers to use for that test based on the consumer’s geographical location. That means, for example, a Call of Duty test run in Houston, Texas, will be pinging a completely different set of servers to a Call of Duty test being run from Riyadh in Saudi Arabia. That, according to Jamie, “enables us to say with a great deal of confidence that this is a really accurate portrayal of end-user experience”.
The tests are run to a schedule, but with millions of devices in homes all around the world, SamKnows takes care to ensure that the live game servers aren’t being hit by a flood of tests all being performed at the same time. “Generally speaking, when we want tests to run every hour, it’s the device itself that uses a particular algorithm to determine what time that actually translates to,” said Jamie.
“So, my device might run a test at 16 minutes and 32 seconds past the hour, and then the device for my neighbour might run at 42 minutes and 12 seconds past the hour. There’s a bit more randomness, but that is essentially how it happens: a Whitebox or an embedded Agent has a test schedule, which is a list of tasks that it has to run to a particular frequency.”
We could have our device in an empty home and still be able to run all of our tests
However, before a test is performed, another crucial check happens. “When the time comes to actually run this test, we don’t run it straight away. We’ll check it’s appropriate for this test to be running, and by that I mean are we about to interfere with someone’s internet connection? Alternatively, is the use of the internet connection about to interfere with our test? It works both ways.” This is called cross-traffic detection, and it ensures the volume of traffic being carried over the home network won’t disrupt the tests, or the consumer’s experience.
“All we’re doing here is checking to see how busy the network is,” said Jamie. “We don’t care what the traffic actually is. Sometimes, when we’re explaining how this works to clients or consumers, they ask ‘how do you know when I’m running Netflix so that your Netflix test picks this up?’ We have to explain that, actually, it doesn’t matter.
We can run our Netflix test and it has nothing to do with whether you’re watching Netflix or whether you even have a Netflix account. We could have our device in an empty home with no users who don’t have accounts for anything, and we’d still be able to run all of our tests.”
Sending back the results
Once a set of tests is completed, SamKnows is naturally keen to get those results into its data pipeline as soon as possible.
One of the strengths of the SamKnows system is that it can discover faults in a broadband provider’s network before the provider’s own systems even report a problem; but, again, consideration is given to the amount of traffic being run over consumers’ networks. “When a test runs, the output is stored on the local device and, if possible, sent back to us as quickly as it can be,” said Jamie. “Sometimes, we’ll store the results for a short while, especially if we’re running a lot of tests. On a particularly poor line, where the device can’t actually connect to our back- end systems, it will store the test results for longer and we’ll keep trying to send that data back to us.”
Similarly, SamKnows has to be careful not to effectively create a denial-of-service attack on its own network! “We have a huge number of Agents, tens of millions of devices, and they’ve been designed to not overwhelm us with communication, but they do need to be sending us test results as soon as they realistically can, because we want to provide a near real- time view of what’s going on.”
The results are sent back to SamKnows’ edge servers in a specifically crafted format, where they enter the data pipeline. “Our data pipeline in 2023 is a very interesting and involved thing,” said Jamie. “There are a lot of APIs that are now part of this process. We make heavy use of tools such as Kafka and Flume in order to manage and manipulate the data in certain ways, to ensure that we’re getting all the metadata we need and that everything is formatted correctly. Then, ultimately, it all ends up in our big data storage and we use Google’s BigQuery for this. This is essentially the final resting place for this data. The data isn’t going to change anymore.”
Test results are sent back quickly “to provide a near real-time view of what’s going on
The game data’s return journey
The flowchart below shows how data moves from the Agents performing the tests on consumers’ devices, through multiple steps in SamKnows cloud infrastructure, before it’s finally accessible to customers via the SamKnows One product or via the Data API.