Although it is quite literally all fun and games, the world of gaming is serious business. It’s likely that you know at least one person that plays on the regular and just as likely that they dedicate a lot of time to it. And they’re right to! Not only are online games a generally fun and immersive experience, but gaming is a huge industry, with massive potential. The gaming industry itself is twice as big as the movie industry, and ten times bigger than the music industry, raking in $180 billion dollars a year and rising. It’s also estimated that 700 million people play online games worldwide, and on average, video gamers spend over 6 hours playing games each week.
“Fast performance is critical to gamers, being cited as the most important aspect of game play across all age groups”
Online Gaming: where latency is life or death
Online multiplayer gaming is much more sensitive to variation in internet performance than say video streaming. While Netflix buffers several minutes of video in order to smooth out any fluctuations in performance, online gaming requires every action from each player to be transmitted to a central gaming server and then back to all the other players in the game, in milliseconds. So, any spike in latency or packet loss could mean the difference between life and death - for your computer game character. As you can tell, the stakes are pretty high, but for some, they’re even higher. In the world of competitive gaming, or ‘esports’, there are annual international tournaments with prize pools that rack up to the hundred of thousands for the winners. For competitors practicing at home, latency or packet loss doesn’t just mean virtual life or death, but also a very real threat to potential winnings.
Understanding gaming performance
The only way to understand actual gaming performance is to test against real gaming servers from real homes. Game developers often show this micro information in-game, but to understand gaming performance on a macro level, we needed to start building our own gaming tests. As we began creating game tests, we realized the process was tricky. Anti-cheating protocols meant finding the correct servers to test against were not so straightforward.
Gaming servers are spread around the world in zones
While observing games and the traffic they generate, sometimes we’d find special packets that the game publishers use to monitor their own infrastructure, and by mimicking these packets we were able to test performance all the way to the application layer inside the gaming infrastructure. Otherwise we can use a standard ICMP (ping) latency test to measure the network infrastructure directly in front of the game servers.
Our gaming tests
As you may know, we run Quality of Service tests (QoS) to test your ISP’s network performance, and Quality of Experience tests (QoE) that test your ISP + a specific application’s performance. Such as Zoom, Netflix, Instagram, etc. For gaming, we use the same targeted QoE approach. Each test calculates which gaming server would be used in actual gameplay from the test agents location, then measures round-trip latency and packet loss from the home to the gaming server hourly.
The difference between gaming and speed tests
Our standard ICMP (ping) latency test measures performance from the home to the nearest test server. An online game uses gaming servers that are spread around the globe, but depending on your location and the gaming infrastructure of your game, they may be many miles away or even in a different country. Even if your standard latency results are good, it doesn't mean your online gaming performance will be the same. It can be useful to compare game latency and loss to test server latency and loss, if there's any change in the comparative difference it can help diagnose if there's a problem with the game's performance or the internet service.
Visualising game test data
Each game produces multiple metrics. At the moment we can individually chart these metrics in SamKnows One, but we’re working on new widgets and ways to visualise all of each game’s metrics in one place.
On average we’re building around 2 more game tests each month, so we’re very involved in the development of gaming. To show how hands-on our gaming tests team is, we spoke to them about their personal experience with building gaming tests and their relationship with gaming as a result.
One of our Test Engineers, explained that one of the most annoying game tests to develop was League of Legends. Every time he started a game to compare in-game telemetry to out test measurements, he had to actually play the game. “You cannot leave a League of Legends game until it ends, otherwise they ban you from playing for a while, they even recognise if you don't move your character around and ban you.” Alternatively, after testing within PlayerUnknown's Battlegrounds, he loved the game so much he bought it himself to play at home.
These are some of the games we've built so far
Building tests with diligence and care
Streamline maintenance
Gaming infrastructure changes a lot as new servers are added and old ones removed. We automatically detect test failures on the 1000’s of servers we target to make sure we’re hitting the right servers all the time.
Be as accurate as possible
We compare the in-game measurements vs. the results our own tests generate
Documentation
We keep records of how the gaming infrastructure works for each game
Clean Code
We make sure everything adheres to our own set of in-house coding rules
Our unique gaming dataset
SamKnows internet measuring platform has millions of test agents all over the world that can test gaming performance to real game servers. Everyone’s connection to an online game is unique, by measuring from millions of homes and aggregating the gaming data our customers can do some really interesting things with the data.
Government regulators use our gaming data to lobby ISPs and transmit providers to improve connections to popular gaming servers, and also game developers to move servers closer to their country.
Gaming report for Saudi Arabia - View PDF
Regulators publish reports, benchmarking the comparable performance of gaming services and the gaming performance of major ISPs and regional variations.
For major ISPs, good gaming performance is good for business, we’ve seen our gaming reports impact ISP share prices when they’re released. We encourage our ISP customers to start rolling out their own gaming test suite to optimise the gaming experience for their customers!
As a testament to how involved we are in gaming, one of our other Test Engineers developed a game of his own. A sweet counting dinosaur game for his son who loves dinosaurs. https://www.youtube.com/watch?v=WZyuaUp8wBc