Beyond speed tests: Steam performance
Today we’re taking a look at how well the UK’s largest ISPs perform when users download games from Steam.
Anyone familiar with PC gaming will have come across Steam, the game distribution network operated by Valve. Steam was originally born out of the need to deliver patches to Valve’s own games in 2002, but rapidly expanded into an online store where games produced by Valve and other publishers were sold and distributed. And business is booming — as of 2017, approximately 18% of PC game sales were made using Steam, representing around $4.3 billion in revenue.
Steam’s rise to becoming the leading online game distribution platform was made possible thanks to the global rollout of high-speed broadband services. But the games delivered via Steam have increased in size dramatically over the years, with some games now exceeding 75GB! Even on a 100Mbps broadband connection, a 75GB game would take you around 2 hours to download (assuming you could consistently hit full available speed).
With over 150 million registered users and games measuring in tens of gigabytes, it is easy to see how Steam generates a lot of traffic. At the time of writing, Steam’s peak network utilisation was 5.7Tbps (5,700Gbps) over the past week. Some ISPs, such as MyRepublic in Australia, recognise the importance of Steam to gamers and use average Steam download speeds in their advertising.
How Steam delivers content
As might be expected, Steam makes extensive use of CDN (Content Delivery Networks) to deliver games to end users. In fact, Steam uses multiple different CDNs simultaneously. In our testing so far, we’ve found that Steam uses Level3, Akamai, EdgeCast (Verizon), Stackpath (formally Highwinds) and their own CDN for content delivery.
Steam breaks the world up into “cells” and maps countries and regions to these cells. For the curious among you, this mapping can be found here, and reveals a lot about their network topology. Interestingly, customisations exist for specific ISPs like Verizon, Telus, and Shaw. These ISP-specific customisations appear to bias users on these ISPs towards specific CDN locations rather than introduce caches inside their networks (like Netflix and YouTube do).
Each cell is allocated a list of approximately 20 servers, which may be distributed amongst multiple CDNs. For example, the UK is split into two cells — London and Manchester. Both are primarily served by Valve’s own CDN, with what appear to be backups using Stackpath (in Amsterdam) and Edgecast (in London).
The Steam client downloads a game in “chunks” using multiple parallel TCP sessions to a selection of the servers in the user’s cell. In our testing we found that the Steam client used six parallel TCP sessions. Using multiple parallel connections to multiple CDNs is a good way of working around congested paths between a user’s ISP and any one specific CDN.
How we measure Steam
We’ve developed a measurement client for Steam that can run on SamKnows Whiteboxes and routers running the SamKnows Router SDK. Our Steam client mimics the behaviour of the real Steam client, by logging into the Steam network, retrieving its local Cell ID, and then looking up a list of available servers and chunks for the cell the user resides in. We made use of an offload server to centrally collate server lists and chunk lists for each cell. Once the server lists and chunk lists have been obtained, the client fetches the chunks using six parallel TCP sessions for a fixed duration of ten seconds.
Our Steam measurement client captures the following information on each test run:
- The cell that served content for the user
- The average DNS lookup time for the servers in the cell
- The average round-trip latency to servers in the cell
- The total download speed across all parallel TCP sessions
How we used SamKnows One to analyse the results
As always, we used SamKnows One to analyse the measurement data we collect and then compare it to our existing measurements. SamKnows One is our powerful internet performance measurement platform that sits in the cloud, with the capacity to analyse, visualise, and store vast quantities of data. In this instance, we were interested in comparing the download speed achieved to Steam’s CDN against the download speed achieved to our own high-capacity test servers. This kind of analysis is easy to perform in SamKnows One, as multiple independent metrics can be added to a single chart.
How Steam measures up in the UK
We carried out measurements from approximately 200 Whiteboxes distributed across the fastest 76Mbps FTTC (VDSL) from BT, Sky, TalkTalk, and Vodafone, and 200Mbps and 350Mps cable products from Virgin Media. Measurements were carried out over a 4-day period in August 2018.
All of the major UK ISPs performed similarly when downloading from Steam. The Steam measurements for all ISPs recorded 50–70% of the speed of the regular download speed test. This suggests that the Steam CDN can struggle to saturate the link completely at higher speeds. Of course, in terms of absolute speeds, Virgin unsurprisingly rules the roost with their 200Mbps and 350Mbps products (50% of 350Mbps is a lot faster than 50% of 76Mbps!).
Figure 1 below shows the average download speed over Steam versus the average download speed to our dedicated servers.
Figure 1: SamKnows One
Similarly, there is no indication of any routing anomalies to the Steam CDNs, with all ISPs seeing round-trip latency in the low double digits. This can be seen in Figure 2 below, with a comparison to our latency measurement to our dedicated test servers.
Figure 2: SamKnows One
Measurements were distributed evenly between the London and Manchester Steam “cells”, and there was no discernible difference in performance between the two.
The sheer size of modern games means that the download speed that can be achieved for Steam will be something that gamers care a lot about. ISPs are already recognising this, and some have started to advertise on the back of high-level data that Valve publishes about performance to Steam.
Like other services we’ve looked at lately, Steam makes extensive use of CDNs to deliver content. Steam takes a subtly different approach to that of the video stream providers though — it downloads data from multiple different CDNs simultaneously. This will have little practical impact in a market like the UK, where peering between ISPs and content providers is generally open and very good. However, in other markets with less open peering arrangements (still common in South East Asia and South America), downloading from multiple different CDNs simultaneously may help offset poor performance to any one of them.
Here at SamKnows we will continue developing measurements that go beyond simple speed tests and measure the performance of real applications. It’s only by measuring the performance of real applications that we can accurately characterise user experience. Keep your eyes peeled for the next instalment in this series!