Just like how you update your phone and your applications, routers need updating too. As we have Agents in tens of millions of routers worldwide, in order to push new tests and test enhancements, we used to have to wait for large-scale firmware updates. ISPs upgrade the firmware on their routers by a fixed schedule, usually dictated by wider concerns of the manufacturer – who is also trying to release new features.
These upgrades typically occur only once or twice a year, which causes a problem for us because we release new tests and enhancements very quickly, often daily. Think of an agile company trying to fit into a waterfall world! Waiting for a firmware update to come around means missing out on months of interesting quality of experience data!
We realized we needed to develop a solution so we could continuously release our own features out-of-cycle with the manufacturers.
Getting ahead of schedule
We looked at how other teams had approached this problem when faced with similar challenges. Smartphone apps, which are remotely managed, stuck out to us. A distributed population of devices running on different hardware and operating systems, but capable of being updated at any time? That’s what we were after! We looked to emulate this model, and to think about our Agent as an App which sits on the router. We rewrote our tests so we can push updates as we develop them, without having to wait for the upgrade cycle to come around – which is especially important for Quality of Experience (QoE) measurements.
Many other speed tests on the market don’t ever change, even if they’re incorrect! We consistently maintain all of our tests to ensure they remain perfectly accurate, never compromising on quality. But for QoE measurements we need to be developing (and releasing) tests very quickly. We’re fortunate to have relationships with application providers, video streaming companies, video conferencing providers, gaming companies, etc., which means we get a heads-up about significant updates to make sure we’re synchronizing our development schedule for important releases and launches. Tomorrow’s most popular game may not even have been launched yet! But, when it’s available and the whole world starts playing, we need to have a test ready to go ASAP so we capture its performance from day one.
Single gaming test framework
To stay ahead of the curve, we created a way to embed a single gaming test framework inside of routers. This means, instead of developing a new program update for each game, we have one program that automatically fetches a configuration file from our servers, listing all of the endpoints to measure against for all our gaming tests. This means we can update end points when gaming infrastructure changes, and add new games almost instantly.
It now takes us longer to download a game than it takes to make a new game test and push it out to millions of SamKnows enabled routers.
Key advantages:
- Add new games instantly
- Update game tests if gaming infrastructure changes
- The game tests can now be coded in JSON which is less error prone than C++
Future plans
Currently, we’re porting this new gaming test framework to our Android and iOS Agents, to stay on top of crucial performance measurements on the go.
Next, we’re going to expand the scope to include all our other latency tests, allowing us to configure unique and complex latency studies on behalf of both our ISP and government clients.
In the future, we’re working towards a Super Agent. 100% updated dynamically across all the ~40 million devices it’s installed in around the world as we release new tests on a daily basis.. Our work never stops for an earth that never shuts off!