Spotlight: From startup to global leader, SamKnows' data journey

Where did the Agent come from?

When SamKnows first burst into life in 2008, it was hard to find one reliable measurement of broadband performance. One that wasn’t tarnished by other traffic on the network, the speed-sapping effects of Wi-Fi, or simply because it was testing only the speed between the user and a test server a few miles down the road.

Skip ahead 15 years and SamKnows is taking so many real-world performance measurements every day that chief technology officer Sam Crawford has lost count. “Embarrassingly, I don’t know the current volume of collections per day,” he admits. “It’s well into the billions of measurements per day, though.”

What’s more, each of those billions of performance tests is measuring something real: how long it takes to download an image from Facebook, the bitrate of a video streamed from YouTube, the packet loss on the Call of Duty servers. It isn’t a piece of educated guesswork, based on a synthetic test to a dedicated test server somewhere; it’s a carefully calibrated analysis of real-world application performance.

SamKnows has come a long way since 2008

All that’s possible because of a piece of software: the SamKnows Agent. It’s now found in tens of millions of broadband routers across the globe, unobtrusively taking performance measurements 24 hours a day.

This is the story of the SamKnows Agent: how it came to be, how it evolved, and how it transformed the way broadband performance is measured around the world.

A different type of testing

Today’s broadband market is very different to the one that existed in 2008, when there was almost no independent verification that consumers were getting what they paid for.

“I remember driving somewhere and seeing a billboard on the side of the motorway advertising very low-cost broadband from some provider, and there were no caveats in the small print about the kind of speeds you should expect during peak hours,” said Sam. “Nowadays, you can take it for granted that ou’re going to generally get pretty good speeds all of time from a decent ISP in a well- provisioned area. But at that time, you definitely couldn’t take that for granted. There were lots of capacity problems all of the time.”

“I thought wouldn’t it be great if there was some way of shining a light on this,” Sam adds.

Sam examined the broadband performance tests that were available at the time, but wasn’t impressed with what he found. “Everything was very focused on speed,” he said. “There wasn’t really any consideration of latency or packet loss, or even variation by time of day or looking at different protocols.

Real-world performance is key to SamKnows

No one was measuring anything like BitTorrent or web page loading times. There were still problems with DNS response times from ISPs’ DNS servers, which were all over the place. So, just running pure speed tests alone – I didn’t think that was enough.”

With a background in computer science and the desire to ensure that any tests he created were defensible, Sam devised a test methodology that was different to anything else available at the time.

Passive vs active

Broadband performance tests can be broadly broken down into two categories: passive and active. Passive tests might use deep packet inspection to infer performance characteristics from real user traffic.

For example, your ISP might see that you’re using 3Mbits/ sec and you’re connecting to servers in Netflix’s IP range, so you can infer some metrics about latency and bandwidth consumption based on that data. On the plus side, such passive measurement is easy to deploy, doesn’t require any user interaction and, potentially, gives providers huge swathes of data.

But it’s a long way from perfect. You can only measure what the user’s actually doing on their broadband connection, at the time they’re doing it. If you’re trying to draw a distinction between off-peak and peak-time usage and the user only uses broadband when they get in from work in the evenings, then you’re out of luck. Neither do you have any idea if the user is connecting to Netflix from a wired Ethernet connection or from their phone on the garden patio, which can have a massive impact on performance, along with a host of other in-home factors. Also, as Sam points out, “it’s a privacy minefield”.

Sam decided that active measurement was the way forward, and while there were no shortage of active speed tests on the market at the time, they again had flaws. We’re all familiar with the type of test where you press a button on a website or app and wait for the results to appear, but that approach has its problems too.

SamKnows’ Agent is now inside routers

“It isn’t a controlled environment,” said Sam. “What type of machine are you running it on? Is it a laptop or a desktop? Is it a phone, because these will have different capabilities? What else is running on the machine at the same time? Which browser are you running, because different browsers have different performance characteristics? The big one is how is the machine connected? Is it over Ethernet or Wi-Fi, what generation Wi- Fi, how noisy is it, what’s your signal strength? All of these factors have a huge impact on the measurements.”

Introducing the Whitebox

That many known unknowns made Sam’s mind up: if SamKnows was going to measure home performance accurately, it would need its own hardware. “The most scientifically robust way for us to have a good measurement methodology was to have schedule tests running on a hardware device connected via Ethernet inside the user’s home, running tests 24/7 and taking into account cross traffic,” he said. “That led us to build the first Whitebox.”

The Whitebox was a small piece of hardware, posted to broadband customers, which they simply plugged into their existing router and let it go about its business. The quid pro quo was that the consumer was given access to a whole host of metrics about their own broadband connection that they’d never seen before.

Having dedicated hardware to run tests also provided greater flexibility in the type of tests SamKnows could deploy. Although it started off running the same sort of tests that others were doing – speed, latency, jitter, packet loss – SamKnows was later able to add a whole swathe of quality of experience (QoE) tests aimed at specific applications.

“There are lots of different things that we might use our internet connections for, so just running speed tests alone isn’t really representative of anything other than running a speed test,” said Sam.

“So back in 2015, we started looking at measuring specific applications – and this isn’t taking our existing tests, such as our speed tests, and then applying some sort of mathematical model to them and saying ‘oh yeah, because we can stream at 2Mbits/sec then we can watch a YouTube SD video’ and do all that hand- waving stuff. We’re not doing that. We’re actually streaming real content and the crucial thing is we’re exchanging real application traffic with the real application servers in order to carry out these measurements. So, for example, with YouTube we’re actually streaming real YouTube videos from the real YouTube servers, which might be deployed inside your ISP’s network.”

The number of those QoE tests SamKnows conducts has grown enormously since 2015 (see the timeline of tests below), and even though SamKnows pioneered this method of testing eight years ago, Sam believes it’s still unique in the industry. “As far as I know, there isn’t really any other internet measurement company that has such a focus on application testing,” he said.

The Agent arrives

The Whiteboxes gave SamKnows and its clients insights into broadband performance that no other company could offer – but what they didn’t offer was scale. Sending out Whiteboxes was an expensive, labour-intensive business, and there are cost limitations on how many can be deployed. The data they delivered was rich, but the sample size was relatively small.

Those limitations prompted SamKnows to develop an idea that once again transformed the business. Instead of deploying hardware boxes to people’s homes, why not write software for the broadband routers that are already there? The software Agent was born.

“I won’t say it’s a silver bullet, but it’s really, really close to a silver bullet, because it overcomes all of those limitations that we have with the Whitebox,” said Sam.

Services such as YouTube are measured

“It’s great for us and it’s great for the ISP, because they can deploy a software update to their routers. No device is posted out anymore, the user doesn’t have to do anything; it’s just an over-the-air software update. They can deploy this to millions of devices overnight if they wish, and it has all the same measurement methodology and functionality as the Whitebox.

“It solves the logistical problems with deployment, it solves the scale problem, and it solves the time to deploy. So, it’s been a really big win for us.”

Making the Agent work

That’s not to say any of this was easy. The hardware inside home broadband routers isn’t hugely powerful and there’s a range of different chipsets used. To get the broadband providers on side, the SamKnows Agent would have to work flawlessly with the router and not interfere with the customer’s broadband connection. That’s something that takes time and expertise.

“You need to write your software in such a way that it can be deployed inside all of these routers,” said Sam. “People often say that Android is quite a fragmented ecosystem on the mobile side. Just wait until you see the router environment! That makes Android look like a super-tidy place.

So, building software for routers can be very painful, and the first time we did it back in 2016, it was quite a painful process; but nowadays, we have a good team and we’re able to turn these out really quickly.”

Sam and his team have also spent many years working with router chipset manufacturers such as Qualcomm and Broadcom, in order to maximise the efficiency of the Agent and ensure that it doesn’t get in the way of the router’s day job. “What those companies have is a way for parties to run software on their dedicated co-processors,” said Sam. “We call this hardware- accelerated measurements. So we’re making use of these dedicated, separate hardware accelerators so we can run our very network- intensive measurements.”

The team works closely with vendors

That doesn’t mean all the tests are run on the co- processors. “Lots of our measurements aren’t super network intensive, so measuring Google Meet or Fortnite, for example, doesn’t really require much bandwidth. As such, there’s no need to use these co-processors.” They’ll run on the router’s main processor instead.

Being able to build software of this complexity “requires good relationships with the chipset vendors for them to actually provide access to this stuff and to spend time with you,” said Sam. “Also essential is a good development team, because these things aren’t easy to work with. But now we can do things like measure ten gigabits per second on relatively low- end commodity hardware and get very reliable and accurate results.”