Introducing the Whitebox
That many known unknowns made Sam’s mind up: if SamKnows was going to measure home performance accurately, it would need its own hardware. “The most scientifically robust way for us to have a good measurement methodology was to have schedule tests running on a hardware device connected via Ethernet inside the user’s home, running tests 24/7 and taking into account cross traffic,” he said. “That led us to build the first Whitebox.”
The Whitebox was a small piece of hardware, posted to broadband customers, which they simply plugged into their existing router and let it go about its business. The quid pro quo was that the consumer was given access to a whole host of metrics about their own broadband connection that they’d never seen before.
Having dedicated hardware to run tests also provided greater flexibility in the type of tests SamKnows could deploy. Although it started off running the same sort of tests that others were doing – speed, latency, jitter, packet loss – SamKnows was later able to add a whole swathe of quality of experience (QoE) tests aimed at specific applications.
“There are lots of different things that we might use our internet connections for, so just running speed tests alone isn’t really representative of anything other than running a speed test,” said Sam.
“So back in 2015, we started looking at measuring specific applications – and this isn’t taking our existing tests, such as our speed tests, and then applying some sort of mathematical model to them and saying ‘oh yeah, because we can stream at 2Mbits/sec then we can watch a YouTube SD video’ and do all that hand- waving stuff. We’re not doing that. We’re actually streaming real content and the crucial thing is we’re exchanging real application traffic with the real application servers in order to carry out these measurements. So, for example, with YouTube we’re actually streaming real YouTube videos from the real YouTube servers, which might be deployed inside your ISP’s network.”
The number of those QoE tests SamKnows conducts has grown enormously since 2015 (see the timeline of tests below), and even though SamKnows pioneered this method of testing eight years ago, Sam believes it’s still unique in the industry. “As far as I know, there isn’t really any other internet measurement company that has such a focus on application testing,” he said.