In September 2018, we were contacted by Virgin Media who were starting to trial a new 10Gbps FTTH EPoN Broadband product, testing both the access technology and in-home connectivity. While our measurement code has no hard upper limit to the speed we can measure, our existing Whitebox 8.0 hardware only has 1Gbps Ethernet ports, which limits us to 1Gbps. So, we needed a new Whitebox to hit 10Gbps.
We had already prototyped a hardware test agent with a few ODMs (original device manufacturers) the year before, so we had a head start. We ultimately settled on a Marvell Armada 8040 based device. This has four ARM 64-bit cores running at 2GHz, and is equipped with two 10Gbps interfaces. This board uses SFP+ slots, allowing us to use more links than just the classic copper ethernet ports. It also has a variety of debugging and management interfaces, making development fairly easy.
SamKnows has been embedding tests in hardware for over 10 years so the integration into the new Whitebox hardware was straightforward. To create the firmware, we used OpenWrt, to which we made few changes to incorporate the Marvell drivers and other components in the board’s toolchain. We also stripped out all non-essential features for our use case. Next, we increased our lab-environment capacity so that we could reach 10Gbps and then tested the new Whitebox. The whole exercise took us only a week from start to finish.
The new Whitebox prototype (10Gbps) on top of the Whitebox 8.0 (1Gbps)
Lab testing
Initially, we were hitting over 9Gbps, and other times we were only hitting 4.5Gbps. After some investigation, we found the issue was related to how the CPU was processing interrupts from the network driver. We were able to resolve this permanently by pinning the measurement applications to the Marvell Armada CPU cores that were not processing interrupts.
Soon, we were measuring over 9.1Gbps, with 1500 byte packets and 9.8Gbps with 9000 byte packets, with about 30% CPU to spare. This was impressive, especially considering real-world speeds of 10Gbps only reach a maximum of ~9.5Gbps, which we can also measure if the ISP has the network capacity. In Virgin Media’s case, the 9.5Gbps (application layer) limit is caused by the fact that packet headers, such as Ethernet, IP, and TCP headers, come out of the 10Gbps layer 2 link speed. The results recorded in the lab were conducted with a single TCP connection, operating in userspace, and using a variety of optimisations to achieve this. But this result alone dispels two myths: you do not need to use hardware acceleration to hit 10Gbps, and you do not need many parallel TCP sessions to hit very high speeds.
Field testing
Following our internal lab testing, we installed the prototype in the Virgin Media Cambridgeshire “hubsite”. It was connected via an ARRIS OLT to a dedicated ARRIS CMTS with new 10Gbps line cards in it. We also provisioned a dedicated 10Gbps test server in London, connected to Virgin Media with well over 10Gbps of peering capacity.
We quickly identified a new problem. We could measure 2.5Gbps, but then seemed to hit a hard ceiling at this point. Network routing analysis showed that traffic was taking far longer to reach the London SamKnows server than expected. Round-trip latency was recording around 13ms, but given the ~70 miles between London and the Cambridgeshire hubsite, we would expect to see only a fraction of that (approximately 1-2ms). Further analysis showed that the return path of traffic from our London test server to Virgin Media in Cambridgeshire passed through Virgin Media’s Manchester POP, adding to the delay. We hypothesised that this was somehow the cause of the poor speeds we were seeing, so we provisioned another test server in Manchester, closer to Virgin Media’s POP. We also added our newly-developed SamKnows/Cloudflare test server platform, which has a Manchester location and sufficient peering capacity for our needs. After making this change, we could see just over 8Gbps from the customers’ home in Papworth.
Figure 1 - Download speeds (blue) are showing consistent speeds of 8Gbps over time, when testing from Cambridgeshire to Cloudflare.
Launching the ‘X’
Being able to deliver over 8Gbps to a consumer’s home is a significant achievement for the Virgin Media team, particularly as this was a real-world set-up and not based on an optimised lab-only testing environment. Virgin Media is continuing to further enhance the new service and fine tune the network to deliver even faster speeds to customers. We are delighted to be able to work with our partners ARRIS and Virgin Media to independently field test and benchmark new, super-fast products. Testing speeds of this magnitude is extraordinarily complex, with many variables that could impact performance. We’re thrilled that Virgin Media were able to adopt the SamKnows certified methodology and hit their target speed quickly, without delays or major expense. We have already implemented enhancements to our Router SDK to make 10Gbps testing functionality available as standard. And we have also ordered our first shipment of 10Gbps Whiteboxes, which we’re calling the ‘X’.