TCP speed test
Measures the download and upload speed of the broadband connection in bits per second. The transfer is conducted over one or more concurrent HTTP connections (using the GET verb for download and the POST verb for uploads).
In the download speed test the client will fetch a portion of an infinitely sized binary (non-zero, randomly generated) payload hosted on an HTTP server on the target test server. The content is discarded as soon as it is received.
In the upload test the client will generate the payload itself (using /dev/urandom as a non-blocking source of random content) to send to the server. The upload speed is measured at the server side (the receiver).
Both download and upload speed tests operate for a fixed-duration by default. We recommend a duration of 10 seconds. Optionally, a transfer size limit (specified in MB) can be applied. A fixed-duration test is preferred as it will cater well for all broadband access speeds. However, a fixed-volume test may be necessary where predictability of bandwidth usage is desired.
Four separate variants of the test are supported:
- Single TCP connection download speed test
- Multiple TCP connection download speed test
- Single TCP connection upload speed test
- Multiple TCP connection upload speed test
For multiple TCP connection tests we typically recommend that eight concurrent connections are used. In some cases (e.g. where the bandwidth x delay product between client and server is very high) it may be necessary to increase this.
The HTTP test can be optionally configured to force the use of the ‘cubic’ TCP congestion control algorithm when testing against the SamKnows HTTP Server application. This is a preferred option and used widely, but is not the application default.
Factors such as TCP slow start are accounted for through the use of a “warm-up” period. This period begins as soon as the test starts and seeks to establish that the throughput has reached a stable rate before starting the real test (which will continue over the same TCP connection(s)).The warmup-period ends after a minimum period (default 1.5 seconds) when either a stable rate is achieved, or when a preconfigured time limit (default 5 seconds) is hit. It is important to note that the data transferred in the warm-up period is excluded from the main test results, but it is still recorded separately as a supplementary metric.
The speed test client captures the following key metrics:
- The speed, in bytes per second
- The amount of data transferred
- The duration of the test
- Details about the warm-up period
- The hostname and IP address of the test server
The client may also record these values at multiple intervals during the test. This is commonly used to help characterise the difference between 'burst' and 'sustained' throughput (where transfer speeds may be inflated at the start of a TCP connection). Speed is measured at the application layer, so on a 100Mbit/s physical link the maximum measurable speed is approximately 95Mbit/s, due to packet overheads.
Finally, our TCP speed test will make use of hardware-specific functionality on some hardware devices. This is typically used on devices where the CPU is insufficiently powerful to saturate the WAN link in software mode alone. The nature of these implementations varies; sometimes it uses a dedicated hardware co-processor, and sometimes it uses a special kernel driver to avoid unnecessary copying data. An example of this is where we have integrated with Broadcom's "BSPEED" kernel driver in their DOCSIS chipsets.
Example
We noticed under performing Fibre Max in our study Measuring Broadband New Zealand. After an investigation the cause was located and a fix rolled out. Our TCP speed test shows a significant improvement in average download speed after 4th of September. Average speeds go from under 750Mbps to over 900Mbps which is in-line with what we would expect to see on Fibre Max lines. RSP Y made their changed on the 19th of October, a similar step change in performance can be seen following these changes.
This chart shows the average download speed from two anonymised ISPs, X and Y, which made changes to their network following SamKnows' Fibre Max investigation in New Zealand.
UDP speed test
The UDP speed test measures downstream and upstream throughput at the application layer using UDP.
This test supports both software and hardware accelerated modes of operation. In the software mode, the traffic is generated and measured using the main CPU in the device, in userspace. In hardware accelerated mode, the test makes use of platform-specific hardware acceleration features to generate or receive UDP packets at a far higher speed than the main CPU can support.
Hardware acceleration is currently supported on certain generations of Broadcom and Qualcomm SOCs. The measurement traffic is not directly handled by the main CPU, and instead uses a separate network co-processor. For Broadcom, measurements of up to 10Gbps are attainable using the licenced 'speed service' API functionality from Broadcom. On Qualcomm, measurements of up to 10Gbps are also attainable using their NSS UDP APIs. SamKnows has performed multiple integrations with each of these implementations.
Note: It is preferred to avoid using hardware acceleration for speed tests, unless unavoidable. The reason is that these features may be subject to additional licensing fees (as with Broadcom) or may make troubleshooting more difficult.
Regardless of whether hardware acceleration is used or not, the measurement operates in the same way. For a download test, the UDP speed test client issues a download test request to the SamKnows test server over TCP-based control channel, requesting that the server begin transmitting UDP test traffic back to the client at a fixed rate. The client then begins to receive the measurement packets, and assesses the throughput received. If the throughput is at least 80% of the requested rate, then the client requests that the server increase its sending rate, and the process restarts. Eventually, the client will stop requesting increases in transmission rates by the server, and the client will measure the received rate at the current transmission rate. This process is controlled by a 'speed ladder', which is remotely configurable.
For an upload test, the UDP speed test client issues a notification to the test server over TCP, informing it that transmission of UDP test traffic is about to commence. The client then begins transmitting measurement packets. Packets are transmitted at full line rate immediately and for the entire duration of the test. The packet size can be configured, and is 1400 bytes by default.
In both the upstream and downstream directions, the throughput is measured over a five second period by default (excluding any ladder ramp-up time). The speed is always measured at the receiver side.
Finally, there is a ‘dual-channel’ extension to the hardware-accelerated mode of transmission. The dual-channel approach uses a software based UDP measurement stream in parallel with the hardware accelerated UDP stream. This mode is to specifically cater for certain legacy hardware that has the main CPU connected via a 1Gbps internal interface, and the network co-processor connected via another 1Gbps internal interface. Used individually, measurements could never exceed 1Gbps, even on a >1Gbps WAN link. However, when used in parallel, measurements over 1Gbps can be achieved.
The following key metrics are captured by the test for both downstream and upstream:
- Measured throughput
- Transmission rate
- Transmission bytes
- Transmission duration
- Server hostname and IP
- Packet size
Lightweight capacity test (UDP)
This test measures the instantaneous capacity of the link using a small number of UDP packets. The test supports both downstream and upstream measurements, conducted independently.
In the downstream mode, the test client handshakes with the test server over TCP, requesting a fixed number of packets to be transmitted back to the client. The client specifies the transmission rate, number of packets and packet size in this handshake. The client records the arrival times of each of the resulting packets returned to it.
In the upstream mode, the client again handshakes with the test server, this time informing it of the characteristics of the stream it is about to transmit. The client then transmits the stream to the server, and the server locally records the arrival times of each packet. At the conclusion of this stream, the client asks the server for its view of the arrival time of each packet.
With this resulting set of arrival times, the test client calculates the throughput achieved. This throughput may be divided into multiple windows, and an average taken across those, in order to smooth out buffering behaviour.
This test can use approximately 99% less data than the TCP speed test and completes in a fraction of the time (100 milliseconds versus 10 seconds). Of course, there are trade-offs with such a vast reduction in data usage. The accuracy is lower than the regular TCP or UDP speed tests, and this varies by access technology. Generally speaking, DSL or fibre connections produce results within a few percent of the TCP or UDP measurements. Results are more varied on DOCSIS connections, due to the increased variability of latency on this access technology.
WebSockets speed test
Measures the download and upload speed of the broadband connection in bits per second. The transfer is conducted over one or more concurrent TCP connections. WebSockets and HTTP (using the Fetch API) are used via HTML5 APIs as the transport for all measurement traffic. This requires the web browser to support HTML5.
In the download speed test the client will repeatedly fetch chunks of data from the target test server. The content is discarded as soon as it is received. The client will determine from the user agent which is the best transport protocol to use; some browsers perform better with WebSockets and others perform better using the Fetch API.
In the upload test the client will generate the payload itself to send to the server. Similarly, the client will repeatedly upload chunks of this data to the target test server, which will immediately discard the content.
Our lab tests have demonstrated that figures of 10Gbps are attainable on reasonably modern desktop hardware (2020, Apple Mac M1) using modern web browsers.
The speed tests (both download and upload) operate for a fixed-duration specified in seconds. A maximum limit on transfer volume may be imposed if data volumes are of concern.
Both the download and upload tests use eight concurrent TCP connections by default. The size of the chunk to transfer will vary between 16KB and 1MB, and will be determined automatically during the warm-up phase of the test.
Factors such as TCP slow start are accounted for using a "warm-up" period. This period begins as soon as the test starts and seeks to remove the impact of TCP slow start from the results and serves to determine the number of TCP connections to be used for the remainder of the test. It is important to note that the data transferred in the warm-up period is excluded from the main test results.
The speed test client will record the throughput, bytes transferred and time taken at the end of the test.
Example
SamKnows RealSpeed uses our Web socket speed test to compare in-home performance on a device to the service provided at the router.
