Metrics

The SamKnows measurement agents support a wide range of tests. Each of these tests will carry out its measurements (typically to dedicated test servers or directly to content providers), and will produce one or more metrics. These documents describe the methodology used to carry out each of the tests SamKnows supports and outlines the metrics that each test captures.

The availability of tests varies by platform. For example, the lightweight capacity test is not available for use in web tests, as the use of UDP is prohibited by web browsers.

Tests and metrics supported by platform

Test Metric Platforms Description
Download (TCP) Download speed Whiteboxes, Routers, Android, iOS The download speed in Mbps when downloading (using TCP) random bytes from a test server
  TCP Retransmissions Whiteboxes, Routers The number of retransmitted TCP segments/packets
  Burst download speed Whiteboxes, Routers The download speed during the first 5 seconds of a test
  Sustained download speed Whiteboxes, Routers The download speed of the test during the last 5 seconds
  Percentage of Best Whiteboxes, Routers Download speed result as a percentage of the user's best ever result
  Percentage of Advertised Whiteboxes, Routers Download speed result as a percentage of their package's advertised downstream speed
Download (HTML5) Download speed Web The download speed in Mbps when downloading (using TCP) random bytes from a test server using HTML5 APIs (WebSockets and Fetch)
Download (Lightweight UDP) Download speed Whiteboxes, Routers The download speed in Mbps when downloading (using UDP) from a test server, using less data than the TCP test
Download (Hardware accelerated UDP) Download speed Broadcom-based Routers The download speed in Mbps when downloading (using UDP) random bytes from a test server
Upload (TCP) Upload speed Whiteboxes, Routers, Android, iOS The upload speed in Mbps when uploading (using TCP) random bytes to a test server
  Percentage of Best Whiteboxes, Routers Upload speed as a percentage of their best ever upload speed
  Percentage of Advertised Whiteboxes, Routers Upload speed as a percentage of their advertised upload speed
Upload (HTML5) Upload speed Web The upload speed in Mbps when uploading (using TCP) random bytes to a test server using HTML5 APIs (WebSockets and Fetch)
Upload (Lightweight UDP) Upload speed Whiteboxes, Routers The upload speed in Mbps when transmitting (using UDP) to a test server, using less data than the TCP test.
Upload (Hardware accelerated UDP) Upload speed Broadcom-based Routers The upload speed in Mbps when transmitting (using UDP) random bytes to a test server
Latency Latency Whiteboxes, Routers, Android, iOS Average round-trip time of a series of echo packets transmitted over UDP
  Packet Loss Whiteboxes, Routers, Android, iOS Percentage of packets lost not received within two seconds of sending
  Jitter Up Whiteboxes, Routers, Android, iOS Variability of packet latency upstream
  Jitter Down Whiteboxes, Routers, Android, iOS Variability of packet latency downstream
  Contiguous Packet Loss (Availability/Disconnections) Whiteboxes, Routers Periods of time where more than 2 consecutive packets are lost
  MOS (Mean Opinion Score) Whiteboxes, Routers, Android, iOS Grading of the quality of a voice call, on a scale of 1 (worst) to 5 (perfect)
Latency (HTML5) Latency Web Round trip time for a series of packets sent to our test server over WebSockets
Latency and packet loss (ICMP) Latency Whiteboxes, Routers Average round-trip time of a series of ICMP echo packets
  Packet Loss Whiteboxes, Routers Percentage of ICMP packets lost not received within two seconds of sending
Web-browsing Response Time Whiteboxes, Routers, Android, iOS Measures the time taken to fetch the HTML and referenced resources from a page of a popular website.
  Failure Rate Whiteboxes, Routers, Android, iOS The percentage of failures to fetch the HTML of a website
DNS DNS Resolution Time Whiteboxes, Routers The DNS resolution time of a selection of common website domain names
  DNS Failure Rate Whiteboxes, Routers The rate of DNS resolution failures
Netflix Netflix Downstream Speed Whiteboxes, Routers The download speed when downloading content from Netflix
  Netflix Bitrate Reliably Streamed Whiteboxes, Routers The bitrate that can be reliably streamed without stalls
  Netflix Startup Delay Whiteboxes, Routers The amount of time for the video to fill the buffer with two seconds of video / start playing
  Netflix TCP Connect Time Whiteboxes, Routers The TCP connect time to the Netflix server
  Netflix Bitrate % of Max Whiteboxes, Routers The bitrate as a percentage of the max bitrate attempted/possible
YouTube YouTube Bitrate Reliably Streamed Whiteboxes, Routers, Android, iOS The bitrate that can be reliably streamed without stalls
  YouTube Startup Delay Whiteboxes, Routers, Android, iOS The amount of time for the video to fill the buffer with two seconds of video / start playing
  YouTube TCP Connect Time Whiteboxes, Routers, Android, iOS The TCP connect time to the YouTube server
  YouTube Bitrate % of Max Whiteboxes, Routers The bitrate as a percentage of the max bitrate attempted/possible
BBC iPlayer iPlayer Bitrate Reliably Streamed Whiteboxes, Routers The bitrate that can be reliably streamed without stalls
  iPlayer Startup Delay Whiteboxes, Routers The amount of time for the video to fill the buffer with two seconds of video / start playing
  iPlayer TCP Connect Time Whiteboxes, Routers The TCP connect time to the iPlayer server
  iPlayer Bitrate % of Max Whiteboxes, Routers The bitrate as a percentage of the max bitrate attempted/possible
Hulu Hulu Bitrate Reliably Streamed Whiteboxes, Routers The bitrate that can be reliably streamed without stalls
  Hulu Startup Delay Whiteboxes, Routers The amount of time for the video to fill the buffer with two seconds of video / start playing
  Hulu TCP Connect Time Whiteboxes, Routers The TCP connect time to the Hulu server
DOTA2 Latency Whiteboxes, Routers ICMP Round-trip time latency to a DOTA2 server
  Packet Loss Whiteboxes, Routers ICMP Percentage of packets lost not received within two seconds of sending
League of Legends Latency Whiteboxes, Routers ICMP Round-trip time latency to a League of Legends server
  Packet Loss Whiteboxes, Routers ICMP Percentage of packets lost not received within two seconds of sending
FIFA 2018 Latency Whiteboxes, Routers ICMP Round-trip time latency to a FIFA server
  Packet Loss Whiteboxes, Routers ICMP Percentage of packets lost not received within two seconds of sending
Multicast IPTV Jitter Whiteboxes, Routers The jitter of the incoming video stream
  Channel change time Whiteboxes, Routers How long it takes to join a new multicast group (change TV channel)
CDN Object fetch time Whiteboxes, Routers How long it takes to fetch a small object from the CDN
  Object fetch speed Whiteboxes, Routers How fast an object can be fetched from the CDN
  TCP connection establishment time Whiteboxes, Routers The latency to the CDN
VOIP SIP Jitter Up Whiteboxes, Routers Variability of packet latency upstream for a SIP connection
  Jitter Down Whiteboxes, Routers Variability of packet latency downstream for a SIP connection
  Mean Opinion Score (MOS) Whiteboxes, Routers Grading of the quality of a voice call, on a scale of 1 (worst) to 5 (perfect)
Traceroute None Whiteboxes, Routers No metrics at present, currently just used internally for debugging
Data usage Wired LAN traffic Whiteboxes, Routers Volume of customer traffic generated and received by wired LAN devices
  Wi-Fi traffic Whiteboxes, Routers Volume of customer traffic generated and received by Wi-Fi devices
  SamKnows traffic Whiteboxes, Routers Volume of SamKnows test traffic
Online Storage Download speed Whiteboxes, Routers The download speed in Mbps when downloading content from Google Drive, Amazon, Dropbox, OneDrive and Sharepoint
Steam Download speed Whiteboxes, Routers The download speed in Mbps when downloading a game from Steam's network
  TCP connection time Whiteboxes, Routers The TCP connection establishment time to Steam's network