Download PDF

Metrics.

The SamKnows measurement agents support a wide range of tests. Each of these tests will carry out its measurements (typically to dedicated test servers or directly to content providers), and will produce one or more metrics. These documents describe the methodology used to carry out each of the tests SamKnows supports and outlines the metrics that each test captures.

The availability of tests varies by platform. For example, the lightweight capacity test is not available for use in web tests, as the use of UDP is prohibited by web browsers.

Tests and metrics supported by platform

TestMetricPlatformsDescription
Download (TCP)Download speedWhiteboxes, Routers, Android, iOSThe download speed in Mbps when downloading (using TCP) random bytes from a test server
 TCP RetransmissionsWhiteboxes, RoutersThe number of retransmitted TCP segments/packets
 Burst download speedWhiteboxes, RoutersThe download speed during the first 5 seconds of a test
 Sustained download speedWhiteboxes, RoutersThe download speed of the test during the last 5 seconds
 Percentage of BestWhiteboxes, RoutersDownload speed result as a percentage of the user's best ever result
 Percentage of AdvertisedWhiteboxes, RoutersDownload speed result as a percentage of their package's advertised downstream speed
Download (HTML5)Download speedWebThe download speed in Mbps when downloading (using TCP) random bytes from a test server using HTML5 APIs(WebSockets and Fetch)
Download (Lightweight UDP)Download speedWhiteboxes, RoutersThe download speed in Mbps when downloading (using UDP) from a test server, using less data than the TCP test
Download (Hardware acceleratedUDP)Download speedBroadcom-based RoutersThe download speed in Mbps when downloading (using UDP) random bytes from a test server
Upload (TCP)Upload speedWhiteboxes, Routers, Android, iOSThe upload speed in Mbps when uploading (using TCP) random bytes to a test server
 Percentage of BestWhiteboxes, RoutersUpload speed as a percentage of their best ever upload speed
 Percentage of AdvertisedWhiteboxes, RoutersUpload speed as a percentage of their advertised upload speed
Upload (HTML5)Upload speedWebThe upload speed in Mbps when uploading (using TCP) random bytes to a test server using HTML5 APIs (WebSocketsand Fetch)
Upload (Lightweight UDP)Upload speedWhiteboxes, RoutersThe upload speed in Mbps when transmitting (using UDP) to a test server, using less data than the TCP test.
Upload (Hardware acceleratedUDP)Upload speedBroadcom-based RoutersThe upload speed in Mbps when transmitting (using UDP) random bytes to a test server
LatencyLatencyWhiteboxes, Routers, Android, iOSAverage round-trip time of a series of echo packets transmitted over UDP
 Packet LossWhiteboxes, Routers, Android, iOSPercentage of packets lost not received within two seconds of sending
 Jitter UpWhiteboxes, Routers, Android, iOSVariability of packet latency upstream
 Jitter DownWhiteboxes, Routers, Android, iOSVariability of packet latency downstream
 Contiguous Packet Loss (Availability/Disconnections)Whiteboxes, RoutersPeriods of time where more than 2 consecutive packets are lost
 MOS (Mean Opinion Score)Whiteboxes, Routers, Android, iOSGrading of the quality of a voice call, on a scale of 1 (worst) to 5 (perfect)
Latency (HTML5)LatencyWebRound trip time for a series of packets sent to our test server over WebSockets
Latency and packet loss(ICMP)LatencyWhiteboxes, RoutersAverage round-trip time of a series of ICMP echo packets
 Packet LossWhiteboxes, RoutersPercentage of ICMP packets lost not received within two seconds of sending
Website PerformanceTest Time to visually complete Whiteboxes, Routers The time taken to render the complete web page, with the page 100% loaded visually
  Time to engine complete Whiteboxes, Routers The time taken for the web browser's engine to say that the web page had finished loading (note that this maydiffer from visual completeness)
  Time to first visual Whiteboxes, Routers The time taken to render the first visual element of a web page
  Average object transfer time Whiteboxes, Routers The average time taken to transfer each object referenced by the web page
  Average object TCP connection time Whiteboxes, Routers The average time taken to establish a TCP connection to the origin servers that host the web page content
  Average object DNS resolution time Whiteboxes, Routers The average time taken to resolve DNS for the content servers that provide the web page content
  Failure Rate Whiteboxes, Routers The failure rate when loading a web page
Web browsing lite Page Load Time Whiteboxes, Routers, Android, iOS Measures the time taken to fetch the HTML and referenced resources from a page of a popular website
  Failure Rate Whiteboxes, Routers, Android, iOS The percentage of failures to fetch the HTML of a website
DNSDNS Resolution TimeWhiteboxes, RoutersThe DNS resolution time of a selection of common website domain names
 DNS Failure RateWhiteboxes, RoutersThe rate of DNS resolution failures
  DNS-over-HTTPS Resolution Time Whiteboxes, Routers The DNS resolution time of a selection of common website domain names using DNS-over-HTTPS
  DNS-over-HTTPS Failure Rate Whiteboxes, Routers The rate of DNS resolution failures when using DNS-over-HTTPS
NetflixNetflix Downstream SpeedWhiteboxes, RoutersThe download speed when downloading content from Netflix
 Netflix Bitrate Reliably StreamedWhiteboxes, RoutersThe bitrate that can be reliably streamed without stalls
 Netflix Startup DelayWhiteboxes, RoutersThe amount of time for the video to fill the buffer with two seconds of video / start playing
 Netflix TCP Connect TimeWhiteboxes, RoutersThe TCP connect time to the Netflix server
 Netflix Bitrate % of MaxWhiteboxes, RoutersThe bitrate as a percentage of the max bitrate attempted/possible
YouTubeYouTube Bitrate Reliably StreamedWhiteboxes, Routers, Android, iOSThe bitrate that can be reliably streamed without stalls
 YouTube Startup DelayWhiteboxes, Routers, Android, iOSThe amount of time for the video to fill the buffer with two seconds of video / start playing
 YouTube TCP Connect TimeWhiteboxes, Routers, Android, iOSThe TCP connect time to the YouTube server
 YouTube Bitrate % of MaxWhiteboxes, RoutersThe bitrate as a percentage of the max bitrate attempted/possible
BBC iPlayeriPlayer Bitrate Reliably StreamedWhiteboxes, RoutersThe bitrate that can be reliably streamed without stalls
 iPlayer Startup DelayWhiteboxes, RoutersThe amount of time for the video to fill the buffer with two seconds of video / start playing
 iPlayer TCP Connect TimeWhiteboxes, RoutersThe TCP connect time to the iPlayer server
 iPlayer Bitrate % of MaxWhiteboxes, RoutersThe bitrate as a percentage of the max bitrate attempted/possible
HuluHulu Bitrate Reliably StreamedWhiteboxes, RoutersThe bitrate that can be reliably streamed without stalls
 Hulu Startup DelayWhiteboxes, RoutersThe amount of time for the video to fill the buffer with two seconds of video / start playing
 Hulu TCP Connect TimeWhiteboxes, RoutersThe TCP connect time to the Hulu server
DOTA2LatencyWhiteboxes, RoutersICMP Round-trip time latency to a DOTA2 server
 Packet LossWhiteboxes, RoutersICMP Percentage of packets lost not received within two seconds of sending
League of LegendsLatencyWhiteboxes, RoutersICMP Round-trip time latency to a League of Legends server
 Packet LossWhiteboxes, RoutersICMP Percentage of packets lost not received within two seconds of sending
FIFA 2018/2019LatencyWhiteboxes, RoutersICMP Round-trip time latency to a FIFA server
 Packet LossWhiteboxes, RoutersICMP Percentage of packets lost not received within two seconds of sending
Apex Legends Latency Whiteboxes, Routers ICMP Round-trip time latency to the Apex Legends servers
  Packet Loss Whiteboxes, Routers ICMP packet loss (as a percentage) for packets sent to the Apex Legends servers
Fortnite Latency Whiteboxes, Routers ICMP Round-trip time latency to the Fortnite servers
  Packet Loss Whiteboxes, Routers ICMP packet loss (as a percentage) for packets sent to the Fortnite servers
Multicast IPTVJitterWhiteboxes, RoutersThe jitter of the incoming video stream
 Channel change timeWhiteboxes, RoutersHow long it takes to join a new multicast group (change TV channel)
CDNObject fetch timeWhiteboxes, RoutersHow long it takes to fetch a small object from the CDN
 Object fetch speedWhiteboxes, RoutersHow fast an object can be fetched from the CDN
 TCP connection establishment timeWhiteboxes, RoutersThe latency to the CDN
VOIP SIPJitter UpWhiteboxes, RoutersVariability of packet latency upstream for a SIP connection
 Jitter DownWhiteboxes, RoutersVariability of packet latency downstream for a SIP connection
 Mean Opinion Score (MOS)Whiteboxes, RoutersGrading of the quality of a voice call, on a scale of 1 (worst) to 5 (perfect)
Latency (HTML5)LatencyWebRound trip time for a series of packets sent to our test server over WebSockets
Latency and packet loss(ICMP)LatencyWhiteboxes, RoutersAverage round-trip time of a series of ICMP echo packets
 Packet LossWhiteboxes, RoutersPercentage of ICMP packets lost not received within two seconds of sending
TracerouteNoneWhiteboxes, RoutersNo metrics at present, currently just used internally for debugging
Data usageWired LAN trafficWhiteboxes, RoutersVolume of customer traffic generated and received by wired LAN devices
 Wi-Fi trafficWhiteboxes, RoutersVolume of customer traffic generated and received by Wi-Fi devices
 SamKnows trafficWhiteboxes, RoutersVolume of SamKnows test traffic
Online StorageDownload speedWhiteboxes, RoutersThe download speed in Mbps when downloading content from Google Drive, Amazon, Dropbox, OneDrive andSharepoint
SteamDownload speedWhiteboxes, RoutersThe download speed in Mbps when downloading a game from Steam's network
 TCP connection timeWhiteboxes, RoutersThe TCP connection establishment time to Steam's network

Derived Metrics

Outages (ACCC public reports)

The ACCC use the 'Contiguous packet loss' test to derive an outages metric for use in the Measuring Broadband Australia reports. Please refer to the description of this test to understand how the test operates technically.

The test is configured to measure contiguous loss to the nearest off-net test server located in Australia.

A contiguous packet loss event is considered an 'outage' if and only if:

  1. The contiguous packet loss event lasts for at least 30 seconds; and
  2. It occurs outside of the hours of midnight to 05:59am, local time.

At this point, the total number of outages is computed for each Whitebox. This figure is then divided by the days that the Whitebox was online during the reporting period. This produces an 'average number of outages per day' for the Whitebox.

For example, if a Whitebox was online for 30 days and had experienced 15 outages during this period, then the average number of outages per day would be 0.5.

This per-Whitebox figure is then aggregated by RSP or access technology to produce the final figure that is used in the reports.