Download PDF

Metrics.

The SamKnows measurement agents support a wide range of tests. Each of these tests will carry out its measurements (typically to dedicated test servers or directly to content providers), and will produce one or more metrics. These documents describe the methodology used to carry out each of the tests SamKnows supports and outlines the metrics that each test captures.

The availability of tests varies by platform. For example, the lightweight capacity test is not available for use in web tests, as the use of UDP is prohibited by web browsers.

Tests and metrics supported by platform

TestMetricPlatformsDescription
Download (TCP)Download speedWhiteboxes, Routers, Android, iOSThe download speed in Mbps when downloading (using TCP) random bytes from a test server
TCP RetransmissionsWhiteboxes, RoutersThe number of retransmitted TCP segments/packets
Burst download speedWhiteboxes, RoutersThe download speed during the first 5 seconds of a test
Sustained download speedWhiteboxes, RoutersThe download speed of the test during the last 5 seconds
Percentage of BestWhiteboxes, RoutersDownload speed result as a percentage of the user's best ever result
Percentage of AdvertisedWhiteboxes, RoutersDownload speed result as a percentage of their package's advertised downstream speed
Download (HTML5)Download speedWebThe download speed in Mbps when downloading (using TCP) random bytes from a test server using HTML5 APIs(WebSockets and Fetch)
Download (Lightweight UDP)Download speedWhiteboxes, RoutersThe download speed in Mbps when downloading (using UDP) from a test server, using less data than the TCP test
Download (Hardware acceleratedUDP)Download speedBroadcom-based RoutersThe download speed in Mbps when downloading (using UDP) random bytes from a test server
Upload (TCP)Upload speedWhiteboxes, Routers, Android, iOSThe upload speed in Mbps when uploading (using TCP) random bytes to a test server
Percentage of BestWhiteboxes, RoutersUpload speed as a percentage of their best ever upload speed
Percentage of AdvertisedWhiteboxes, RoutersUpload speed as a percentage of their advertised upload speed
Upload (HTML5)Upload speedWebThe upload speed in Mbps when uploading (using TCP) random bytes to a test server using HTML5 APIs (WebSocketsand Fetch)
Upload (Lightweight UDP)Upload speedWhiteboxes, RoutersThe upload speed in Mbps when transmitting (using UDP) to a test server, using less data than the TCP test.
Upload (Hardware acceleratedUDP)Upload speedBroadcom-based RoutersThe upload speed in Mbps when transmitting (using UDP) random bytes to a test server
LatencyLatencyWhiteboxes, Routers, Android, iOSAverage round-trip time of a series of echo packets transmitted over UDP
Packet LossWhiteboxes, Routers, Android, iOSPercentage of packets lost not received within two seconds of sending
Jitter UpWhiteboxes, Routers, Android, iOSVariability of packet latency upstream
Jitter DownWhiteboxes, Routers, Android, iOSVariability of packet latency downstream
Contiguous Packet Loss (Availability/Disconnections)Whiteboxes, RoutersPeriods of time where more than 2 consecutive packets are lost
MOS (Mean Opinion Score)Whiteboxes, Routers, Android, iOSGrading of the quality of a voice call, on a scale of 1 (worst) to 5 (perfect)
Latency (HTML5)LatencyWebRound trip time for a series of packets sent to our test server over WebSockets
Latency and packet loss(ICMP)LatencyWhiteboxes, RoutersAverage round-trip time of a series of ICMP echo packets
Packet LossWhiteboxes, RoutersPercentage of ICMP packets lost not received within two seconds of sending
Website PerformanceTestTime to visually completeWhiteboxes, RoutersThe time taken to render the complete web page, with the page 100% loaded visually
Time to engine completeWhiteboxes, RoutersThe time taken for the web browser's engine to say that the web page had finished loading (note that this maydiffer from visual completeness)
Time to first visualWhiteboxes, RoutersThe time taken to render the first visual element of a web page
Average object transfer timeWhiteboxes, RoutersThe average time taken to transfer each object referenced by the web page
Average object TCP connection timeWhiteboxes, RoutersThe average time taken to establish a TCP connection to the origin servers that host the web page content
Average object DNS resolution timeWhiteboxes, RoutersThe average time taken to resolve DNS for the content servers that provide the web page content
Failure RateWhiteboxes, RoutersThe failure rate when loading a web page
Web browsing litePage Load TimeWhiteboxes, Routers, Android, iOSMeasures the time taken to fetch the HTML and referenced resources from a page of a popular website
Failure RateWhiteboxes, Routers, Android, iOSThe percentage of failures to fetch the HTML of a website
DNSDNS Resolution TimeWhiteboxes, RoutersThe DNS resolution time of a selection of common website domain names
DNS Failure RateWhiteboxes, RoutersThe rate of DNS resolution failures
DNS-over-HTTPS Resolution TimeWhiteboxes, RoutersThe DNS resolution time of a selection of common website domain names using DNS-over-HTTPS
DNS-over-HTTPS Failure RateWhiteboxes, RoutersThe rate of DNS resolution failures when using DNS-over-HTTPS
NetflixNetflix Downstream SpeedWhiteboxes, RoutersThe download speed when downloading content from Netflix
Netflix Bitrate Reliably StreamedWhiteboxes, RoutersThe bitrate that can be reliably streamed without stalls
Netflix Startup DelayWhiteboxes, RoutersThe amount of time for the video to fill the buffer with two seconds of video / start playing
Netflix TCP Connect TimeWhiteboxes, RoutersThe TCP connect time to the Netflix server
Netflix Bitrate % of MaxWhiteboxes, RoutersThe bitrate as a percentage of the max bitrate attempted/possible
YouTubeYouTube Bitrate Reliably StreamedWhiteboxes, Routers, Android, iOSThe bitrate that can be reliably streamed without stalls
YouTube Startup DelayWhiteboxes, Routers, Android, iOSThe amount of time for the video to fill the buffer with two seconds of video / start playing
YouTube TCP Connect TimeWhiteboxes, Routers, Android, iOSThe TCP connect time to the YouTube server
YouTube Bitrate % of MaxWhiteboxes, RoutersThe bitrate as a percentage of the max bitrate attempted/possible
BBC iPlayeriPlayer Bitrate Reliably StreamedWhiteboxes, RoutersThe bitrate that can be reliably streamed without stalls
iPlayer Startup DelayWhiteboxes, RoutersThe amount of time for the video to fill the buffer with two seconds of video / start playing
iPlayer TCP Connect TimeWhiteboxes, RoutersThe TCP connect time to the iPlayer server
iPlayer Bitrate % of MaxWhiteboxes, RoutersThe bitrate as a percentage of the max bitrate attempted/possible
HuluHulu Bitrate Reliably StreamedWhiteboxes, RoutersThe bitrate that can be reliably streamed without stalls
Hulu Startup DelayWhiteboxes, RoutersThe amount of time for the video to fill the buffer with two seconds of video / start playing
Hulu TCP Connect TimeWhiteboxes, RoutersThe TCP connect time to the Hulu server
DOTA2LatencyWhiteboxes, RoutersICMP Round-trip time latency to a DOTA2 server
Packet LossWhiteboxes, RoutersICMP Percentage of packets lost not received within two seconds of sending
League of LegendsLatencyWhiteboxes, RoutersICMP Round-trip time latency to a League of Legends server
Packet LossWhiteboxes, RoutersICMP Percentage of packets lost not received within two seconds of sending
FIFA 2018/2019LatencyWhiteboxes, RoutersICMP Round-trip time latency to a FIFA server
Packet LossWhiteboxes, RoutersICMP Percentage of packets lost not received within two seconds of sending
Apex LegendsLatencyWhiteboxes, RoutersICMP Round-trip time latency to the Apex Legends servers
Packet LossWhiteboxes, RoutersICMP packet loss (as a percentage) for packets sent to the Apex Legends servers
FortniteLatencyWhiteboxes, RoutersICMP Round-trip time latency to the Fortnite servers
Packet LossWhiteboxes, RoutersICMP packet loss (as a percentage) for packets sent to the Fortnite servers
Multicast IPTVJitterWhiteboxes, RoutersThe jitter of the incoming video stream
Channel change timeWhiteboxes, RoutersHow long it takes to join a new multicast group (change TV channel)
CDNObject fetch timeWhiteboxes, RoutersHow long it takes to fetch a small object from the CDN
Object fetch speedWhiteboxes, RoutersHow fast an object can be fetched from the CDN
TCP connection establishment timeWhiteboxes, RoutersThe latency to the CDN
VOIP SIPJitter UpWhiteboxes, RoutersVariability of packet latency upstream for a SIP connection
Jitter DownWhiteboxes, RoutersVariability of packet latency downstream for a SIP connection
Mean Opinion Score (MOS)Whiteboxes, RoutersGrading of the quality of a voice call, on a scale of 1 (worst) to 5 (perfect)
Latency (HTML5)LatencyWebRound trip time for a series of packets sent to our test server over WebSockets
Latency and packet loss(ICMP)LatencyWhiteboxes, RoutersAverage round-trip time of a series of ICMP echo packets
Packet LossWhiteboxes, RoutersPercentage of ICMP packets lost not received within two seconds of sending
TracerouteNoneWhiteboxes, RoutersNo metrics at present, currently just used internally for debugging
Data usageWired LAN trafficWhiteboxes, RoutersVolume of customer traffic generated and received by wired LAN devices
Wi-Fi trafficWhiteboxes, RoutersVolume of customer traffic generated and received by Wi-Fi devices
SamKnows trafficWhiteboxes, RoutersVolume of SamKnows test traffic
Online StorageDownload speedWhiteboxes, RoutersThe download speed in Mbps when downloading content from Google Drive, Amazon, Dropbox, OneDrive andSharepoint
SteamDownload speedWhiteboxes, RoutersThe download speed in Mbps when downloading a game from Steam's network
TCP connection timeWhiteboxes, RoutersThe TCP connection establishment time to Steam's network
Social MediaLatencyWhiteboxes, RoutersThe round-trip latency to the video, audio, text and image servers for each different social media network. Latency is measured independently for downloading and uploading content, as they can use different endpoints.
Packet LossWhiteboxes, RoutersThe packet loss to the video, audio, text and image servers for each different social media network. Packet loss is measured independently for downloading and uploading content, as they can use different endpoints.
Hop countWhiteboxes, RoutersThe number of hops to the video, audio, text and image servers for each different social media network.

Derived Metrics

Outages (ACCC public reports)

The ACCC use the 'Contiguous packet loss' test to derive an outages metric for use in the Measuring Broadband Australia reports. Please refer to the description of this test to understand how the test operates technically.

The test is configured to measure contiguous loss to the nearest off-net test server located in Australia.

A contiguous packet loss event is considered an 'outage' if and only if:

  1. The contiguous packet loss event lasts for at least 30 seconds; and
  2. It occurs outside of the hours of midnight to 05:59am, local time.

At this point, the total number of outages is computed for each Whitebox. This figure is then divided by the days that the Whitebox was online during the reporting period. This produces an 'average number of outages per day' for the Whitebox.

For example, if a Whitebox was online for 30 days and had experienced 15 outages during this period, then the average number of outages per day would be 0.5.

This per-Whitebox figure is then aggregated by RSP or access technology to produce the final figure that is used in the reports.