Dream to stream
In 2022, video streaming is a normal part of daily life for many people, just like eating, walking, or talking. Anything from news broadcasts to entertainment, we watch on phones, tablets, computer screens, to large TVs and projector screens. But this also means that when video playback freezes, it’s one of the most frustrating user experiences.
When there’s a problem with video streaming, customers often call their broadband provider, assuming it’s an issue with their broadband. This is not necessarily the correct assumption, and can result in effort being focused in the wrong place and the problem not getting fixed. And the customer continues to have a less than perfect streaming experience.
SamKnows has developed many unique streaming video tests for the most popular services, but there's countless more services that we couldn't measure and monitor, until now.
Request a demo now, or read on to find out all the details.
Let’s get into the nitty gritty
Your streaming technologies handle varying bandwidth and quality levels
Streaming technologies have evolved to support multiple quality levels, often automatically selecting a suitable level based on the current streaming ability (240p, 1080p, etc.). The speed of the internet connection is not always constant. It can be influenced by several factors, including connection speed type (DSL, Fiber, Cable, etc.), sharing an internet connection, congestion on a network, or CDN (Content Delivery Network) limitations that may rarely cause an overloaded server.
To lower the impact of bandwidth issues, platforms and devices have taken certain actions, such as improving the efficiency of video compression, e.g. a new codec, which lowers the bandwidth requirement for a given quality level, automatically selecting the correct compression level based on your customer expectations, for example watching the news on the train would be different to watching an HD movie at home on a big screen while stationary, and finally, the increase of size for the playback buffer is adjusted. If you’re ever watching something and it buffers for a long time before playing, that’s because it is trying to reduce speed fluctuations and ensure it doesn’t play just to buffer again shortly.
Netflix created VMAF, their own visual quality solution
The key measurement of streaming is the video quality. Historically, the bitrate (or screen resolution) has been used to describe this. However, with different codecs and encoder settings, it doesn’t always capture the user experience very well. For example, using a higher bitrate with an old codec may be visually inferior to a new codec.
Netflix has created a solution to this problem with their perceptual quality metric, also known as VMAF, or Video Multimethod Assessment Fusion. It’s described in detail on their website. Overall, it provides a score on how close the picture is compared to the original, from the perspective of a human observer.
VMAF Score Example
VMAF scores have been calculated with varying screen resolutions. The relative difference from the highest quality have been highlighted below:
The human eye might be forgiving with a lower quality setting if the playback is smooth, but when lowering the quality levels too much, especially depending on the screen size and distance to the screen, the user experience will suffer.
Analyzing thousands of connections with the SamKnows Media Streaming Test
Capturing the video streaming capabilities of a single internet connection is useful. However, being able to look across hundreds or thousands of broadband connections when scaling up would allow us to compare much more.
This is why we have come up with the SamKnows Media Streaming Test.
You can use this test to monitor the performance of any HLS or DASH based streaming service, to see how they perform under heavy traffic, during a live event for example. If your streaming service is distributed across multiple content delivery networks you could benchmark the relative performance of each CDN. This will also allow you to compare the following:
- Streaming performance between different CDN providers, e.g. Akamai, Limelight, Cloudflare
- Streaming performance between you and other ISPs, and various speed tiers
- Geographical comparisons, e.g. countries, or urban/rural comparison
By monitoring your streaming services, you could use the data to optimize and improve your service, alert and inform your support teams if there's a problem with a service. Allowing your support staff to respond to customer queries with accurate informed information.
- - - - - - - - -
How the test works
The new SamKnows Media Streaming Test is built around FFmpeg and therefore supports a wide range of protocols, but it is primarily focusing on the following:
- HTTP Live Streaming (HLS)
- Dynamic Adaptive Streaming over HTTP (DASH):
Since the test has the same streaming/playback characteristics in every single device, the results can be compared fairly. The test supports both Video On Demand (VOD) as well as Live streaming manifests. Both technologies rely on storage of media content on a regular HTTP server, which is often powered by large content delivery providers, e.g. Akamai, Limelight or Cloudflare. These providers have extensive infrastructure in all corners of the world.
The test can run in two different modes:
1. Find reliably streamed bitrate: Determine the highest quality level for the internet connection. A quality ladder is used that is stepped from top to bottom, where a lower quality is selected if a stall event occurs.
The test can make use of previous speed measurements for an internet connection to give a hint to the test so that irrelevant quality levels can be skipped. This avoids wasting bandwidth and lowers the test duration.
2. Fixed bitrate: A specific quality level is selected with focus on reporting the degree of stalling. The number of stalls and time measurements.
Summary Of Test Results:
The test captures a large number of metrics, such as:
1. Connection establishment: name resolution, TCP & TLS connection time
2. For each media type (audio and/or video):
- Metadata, e.g. screen resolution, framerate, codec
- Initial buffering delay
- Average download speed
- Quality level (bitrate reliably streamed) and perceptual quality score (VMAF)
- Stall statistics
Running the test with an HLS manifest:
To verify our new test’s behavior, we’ve carried out some measurements inside our lab. We use a 2160p reference video, delivered over HLS in a video-on-demand fashion.
When delivered at different resolutions, our reference video yields the VMAF scores shown in the chart below:
Comparison of Streaming Test Service: HLS VOD quality levels (2160p reference)
To show how different internet connection speeds affect streaming quality, we have throttled the bandwidth with the help of a traffic shaper in our lab. The rates are selected slightly above each quality level to capture the full range of available video streams. They would represent speeds of internet connections, wired and wireless.
The test goes through the quality ladder until it finds a level that does not lead to stalling.
Find reliably streamed bitrate results:
Fixed bitrate results:
When the goal of the test is to assess the amount of stalling that occurs for a certain quality level, the fixed bitrate mode is used.
To face significant stall events, we picked 6 Mbps traffic shaping against an 18 Mbps stream.
While the test was running, additional measurements were taken of the download progress, also known as buffering progress:
When video is streaming successfully without stalling, the blue line typically stays around the 5 second line. When stalling occurs, download speed is not enough to keep up with the playback timeline. In general, the measurement of total stall duration is most important.
- - - - - - - - -
Interested in understanding your customers’ streaming experience?
Our sales team at SamKnows would be happy to talk you through the test and how it can better help you understand your customers’ streaming experience.