Connect America Fund reporting requirements

Connect America Fund Order

Back in July 2018 the FCC released an updated order specifying the auditing requirements for ISPs taking part in the Connect America Fund (CAF). In short, this order details how ISPs receiving CAF funds need to measure and report on their performance.

In this post we will summarise the CAF measurement requirements and how ISPs can meet them.

Summary of the CAF requirements

  • Download and upload speed to be measured hourly, and latency every minute.
  • Measurements should be carried out daily between 6pm and midnight, local time.
  • Measurements need to be conducted from hardware inside real customer’s homes (such as the router or a device connected directly to the router).
  • Measurements need to be conducted to (or via) one of 16 FCC-designated cities.
  • Minimum sample size requirements are enforced for the number of homes to be measured on each speed tier in each state.
  • At least 80% of download and upload speed measurements must hit 80% of the advertised rate.
  • At least 95% of latency measurements should be at or below 100ms.
  • Data must be sampled for one week per quarter, starting Q3 2019.
  • Data must be reported to the FCC annually on July 1st.

It is worth noting that the requirements differ slightly for ISPs that are over 500 miles from one of the FCC-designated cities. This would broadly include Alaska and Hawaii. For the sake of brevity, we won’t cover these in depth here. For more information, see pages 28–29 of the FCC order.

Measurement devices

The CAF requirements call for measurements to be carried out from “software installed on residential gateways or in equipment attached to residential gateways”.

SamKnows has an ISP solution for both of the options the FCC put forward.

Router SDK

SamKnows offers a Router SDK for the purposes of embedding its measurements inside residential gateways. This is already used on millions of CPE around the world with some of the world’s largest ISPs. Whilst it offers unrivalled flexibility, it is often not a viable option for smaller ISPs, who do not have control of the software running in their residential gateways (or perhaps do not even supply them to customers).

The Whitebox

The alternative option for small-to-mid size ISPs is the SamKnows Whitebox. The Whitebox is a purpose-built hardware measurement device manufactured by SamKnows, capable of measuring fixed-line broadband connections of up to 1Gbps. It supports a wide range of network and application-specific measurements, and can defer measurements in the presence of user-generated cross traffic.

“Cross-traffic” refers to traffic that users will generate whilst using their internet connection at the same time as we want to run our measurements. This is, of course, a bad thing. If measurements run whilst a user is using the internet connection then (1) the measurements will be impaired and (2) the user’s experience of whatever application they’re using may be harmed. The SamKnows Whitebox is fully able to take cross-traffic into account. It uses a pause-and-retry approach in the presence of cross-traffic that exceeds a configurable threshold (64kbit/s as standard).

The Whitebox can also automatically determine its installation location. This is not using IP address geolocation, which can be notoriously inaccurate. Instead, the Whitebox observes all neighbouring Wi-Fi MAC addresses and uses third-party Wi-Fi geolocation services to accurately pinpoint the device. This approach is commonly referred to as “Wi-Fi assisted GPS”.

The Whitebox has the following features:

  • Carries out automatic scheduled performance measurements.
  • Measures download and upload speeds of up to 1Gbps.
  • Cross-traffic detection of 802.11b/g/a/n/ac Wi-Fi traffic and wired Ethernet traffic.
  • LAN-to-WAN throughput of 1Gbps.
  • Operates as a bridge, does not introduce double-NAT’ing.
  • Plug-and-play installation, no configuration or engineer required.
  • Remote configuration and firmware updates.
  • Automatic detection of the installation address, based upon Wi-Fi geolocation

For a small deployment of the kind that the CAF requirements call for, the Whitebox is an ideal solution.

Measurements

Download and upload speed

The requirements call for download speed and upload speed to be measured for a duration of 10–15 seconds. There is no further detail in the document regarding the protocol (TCP or UDP), the number of concurrent connections, or anything else.

We suggest using a TCP-based speed test, with eight concurrent TCP connections, operating for a fixed-duration of 10 seconds. This mirrors the configuration used in the FCC’s Measuring Broadband America (MBA) program, which is cited in the CAF order as being sufficient to characterise performance.

The full description of the SamKnows TCP-based download and upload speed test can be found below:

The speed test measures the download and upload speed of the broadband connection in bits per second. The transfer is conducted over one or more concurrent HTTP connections (using the GET verb of download and the POST verb for uploads).

In the download speed test the client will fetch a portion of an infinitely sized binary (non-zero, randomly generated) payload hosted on an HTTP server on the target test server. The content is discarded as soon as it is received.

In the upload test the client will generate the payload itself (using /dev/urandom as a non-blocking source of random content) to send to the server. The measure of throughput may be optionally carried out on the server side (the receiver) in the upload test.

The speed tests (both download and upload) operate for either a fixed-duration (specified in seconds) or a fixed-volume (specified in MB). Where possible, a fixed-duration test is preferred as it will cater well for all broadband access speeds. However, a fixed-volume test may be necessary where predictability of bandwidth usage is desired.

For multiple TCP connection tests we typically recommend that eight concurrent connections are used. In some cases (e.g. where the round-trip time between client and server is very high) it may be necessary to increase this.

Factors such as TCP slow start are accounted for through the use of a “warm-up” period. This period begins as soon as the test starts and seeks to establish that the throughput has reached stable rate before starting the real test (which will continue over the same TCP connection(s)). It is important to note that the data transferred in the warm-up period is excluded from the main test results, but it is still recorded separately as a supplementary metric.

The speed test client will record the throughput, bytes transferred and time taken at the end of the test. It may also record these values at multiple intervals during the test. This is commonly used to help characterise the difference between ‘burst’ and ‘sustained’ throughput (where transfer speeds may be inflated at the start of a TCP connection).

Latency

The CAF requirements define a latency measurement as “a single measurement of latency, often performed using a single User Datagram Protocol (UDP) packet or a group of three Internet Control Message Protocol (ICMP) or UDP packets”. Later references in the document make clear that they are seeking round-trip latency measurements, not one-way.

SamKnows supports both ICMP and UDP latency measurements. For the CAF measurements, SamKnows recommends the use of ICMP, as this allows us to use measurement server infrastructure that is both cost-effective and immediately available across the US.

It is important to dispel the myth that ICMP latency measurements are worse than UDP ones because “routers treat ICMP with a lower priority”. Routers will indeed treat ICMP with a lower priority, but only if the router itself is the destination of the packet (i.e. you are pinging a router and hitting its CPU). If the router is simply forwarding the packet (as it is when we are pinging a server on the internet), then the router treats it with equal priority to every other packet (the packet never hits the router’s CPU). Servers do not treat ICMP with a lower priority.

The SamKnows ICMP latency measurement is defined as follows:

This test measures the mean round-trip time (RTT) of ICMP echo requests in microseconds from the Whitebox to a target test node. The client sends a minimum of five ICMP echo requests of 56 bytes to the target test node waiting up to three seconds for a response to each. Packets that are not received in response are treated as lost. The minimum, maximum and standard deviation of successful results are also recorded.

Measurement servers

The FCC require that ISPs “test speed and latency from the customer premises of an active subscriber to a remote test server located at or reached by passing through an FCC-designated IXP”. The sixteen “FCC-designated IXP” locations are as follows:

  • New York, NY
  • Washington, DC
  • Atlanta, GA
  • Miami, FL
  • Chicago, IL
  • Dallas-Fort Worth, TX
  • Los Angeles, CA
  • San Francisco, CA
  • Seattle, WA
  • Denver, CO
  • Salt Lake City, UT
  • St. Paul, MN
  • Helena, MT
  • Kansas City, MO
  • Phoenix, AZ
  • Boston, MA

SamKnows has devised a solution with Cloudflare, a leading CDN operator, to use Cloudflare’s edge nodes as measurement servers. SamKnows measurement server software runs on all of the Cloudflare edge nodes globally. This allows us to conduct download speed, upload speed and ICMP latency measurements directly to Cloudflare’s 33 different US-based locations. As you will see from Cloudflare’s network map, their server locations cover all of the FCC’s designated IXP cities (with the sole exception of Montana).

This approach allows us to begin carrying out CAF-compliant measurements all over the US, both very quickly (because the infrastructure already exists) and cost effectively.

However, this approach does not suit all deployments. The CAF requirements are quite strict: measurements must be conducted to or via one of the sixteen FCC designated IXP cities. With Cloudflare’s anycast-based network, we do not get to choose where our measurement traffic goes; it will simply be directed to the nearest Cloudflare location. So, if you’re an ISP in Pittsburgh and you have peering/transit in Pittsburgh, it is likely that your users will be hitting the Cloudflare Pittsburgh location, and there’s no way to control that. Pittsburgh is not an FCC-designated IXP location, so measurements to this location would not fulfil the CAF requirements.

In light of the above, SamKnows will perform some initial diagnosis with the ISP to determine whether the Cloudflare-based test servers are suitable. If not, then additional test servers (which SamKnows will purchase, provision and manage) will be required.

Sampling

The CAF sampling requirements state that a random selection of customers per-tier per-state need to be measured. The number of customers per-tier per-state varies according to the ISP’s customerbase:

  1. If the ISP has 50 or fewer customers on a tier in a state: 5 customers need to be sampled.
  2. If the ISP has 51–500 customers on a tier in a state: 10% of customers need to be sampled.
  3. If the ISP has over 500 customers on a tier in a state: 50 customers need to be sampled

Measurements need to be carried out during “peak hours” (defined at 6pm to midnight, seven days per week). Download and upload tests need to be run once per hour, whilst latency tests need to be run once per minute. Clearly, only automatic, scheduled measurements make sense to meet this requirement — manual user-driven measurements could not realistically deliver this.

It is important to note that the above are the _minimum_ sample sizes that need to be maintained. In our experience it is crucial to over-sample, to allow for customers who may go on holiday and switch the Whitebox off, or one of a dozen other reasons that means they may stop collecting measurements for a short period.

Reporting

The CAF documentation defines the two metrics by which ISPs will be measured:

  1. Download and upload speed: At least 80% of measurements must be at or above 80% of the advertised speed.
  2. Latency: At least 95% of measurements must be at or below 100ms round-trip latency.

Please note that the latency requirements differ for “high-latency bidders”. Latency is measured against a threshold of 750ms, but an additional requirement of meeting a minimum MOS (Mean Opinion Score) value is also introduced. For more information, see pages 28–29 of the FCC order.

Measurements must be sampled for one week per quarter, and then submitted to the FCC. Typically SamKnows recommends collecting measurements continuously, rather than just carrying out measurements for a short period each quarter.

Crucially, the first batch of results are due July 1, 2020. This must include data from Q3 2019 and Q4 2019. Effectively, this means that an ISP would need to start collecting measurements well before July 2019.

Effectively, this means that an ISP would need to start collecting measurements well before July 2019.

SamKnows offers a solution called SamKnows One that directly meets the FCC’s reporting requirements.

SamKnows One

SamKnows One is the web-based analytics, visualisation and management platform used by all SamKnows clients. This allows clients to view and analyse the collected measurement data in near-realtime.

We have recently developed new functionality inside SamKnows One to account for the CAF download, upload and latency reporting requirements. This allows ISPs that are using SamKnows One to directly export the measures that the FCC require, without having to do any analysis or number crunching themselves.

ISPs using SamKnows One also get access to all of the other functionality SamKnows One offers, including:

  • Advanced analytics and reporting, allowing both consumers and enterprise customers to analyse collected measurements.
  • Configurable dashboards providing an at-a-glance view of the state of the network.
  • Network monitoring tooling such as alerting to allow you to dive into analysing performance changes and monitor your network on a continuous basis.
  • A fully-featured management suite allowing users to be self-sufficient for most common tasks and control their measurement platform.
  • An advanced permissions and access control system.
  • A series of APIs allowing you to interact with SamKnows One systems from inside internal or automated systems.

SamKnows CAF solution

SamKnows offers the following solution to ISPs which fulfils all of the CAF requirements:

  • SamKnows Whitebox 8.0 hardware measurement agents or Router SDK.
  • Download, upload and ICMP latency measurements.
  • SamKnows test servers, running on the Cloudflare infrastructure. Or if this is unsuitable due to routing, then a dedicated test server in the nearest FCC-designated IXP location.
  • SamKnows One, for realtime reporting of results, including metrics tailored specifically to meet the FCC’s reporting requirements.

For more information and a quote please contact us at community@samknows.com.