Data usage.

Bandwidth consumption by users is growing rapidly. Our data usage test provides a way to measure this growth.

Back to top

Data usage

Supported clients: Whiteboxes, Routers

Passively measures the amount of data used on the user's broadband connection. This test does not inspect or record the contents of packets; it only records volumes. Volumes are reported hourly and are broken down by customer wired traffic, customer Wi-Fi traffic and SamKnows test traffic (i.e. the volume of traffic used in performing all of the SamKnows measurements). All volumes are reported independently (one value is not inclusive of any other) and are captured at the frame level, meaning that they include Ethernet, IP and TCP/UDP header overheads.

The precise mechanism used to record data usage volumes varies by the hardware platform in use. In some devices a managed hardware Ethernet switch is present; in these cases the port counters are simply interrogated and recorded (the CPU does not even see the traffic). In other devices only a software switch is present, and in these cases the counters from the software bridge in the Linux kernel are taken. For Wi-Fi, if the device in question is a Whitebox then wireless traffic is monitored passively, and the volumes of the encrypted packets will be recorded. If the device in question is a CPE, then the wireless traffic will be visible locally in unencrypted form, and the volumes will simply be recorded from the network interface's counters.

The measurement client records the following each hour:

  • SamKnows test traffic received
  • SamKnows test traffic transmitted
  • Customer wired LAN traffic received
  • Customer wired LAN traffic transmitted
  • Customer Wi-Fi traffic received
  • Customer Wi-Fi traffic transmitted

Output format

bgtrafmon outputs the result in the following format at the specified frequency:

DATAUSAGE.1;<dtime>;<sk_tx_bytes>;<sk_rx_bytes>;<cust_wired_tx_bytes>;<cust_wired_rx_bytes>;<cust_wifi_tx_bytes>;<cust_wifi_rx_bytes>;

Examples

DATAUSAGE.1;<dtime>;<sk_tx_bytes>;<sk_rx_bytes>;<cust_wired_tx_bytes>;<cust_wired_rx_bytes>;<cust_wifi_tx_bytes>;<cust_wifi_rx_bytes>;
Examples

DATAUSAGE.1;1529665986;2629462;102265170;441;6342;0;1232 DATAUSAGE.1;1529665996;1225597;46523156;646;5673;0;462

These fields have the following definitions:

Number Field Type Notes
1 dtime Integer Time report printed in UTC (all other fields refer to the preceding 60 mins)
2 sk_tx_bytes Integer Bytes transmitted by SamKnows tests in the report interval
3 sk_rx_bytes Integer Bytes received by SamKnows tests in the report interval
4 cust_wired_tx_bytes Integer Bytes transmitted by customer's wired LAN devices in the report interval
5 cust_wired_rx_bytes Integer Bytes received by customer's wired LAN devices in the report interval
6 cust_wifi_tx_bytes Integer Bytes transmitted by customer's wired LAN devices in the report interval
7 cust_wifi_rx_bytes Integer Bytes received by customer's wired LAN devices in the report interval

All byte counts include Ethernet and IP headers.

Supported devices

  • SK WB8
  • AC1750v2
  • WDR3600
  • WR1043ND
  • WR741ND
  • WR741NDv4
  • WNR3500L

Platform limitations

In an ideal scenario, "customer's Ethernet devices' data usage" should be limited to LAN <-> Internet traffic, "customer's Wi-Fi devices' data usage" should be limited to WLAN <-> Internet traffic and "SamKnows Test data usage" should be separated from OAM traffic. None of the platforms are able to provide these isolated counts.

  • "customer's Ethernet devices' data usage" includes LAN <-> WLAN traffic on all devices.
  • "customer's Wi-Fi devices' data usage" also includes LAN <-> WLAN traffic on all devices except the WNR3500L.
  • "SamKnows Test data usage" includes OAM traffic on all devices. Counters are only available for all CPU tx and rx traffic which also includes all OAM traffic.

The wired data usage and SamKnows test counters are derived from Ethernet layer counters available on the device, while the Wi-Fi statistics are derived from pcap capture filters (except on the WNR3500L). The accuracy of the counters provided by different platforms and what types of traffic they include varies, and the details can be found in the device sections that follow.

Based on source and destination, the network traffic on a Whitebox can be split into the following categories.

LAN <-> Internet

Customers' Internet usage from devices with a wired connection

No isolated counter is available for this traffic type on any of the platforms. Where a platform provides WAN port and CPU port counters, the difference of the WAN port counter and CPU port counter is used as a substitute.

LAN <-> LAN

Data usage between customers' devices within the LAN for file sharing between PCs, phones, NAS, etc where both devices have a wired connection to the CPE/Whitebox. This traffic should be excluded from the Wired Data Usage counters.

All platforms exclude this from WAN port counters.

LAN <-> WLAN

Data usage between customers' devices for file sharing between PCs, phones, NAS, etc where one device has a wired connection to the CPE/Whitebox and the other has a wireless connection. This traffic should be exclude from Wired and Wireless Data Usage counters.

No isolated counter is available for this traffic type on any of the platforms. This traffic can not be excluded from Wired and Wireless Data Usage counters.

Whitebox <-> Internet

OAM traffic and SamKnows test traffic

No isolated counter is available for this traffic type on any of the platforms. On platforms that provide a CPU port counter, that is as used as an alternative, but that includes all OAM traffic between the CPU and the Internet and the CPU and all devices on LAN and WLAN.

Whitebox <-> LAN

OAM traffic

No isolated counter is available for this traffic type on any of the platforms. On platforms that provide a CPU port counter, it includes this traffic type.

WLAN <-> Internet

Customers' Internet usage from devices with a wireless connection

No isolated counter is available for this traffic type on any of the platforms. The Nl80211Counter or BCMWlCounter is used as an alternative.

WLAN <-> WLAN

Data usage between customers' devices within the WLAN for file sharing between PCs, phones, NAS, etc where both devices have a wireless connection to the CPE/Whitebox

No isolated counter is available for this traffic type on platforms other than the WNR3500L. More details can be found in the BCMWlCounter section.

Whitebox <-> WLAN

There's no WLAN traffic sent to or from the Whitebox on devices other than the WNR3500L. Non-WNR3500L devices do not connect to the wireless access point that the customers' devices connect to, but use the WLAN interface in monitor mode on the selected access point's frequency to capture the packet headers and extract packet sizes. The WNR3500L on the other hand works as an access point and all wireless devices connect to it.

Counter types

SwLibCounter - libsw

On devices that have a hardware switch controller the libsw API provides some of the functionality of the swconfig utility with which we can obtain the counters for individual switch ports. The SamKnows Test usage + OAM data usage counts are taken from the CPU-port counter and wired data usage is calculated by subtracting the CPU-port counter from the WAN-port counter.

For all the devices where this counter is used, customer's LAN devices are connected to the LAN ports of the Whitebox, and the Whitebox's WAN-port is connected to one of the LAN ports of the CPE. Since the CPE is the WLAN access point, all LAN <-> WLAN traffic must pass through the Whitebox's WAN-port to the CPE's LAN port and then routed to the CPE's WLAN. So the WAN-port counters always include LAN <-> WLAN traffic.

SysClassCounter - /sys/class/net interface

On devices that do not have a hardware switch controller or no libsw support, the counters from the /sys/class/net interface are used. The calculations for each device type are different and are detailed in the device sections that follow.

On the WR741ND and WR741NDv4, customer's LAN devices are connected to the LAN ports of the Whitebox, and the Whitebox's WAN-port is connected to one of the LAN ports of the CPE. Since the CPE is the WLAN access point, all LAN <-> WLAN traffic must pass through the Whitebox's WAN-port to the CPE's LAN port and then routed to the CPE's WLAN. So the WAN-port counters always include LAN <-> WLAN traffic.

Nl80211Counter - iw utility and libpcap

This counter first scans for all available networks and selects one based on a partial MAC address match or signal strength. It then sets up the interface in monitor mode and captures packet headers using libpcap. The Rx bytes filter captures packers where the fromDS field is set to true and address 2 matches the selected access point's mac address. The Tx bytes filter captures packets where the toDS field is set to true and address 1 matches the selected access point's mac address.

The tx and rx bytes may be lower than the expected count when not all the 802.11 frames are received by the Whitebox. This can happen due to several reasons:

  • The access point and WLAN endpoint communicate using a standard (a/b/g/n/ac) not supported by the Whitebox.
  • The access point and WLAN endpoint communicate in a 5GHz band but the Whitebox model only supports 2.4GHz.
  • The signal level received by the Whitebox can be too low. This is common when the access point uses Beamforming.
  • The access point and WLAN endpoint communicate using a higher bandwidth (40/80/160 MHz) but the Whitebox may not support that bandwidth.

BcmWlCounter - Broadcom's wl utility

Uses the best effort category wme counters from wl. The wl wme_counters output gives separate counters for the four WME access categories - voice, video, best effort and background. The best effort category (AC_BE) counters are used as wireless traffic counters. The other categories show negligible counts and are excluded.

The AC_BE counters include rx, tx and fwd byte counts. All traffic between devices on the WLAN is reported as fwd_bytes. WiFi Rx and Tx byte counts are calculated by subtracting the AC_BE fwd_bytes count from the AC_BE rx_bytes as well as AC_BE tx_bytes.

Handling counter overflows

Since the units keep running for extended periods of times, the raw counters obtained by the different counter types can be quite large and can roll over as well. To minimise the complexity in handling overflows when adding large numbers and in detecting rolled over counters, all counter types first obtain the deltas between the current and previous values of all raw counters. All counter arithmetic calculations are limited to two operands and overflows are handled for each calculation.

SK WB8

Customer ethernet usage counter

  • Type: SwLibCounter
  • Includes:
  • Limitations

Customer Wi-Fi usage counter

  • Type: Nl80211Counter
  • Includes:
  • Limitations

SK test usage counter

  • Type: SwLibCounter
  • Includes:

AC1750v2

Customer ethernet usage counter

  • Type: SwLibCounter
  • Includes:
  • Limitations

Customer Wi-Fi usage counter

  • Type: Nl80211Counter
  • Includes:
  • Limitations

SK test usage counter

  • Type: SwLibCounter
  • Includes:
  • Limitations

WDR3600

Customer ethernet usage counter

  • Type: SwLibCounter
  • Includes:
  • Limitations

Customer Wi-Fi usage counter

  • Type: Nl80211Counter
  • Includes:
  • Limitations

SK test usage counter

  • Type: SwLibCounter
  • Includes:
  • Limitations

WR1043ND

Customer ethernet usage counter

  • Type: SwLibCounter
  • Includes:
  • Limitations

Customer Wi-Fi usage counter

  • Type: Nl80211Counter
  • Type: Nl80211Counter (2.4 GHz only)
  • Includes:
  • Limitations

SK test usage counter

  • Type: SwLibCounter
  • Includes:
  • Limitations

WR741ND and WR741NDv4

Customer ethernet usage counter

  • Type: SysClassCounter
  • Includes:
  • Limitations

Customer Wi-Fi usage counter

  • Type: Nl80211Counter
  • Includes:
  • Limitations

SK test usage counter

  • Type: SysClassCounter
  • Includes:
  • Limitations

WNR3500L - bridge mode

Customer ethernet usage counter

  • Type: SysClassCounter
  • Includes:
  • Limitations

Customer Wi-Fi usage counter

  • Type: BcmWlCounter
  • Includes:
  • Limitations

SK test usage counter

  • Type: SysClassCounter
  • Includes:
  • Limitations

WNR3500L - router mode

Customer ethernet usage counter

  • Type: SysClassCounter
  • Includes:
  • Limitations

Customer Wi-Fi usage counter

  • Type: BcmWlCounter
  • Includes:
  • Limitations

SK test usage counter

  • Type: SysClassCounter
  • Includes:
  • Limitations

SK WB8

2 GHz

  • Capabilities
  • Frequencies

5GHz

  • Capabilities
  • VHT Capabilities
  • Frequencies

AC1750v2

2 GHz

  • Capabilities:
  • Frequencies:

5 GHz

  • Capabilities:
  • VHT Capabilities:
  • Frequencies:

WDR3600

2 GHz

  • Capabilities: 0x11ef
  • Frequencies:

5 GHz

  • Capabilities: 0x11ef
  • Frequencies:

WR1043ND

2 GHz

  • Capabilities
  • Frequencies

WR741ND

2 GHz

  • Capabilities
  • Frequencies

WR741NDv4

2 GHz

  • Capabilities
  • Frequencies

WNR3500L

2 GHz

  • Capabilities
  • Frequencies