Linux Plumbers Conf 2011

One Line Summary

Abstract

Byte queue limits are a mechanism to limit the size of the transmit hardware queue on a NIC by number of bytes. The goal of these byte limits is too reduce latency caused by excessive queuing in hardware without sacrificing throughput.

To avoid starvation, when the host is streaming data the hardware needs to queue enough data so that the queue does not completely drain between successive packet transmissions. In a canonical network driver model this equates to queuing enough data to span the time that the hardware transmit queue becomes full, the driver removes completed packets from the queue (usually at interrupt), and new packets are queued.

While arbitrarily large hardware queues could be used to prevent starvation, they induce high latency which has undesirable consequences. High priority packets sent by the host will incur large latency due to head-of-line-blocking. The work conserving queuing disciplines in Linux (like Stochastic Fairness Queuing (SFQ) and PRIO) need to have a backlog on the host before they can be effective; large hardware queue makes producing this backlog harder to achieve thus reducing the usefulness of these queuing disciplines.

Hardware queuing limits are typically specified in terms of a number hardware descriptors, each of which has a variable size. The variability of the size of individual queued items can have a very wide range. For instance with the e1000 NIC the size could range from 64 bytes to 4K (with TSO enabled). This variability makes it next to impossible to choose a single queue limit that prevents starvation and provides lowest possible latency.

The amount of queuing required to prevent starvation of the transmitter is much more dependent on the serialization time of the bytes as opposed to the number of packets being transmitted. Limiting the hardware transmission queue by number of bytes takes this fact into account to provide low latency without starvation.

The objective of byte queue limits is to set the limit so that it is the minimum needed to prevent starvation between successive transmissions to the hardware. The latency between two transmissions can be variable in a system. It is dependent on interrupt frequency, NAPI polling latencies, scheduling of the queuing discipline, lock contention, etc. Therefore we propose that byte queue limits should be dynamic and change in accordance with networking stack latencies a system encounters.

Tags

network, queue, buffer bloat

Speaker

• Tom Herbert

  Google
  Biography

  Manger of Host Networking group at Google.