Linux Plumbers Conf 2010

One Line Summary

Abstract

Data center traffic is sensitive to high network latencies, even if it only affects a small percentage of the overall traffic. For example, many data center applications break complex operations into hundreds, or thousands, of simpler ones that are distributed to individual hosts which then return their results. The latency of the complex operation is typically determined by the highest latency of the distributed simpler ones. That is, large network latencies that only affect 0.01% or 0.001% percent of the packets result in higher latencies for 1 to 10% of the application’s operations. High network latencies are the result of queue build-up or packet losses. Both of these arise as side effects of TCP’s congestion control mechanism where the congestion window increases until the network becomes congested and packet losses occur.

Instead, TCP-NV attempts to avoid congestion by decreasing its congestion window when persistent queue build-up is detected. It is based on TCP-BIC and behaves like TCP-BIC while no queue build-up is detected. Once it is detected, TCP-NV decrease its congestion window by an amount proportional to the amount of queue build-up detected. TCP-NV detects queue build-up through measurements of the RTT and rate instead of depending on explicit notifications by the network. The advantage of this approach is that no modifications to the networking infrastracture are required. The disadvantage is that noise in the measurements can result in incorrect signals. This is especially true in modern hardware, where mechanisms like TSO, LRO and interrupt coalescence can affect these measurements, especially with 10G NICs. In order to deal with these effects, TCP-NV incorporates various techniques to filter affected measurements.

TCP-NV is implemented in Linux and has been tested in simulation and with real hardware. Our tests indicate that NV is successful at reducing queue build-up and packet losses. For example, when 2 1Gbps hosts send 1 MB RPCs to a third host using TCP-BIC, packet losses are 1.5% to 3.8%. With TCP-NV the losses are less than 0.008%. Enabling Ethernet flow control eliminates packet losses for both BIC and NV. However, packet RTT increases up to 50ms when using BIC due to queuing, but stays under 1ms for NV. Finally, in simulations with 256 10 Gbps hosts, 99 percentile latencies for TCP-BIC grow up to 21ms, but stay under 1ms for TCP-NV.

Tags

Linux, kernel, networking, TCP

Presentation Materials

Speaker

• Lawrence Brakmo

  Google
  • Website: http://www.google.com/

  Biography

  Larry has been at Google for the last 4 years and is a member of the host networking team where he works on TCP related issues. Prior to joining Google he was a researcher at DoCoMo USA Labs, HP Labs and Digital’s Western Research Lab. As part of his Ph.D. thesis he was the main developer of TCP Vegas.