Experiences in Design and Implementation of a High Performance Transport Protocol Yunhong Gu, Xinwei Hong, and Robert L Grossman National Center for Data Mining Outline • • • • • TCP’s inefficiency in grid applications UDT Design issues Implementations issues Conclusion and future work TCP and AIMD • TCP has been very successful in the Internet – AIMD (Additive Increase Multiplicative Decrease) • Fair: max-min fairness • Stable: globally asynchronously stable • But, inefficient and not scalable – In grid networks (with high bandwidth-delay product) • RTT bias Efficiency of TCP Gb/s link, 200ms RTT, between Tokyo and Chicago 28 minutes On 10 Gb/s link, 200ms RTT, it will take hours 43 minutes to recover from a single loss TCP’s throughput model: S RTT 2p It needs extremely low loss rate on high bandwidth-delay product networks Fairness of TCP Amsterdam 100ms Gb/s Chicago 1ms Chicago 1Gb/s Merge two real-time data streams From Chicago to Chicago 2: 800Mbps From Amsterdam to Chicago 2: 80Mbps The throughput is limited by the slowest stream! UDT – UDP-based Data Transfer Protocol • Application level transport protocol built above UDP • Reliable data delivery • End-to-end approach • Bi-directional • General transport API; not a (file transfer) tool • Open source UDT Architecture  Pkt Scheduling Timer Sender Sender DATA ACK Recver NAK  Retransmission Timer  Rate Control Timer ACK2  ACK Timer  NAK Timer Recver UDT – Objectives • Goals – – – – Easy to install and use Efficient for bulk data transfer Fair Friendly to TCP • Non-goals – TCP replacement – Messaging service Design Issues • Reliability/Acknowledging • Congestion/Flow Control • Performance evaluation – Efficiency – Fairness and friendliness – Stability Reliability/Acknowledging • Acknowledging is expensive – Packet processing at end hosts and routers – Buffer processing • Timer-based selective acknowledgement – Send acknowledgement per constant time (if there are packets to be acknowledged) • Explicit negative acknowledgement Fairness and Friendliness 50 TCP flows and UDT flows between SARA and StarLight Realtime snapshot of the throughput The UDT flows have similar performance and leave enough space for TCP flows TCP Friendliness • Impact on short life TCP flows – 500 1MB TCP flows with 1-10 bulk UDT flows, over 1Gb/s link between Chicago and Amsterdam TCP Throughput (Mbps) 80 70 60 50 40 30 20 Number of UDT flows 10 Stability • Stability index of UDT and TCP – Stability: average standard deviation of throughout per unit time – 10 UDT flows and 10 TCP flows with different RTTs Stability Index 0.8 0.6 UDT TCP 0.4 0.2 -2 10 10 -1 10 10 RTT (ms) 10 10 Implementations Issues • • • • • Efficiency and CPU utilization Loss information processing Memory management API Conformance Efficiency and CPU utilization • Efficiency = Mbps/MHz • Maximize throughput – Use CPU time as little as possible, so that CPU won’t be used up before network bottleneck is reached – Remove CPU burst, which can cause packet loss: even distribution of processing • Minimize CPU utilization Loss Processing • On high BDP networks, the number of lost packets can be very large during a loss event • Access to the loss information may take long time • Acknowledge may take several packets Number of Loss Packets 3000 2000 1000 0 10 20 30 40 50 Loss Events 60 70 80 90 100 Loss Processing • UDT loss processing – Most loss are continuous – Record loss event other than lost packets – Access time is almost constant Access Time (us) 0 10 20 30 40 50 60 Loss Events 70 80 90 100 Memory Processing • • • • Memory copy avoidance Overlapped IO Data scattering/gathering Speculation of next packet User Buffer Data Protocol Buffer Protocol Buffer New Data API • Socket-like API • Support overlapped IO • File transfer API – sendfile/recvfile • Thread safe • Performance monitoring API - Example int client = socket(AF_INET, SOCK_STREAM, 0); connect(client, (sockaddr*)&serv_addr, sizeof(serv_addr)); If (-1 == send(client, data, size, 0)) { //error processing } UDTSOCKET client = UDT::socket(AF_INET, SOCK_STREAM, 0); UDT::connect(client, (sockaddr*)&serv_addr, sizeof(serv_addr)); If (UDTERROR == UDT::send(client, data, size, 0)) { //error processing } Implementation Efficiency • CPU usage of UDT and TCP – UDT takes about 10% more CPU than TCP – More code optimizations are still on going CPU Usage (%) 60 50 40 30 20 udt sending udt receiving tcp sending tcp receiving 10 0 10 15 20 25 30 Sample Event 35 40 45 50 Conclusion • TCP is not suitable for distributed data intensive applications over grid networks • We introduced a new application level protocol named UDT, to overcome the shortcomings of TCP • We explained the design rationale and implementations details in this paper Future Work • Bandwidth Estimation • CPU utilization – Self-clocking – Code optimization • Theoretical work References • More details can be found in our paper • UDT specification – Draft-gg-udt-01.txt • Congestion control – Paper on Gridnets '04 workshop • UDT open source project – http://udt.sf.net Thank you! Questions and comments are welcome! For more information, please visit Booth 653 (UIC/NCDM) at Exhibition Floor UDT Project: http://udt.sf.net NCDM: http://www.ncdm.uic.edu ... Efficiency and Fairness Characteristics • Takes 7.5 seconds to reach 90% of the link capacity, independent of BDP • Satisfies max-min fairness if all the flows have the same end-to-end link capacity... (Additive Increase Multiplicative Decrease) • Fair: max-min fairness • Stable: globally asynchronously stable • But, inefficient and not scalable – In grid networks (with high bandwidth-delay product)... shortcomings of TCP • We explained the design rationale and implementations details in this paper Future Work • Bandwidth Estimation • CPU utilization – Self-clocking – Code optimization • Theoretical