![\"Compare](\"https:\/\/lswp.store\/wp-content\/uploads\/2020\/05\/QUIC-UDP-XDP-1024x512.png\")
<\/p>\n<\/p>\n
HTTP\/3 uses QUIC which in turn uses UDP (User Datagram Protocol) rather than TCP\/IP for communications. XDP<\/a> (eXpress Data Path) has been added recently to several operating systems to provide User Space applications performance comparable to direct kernel applications.<\/p>\n The goal of this blog is to determine if QUIC can gain significant enough performance to justify implementing XDP where it is available. Understanding performance gains usually requires detail so besides a simple test, a profiler will be used to see exactly where performance gains occur.<\/p>\n XDP can be run with hardware support or can be run entirely in software. The goal is to have an XDP program be usable in the widest possible range of computing environments. This is best accomplished by targeting virtual machines<\/a>.<\/p>\n The QUIC engine used was the popular open-source LSQUIC library available from LiteSpeed Technologies. Only small modifications were made to the base library to support XDP. The XDP functionality was provided by a separate module, which implemented LSQUIC callbacks to send, receive, and allocate packets. Testing was performed using the The client machine ran a released version of the The server machine ran the modified version of the Simple volume tests of 2GB were executed to show performance differences between native UDP and XDP UDP. A profiler was used to examine the effect of XDP UDP versus native UDP in the program to help identify specific bottlenecks.<\/p>\n NOTE: A non-VM machine was chosen with a high speed data path so as to best simulate access to the wider internet in a controlled environment. Other works showing performance include this one<\/a>, which shows runs simply tossing away or moving data without any real involvement. In contrast, we believe our comparison is the first where a real application could be used.<\/p>\n 3 runs in a row were performed by the client machine pointing to the server machine which was configured to run in native UDP mode. Results reported were:<\/p>\n Resulting in an average of 102,077,709 bytes per second.<\/p>\n The server was restarted to run in XDP UDP mode and the client tests were repeated:<\/p>\n Resulting in an average of 146,116,331 bytes per second.<\/p>\n This is a 43% performance improvement by changing from native UDP sends to XDP UDP sends.<\/p>\n By examining the Since there\u2019s such a significant difference in performance a profiler was used to see exactly where the program was spending its time. In particular the profiler Processing the perf output with FlameGraph produces:<\/p>\n In a flamegraph<\/a> the widest (horizontal) bars take the most CPU. As you go up the graph, you go deeper into the call stack. At the bottom is Running the same perf run and FlameGraph with XDP results in:<\/p>\nThe Test Environment<\/h2>\n
LSQUIC<\/h3>\n
http_server<\/code> and http_client<\/code> programs provided by LSQUIC, modified as necessary.<\/p>\nhttp_client<\/code> program from the LSQUIC library and was always the receiver. It was specified to use the Q046 QUIC protocol version.<\/p>\nhttp_server<\/code> program from the LSQUIC library and was run with command line parameters to use either the native UDP or the XDP UDP. It ran in Interop mode, which is a mode of the program where data is transmitted without having to be read from disk.<\/p>\nHardware and Software<\/h3>\n
Server machine:<\/h4>\n
\n
Client machine:<\/h4>\n
\n
The Tests and Results<\/h2>\n
\n
\n
XDP Gets the Win<\/h3>\n
top<\/code> program on all runs we could see that the server program was running at 100% or nearly so. On the client machine the CPU differences were significant. When running in XDP UDP mode the CPU utilization was close to or exceeding 70%. When running in native UDP mode the CPU utilization was less than 60%. This implies that the limit was in the CPU on the server system, which is where the UDP transmissions were occurring.<\/p>\nFlame Graph from perf Profiling<\/h2>\n
perf<\/code> was used as it provides great detail.<\/p>\nFlame Graph for native UDP<\/h3>\n
![\"\"](\"https:\/\/lswp.store\/wp-content\/uploads\/2020\/06\/UDP-1024x1015.png\")
<\/a><\/p>\nhttp_server<\/code>. As you go up, you go into the program. The bar stays very wide until send_packets_out<\/code>, an LSQUIC function that has two expensive children, gquic_encrypt_packet<\/code> and send_batch<\/code>, the latter being the more expensive of the two. Following send_batch<\/code> it continues staying very wide into __libc_sendmsg<\/code> where it leaves the program entirely and is routed into the kernel. From this we can determine that the kernel overhead is by far the most CPU intensive cost.<\/p>\nFlame Graph for XDP<\/h3>\n
![\"QUIC](\"https:\/\/lswp.store\/wp-content\/uploads\/2020\/06\/XDP-576x1024.png\")
<\/a><\/p>\n![flamegraph<\/a> has a much faster separation into narrower functions indicating that there are more contributors to CPU cost, in other words, not a single bottleneck. As above the widest row stays wide going into the program until send_packets<\/code> out where it breaks into gquic_encrypt_packet<\/code> and send_batch<\/code>, where send_batch<\/code> is the slightly wider function. The entry point into the kernel is __libc_sendto<\/code> where it is barely the widest row at that depth indicating that CPU overhead of encryption is nearly as expensive as transmission.<\/p>\nThis gives us an indication that the kernel overhead is substantially reduced in XDP and explains the performance improvements.<\/p>\nConclusions<\/h2>\nThe goal of this blog post was to determine where the bottlenecks in QUIC and HTTP\/3 are and to see if the XDP UDP interface could help in improving the overall performance. The results of the tests show quite convincingly that XDP reduces the kernel penalty of native UDP.<\/p>\nWhile XDP UDP is not a perfect solution for many environments as it is Linux specific it appears that for Linux web servers implementing XDP will result in higher overall performance.<\/p>\n","protected":false},"excerpt":{"rendered":"Goal: Determine whether any performance gain can be achieved with XDP UDP versus kernel UDP HTTP\/3 uses QUIC which in turn uses UDP (User Datagram Protocol) rather than TCP\/IP for communications. XDP (eXpress Data Path) has been added recently to several operating systems to provide User Space applications performance comparable to direct kernel applications. The […]<\/p>\n","protected":false},"author":1,"featured_media":11326,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[20],"tags":[171,238,289,322],"class_list":["post-11324","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-performance","tag-lsquic","tag-quic","tag-udp","tag-xdp"],"jetpack_featured_media_url":"https:\/\/lswp.store\/wp-content\/uploads\/2020\/05\/QUIC-UDP-XDP.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts\/11324","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/comments?post=11324"}],"version-history":[{"count":0,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/posts\/11324\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/media\/11326"}],"wp:attachment":[{"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/media?parent=11324"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/categories?post=11324"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lswp.store\/index.php\/wp-json\/wp\/v2\/tags?post=11324"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}](\"https:\/\/lswp.store\/wp-content\/uploads\/2020\/05\/XDP.svg\"){kind=link}