Get all of the QUIC news, tips, and thoughts from our developers here.
LiteSpeed believes in QUIC. We have publicly released our open source QUIC Client Library, we are participating in the QUIC standardization process alongside Google, Microsoft, Facebook, and others, and we currently power more than 97% of sites worldwide that run QUIC.
“I apologize for being Martin Thomson.” – Martin Thomson, DoH WG Meeting, IETF 104, 3/26/2019
IETF 104 has come and gone. Held in Prague last week, the conference had plenty of interesting meetings. I wanted to follow the progress of the QUIC, TLS, HTTPbis, and DoH Working Groups. I attended the meetings remotely despite the inconvenient time difference.
Where are the browsers?
It takes two to tango. While there are many server implementations at various stages of development (for example, did you know that www.litespeedtech.com can be fetched using ID-18 version of the IETF QUIC?), there are no browsers that use IETF QUIC protocol. David Schinazi (Google) reported that Chromium only supports the QUIC handshake for now, while Eric Kinnear (Apple) said that Safari has a functional IETF QUIC branch.
Discarding Old Keys
As QUIC uses TLS 1.3, it must somehow support key updates. There have been several proposals how to do this. Martin Thomson (Mozilla) summarized them in his presentation to the Working Group (see slides). The august body, predictably, failed to reach consensus and thus compelled the chairs to empanel YADT (Yet Another Design Team) headed by David.
Network Tools
A trifecta of presentations on QUIC tooling engendered much enthusiasm among the TSVAREA meeting attendees. The presentations were
Logging, Tooling and Debugging for Modern Network Protocols. Robin Marx (Hasselt University) has been researching how and what information to log in order to be able to generate informative pictorial representations of various connection events. Robin proposed to create a new, structured log format (tentatively named qlog), which, when used by both endpoints, will facilitate debugging or troubleshooting network issues. The collection of use cases he has compiled is impressive. The audience even proposed creating a dedicated qlog Working Group to log everything – and it just may happen. Stay tuned for updates.
QUIC tracing. Victor Vasiliev (Google) has been working on logging and tracing network events since he was an intern at Google (while still an intern, he was the one who spotted a symptom of the Linux Cubic quiescence bug). Victor’s tooling focuses on connection performance and also relies on rich data logging.
QUIC Logging: The In-Network View. Jari Arkko’s (Ericsson) presentation described spindump – a tool to observe and display characteristics of a QUIC connection using the QUIC spin bit. The spin bit is a controversial property of the QUIC short packet header and it was the subject of much contentious debate. (LiteSpeed position from the start was to support the inclusion of the spin bit into the QUIC protocol. We implemented spin bit support a few weeks ago and deployed it on our test endpoints as well as www.litespeedtech.com).
Part of a slide from Robin’s presentation.
Mind the Gap
The QUIC WG Interim meeting scheduled at the end of May in London may become the last our WG holds. As usual, the official IETF WG business will be preceded by two days of interop, which will target the (yet unpublished) 20th revision of the IETF QUIC Draft.
The QUIC WG chairs, Lars Eggert and Mark Nottingham, instructed the editors to plan for submitting the drafts to the IESG in July. In the next sentence, they qualified that date as “aspirational…”
We are excited to announce our brand new QUIC exploration service: HTTP3Check.
As you may know, LiteSpeed is committed to the new QUIC transport protocol as well as HTTP/3, the latest iteration of HTTP. gQUIC support comes standard with LiteSpeed Web Server and LiteSpeed Web ADC. Our own site features IETF-QUIC ID-18 and we are currently working hard on supporting HTTP/3 as well.
As these new protocols are rolled into our products as well as servers across the Internet, it is helpful to have tools for searching, checking, and troubleshooting. For this reason, we developed a hosted QUIC and HTTP/3 search tool called HTTP3Check.
Exploring QUIC With HTTP3Check
With this service, users can query servers by hostname to check for QUIC and HTTP/3 support.
Try searching our site, www.litespeedtech.com, by following this link: https://www.http3check.net/?host=www.litespeedtech.com.
See popular searches to get an idea of what sites users are searching.
HTTP3Check provides various items and metrics about the server’s QUIC support. With its first connection, the tool asks the given endpoint for all of the QUIC versions that it supports. After a successful first connection, the tool will open a second connection to test the server’s 0-RTT support. Along with the HTTP Header and the server’s certificate chain, the tool keeps track of packet and handshake timestamps to give the user an idea of the endpoint’s performance and latency.
Finally, HTTP3Check provides the user with the QLog results for the attempted connections. QLog is a standard under development for QUIC Connection debugging and visualization. Since the QLog contains information about all incoming packets and other important events, it is a great way to compare and troubleshoot servers.
As the Internet prepares for new and improved protocols, prepare yourself with the tools necessary to search, query, and troubleshoot QUIC and HTTP/3.
As we develop our own IETF QUIC implementation, we are focused on protocol correctness and robust performance. To get a better idea of the extent of our implementation comparative performance, we decided to benchmark it against other publicly available implementations.
Since there are few stable IETF QUIC implementations, we have restricted the list of servers tested to only mature GQUIC implementations. We have made the assumption that the performance for an IETF QUIC implementation and its GQUIC counterpart should match closely.
Benchmark
We have designed a flexible QUIC benchmarking program and included a testing script for easy test automation. This open-source program is available in the lsquic repository on Github. It has the ability to request an array of files over many concurrent connections using many concurrent streams. This flexibility makes it an essential tool for understanding the performance of a QUIC implementation.
We used this program and testing script to run a series of benchmark tests to understand the strengths and weaknesses of our own QUIC implementation, as well as other publicly available implementations. The benchmarking tests are meant to push servers to their limit and simulate realistic QUIC traffic. They range from requesting many small files to requesting a few large files.
Methods
The benchmark tests follow the client-server model and take place on a private network with a 10G link between the two testing machines. The client testing program used is the http_client sample client distributed with the lsquic-client repository.
The independent variables in this benchmark are server type, the number of requests, the size of requests, the number of concurrent clients, the number of concurrent connections per client, and the number of concurrent streams per client.
The server types tested in this experiment are lsws, proto-quic, caddy, and quic-go. Not all server types will be included in each test.
The dependent variables in this benchmark are client time, client CPU usage, server CPU usage, and in some tests, packets received on client machine.
The testing script runs five trials and calculates the average client time as well as the standard deviation across trials. Using the Linux top(1) command, the tester manually monitors client and server CPU usage and reports an average value, rounded to the nearest factor of 5.
Results
Round 1: Many Small Requests
Description
With the first test in this series of benchmarks, looked at how each server would handle a high number of small requests. A single test client would open one hundred concurrent connections, with a maximum of ten concurrent request streams. The client would make one million requests; each request was for a 630-byte file.
Results
server
Client_time_avg (sec)
client_time_stdev
Client_cpu (%)
Server_cpu (%)
lsws
12.829
0.079
90
100
proto-quic
42.462
2.187
90
100
quic-go
60.421
0.407
85
295
caddy
67.706
0.380
80
335
Discussion
The server with the lowest average client time was the LiteSpeed Web Server. It reached full server CPU utilization for a single thread of execution. Next, Google’s proto-quic server had the second lowest average client time and reached full server CPU utilization for a single thread of execution. Finally, the Golang-based implementations, quic-go and Caddy, reported the highest average client time. Additionally, both quic-go and Caddy used more than a single thread, consuming more resources that the previously reported servers.
Round 2
Description
A single test client opens ten concurrent connections, with a maximum of ten concurrent request streams. The client makes four hundred requests; each request is for a 10-megabyte file.
The LiteSpeed Web Server (lsws) trial with the lowest average client time is the HTTP/2 protocol. Next, the HTTP/1.1 protocol had the second lowest average client time. Finally, in comparison with the other LiteSpeed Web Server trials, the QUIC protocol had the highest average client time.
When compared to other QUIC trials, LiteSpeed Web Server (lsws) using QUIC showed the lowest average client time. It is followed by Google’s proto-quic server and the Golang-based implementations, quic-go and caddy.
Last week, IETF 103 was held in Bangkok. Those lucky enough to attend in person had fun eating durians, attending the Diwali celebration at a famous local temple, or getting patted down by a few sullen Thai law enforcement officers. Since I was attending remotely, I had to content myself with eating tomato soup with sardines at 3 o’clock in the morning… and not getting harassed by the Thai police.
The new HTTP/3 logo from Mike Bishop’s HTTP/QUIC presentation. Just kidding, this won’t be the logo! …I think.
HTTP/3
The unwieldy “HTTP over QUIC” (or HQ for short) is now known as HTTP/3 (or H3). This big change was sudden to many and set the Internet abuzz. Undoubtedly, this step will be considered controversial. And yet, the name change was approved by both QUIC and HTTPbis Working Groups. (See relevant minutes: 1, 2.) Concurrently, it was decided that the HTTPbis WG will take on maintenance of HTTP/3 and QPACK documents after they are published as RFCs.
LiteSpeed first with functional HTTP/3 server
During the QUIC Interop Hackathon that accompanied IETF 103, LiteSpeed Technologies became the first entity to operate a functional HTTP/3 server. On the night of November 6th, a Facebook HTTP/3 client made several successful GET and POST requests to the LiteSpeed HTTP/3 server. After some fixes to the Facebook server, we were able to return the favor on the afternoon of November 8th when the LiteSpeed client fetched resources from the Facebook server.
Or so we’ve been told by Martin Thomson and other QUIC luminaries. Here is a (likely incomplete) list of changes which will make the new version incompatible with the previous version:
First byte (a.k.a. octet zero) of both long and short headers will change.
A slide from Martin Thomson’s First Octet presentation.
Farewell to Q043 and earlier
The octet zero change means that servers will not be able to support older versions of Google QUIC — Q043 and earlier — and IETF QUIC at the same time. Ian Swett informed us that Google plans to deploy IETF QUIC Draft-17 at scale at which point they, too, will have to drop support for pre-Q044 QUIC.
The spin bit is in
A diagram from Marcus Ihlar’s Spin Bit presentation
The QUIC Working Group voted to include the spin bit into the specification. Having been the subject of a rather contentious debate at IETF 101 in London, the spin bit is an optional protocol feature.
Chrome’s and Firefox’s stated intent of not implementing the spin bit makes its value questionable. On the other hand, Microsoft, LiteSpeed Technologies, and F5 pledged to support it in both their server and client products and Apple plans to implement it on the client.
Is there life after QUIC?
At the end of the second (and last) QUIC session, we discussed the road ahead. The plan it to freeze the 17th version of the draft family and get many implementations going. The finish line has been moved again, this time to July 2019.
Shall we disband after QUIC is released? There has been talk of rechartering the Working Group to take on new challenges, such as multipath and partial reliability.
Given the rate of protocol changes that are still being worked on, that point still seems pretty far off to me. Now, it’s time to roll up the sleeves and finish our server implementation. Until next time!
Last week, I attended the QUIC Working Group Interim meeting in New York City. Our group is one of many at the IETF. We are working on standardizing the IETF QUIC protocol. The QUIC protocol that is currently used on the Internet is called Google QUIC. It started as a Google experiment between the Chrome browser and some of the Google services a few years ago. We added support for Google QUIC to the LiteSpeed Web Server and ADC products in 2017. At around the same time, Google handed off the protocol to IETF for standardization. This way, everyone can implement QUIC and reap the benefits of a more performant web. I’ve blogged about previous meetings.
This meeting lasted four days: Monday and Tuesday were dedicated to the Interop Hackathon, while Wednesday and Thursday we worked on the QUIC drafts (agenda; minutes). What follows is a recap of the proceedings as well as some observations.
I have been looking forward to this Interim. It was to be the second QUIC Interim Meeting that I attended in person. Unlike the first meeting in Seattle, this one would be a cinch to attend: jump on a train and I am there!
On the morning of Monday, September 17th, I joined thousands of other commuters making their daily trip to the city. Young and old, white and black, skinny and well-rounded: all were on their way to earn their living. In this strange zone, where one is surrounded by people, but yet all alone, my mind wandered as I looked out the train window at houses, roads, stacks of railroad ties, puddles, junkyards, flags, trucks, bridges, baseball fields, and an occasional tent of a homeless person. Two hours door-to-door.
Lars Eggert’s employer, NetApp, was the host of the meeting. Each day at 9:30 am, we assembled in a comfortable conference room on the 19th floor of 285 Madison Ave.
View from the 19th floor (Madison and 41st Street)
The Interoperability Hackathon — or the Interop — is an opportunity for the different implementers of QUIC to meet face-to-face and to find and fix bugs quickly. This time, there were two interop events at once: the QUIC transport interop and the QPACK interop. I participated in the second one.
QPACK is the HTTP header compression protocol. Like most of our software, the LiteSpeed QPACK library is open-source. There are a few other open-source as well as closed-source implementations. The way to test for compatibility is to encode a list of headers using one QPACK implementation, decode the result using another QPACK implementation, and check whether the result matches the original input.
A snapshot of the QPACK Interop Matrix
By the end of the second day, the participants — Alan Frindell (Facebook), Martin Thomson (Mozilla), Mike Bishop (Akamai), Roma Kane (F5), Masakazu Kitajo (Apache Traffic Server), and yours truly — found and fixed dozens of bugs.
Spin Bit Experiments
On Wednesday, the official part of the QUIC Interim commenced with three spin bit experiments presentations. If you remember, the latency spin bit is meant to aid network operators — in other words, so oft-maligned “middleboxes” — to debug issues with QUIC connections. Because QUIC uses UDP and encrypts almost everything in its packets, the observer lacks most of the information that is present in a TCP connection.
Alexandre Ferrieux of Orange was first with his Spin bit and beyond @ Orange. He hit rough terrain early, as several people, including the chairs, interrupted him. The Working Group consensus, they said, is only to expose the spin bit. Alexandre’s experiments included additional square sequence and end-to-end bits to provide information on upstream loss and downstream loss, respectively. Thus, he did not have a chance to present all the reasons why one would want those three bits of information in each packet. Nevertheless, the reasons are listed in his presentation and well worth a read.
Next was Marcus Ilhar of Ericsson. His presentation was subtitled Testing the Robustness of the 1-bit Signal. He modified quic-go to add the spin bit and wrote a middlebox simulator that both introduced reordering and loss and measured the spin bit. The conclusion is that the spin bit is indeed useful for measuring the RTT of a connection.
A portion of a slide from Brian Trammell’s presentation
As he was waiting to board his plane, so went Brian’s story, he wondered why uploading images from his laptop to his home computer was so slow. He broke out tcptrace and analyzed the information it presented. The main insight is that one does not have to have information about congestion window to debug network problems: the RTT information is enough. The RTT information would be provided by the spin bit.
Gutenberg Bible
I craved soul food more than the earthly sustenance and so I visited the Morgan Library & Museum during lunch. Among other rare items, its Treasures from the Vault exhibition featured a real Gutenberg Bible.
Gutenberg Bible at the Morgan Library & Museum
Only 180 copies (120 on paper and 60 on vellum) were printed in 1455. Of those, just 49 are extant, 3 of them at the Morgan Library. The paper copy on display is a large, beautifully typeset and bound, tome.
Speed Dating
Martin Thomson’s selection of GitHub issues was subjected by what was called “Speed Dating.” Each of the issues was given three minutes for discussion by the group. Simplification of PNE (packet number encryption) was picked up by EKR (Eric Rescorla) to research, while most of the other issues were closed with no action.
Interop Results
Christian Huitema (Microsoft) and Alan Frindell updated the participants with the results of the interop. Among notable developments, Christian highlighted that there were two successful connection migrations and that Martin Thomson and Kazuho Oku (Fastly) are close to having interoperable HTTP stacks. Alan summarized the QPACK Interop and informed the group that Google, too, has a QPACK implementation close to shipping.
This is because, I said, HTTP is not just for the web. Long lived connections or connections with high volume of requests (or both) may show significant degradation in compression performance. I ran a simulation of a scenario wherein server reply contains three headers — never-changing :status and content-type, and cookie that changes every ten requests. 10,000 responses have 0.13 compression ratio when absolute index wraps versus 0.15 when it does not (14% worse compression ratio). 100,000 responses is 0.13 compression ratio versus 0.16 for 19% deterioration. So the difference is not insignificant at all.
The room had to concur. Alan presented us with three options:
Do nothing;
Wrap the encoded value of the absolute index; or
Change the definition of the absolute index.
(3) is the largest change, one that I originally proposed. There was no clear consensus in the room, with a bunch of people, including Martin Thomson, saying that we should do nothing. On the other hand, Brian Trammell and others argued in favor of wrapping. This change, said Brian, reflects the spirit of QPACK. We had to go to hums. The first hum would be whether to do nothing (1) or do something — (2) or (3). If the first hum resulted in doing something, then we’d hum to select (2) or (3).
The first hum was close, with one of the chairs, Mark Nottingham (Fastly), having to interpret it. His interpretation was that we should do something. It was decided that instead of humming, Alan, who is the editor of the QPACK draft, would select between (2) and (3) himself. The humble man was not particularly happy with the task assigned to him, but resigned himself to carrying the burden.
Designer Quadrupeds
Camel: Where a man belongs.
They say that the camel is a horse designed by a committee. The Working Group has a pattern of delegating a group of people to work on a component of the protocol and then subjecting their findings to on-the-spot (read: shallow) analysis. This is like the inverse of bikeshedding, where the body is eager to argue about, and change, the design of a nuclear power plant. This happened in Melbourne to the Compression Design Team, where weeks of work were discarded in twenty minutes and we ended up with the suboptimal solution that is QPACK. The index-wrapping proposal was close to suffering the same fate at the hands of the people who hummed for option (1). I understand the desire to complete the work more quickly, but we should not settle for shabby work!
Connection ID Management
Mike Bishop presented the output of the Connection ID Design Team. Because I participated in this design team, many of the issues raised by the working group sounded familiar to me; the design team had been over them several times.
Header Bits
Martin Thomson talked about the bits in the first byte of the QUIC packet. It would be nice to be able to multiplex QUIC and DTLS, for example.
QUIC Trace Tool
Victor Vasiliev (Google) described the quic-trace tool developed by Google. A QUIC peer can make a recording of a QUIC connection in a special format. quic-trace creates a visualization of the connection using this recording. The visualization can be used to analyze connection performance.
Wayne Thiebaud
Wayne Thiebaud is an American painter. During Thursday’s thankfully long — 90 minutes! — lunch, I again rushed off to the Morgan Library & Museum to view an exhibition of some of his works. While mostly known for his paintings of everyday objects such as cakes and gumball machines, I liked his charcoal drawings the most.
Three Roads, 1983
Flow Control Deadlocks
As part of my work on the Compression Design Team, I discovered a deadlock condition in QCRAM in January of this year. QPACK solves the problem by cautioning the implementers to write to encoder and data streams in specific order. This was part of Mike Bishop’s Flow Control Gotchas presentation. It was interesting to observe that one could also deadlock with just one stream, for example, if a non-streamable frame is preceded by its length and the application does not read the frame until all of it is available.
I asked for an example of a non-streamable frame. Mike replied that the header block is one such example, to which I nodded. Later, however, I changed my mind: it is not possible to write the header block as a stream, because it must be followed by length, but it is possible to read the header block in streaming mode. This is exactly what our QPACK library does. I will bring this up on the mailing list next week.
Upcoming Meetings
The last bit of business was to plan the next meetings. After a bit of haggling, we agreed that the pre-Bangkok (IETF 103) Interop target will be based on Internet Draft 15 (this Interop was targeting ID-14). The January Interim meeting is likely to be held in Tokyo.
The Road Ahead
QPACK is almost done. My suggestion that the encoder stream framing be removed was accepted and made it into the current QPACK draft. The wrapping of the absolute index just squeaked through this week, and this is the last non-editorial change that I foresee.
On the other side of the spectrum, the QUIC transport draft is still rather shaky. Version negotiation, first header bytes, and packet number encryption are all liable to change. I think there is virtually no chance that we make the November deadline.
The good news is that there is definitely progress. If only we can stop ourselves from changing the transport every few months, we will be in good shape.
The second day of the Kista Interim Meeting began with the Explicit Congestion Control (ECN) presentation by Magnus Westerlund. The proposal to add ECN to QUIC has been around since at least January of 2017. A year and a half is a long time in the context of an IETF Working Group! In January of this year, Ingemar Johansson presented the proposal to the WG at the Melbourne Interim but it was still deemed to be lacking. This time around, the result was different. Over the last several months, Magnus and Ingemar (colleagues at Ericsson, our gracious host) prepared a comprehensive presentation along with a GitHub pull request that incorporates ECN into the QUIC protocol.
The Working Group adopted the proposal. Of course, some parts will have to be changed. This is inevitable in a large proposal such as this one. For example, there was a marked pushback against adding a second frame type to carry acknowledgements (named ACK_ECN in the proposal). It looks like the existing ACK frame will be augmented to support additional ECN information. At one point in the discussion, Brian Trammell suggested that we add SMACK (small ACK) and SHACK (short ACK) frames to the protocol. But he was only joking. I think.
QUIC @Facebook
Subodh Iyengar shared experiences implementing and deploying IETF QUIC at Facebook. It is rather impressive that Facebook is running IETF QUIC now, while the protocol is still shifting under our feet. Since HTTP/QUIC is not yet ready, the folks at Facebook use QUIC as the transport for HTTP/1.1. The setup carries 100 billion HTTP requests per day.
Facebook Load Balancer Architecture, taken from Subodh’s slides
I already knew about Facebook’s pioneering ways. During the Seattle Interim in 2017, Alan Frindell told us that Facebook has been running experiments (in production, of course) with the different HPACK successors. Having worked a bit with the Facebook code, I know that they are not afraid to use the hot-off-the-presses C++14 features.
This sort of “lessons learned” disclosure is a very valuable contribution both to the working group and to the implementers of the nascent protocol. The Working Group can see what shortcomings there are in the protocol, so that we can address them early. The implementers avoid committing the same errors, saving valuable time.
Well done, Facebook.
QUIC Load Balancers
Martin Duke presented his QUIC Load Balancers draft. The draft proposes a way for QUIC load balancers to update each other’s state to allow them operating correctly even if there are connection migrations. The load balancers would talk to each other using (naturally) QUIC-LB protocol.
Implicit Stream Open is In
In my first Kista Interim Summary post, I stated that the requirement to open streams in order is without merit. While I still oppose the implicit open, I have come to see the other side’s perspective. Yes, the implicit open carries more information with each STREAM frame, making it simpler to keep track of some things. I still oppose it, for the drawbacks are more significant.
This time around, this issue came to a head and we hummed. Outhummed I was. The consensus is to have implicit open. Well, what’s done is done and life goes on. This issue is comparatively small, and it is not like we were assigning stream IDs out of order anyway.
PNE Negotiation
Packet number encryption (PNE) negotiation, championed by Praveen Balasubramanian, was rejected by the Working Group. Praveen advocated the data center use case, where PNE is unnecessary and only adds processing overhead. The Working Group’s consensus was that PNE is an intrinsic part of QUIC v1. Not having it means a different protocol, which cannot be called QUIC v1.
The Bad
The audio. The audio was bad. The audio was bad as usual, which is also bad. I do not expect a United Nations level of audio setup, but a little extra effort would pay off a lot.
While the chairs’ microphones were good, the two omnidirectional microphones used to pick up comments from the two or so dozen of the other participants failed to serve adequately. Many people are not used to the idea of projecting their voice. Indeed: when you are speaking to people who are fifteen feet away from you, raising your voice is unnecessary. The end result is that I at times had trouble following the discussion.
Next Interim: NYC!
The current plan is to hold the next Interim Meeting in New York City. How convenient: the QUIC brain trust just a train ride away from my house!
The IETF QUIC Working Group has come together for another Interim meeting. This time, the meeting sponsor is Ericsson: they are providing a conference room and networking at their office in Kista, Sweden. I am participating remotely, which means that I woke up at 3 o’clock this morning to be ready for the 9:30 am meeting held across the ocean.
Every item on the agenda is pressing, as the Working Group is trying to make the November deadline. The way QUIC transport layer interacts with the TLS stack is changing. The HTTP/QUIC draft is still quite raw (if it were a steak, you’d send it back). We are trying to reach consensus on as many issues as possible and nail down more stuff than we will add this time around, thereby making progress.
Editors’ Update
Martin Thomson’s report was optimistic. Since our last meeting at the IETF 101, the editors closed many issues in GitHub and marked others v2. The latter means that these issues are postponed until the second version of IETF QUIC, whenever that turns out to be.
Stream 0 Redesign
Eric “EKR” Recorla’s DTLS presentation at the IETF 101 created a lot of discussion. EKR showed that the current QUIC drafts have a fundamental flaw: the transport part of QUIC depends on the crypto layer and the crypto layer depends on the QUIC transport. This makes several things quite painful. Eric’s own DTLS proposal, while not accepted by the Working Group at the time, resulted in the Chairs (Mark Nottingham and Lars Eggert) empaneling a special Stream 0 Design Team. This team, led by EKR and Ian Swett, worked out a novel, consensus approach to address the stream 0 mess.
Bye, Stream 0!
Remember the “CREAMs and CRACKs?” The new design uses dedicated frame types (instead of a special QUIC stream) to carry handshake messages. This allows for significant layering simplification, going from this:
to this:
(Both diagrams are borrowed from the design document linked above.)
Besides the new CRYPTO_HS frame type (I guess they did not have the chutzpah to call the new type CREAM), we see that the TLS record layer is missing. This is the novel part of the design. Since QUIC frames are themselves a type of a record — in the sense that they have boundaries — the TLS record structure can be skipped and TLS messages are carried directly in the new QUIC frames.
The risk of this approach is dependency on TLS libraries exposing a new API to be able to access TLS messages outside of records. The good news is that OpenSSL and BoringSSL have agreed to accommodate us, while a few smaller implementations have already added such support.
Packet Number Spaces
Another interesting part of the Design Team’s proposal is that each encryption level (there are three) gets its own packet numbers. This is something new and was met with scepticism by some.
The proposal — or rather, its direction — was adopted with strong support, modulo the packet number discussion, which is to be revisited.
HTTP
The state of HTTP over QUIC Internet Draft was presented by Mike Bishop, the draft’s author. Mike covered three topics: changes to HTTP frame types, self-describing streams, and stream priority trees.
Frame Type Changes
Out of all HTTP frame types in HTTP over QUIC (HQ), only the PRIORITY frame uses flags. Thus, the flags field is removed from the generic HTTP frame.
Self-Describing Streams
HQ uses three special unidirectional streams:
Control stream;
QPACK encoder stream; and
QPACK decoder stream.
Multiply this by two (client and server) and you get six streams that an HQ implementation deals with in a special manner. Mike’s proposal is to have unidirectional streams in HQ have a leading type byte. Based on its value, the application will know what stream type it is: a control stream, a decoder stream, and so on. This would make it, in the presenter’s own words, simpler. This is to satisfy the principle that
Application layer shouldn’t need to “grab” a particular stream by ID.
I don’t know whence comes this principle. It sounds rather meritless to me. I had to speak up. My argument against this proposal went as follows.
The HQ layer installs different stream handlers based on stream type. The control stream does one thing, the encoder stream another, while the regular HTTP message streams are handled in yet another way. A special stream ID — 2, 3, 6, 7, 10, or 11 — allows to initialize a stream with the correct handler as soon as the first stream frame arrives. On the other hand, having to examine the first byte to initialize the stream forces one to handle the case when the first STREAM frame for a particular stream does not have the first byte. In other words, a stream is instantiated, but has to be finalized later. This is certainly not simpler.
Martin Thomson replied that it made his code simpler. I don’t know about Martin’s code, but it would not make our code simpler. A bit later in the discussion, I offered the following analogy:
Think of a networked computer. Port 80 is HTTP; port 22 is SSH; and so on. The kernel does not examine the first byte of an incoming TCP connection to see what service it wants. Instead, the application is allowed to use port numbers. TCP connections have ports; QUIC streams have IDs. We can use them.
This reminded me of a related discussion we had on the mailing list. There, I also argued against hiding stream IDs. I will quote the last communication I authored on that thread at some length:
The API problem Marten describes seems like a non-issue: There are
two sets of APIs:
- User code / HQ; and
- HQ / QUIC.
Stream IDs do not have to be exposed in the former: The user does not
care about them, as these things are just your regular HTTP messages.
In the HQ / QUIC API, on the other hand, expose away! Streams have
IDs, the QUIC transport draft talks about them, the IDs have meanings --
it only makes sense that the layer that uses the QUIC objects directly
can use Stream IDs. Why not?!
Why do some people high on the IETF QUIC totem pole advocate for not using stream IDs in HQ without giving a good technical reason? I can only guess. (A related issue — in the sense that it is similarly groundless — is the desire to mandate that streams be opened in sequence. This discussion is being kept alive and tomorrow will be revisited again. Watch this space.)
The “no-grab” principle above is fine and dandy, but QUIC streams do not have intrinsic types. Using the first payload byte as a substitute is a kludge.
Roberto Peon pointed out that control streams without fixed IDs would make debugging difficult. Certainly, the proposal’s momentum was slowed, but the consensus was to “continue discussing” it. I suppose it means I have to remain vigilant!
Stream Priorities
HQ has inherited the HTTP/2 stream priority model… and associated problems. The short version is that streams can depend on one another and just because a stream is finished does not mean that it can be removed from the dependency tree. In some scenarios, this could mean that the dependency tree keeps on growing. (Few HTTP/2 implementations do stream dependencies properly. See Are HTTP/2 Servers Ready Yet? Salient quote: “the priority mechanism has not been well designed and deployed.”)
A few ideas were tossed around:
Is this really our problem? Since HQ uses HTTP/2 semantics, shouldn’t we keep priorities as-is?
Should we drop support for priorities altogether?
Should priorities only affect idle streams?
Ultimately, the Working Group resolved to come back to this issue later.
QPACK
Alan Frindell, the editor of the QPACK draft, described what the current version of QPACK looks like. Other than being broken by design, QPACK is almost complete. So much so that the last slide of Alan’s presentation talked about an interop.
I objected to using framing on the QPACK encoder stream, as it renders some optimizations impossible. For example, why can the encoder not write to the packet directly? If one has to know the length, it means that the encoder instructions must first be buffered. The reason for framing, Alan explained, is to aid the decoder. Some HPACK decoders do not deal well with incomplete instructions; they do not know how to resume.
I think that this is not applicable in this case. The table update stream processed by a QPACK library only affects a single entity, the dynamic table, and can save the state easily. It is the processing of partial HEADERS block that HPACK decoders have historically had trouble with. These blocks are framed and carried on other, regular, streams. (I will likely present this argument to the QPACK editor and authors.)
While Alan proceeded to demonstrate a simple QPACK session walk-through, Martin Thomson filed a bug titled “Encoder stream can deadlock.” And so it can! Due to framing. Thanks, Martin.
***
The second part of the Interim debates is tomorrow. Stay tuned!
Nothing is more permanent than a temporary solution.
– Folk wisdom
The Melbourne Interim Meeting has come and gone. I participated remotely, which means that my workday started at 5 pm and wrapped up at 1 am. The whopping 16-hour difference turned out not to be so inconvenient after all.
The three-day schedule was packed tight with topics (see Agenda; Minutes) as the Working Group strives to converge quickly in order to hit the November 2018 deadline for QUIC v1. In this article I will touch on some of these topics.
Header Compression
The Working Group rejected the conservative approach to blocking outright and selected QCRAM (with a “tweak”) as the native QUIC header compression protocol.
Out of the three working proposals put forth by the Header Compression Design Team, QMIN is the most conservative: the protocol solves the head-of-line blocking (HoLB) problem at the cost of compression efficiency. The Working Group looked at the compression performance comparison graph (ideal network conditions: zero packet loss) and decided that compression performance trumps all other considerations:
The flat purple line is hpack0: that is HPACK with the dynamic table size of zero. This is the baseline. The other four lines are HPACK (as it would perform in GQUIC), QMIN, QPACK, and QCRAM-03. One can observe the initial penalty incurred by QMIN.
Of the two proposals that seek to minimize HoLB rather than avoid it, QCRAM is simpler than QPACK, as the latter uses multiple transient control streams to transmit updates. The vanilla QCRAM, however, risks a deadlock.
On the spot, the Working Group came up with the fix: “just use a single control stream,” they said. With that, Buck Krasic and Mike Bishop went off to write up the QCRAM-04 specification. The use of a single control stream makes QCRAM unable to address a whole class of HoLB scenarios. In other words, the deadlock fix makes QCRAM ineffective.
Before the adoption vote was taken on the last day of the meeting, I pointed this out to the Working Group and advocated for selecting a more robust HoLB avoidance mechanism. Despite my objections, the motion to adopt QCRAM passed with overwhelming support. The main line of reasoning behind picking a worse solution was that we can always go back and make improvements or replace it with something better. I am skeptical.
On the bright side, having settled on the header compression protocol, the Working Group has eliminated a large roadblock on the way to hit its November 2018 deadline for QUIC v1. Furthermore, QCRAM has a knob to use blocking avoidance instead of minimization.
In the avoidance mode, QCRAM is structurally similar to QMIN — non-modifying header blocks in individual HTTP message streams and a single control stream to carry updates with ACKs — and thus should have similar performance characteristics.
Encrypted Packet Numbers
Martin Thomson presented his idea of greasing the protocol. Greasing means designing the QUIC packet format (the stuff that is actually transmitted over the network) in such fashion so as to allow future changes to the protocol. What greasing tries to prevent is ossification: inability to deploy new protocol or protocol extensions due to the unchangeable nature of infrastructure components that have come to rely on a particular feature of the current protocols. (As a matter of fact, deployability was the number one original design goal of QUIC.)
Among other things, Martin’s original proposal obfuscates almost everything in the QUIC packet header, including packet numbers. This is quite radical. The proposal engendered lively debate at the Interim Meeting. The subsequent, more modest proposal only aims to encrypt packet numbers. There is an interesting (and long!) email thread where several aspects of this proposal are discussed.
Variable-Length Connection IDs
In our code, the QUIC connection ID is represented as uint64_t. Easy-peasy! You’d think that eighteen quintillion is enough for everybody, right? It turns out, however, that there are uses for both longer and shorter connection identifiers. The longer IDs can be useful to load balancers: the load balancing information (e.g., which backend server to forward the connection to) can be embedded into the connection ID itself. Eight bytes is simply not enough for this. On the other hand eight-byte IDs may be an overkill when used, for example, in a P2P setup.
A new Design Team to hash this out has been formed. The team is to present its findings, potentially with a proposal, at the next meeting.
Explicit Congestion Notification (ECN)
Explicit Congestion Notification (ECN) is a way for the network (read: routers) to notify the endpoints about incipient congestion. (See RFC 3168 for details. It is impressive how much complexity can fit into just four bits.) ECN is meant to improve network performance of any transport protocol on top of IP, not just TCP, and thus is of interest to QUIC. The benefits of ECN were deemed significant enough that a dedicated Design Team has been empaneled. Headed by Ingemar Johansson, it is expected to produce a proposal by IETF 101 in London. One can track their progress by looking at the working document.
Summary
The deadline for QUIC v1, which has already been moved once, is only nine months away. For the Working Group not to miss it, many large pieces have to fall into place, and quickly. Each new conference of the Working Group will be more and more consequential. Here is to our success!
At the Melbourne Interim meeting, the IETF QUIC Working Group decided to use a version of the QCRAM header compression protocol with the following modification: use a dedicated control stream to carry all dynamic table updates. The result is QCRAM-04. This modification is one way to prevent deadlock hazard (the subject of last week’s blog post) present in the previous QCRAM proposals. While it certainly addresses the deadlock, the single control stream introduces ordering to otherwise independent operations, making it unable to minimize HoLB for a class of applications.
The Original HoLB Problem
The original HoLB problem that we are trying to solve is as follows. HPACK/GQUIC delivers all HTTP message headers as header blocks on a separate control stream. HTTP message payload (bodies) are sent on different streams, one message per stream. The application (HTTP) cannot process message bodies without the headers; it must have the headers before it starts reading from the message stream.
Figure 1: HPACK/GQUIC: Control stream carrying two header blocks and two streams carrying message bodies.
If a packet carrying part of the Header Block A is lost, Header Block B and, therefore, the whole of Message B cannot be processed, even if all of the packets carrying Header Block B and Message B Body have arrived. Message B is stalled until the lost packet is retransmitted successfully and the control stream is unblocked. This is the HoLB problem.
Proposed Solutions
The solutions to the HoLB problem fall into two categories:
Avoidance: Disallow vulnerable references. This approach is robust and simple to implement, but carries a compression performance penalty.
Minimization: Allow vulnerable references, but minimize their impact. This approach looks to achieve maximum compression at the cost of protocol and implementation complexity.
QCRAM and QPACK belong to the second category. Until QCRAM-04, both carried table updates on multiple streams, for the following reason:
Coping with False Dependencies
(Refer to Figure 1 again). If Header Block B uses a dynamic entry inserted by Header Block A, then B cannot be processed until A has been processed; this is a true dependency. If, however, Header Block B does not depend on any insertions performed by Header Block A, placing them in order on a single control stream introduces a false dependency. The design choice of placing dynamic table updates on separate streams is driven by desire to eliminate false dependencies and, thereby, to minimize HoLB.
How QCRAM-04 is Different From HPACK/GQUIC
QCRAM-04 moves parts of header blocks from the control stream to the individual message streams. This means that if a particular message’s headers do not cause an insertion into the dynamic table, this message does not have to write to the control stream. This eliminates some false dependencies and is an improvement over HPACK/GQUIC.
Figure 2: QCRAM-04: control stream carrying table modifications upon which Header Block A depends and message streams for messages A and B. Message B can be processed if parts of message A are lost.
False Dependencies Remain
If messages A and B cause independent updates to the dynamic table, they are written to the control stream. This fact makes message B depend on message A: a false dependency.
Figure 3: QCRAM-04: control stream carrying independent table modifications which, nevertheless, make Message B depend on Message A: a false dependency.
Note that Figure 3 and Figure 1 are quite similar.
Three Application Classes
To analyze how QCRAM-04 improves over HPACK/GQUIC, we should consider three application classes (basically, usage patterns):
Dependent header blocks;
Independent header blocks without table modification; and
Independent header blocks with table modification.
I. Dependent Header Blocks
If Header Block B uses table entries added by Header Block A, there is not much one can do about that: A must be processed before B, no matter how the header blocks are transmitted. QCRAM-04 and HPACK/GQUIC do not introduce HoLB: this is the property of the application itself.
II. Independent header blocks without table modification
Header blocks that do not modify the dynamic table do not require the use of the control stream in QCRAM-04 (see Figure 2). This is a clear improvement over HPACK/GQUIC, which introduces a false dependency and risks HoLB by delivering all header blocks over the same stream.
III. Independent header blocks with table modification
If Header Blocks A and B insert different entries into the dynamic table and Header Block B does not use any entries inserted by A, both QCRAM-04 and HPACK/GQUIC introduce a false dependency by delivering unrelated table updates associated with A and B on the same stream (see Figure 3).
Quantitatively, QCRAM-04 is a little better than HPACK/GQUIC here. This is because it sends less data on the control stream, which means that the chances of packet loss are smaller. However, the improvement is small: the control stream carries table updates, which is most of the header block data.
Summary
QCRAM-04 achieves HoLB minimization in one class of applications, but not in the other, where it introduces a false dependency. This means that the answer to the question “Will my application suffer from protocol-induced HoLB were I to switch to HTTP/QUIC?” is “It depends.” That is not good enough. In the HoLB minimization approach, independent dynamic table updates must be sent on different streams.
To achieve the best header compression performance, it is tempting to have the compression protocol allow potential head-of-line blocking (HoLB). The idea is that HoLB will not occur most of the time, while the compression gain is significant. The first article of this series exposes priority inversion caused by HoLB and challenges the assertion that HoLB is rare. The second problem is that HoLB is a potential attack vector, wherein the decoder is tricked into using a lot of memory. One way to prevent such attack is to not read from the stream until the decoder can process the stream immediately. This approach creates a deadlock hazard. This problem is examined in this article.
Note on Terminology
If a header block B depends on an entry inserted by header block A, we say that B depends on A. Together, they are dependent header blocks.
Problem Statement
Writing dependent header blocks to stream out of order introduces the risk of a deadlock. There are two types of a deadlock.
Single-stream Deadlock
When dependent header blocks are written out of order to the same stream, the first header block cannot be parsed because it depends on the block that follows it.
Multiple-stream Deadlock
This type of deadlock happens when the out-of-order header block prevents the receiver from reading stream data that follows the header block. What makes it different from the regular HoLB case is that the sender cannot put the resolving header block onto the stream due to the limits imposed by the QUIC connection flow control.
Scenario 1: Resend After a Reset
One way a multi-stream deadlock can occur is when a header block must be resent after a stream reset. By the time the sender figures out that it needs to resend a header block, it may have filled all existing streams to the brim — which the receiver cannot drain — and is not allowed to stream more data.
Possible Mitigation: Don’t Fill Flow Control Window
One potential solution is always to leave some room in the connection flow control window. That is, leave at least the number of bytes required to resend all unacknowledged header blocks.
There are downsides to this approach:
It is not clear that the connection flow control window information is available at HTTP/QUIC level.
The code complexity of the HTTP/QUIC layer increases — the user code can no longer write directly to the stream. These calls must be intercepted at HTTP/QUIC layer to ensure that the threshold is not exceeded.
Most of the time, these are just wasted bytes that could otherwise be used. In other words, throughput suffers.
Not filling the connection window may prevent the peer from sending WINDOW_UPDATE, resulting in a stall.
Scenario 2: Out-of-order Write
In general, a multi-stream deadlock can occur any time dependent header blocks are written to different streams — consuming flow control credits — out of order. If header block B depends on header block A, but is written to stream first, it may happen that header block A can never be written: If the receiver does not process stream B because of its application protocol rules (in our case, the receiver is avoiding memory exhaustion attacks), it is possible that the connection flow control prevents the sender from sending more data.
Possible Mitigation: Don’t Do That
“Well, don’t do that, then!” – might one say. This is a sound point of view: since the problem is of the application’s own making, it is up to the application to solve it. Nevertheless, the deadlock risk bears being pointed out, as it affects how a QUIC/HTTP implementation may be designed. For example, if the stream interface offered by a QUIC implementation performs internal buffering, can the application make sure that data on one stream gets sent before dependent data on another stream?
Possible Mitigation: Make the Transport Do It
My initial report to the Design Team prompted Martin Thomson to share the finding on the common IETF QUIC WG Mailing list. The resulting thread is large and makes for an interesting read. It contains several ideas on how such deadlock could be prevented by the transport, either by making the current priority implications stricter; or by introducing extra transport rules, exceptions, or overrides of various sorts.
Summary
It is natural to write dependent header blocks to streams in order. Not doing so may result in a deadlock. In particular, resetting streams forces the encoder to write header blocks out of order. Hence, the encoder must address the possibility of a deadlock in some fashion.
P. S.
The “don’t do that” versus “let the transport solve it” discussion is likely to resume at the upcoming Interim Meeting (Jan 23 – 25) in Melbourne, Australia. Watch this space for updates.
