LiteSpeed conference reports from industry meetings and events that LiteSpeed has attended or will attend. CloudFest, QUIC Working Group, cPanel Conference, WHD.Global, Hostingcon, and more!
The second day of the Kista Interim Meeting began with the Explicit Congestion Control (ECN) presentation by Magnus Westerlund. The proposal to add ECN to QUIC has been around since at least January of 2017. A year and a half is a long time in the context of an IETF Working Group! In January of this year, Ingemar Johansson presented the proposal to the WG at the Melbourne Interim but it was still deemed to be lacking. This time around, the result was different. Over the last several months, Magnus and Ingemar (colleagues at Ericsson, our gracious host) prepared a comprehensive presentation along with a GitHub pull request that incorporates ECN into the QUIC protocol.
The Working Group adopted the proposal. Of course, some parts will have to be changed. This is inevitable in a large proposal such as this one. For example, there was a marked pushback against adding a second frame type to carry acknowledgements (named ACK_ECN in the proposal). It looks like the existing ACK frame will be augmented to support additional ECN information. At one point in the discussion, Brian Trammell suggested that we add SMACK (small ACK) and SHACK (short ACK) frames to the protocol. But he was only joking. I think.
QUIC @Facebook
Subodh Iyengar shared experiences implementing and deploying IETF QUIC at Facebook. It is rather impressive that Facebook is running IETF QUIC now, while the protocol is still shifting under our feet. Since HTTP/QUIC is not yet ready, the folks at Facebook use QUIC as the transport for HTTP/1.1. The setup carries 100 billion HTTP requests per day.
Facebook Load Balancer Architecture, taken from Subodh’s slides
I already knew about Facebook’s pioneering ways. During the Seattle Interim in 2017, Alan Frindell told us that Facebook has been running experiments (in production, of course) with the different HPACK successors. Having worked a bit with the Facebook code, I know that they are not afraid to use the hot-off-the-presses C++14 features.
This sort of “lessons learned” disclosure is a very valuable contribution both to the working group and to the implementers of the nascent protocol. The Working Group can see what shortcomings there are in the protocol, so that we can address them early. The implementers avoid committing the same errors, saving valuable time.
Well done, Facebook.
QUIC Load Balancers
Martin Duke presented his QUIC Load Balancers draft. The draft proposes a way for QUIC load balancers to update each other’s state to allow them operating correctly even if there are connection migrations. The load balancers would talk to each other using (naturally) QUIC-LB protocol.
Implicit Stream Open is In
In my first Kista Interim Summary post, I stated that the requirement to open streams in order is without merit. While I still oppose the implicit open, I have come to see the other side’s perspective. Yes, the implicit open carries more information with each STREAM frame, making it simpler to keep track of some things. I still oppose it, for the drawbacks are more significant.
This time around, this issue came to a head and we hummed. Outhummed I was. The consensus is to have implicit open. Well, what’s done is done and life goes on. This issue is comparatively small, and it is not like we were assigning stream IDs out of order anyway.
PNE Negotiation
Packet number encryption (PNE) negotiation, championed by Praveen Balasubramanian, was rejected by the Working Group. Praveen advocated the data center use case, where PNE is unnecessary and only adds processing overhead. The Working Group’s consensus was that PNE is an intrinsic part of QUIC v1. Not having it means a different protocol, which cannot be called QUIC v1.
The Bad
The audio. The audio was bad. The audio was bad as usual, which is also bad. I do not expect a United Nations level of audio setup, but a little extra effort would pay off a lot.
While the chairs’ microphones were good, the two omnidirectional microphones used to pick up comments from the two or so dozen of the other participants failed to serve adequately. Many people are not used to the idea of projecting their voice. Indeed: when you are speaking to people who are fifteen feet away from you, raising your voice is unnecessary. The end result is that I at times had trouble following the discussion.
Next Interim: NYC!
The current plan is to hold the next Interim Meeting in New York City. How convenient: the QUIC brain trust just a train ride away from my house!
The IETF QUIC Working Group has come together for another Interim meeting. This time, the meeting sponsor is Ericsson: they are providing a conference room and networking at their office in Kista, Sweden. I am participating remotely, which means that I woke up at 3 o’clock this morning to be ready for the 9:30 am meeting held across the ocean.
Every item on the agenda is pressing, as the Working Group is trying to make the November deadline. The way QUIC transport layer interacts with the TLS stack is changing. The HTTP/QUIC draft is still quite raw (if it were a steak, you’d send it back). We are trying to reach consensus on as many issues as possible and nail down more stuff than we will add this time around, thereby making progress.
Editors’ Update
Martin Thomson’s report was optimistic. Since our last meeting at the IETF 101, the editors closed many issues in GitHub and marked others v2. The latter means that these issues are postponed until the second version of IETF QUIC, whenever that turns out to be.
Stream 0 Redesign
Eric “EKR” Recorla’s DTLS presentation at the IETF 101 created a lot of discussion. EKR showed that the current QUIC drafts have a fundamental flaw: the transport part of QUIC depends on the crypto layer and the crypto layer depends on the QUIC transport. This makes several things quite painful. Eric’s own DTLS proposal, while not accepted by the Working Group at the time, resulted in the Chairs (Mark Nottingham and Lars Eggert) empaneling a special Stream 0 Design Team. This team, led by EKR and Ian Swett, worked out a novel, consensus approach to address the stream 0 mess.
Bye, Stream 0!
Remember the “CREAMs and CRACKs?” The new design uses dedicated frame types (instead of a special QUIC stream) to carry handshake messages. This allows for significant layering simplification, going from this:
to this:
(Both diagrams are borrowed from the design document linked above.)
Besides the new CRYPTO_HS frame type (I guess they did not have the chutzpah to call the new type CREAM), we see that the TLS record layer is missing. This is the novel part of the design. Since QUIC frames are themselves a type of a record — in the sense that they have boundaries — the TLS record structure can be skipped and TLS messages are carried directly in the new QUIC frames.
The risk of this approach is dependency on TLS libraries exposing a new API to be able to access TLS messages outside of records. The good news is that OpenSSL and BoringSSL have agreed to accommodate us, while a few smaller implementations have already added such support.
Packet Number Spaces
Another interesting part of the Design Team’s proposal is that each encryption level (there are three) gets its own packet numbers. This is something new and was met with scepticism by some.
The proposal — or rather, its direction — was adopted with strong support, modulo the packet number discussion, which is to be revisited.
HTTP
The state of HTTP over QUIC Internet Draft was presented by Mike Bishop, the draft’s author. Mike covered three topics: changes to HTTP frame types, self-describing streams, and stream priority trees.
Frame Type Changes
Out of all HTTP frame types in HTTP over QUIC (HQ), only the PRIORITY frame uses flags. Thus, the flags field is removed from the generic HTTP frame.
Self-Describing Streams
HQ uses three special unidirectional streams:
Control stream;
QPACK encoder stream; and
QPACK decoder stream.
Multiply this by two (client and server) and you get six streams that an HQ implementation deals with in a special manner. Mike’s proposal is to have unidirectional streams in HQ have a leading type byte. Based on its value, the application will know what stream type it is: a control stream, a decoder stream, and so on. This would make it, in the presenter’s own words, simpler. This is to satisfy the principle that
Application layer shouldn’t need to “grab” a particular stream by ID.
I don’t know whence comes this principle. It sounds rather meritless to me. I had to speak up. My argument against this proposal went as follows.
The HQ layer installs different stream handlers based on stream type. The control stream does one thing, the encoder stream another, while the regular HTTP message streams are handled in yet another way. A special stream ID — 2, 3, 6, 7, 10, or 11 — allows to initialize a stream with the correct handler as soon as the first stream frame arrives. On the other hand, having to examine the first byte to initialize the stream forces one to handle the case when the first STREAM frame for a particular stream does not have the first byte. In other words, a stream is instantiated, but has to be finalized later. This is certainly not simpler.
Martin Thomson replied that it made his code simpler. I don’t know about Martin’s code, but it would not make our code simpler. A bit later in the discussion, I offered the following analogy:
Think of a networked computer. Port 80 is HTTP; port 22 is SSH; and so on. The kernel does not examine the first byte of an incoming TCP connection to see what service it wants. Instead, the application is allowed to use port numbers. TCP connections have ports; QUIC streams have IDs. We can use them.
This reminded me of a related discussion we had on the mailing list. There, I also argued against hiding stream IDs. I will quote the last communication I authored on that thread at some length:
The API problem Marten describes seems like a non-issue: There are
two sets of APIs:
- User code / HQ; and
- HQ / QUIC.
Stream IDs do not have to be exposed in the former: The user does not
care about them, as these things are just your regular HTTP messages.
In the HQ / QUIC API, on the other hand, expose away! Streams have
IDs, the QUIC transport draft talks about them, the IDs have meanings --
it only makes sense that the layer that uses the QUIC objects directly
can use Stream IDs. Why not?!
Why do some people high on the IETF QUIC totem pole advocate for not using stream IDs in HQ without giving a good technical reason? I can only guess. (A related issue — in the sense that it is similarly groundless — is the desire to mandate that streams be opened in sequence. This discussion is being kept alive and tomorrow will be revisited again. Watch this space.)
The “no-grab” principle above is fine and dandy, but QUIC streams do not have intrinsic types. Using the first payload byte as a substitute is a kludge.
Roberto Peon pointed out that control streams without fixed IDs would make debugging difficult. Certainly, the proposal’s momentum was slowed, but the consensus was to “continue discussing” it. I suppose it means I have to remain vigilant!
Stream Priorities
HQ has inherited the HTTP/2 stream priority model… and associated problems. The short version is that streams can depend on one another and just because a stream is finished does not mean that it can be removed from the dependency tree. In some scenarios, this could mean that the dependency tree keeps on growing. (Few HTTP/2 implementations do stream dependencies properly. See Are HTTP/2 Servers Ready Yet? Salient quote: “the priority mechanism has not been well designed and deployed.”)
A few ideas were tossed around:
Is this really our problem? Since HQ uses HTTP/2 semantics, shouldn’t we keep priorities as-is?
Should we drop support for priorities altogether?
Should priorities only affect idle streams?
Ultimately, the Working Group resolved to come back to this issue later.
QPACK
Alan Frindell, the editor of the QPACK draft, described what the current version of QPACK looks like. Other than being broken by design, QPACK is almost complete. So much so that the last slide of Alan’s presentation talked about an interop.
I objected to using framing on the QPACK encoder stream, as it renders some optimizations impossible. For example, why can the encoder not write to the packet directly? If one has to know the length, it means that the encoder instructions must first be buffered. The reason for framing, Alan explained, is to aid the decoder. Some HPACK decoders do not deal well with incomplete instructions; they do not know how to resume.
I think that this is not applicable in this case. The table update stream processed by a QPACK library only affects a single entity, the dynamic table, and can save the state easily. It is the processing of partial HEADERS block that HPACK decoders have historically had trouble with. These blocks are framed and carried on other, regular, streams. (I will likely present this argument to the QPACK editor and authors.)
While Alan proceeded to demonstrate a simple QPACK session walk-through, Martin Thomson filed a bug titled “Encoder stream can deadlock.” And so it can! Due to framing. Thanks, Martin.
***
The second part of the Interim debates is tomorrow. Stay tuned!
The second meeting of the TLS working group covered position of CIDs in DTLS 1.3, DNSSEC chain extension, and others.
Alessandro Ghedini (Cloudflare) presented his proposal to compress certificate chains using Brotli. Up to 30% savings could be achieved. This optimization is particularly germane to QUIC, as it reduces the reflection attack risk.
Christian Huitema (Microsoft) talked about encrypted SNI. This is an early proposal and will require a security analysis.
Lunch – Africa Initiative
Instead of lunch, I attended the IETF Africa Initiative session organized by Kevin Chege. The initiative aims to increase involvement in the IETF by participants from Africa. Nesredien Suleiman, a professor from at the University of Addis Ababa, presented the results of his own efforts in Kenya. This was followed by a discussion with several insightful comments offered by several African attendees — a professor from the University of Ilorin (Nigeria), an advisor to the Minister of Education of Mozambique, and others. The role of government in effecting positive change was discussed, with Kenya being an example of this working.
Third Session – ACME
The topics discussed at the Automated Certificate Management Environment WG meeting were somewhat obscure to me. ACME is the protocol behind LetsEncrypt — a push to convert all HTTP servers to use TLS. This effort shows no signs of slowing down.
Fourth Session – IETF Plenary
The plenary session began at 5:15 pm and lasted almost four hours. The auditorium was large enough to hold 500-600 people, but not all seats were taken. The various boards, committees, and chairs of the IETF of associated organizations took turns taking the stage. Awards were presented and speeches were made. Practical matters were also discussed: budget; diversity; technical leadership; cost of registration and visas for people from developing nations. In a memorable exchange, Kyle Rose chastised the IAOC for the lack of cookies (yes, the edible kind) at the conference. Quoth Kyle: “I was reduced to buying cookies with cash!”
I realized what a diverse, yet, at the same time, close-knit community of people IETF really is. It is quite astonishing how this self-organized, self-appointed, and mostly unpaid body of techies is dictating — most of the time successfully — the rules of the Internet game to the world.
Thursday 3/22/18
First Session – QUIC: The Spin Bit Debate
ECN – Explicit Congestion Notification
First, the working group reviewed some outstanding items of the ECN proposal. It was decided that the ECN will make it into the main draft around June of this year.
Latency Spin Bit
Brian Trammell presented the latency spin bit. This produced two hours of rather heated debate as many people were unwilling to give up even one bit for the spin bit, afraid of information leakage. Network operators — Orange, Verizon, AT&T — insisted that the spin bit would be very useful, especially with the looming packet number encryption. At one point, the microphone queue was over twenty people long.
The who-is-who of the QUIC WG waiting to express their opinions
Spencer Dawkins had to intervene a few times to get the working group focussed again. At the end, it was hummed that Brian’s version of the spin bit was wanted, albeit in a separate draft. Three bits for the spin bit are to be reserved in the current draft and, when and if the spin bit draft is deemed finished, the reference to it will be made.
Second Session – DOH
DOH stands for DNS over HTTP. The DOH Internet Draft lists several reasons when DNS over HTTP may be preferable.
D. K. Gillmor (DKG) presented his Opportunistic DNS idea. It is possible for a web server to push DNS records (of the content type specified in the DOH protocol) if it thinks that the client might need them!
One of the slides from DKG’s presentation
This may allow the client to skip sending DNS requests needed to render the page, thereby loading it faster. Some interest was shown in the room, in particular by Patrick McManus and Mark Nottingham. The latter noted that something like this had already been discussed in the httpbis WG.
During Gorry Fairhurst’s presentation, Spencer Dawkins queried Lars Eggert whether QUIC does any sort of path MTU discovery. Lars replied that this is currently not in scope, because he does not have a good feeling of how complicated this is. Gorry said that it would be nice if someone from the QUIC WG reviewed this. David Black (one of the TSVWG chairs) said that he would like QUIC to be upwards compatible with this.
Michael Schapira stole the show with his presentation. I thought it was the best presentation — both in content and in style — of the whole IETF 101. The video recording is also available — it is well worth watching.
The PCC approach is both novel and logical at the same time. The main idea of PCC is that a packet loss does not necessarily mean congestion — the assumption that underlies all TCP flavors. There are four main causes of packet loss:
Current flow is the cause of the congestion. The current flow should back off.
Shallow buffer is overflown. The current flow should back off a little.
Some other flow is the cause of the congestion. That other flow should back off.
Loss is random. No action should be taken.
Note that TCP treats all four events the same, whereas only (1) should cause the sender substantially to reduce its CWND.
PCC uses game theory to calculate a utility function value. Experiments are run continuously to see how reducing or increasing CWND affects the utility value. Based on the experiment results, the CWND may be adjusted.
The approach PCC takes is totally unlike TCP — I don’t even think it can be called TCP. On the other hand, PCC can be deployed now, as only the sender logic is modified. Otherwise, it is compatible with TCP.
Conclusion
I would be lying if I said that I understood everything I had heard in all of the presentations. Each working group is highly specialized and it requires a significant amount of involvement to understand, let alone comment on, the particulates. I felt satisfied if I grasped the gist of the discussion — which was most of the time.
The IETF is alive, well, and growing. The Task Force’s work scope is tremendous: from fixing 37-year-old bugs in TCP specification to theorizing about the evolution of the networks; from intra-datacenter protocol improvements to providing satellite Internet access to the developing world. And anyone can get involved!
When it comes to QUIC, there are still many unfinished pieces. If our Working Group is to make the November 2018 deadline, we should stop trying to boil the ocean. Baby steps: let’s start with the Great Lakes.
I attended the IETF 101 Conference in London a couple of weeks ago. This was my first time attending this conference. What follows are some of my recollections of people, talks, and events.
Sunday 3/18/2018
My trip to the IETF 101 began with being stuck at the Frankfurt Airport due to an unseasonable snow.
My plane, just sitting there. And sitting some more.
I landed at the Heathrow Airport several hours later than planned and arrived at the venue a bit after 5 pm instead of 11 am. This resulted in me missing all the newbie-oriented meetings that I had been hoping to attend. I was able to catch most of the reception, which had free beer (plus) but little actual food (minus).
I listened to the Hot RFC lightning talks instead. There were seventeen short (hence “lightning”) talks squeezed into one and a half hours. Between QUIC over Satellite by John Border and Web Packaging by Jeffrey Yasskin, there was something for everyone.
Monday 3/19/2018
Breakfast
I met Alan Frindell (whom I befriended last year at the Seattle Interim) for breakfast at a nearby cafe. After a thought-provoking QUIC discussion, I headed off to the first session of the day. At the IETF Conference, there are six to eight sessions happening in parallel during each time slot, and so one has to choose which meeting to attend.
First Session – TCP Maint
I sat in at the TCP Maintenance and Minor Extensions Working Group (WG) meeting. There were several presentations, a couple of which stood out to me.
It turns out, RFC 793 (that’s the TCP specification itself from 1981) contains three bugs that have been known about for decades. Fernando Gont recommended that RFC 793 be updated with the fixes, while Lars Eggert, whom I heretofore have known only as the chair of the QUIC WG, proposed that an erratum be added to RFC 793 instead.
Praveen Balasubramanian, another QUIC participant, presented TCP Fast Open (TFO) deployment data as collected by Microsoft. TFO works well in some regions much better than in other regions. For example, TFO worked for only 1.5% of devices in China is compared with 21% success rate overall. See Praveen’s presentation for details.
Second Session – QUIC
The first afternoon session was QUIC. After an update from the editors, we got to the much-anticipated meat of today’s debate — the hand grenade that is QUIC over DTLS that Eric Rescorla (EKR) tossed into our midst just several days ago.
In his slides, EKR enumerated the several issues with the Stream 0 approach. The root cause of the problems, according to EKR, is that TLS depends on QUIC, which in turn depends on TLS. One way to address this is to introduce special frames for crypto streams and crypto ACKs (CREAMs and CRACKs!). A cleaner solution is to use DTLS. The downside of using QUIC/DTLS is that it would require significant changes to QUIC, and even some changes to DTLS.
The discussion was framed using the following questions:
What is the right architecture for QUIC?
How do we evaluate the alternatives?
The working group split into two camps: Pro-DTLS Switch, headed by EKR and his fellow Mozillians; and Anti-DTLS Switch, headed by Ian Swett (Google), who at that time joined EKR on the stage. The back-and-forth revolved around two topics: the technical advantages (or disadvantages) of switching to DTLS and the fact that this is quite late in the process to make drastic changes to QUIC. At one point, Spencer Dawkins (Transport Area Director) interrupted the debate to remind the participants that using TLS 1.3 is in the QUIC WG Charter. Eventually, the Anti-DTLS Switch prevailed, as it became obvious that the November 2018 deadline for QUIC would have to slip. As a consolation prize, the chairs promised to appoint a design team potentially to figure out the new architecture.
Martin Thomson (Mozilla) presented his proposal to use two variable-length connection IDs instead of one fixed-length connection ID. This was the result of the work of the design team empaneled at the January Interim meeting. From the current Editor’s Draft:
The long header contains two connection IDs: the Destination Connection ID is chosen by the recipient of the packet and is used to provide consistent routing; the Source Connection ID is used to set the Destination Connection ID used by the peer.
The session wrapped up with two lightning talks: QUIC in the Wild and PLPMTUD. (PL stands for Packetization Layer).
Third Session – Internet Area WG
Among the several presentations, IP Fragmentation Considered Fragile stood out. In it, Ron Bónica highlighted the problems associated with IP fragmentation and proposes a set of common practices for application developers and network operators. To sum it up: avoid IP fragmentation! For details, see the Internet Draft.
Fourth Session – TLS
The last session was the shortest — one hour — but it was the most contentious of all. Under consideration was the TLS Vizability, a proposal to allow a key manager for TLS that would be able to get access to the plaintext exchanged between two endpoints. While first being presented as a solution for data centers, it was eventually acknowledged that state actors may have a good use case for this as well — to snoop. There were several rather heated exchanges that left both Russ Housley (one of the authors) and the WG Chairs a bit exasperated.
Stephen Farrell, the opponent of the proposal, put it this way: “What makes you think it is in the charter of the Working Group to break TLS like this?!”
Even after a plea from the area director, the hum to adopt this draft was inconclusive; if anything, the “Nos” had it. Since the consensus was not reached, any future versions of this draft would have to go through the area director before being put forth before the WG again.
Multicast QUIC Demo
I was able to meet Lucas Pardue at 7 pm at the Hackdemo Happy Hour. Lucas works for BBC Research & Development; his team works on video-related technologies. Multicast QUIC is a technique to transmit video and audio streams to multiple clients using less network resources (fewer streams).
Newcomers’ Dinner
This dinner was conveniently located at a burger joint across the street. I met Jeffrey Yasskin (Google) and Yoav Weiss (Akamai). I talked with Jeffrey about his proposal for web packages, while Yoav shared his expertise in HTTP caching and structured headers. Grizzled veteran programmers and graduate students, we all had a great time saying hello to each other, unashamed of our newbiness.
Tuesday 3/20/2018
First Session – Measurement and Analysis for Protocols
The morning time slot contained several excellent presentations, among them:
Being a research group, panrg’s work has no immediate applications. Rather, the presentations were mostly an exploration of how things might evolve. Spencer Dawkins’ presentation concentrated on what not to do. Because there have been many ideas that turned out to be bad, why not learn from our own mistakes?
The Path Aware Networking Proposed Research Group has voted to become a full-status research group at the end of its session.
Third Session – HTTPbis
A lot is happening in the HTTP space and there are several interesting proposals, among them
Secondary certificates;
Expect-CT header;
Header common structure (aka structured headers);
Variants header;
Cache digests for H2; and
Client hints.
To see which of these proposals have been adopted by the working group, see the httpbis data tracker.
Secondary Certificates
Secondary certificates is a mechanism whereby a single HTTPS connection can be shared between several origins.
Expect-CT header
CT stands for Certificate Transparency. The Internet Draft describes two use cases for this experimental header. In short, the idea is to make the client detect incorrect certificate configuration on subsequent connections.
Header Common Structure
This draft specify a way to define and use HTTP header fields. This seems to be another cache-related improvement: Mark Nottingham (Fastly) is one of its authors and Yoav Weiss (Akamai) advocated strongly for a scheme just like this one during our Newcomers’ Dinner discussion. Yoav’s concern is compression: splitting the headers into smaller pieces should improve compression performance.
Variants header
The Variants and Variant-Key headers are an alternative way to communicate a secondary key for an HTTP resource. The draft was hummed to be adopted by the working group.
This specification defines a HTTP/2 frame type to allow clients to inform the server of their cache’s contents. Servers can then use this to inform their choices of what to push to clients.
The web server should take client hints into account before pushing promises to client.
Client Hints
The client hints are a finer-grained way for a client to specify its preferences — an alternative to the User-Agent header.
London Web Performance Meetup
After the HTTP session wrapped up at 18:15, I caught the Tube to the London Web Performance meetup. The panel consisted of Mark Nottingham (Fastly), Hooman Beheshti (Fastly), Martin Thomson (Mozilla), Jana Iyengar (Fastly), Ian Swett (Google), Mike Bishop (Akamai), and Subodh Iyengar (Facebook).
There were about 20 attendees — among them Lucas Pardue (BBC Research and Development), Yoav Weiss (Akamai), and Barry Pollard (the author of an upcoming book on HTTP/2 (Manning)). The others were web developers for whom, I thought, the panel discussion regarding the various transport-level tradeoffs was a bit over their head. After the discussion, there was beer and pizza, during which I chatted with a few people about QUIC and LiteSpeed. Kazuho Oku gave a lightning talk about client early hints (RFC 8297).
Last Monday, George, Jon, and I met up with Ruikai at the Frankfurt International Airport and headed for Europa-Park, to attend the CloudFest conference (previously known as the World Hosting Days). Europa-Park is situated in Rust, a southern German village on the banks of the river Elz, a tributary of Rhine.
Because Rust is so small, it does not have a train station. CloudFest provides a shuttle bus service between Offenburg and Europa-Park. On the day of our arrival, it tooks us two attempts to board the bus, as the first coach could not fit all the people. We steeled our resolve to get onto the next transport and braved a warm rain under the unexpected rays of sunshine.
When we got to our destination, we went straight to the venue to set up our booth.
The before and after: our isle on Monday afternoon when we were setting up and on Tuesday morning, the first day of the conference.
It was good to meet with current clients, whom we love, and with prospective clients, whom we plan to love. We enjoy telling our story and help people solve their problems. We also took the opportunity to listen to our clients’ concerns and ideas. LiteSpeed wants to make your life easier — if there is something we can do, do tell us!
We were happy to catch up with our partners, cPanel and Plesk.
This has been a wonderful trip and this conference, as always, has been superbly organized. Our quartet — a system admin, a support specialist, a programmer, and the company owner — established yet another bond. We are a team; we are dedicated to moving the web forward.
On the penultimate day of the conference, we made a short excursion to Freiburg im Breisgau — a beautiful city a short train ride away from Rust. Stepping over its Bächle, I could not help but think about the people who first made these channels almost a thousand years ago: Did they know that their creation would survive empires and witness the technological revolution? In the end, we are like them: we, too, are craftsmen, making tools. It is only fitting that we would be at the CloudFest conference — or, to use a German word, a Fachmesse — a trade fair.
Nothing is more permanent than a temporary solution.
– Folk wisdom
The Melbourne Interim Meeting has come and gone. I participated remotely, which means that my workday started at 5 pm and wrapped up at 1 am. The whopping 16-hour difference turned out not to be so inconvenient after all.
The three-day schedule was packed tight with topics (see Agenda; Minutes) as the Working Group strives to converge quickly in order to hit the November 2018 deadline for QUIC v1. In this article I will touch on some of these topics.
Header Compression
The Working Group rejected the conservative approach to blocking outright and selected QCRAM (with a “tweak”) as the native QUIC header compression protocol.
Out of the three working proposals put forth by the Header Compression Design Team, QMIN is the most conservative: the protocol solves the head-of-line blocking (HoLB) problem at the cost of compression efficiency. The Working Group looked at the compression performance comparison graph (ideal network conditions: zero packet loss) and decided that compression performance trumps all other considerations:
The flat purple line is hpack0: that is HPACK with the dynamic table size of zero. This is the baseline. The other four lines are HPACK (as it would perform in GQUIC), QMIN, QPACK, and QCRAM-03. One can observe the initial penalty incurred by QMIN.
Of the two proposals that seek to minimize HoLB rather than avoid it, QCRAM is simpler than QPACK, as the latter uses multiple transient control streams to transmit updates. The vanilla QCRAM, however, risks a deadlock.
On the spot, the Working Group came up with the fix: “just use a single control stream,” they said. With that, Buck Krasic and Mike Bishop went off to write up the QCRAM-04 specification. The use of a single control stream makes QCRAM unable to address a whole class of HoLB scenarios. In other words, the deadlock fix makes QCRAM ineffective.
Before the adoption vote was taken on the last day of the meeting, I pointed this out to the Working Group and advocated for selecting a more robust HoLB avoidance mechanism. Despite my objections, the motion to adopt QCRAM passed with overwhelming support. The main line of reasoning behind picking a worse solution was that we can always go back and make improvements or replace it with something better. I am skeptical.
On the bright side, having settled on the header compression protocol, the Working Group has eliminated a large roadblock on the way to hit its November 2018 deadline for QUIC v1. Furthermore, QCRAM has a knob to use blocking avoidance instead of minimization.
In the avoidance mode, QCRAM is structurally similar to QMIN — non-modifying header blocks in individual HTTP message streams and a single control stream to carry updates with ACKs — and thus should have similar performance characteristics.
Encrypted Packet Numbers
Martin Thomson presented his idea of greasing the protocol. Greasing means designing the QUIC packet format (the stuff that is actually transmitted over the network) in such fashion so as to allow future changes to the protocol. What greasing tries to prevent is ossification: inability to deploy new protocol or protocol extensions due to the unchangeable nature of infrastructure components that have come to rely on a particular feature of the current protocols. (As a matter of fact, deployability was the number one original design goal of QUIC.)
Among other things, Martin’s original proposal obfuscates almost everything in the QUIC packet header, including packet numbers. This is quite radical. The proposal engendered lively debate at the Interim Meeting. The subsequent, more modest proposal only aims to encrypt packet numbers. There is an interesting (and long!) email thread where several aspects of this proposal are discussed.
Variable-Length Connection IDs
In our code, the QUIC connection ID is represented as uint64_t. Easy-peasy! You’d think that eighteen quintillion is enough for everybody, right? It turns out, however, that there are uses for both longer and shorter connection identifiers. The longer IDs can be useful to load balancers: the load balancing information (e.g., which backend server to forward the connection to) can be embedded into the connection ID itself. Eight bytes is simply not enough for this. On the other hand eight-byte IDs may be an overkill when used, for example, in a P2P setup.
A new Design Team to hash this out has been formed. The team is to present its findings, potentially with a proposal, at the next meeting.
Explicit Congestion Notification (ECN)
Explicit Congestion Notification (ECN) is a way for the network (read: routers) to notify the endpoints about incipient congestion. (See RFC 3168 for details. It is impressive how much complexity can fit into just four bits.) ECN is meant to improve network performance of any transport protocol on top of IP, not just TCP, and thus is of interest to QUIC. The benefits of ECN were deemed significant enough that a dedicated Design Team has been empaneled. Headed by Ingemar Johansson, it is expected to produce a proposal by IETF 101 in London. One can track their progress by looking at the working document.
Summary
The deadline for QUIC v1, which has already been moved once, is only nine months away. For the Working Group not to miss it, many large pieces have to fall into place, and quickly. Each new conference of the Working Group will be more and more consequential. Here is to our success!
This morning, I arrived at 351 Elliott Ave around 8:40 am, which was about 20 minutes too early. I plopped down in the lobby and chatted a bit with someone whom I took for a manager of some sort. He was in charge of a large model of the new F5 Networks building. The base of the model had some chips in it that needed to be repaired and two repairmen came to take care of it. The actual building is currently under construction and should be ready by 2019. Boasting 44 floors and standing 660 ft. tall, the new F5 Networks tower is a bona fide skyscraper! F5 is growing and Seattle is growing with it.
The meeting began at 9:33. After a few short preliminaries (note well, etc), Mark Nottingham (Fastly) — the working group’s co-chair — announced that Jeff Pinner (Lyft) and Roberto Peon (Facebook) have been nominated to lead a closed design team to come up with a QUIC API document.
Then Lars Eggert (NetApp) — the other co-chair — departed from his usual MO to deliver a warning to the participants:
There are fifty people in this room and half of you haven’t said a thing in two days! This is not good. The committee keeps on arranging venues that can accommodate this many people, relying on people like Martin [Duke (F5 Networks)] to have their companies host us.
It is hard not to agree with Lars, even though he sounded a bit apologetic in the end: Is there a good case for mute participants? Since they do not participate, are they even participants? Perhaps they participate during breaks? But those aren’t an official part of the conference. Are they?
It turns out, the breaks are a crucial part of moving the standardization process forward. People use the breaks at conferences like this one to discuss things face-to-face and to mingle. In our case, however, the discussions during breaks were officially sanctioned by the chairs for people to hash out their technical issues. There are two problems with this:
The working group’s discussion is supposed to be recorded. There are three official ways to discuss the IETF QUIC drafts:
Mailing list;
GitHub artifacts — issues, PRs, etc; and
Conferences/meetings
The first two are easy: the medium is the record. The meetings have a scribe who is to write everything down. These are not empty words: yesterday’s meeting was stalled while we waited for someone to volunteer to be the scribe.
Yet these rules are suspended for discussions during breaks: the discussions are not recorded.
The remote participants cannot participate. Already second-class citizens, they have no ability to take part in discussions during the breaks: Even if the microphone stays on, there are so many groups of people talking, that it is just a din to those listening online.
On the other hand, these break-based discussions are quite productive. I do not know what, or whether, we are to do anything about this; I am just stating the facts.
Mark Nottingham then noted that there is a different dynamic when there are twenty people in the room versus fifty people:
This is not to say that fifty is bad, but you got to participate… In addition, I must note that this group is not very diverse, especially with regards to gender. I would love to see that change.
Speaking of representation: today, when the sign-in sheet was being passed around, I had the presence of mind to glance over the preceding entries. There were names of several companies that I had not heard mentioned at the meeting. Below I publish the list of attending companies and organizations that I have collected:
F5 Networks
ACLU
Adobe
Akamai
Apple
AT&T
BBC
Cloudflare
Deutsche Telekom
Facebook
Fastly
Google
Huawei
LiteSpeed Technologies
Lyft
Microsoft
Mozilla
NetApp
Symantec
Twitter
Verizon
Around 9:45 am, Ian Swett (Google) began his ACKs and Recovery presentation. The first slide had five QUIC ACK Principles which Ian introduced as something that we can all agree upon. Predictably, disagreement instantly ensued. After some dispute regarding the ACK Principles, we got to the meat of the ACK discussion. Ian recounted how some ACK (mis)behaviors only manifest themselves when the network experiences “ridiculously” high packet loss — ca. 30%.
With regard to the ACK frame structure, it was decided to remove the timestamps section from the ACK until we understand better how we want to aid the congestion control. In addition, deciding which congestion control a connection is to employ should be part of connection negotiation. A separate working group is to make a proposal regarding the use of timestamps and ECN for congestion control.
Martin Duke raised an issue with the term “timestamp,” since it is not the common definition of “timestamp.” As for the ACK delay time format, Jeff had the following comment to offer (here I am, unfortunately, paraphrasing — the original wording was better):
The biggest problem with the float format is that it claims to be something that it isn’t, the spec is wrong.
This elicited a couple of boos (I am assuming from the authors of the format).
Around 11:20, Mike Bishop (Microsoft) discussed the pros and cons of QPACK, of which he is the author, and QCRAM proposed by Buck Krasic (Google). The discussion went on for some time, as both QPACK and QCRAM have some features that are desirable and others that are not — it is just that these sets are flipped in each: like the yin and yang. Jeff, always quick with a simile, posited that what we wanted was a unicorn of a compression mechanism. I asked Buck whether it is valuable to base the new algorithm on HPACK, for the latter is so simple, it can just be thrown out and a new one can be written. Buck replied that that was a secondary consideration: the primary reason for basing QCRAM on HPACK is to keep all the decision making in the encoder, which is one of the desirable features, and here the consensus was near unanimous. (There were many more questions and observations offered by the members — I am only listing mine because I am partial.)
Have you ever hummed? What, do you mean like a tune? – you may ask. No, “hum” as in going “HUMMMM…” Sort of like singing a mantra, but instead of OHMMM… you go HUMMM… You haven’t? I have! This is how (apparently!) IETF working groups vote for a motion. When Mike Bishop posed the first question to be hummed, I was so surprised that I missed the gist of the question completely. However, the second humming (hummer? humster? hummedura?) was stated, approximately, as follows: “Hum if you are in favor of saying that it’s OK to abandon any and all HPACK compatibility or design principles when coming up with the new HTTP/QUIC header compression mechanism.” I hummed ‘yes!’ I am a hamster! Hear me HUM! This hum was judged to be in the affirmative.
During the last break of the day, I talked with Alan Frindell (Facebook) about compression. He had made several insightful comments during the preceding discussion and the opportunity to pick his brain was too good to pass up. Turns out, they have been running modified HPACK code (part of proxygen) in production that supports QCRAM and QPACK! It’s hard to be better prepared than that!
Alan described Roberto’s two-stream framing proposal to me. I replied that yes, it is good, but we are digging into the transport! As long as we are digging into the transport, we could do the same with one stream: specify a byte sequence to be message separators and parse the messages out. Then you do not have to put them into the same packet. Further, I went on, the fact that QPACK uses extra streams is generally considered a strike against it. This is because we do not want to run out of streams. But why are there limitations to begin with? Why is the number of streams limited? (This is related to issue #796.) Had we an unlimited number of streams, the question of special framing for messages would not exist! Of course, lifting those restrictions would dig even deeper into QUIC…
Buck discussed the benefits of reduced HoL blocking as measured in Google’s A/B testing early this summer. Reducing head-of-line blocking is definitely something to strive for.
The meeting wrapped up at 3 pm and I caught Bus #24 to downtown. It was my plan all along to visit the Seattle Art Museum before going to the airport: SAM is open until 9 pm on Thursdays. Walking among works of art, I managed to forget QUIC for a little while. Centuries of civilization were speaking to me.
I only began to reflect upon the events of the past three days while definitely not enjoying the unappetizing dinner served at an airport restaurant. (It seems at the SeaTac airport they threw off all pretense: “Give us your money! Now, eat this nondescript mystery meat with some lettuce on it!”) The broiled crypto-chicken could not spoil my mood: I was happy.
The cast of characters in the wonderful play that is the QUIC standardization process is no longer just a bunch of email addresses and GitHub handles for me: I met Eric Rescorla, I argued with Buck Krasic, I shook Martin Thomson’s hand, I exchanged stories of QUIC’s deployment pitfalls with Jana Iyengar, I picked Alan Frindell’s brain. Now I know these (and other!) guys and, what’s more, I like them.
I gained valuable insight listening to the participants. Observations by Jeff Pinner, Martin Thomson (Mozilla), Roberto Peon, Alan Frindell, and Ian Swett offered me a new perspective on things. It is in human nature to cooperate (the very QUIC standardization process is a good example of this): people will share painfully learned lessons, ideas, problems, solutions — for free! One only has to listen. What a great deal!
Whereas during yesterday’s early morning sally I braved the 48-degree weather to reach the southern end of Lake Union, today I explored the Queen Anne district in a different direction. From my hotel, situated at the corner of 5th and Roy, I walked briskly south toward the Space Needle. The morning was bright and gentle, the air fresh, and the sights of the friendly city entertained my eyes. My mind searched, involuntarily, for some European city — perhaps even a city from my childhood — that Seattle reminded me of. In a daydream, I arrived at the foot of the Space Needle and snapped a picture (the sun was hitting it perfectly). From there, I walked west and down toward the Elliott Bay. From the Olympic Sculpture Park, I headed north along the shore toward the F5 Network towers (this is where the working group meets). After a pleasant (sans dodging one kamikaze cyclist) ten-minute walk I arrived at my destination at 9:06 am.
The attendance was higher than the day before. Very few seats remained empty, as almost fifty people were in the room. During the rushed introductions — I wish people would take their time so that I could see them and write down their and their employers’ names — I heard two more large companies not mentioned in my previous blog post: BBC and Symantec. QUIC is gathering a lot of attention.
Jana Iyengar’s (Google) Connection Termination presentation was first. A mere eight slides, it engendered three hours of debates:
What is the difference between application close and immediate close?
Which two purposes was the PING frame originally meant to serve?
Are we using transport mechanisms to solve application problems?
And so on…
Detailed explanations of the above are beyond the scope of this post. Suffice it to say, there are enough intricacies here to nourish anyone’s inner pedant. Nevertheless, I felt that a lot of the discussion was centered on figuring out what the terms actually mean rather than the exposition of the problems and their possible solutions. It may be useful to have a glossary of terms to go along with a presentation such as this one. The glossary would be learned beforehand by the participants. That would save a lot of time.
At the peak of the Connection Termination debates, the queue that Mark Nottingham — who, along with Lars Eggert, chairs the working group — maintained on the whiteboard at times grew to eight or nine entries! That’s nine people who are waiting in line to speak, be it to ask a new question or to address a previous point.
As the arguments dragged on, Mark treated us to some comic relief. The talk was about what happens to a connection’s buffer queues when it is terminated: Do these queues get drained? If yes, when? The number of people who wanted to speak was large — and so was the queue on the whiteboard. Now, the job of the chair is to keep the meeting focused and on time. To that effect, Mark quipped: “Let’s drain this queue and then continue the [Connection Termination] discussion.” Ha!
I found myself nodding several times when Praveen Balasubramanian (Microsoft) contrasted QUIC behavior with that of TCP. During one of the breaks I chatted with him and Nick Banks (Microsoft) about TCP and QUIC. It is a wonderful sensation to converse with people who understand you half way before you finish your sentence.
We continued to talk about Connection Termination some more after lunch, until about 1:30 pm, at which time the conference was nudged to switch to the next topic: streams.
Now, I don’t know what the heart of QUIC is, but streams are its soul. Without streams, QUIC is as inanimate as a browned, day-old banana peel. First, Mike Bishop (Microsoft) in his presentation argued that we should “more or less keep streams as they are:” that is, bidirectional with a relatively rich set of frames to specify which end should stop sending. Then, Eric Rescorla (Mozilla) — whom most people at the conference address by his nom de guerre, EKR (pronounced Ecker) — reported on unidirectional streams. EKR reimplemented the bidirectional streams of Minq as a pair of unidirectional streams. He reported on some of the gotchas and the apparent pros and cons of each approach.
The discourse then veered into an almost surreal talk of unidirectional streams that are unpaired, paired, and paired to more than one stream. In other words, having no arity, having 1:1 arity, and having 1:N arity, respectively. Our brains went into overdrive and I could see that people were struggling (or, at least, I hope that it was not just me alone) to grasp all the implications of having this arity buffet available to the application.
There were many interesting idea exchanges and back-and-forth arguments. I tried to jot down things I did not know about, so that I can research them later. This author even piped in two or three times and, I dare to say, contributed to the discussion in a meaningful way.
* * *
It has been a long day and it has been a long post. Tomorrow is the final day of the conference. Good night.
P. S. I had the good fortune to be seated next to Alan Frindell (Facebook). Having found out that he was a Seattleite, I hurried to express the way I felt about his city during one of the breaks. I told him about the friendly people, the natural beauty of the place, the bike lanes. To top it all off, I said: “And you have a bookstore, man! A bookstore!” Alan looked at me and asked: “Which one do you mean? We have several.” O, Alan! I don’t like Seattle anymore: I love it.
Today, October 3rd, was the interop day, allocated for the different IETF QUIC implementations to test interoperability: that is, whether they can talk to each other using the QUIC protocol. …But let me take a step back and start at the beginning.
I came to the IETF QUIC Working Group (WG) Interim Meeting in Seattle, WA to participate in discussions to iron out the details of the IETF version of the QUIC protocol, which is to become the standard. The drafts under consideration — QUIC Transport, HTTP over QUIC, QUIC Crypto, and the two compression proposals to replace HPACK — QPACK and QCRAM — differ significantly from the current QUIC specifications. Or, rather, the current gQUIC specifications (let’s be precise!) — g is for Google. QUIC has evolved from being simply a transport to better support SPDY to being a generic transport protocol — the next transport protocol for the Internet.
That sounds somewhat grandiose, but if anyone can pull it off, this group of people can. The interest in QUIC is growing, as evidenced by the attendees: F5 Networks (the conference host); Akamai; Apple; AT&T; Facebook; Google; Microsoft; and several smaller organizations, among which I certainly count ourselves, LiteSpeed Technologies.
Our own LiteSpeed QUIC does not yet support IETF QUIC. Hence, I spent my time building and reading other projects. There are already a few, to borrow a term, inchoate IETF QUIC implementations, and that is very good.
It was great to meet, and talk shop with, Marten Seemann (he of the quic-go project) and Jana Iyengar (Chromium), whose code I’ve studied and used to test our own code. I also met Eric Rescorla, whose SSL book I first read fifteen years ago. (If you did programming fifteen years ago, you’ve read his book). I haven’t caught the names of all the people in attendance, but I hope to rectify that over the next two days, when the protocol is to be debated.
Now it is getting late and it is time to turn down for the night. Perhaps I will re-read a few pages of the new drafts before shutting my eyes. I will dream of QUIC.
P. S. For lunch, a few of us went across the street to get Thai food from a food truck! Of course, it tasted like one would expect food truck Thai food to taste. You can’t win them all.
This week some of our team hit the road and spent a few days in sunny Florida attending the cPanel Conference. If you were there, too, you probably saw our big blue display outside of Grand Ballroom 1, and hopefully you stopped to say hello to Jackson and Jonathan, too.
Conferences like this are a great opportunity for us to meet up with old friends, existing clients, and those who have not yet experienced LiteSpeed. Sometimes this week our existing clients took the opportunity to chat with our potential clients and share what they love about LiteSpeed – how great is that?
We enjoyed the chance to chat with our industry colleagues from all around the country, and some international friends as well. We had conversations with potential clients from Canada and Japan, and we played Cornhole with our friends at the table across the aisle.
Jonathan gave a talk on Wednesday afternoon entitled “Are your servers QUIC?” If you weren’t able to be there, you can download a PDF of the presentation and see what you missed!
This was cPanel’s 11th conference and their 20th Anniversary celebration. Harvey and Irma couldn’t keep this party from happening, though they certainly tried. All in all it was a great few days out of the office. We ate delicious hotel meals, attended a few festive parties, played games, attended talks, and most importantly we met up with old industry friends and made some new ones.
We are already looking forward to the next conference!
By the way, if you liked looking at our pictures, cPanel posted a bunch of their own on their Facebook page (which is where we grabbed that final image from – thanks, cPanel 🙂 ).
