Our Roadmap for QUIC and HTTP/3 Support in NGINX

Liam Crilly of F5

Sr Director, Product Management

July 12, 2021

[Editor: NGINX now officially supports HTTP/3 with QUIC. It is available as part of NGINX 1.25.1 mainline version for open source users and NGINX Plus R30 for enterprise customers.]

[Editor : Binary packages for the NGINX preview implementation of QUIC+HTTP/3 are now available for Red Hat Enterprise Linux 9 and Ubuntu 22.04. Get the details on our blog.]

The Internet refuses to stand still. Even before the standardization of HTTP/2 in 2015, work had already begun on QUIC and what would later become HTTP/3. Now, five years since the first draft specification for QUIC was submitted to the IETF, we have RFC 9000, QUIC: A UDP‑Based Multiplexed and Secure Transport.

Of course, it’s not quite that simple. At the time of writing, a total of four RFCs have been published that describe the core QUIC transport and encryption layer. A further eleven related specifications remain in draft, including the final definition of HTTP/3. However, many client and server implementations are already in widespread use. In fact, 20% of all websites already support HTTP/3 thanks to the efforts of CDN providers and tech giants who have been keeping pace with the draft standards. And so have we.

For an in‑depth look at the history of HTTP and our implementation of HTTP/3 and QUIC, check out this video:

Last year we announced a technology preview for QUIC and HTTP/3 and a lot has happened since then. Here’s a quick rundown of the key development efforts that have gone into the NGINX implementations for both QUIC and HTTP/3:

• We implemented multiple draft versions of HTTP/3, from draft 23 through draft 29 (the most widely implemented version today), and of course the published standard (draft 34).
• We continued to merge all changes made in the NGINX mainline branch into the nginx-quic branch so that all new features (and fixes) can be tested with QUIC.
• We participated in QUIC interop testing so that our implementation is continuously tested with multiple clients, and in comparison with other server implementations.
• We developed an eBPF program to take advantage of NGINX’s multi‑process architecture. NGINX uses multiple worker processes which enables highly scalable performance across multiple CPU cores. For maximum efficiency, we like to handle each connection with the same worker process. QUIC makes this challenging, as a connection is not tied to the client’s IP address, and the Linux kernel doesn’t provide UDP port-to-process affinity. To solve this we implemented an eBPF extension to integrate with SO_REUSEPORT so that we can map the QUIC connection ID to the worker that first handled it. This is neatly integrated into the NGINX core so that NGINX loads the eBPF bytecode into the kernel’s socket selection code.
• We consolidated NGINX’s connection‑handling code to simplify how long‑lived connections are configured. Configuration directives that previously applied only to HTTP/1.1 also work for HTTP/2, and will also apply to QUIC+HTTP/3. You no longer need to use different directives for the same functionality just because the protocol version is different. See the client_header_timeout, keepalive_requests, keepalive_timeout, and large_client_header_buffers directives.
• We refactored the additional code into two parts: a smaller part with changes that affect the NGINX core, and a larger part with the implementation of the transport protocols. By organizing the codebase in this way, we can keep the changes to NGINX internals to a minimum, mitigating the risk of introducing security bugs into NGINX itself.

The NGINX Roadmap

Our roadmap for QUIC+HTTP/3 has two major milestones:

1. Merge the nginx-quic development branch into the NGINX mainline branch
2. Implement performance optimizations

Implementation of the QUIC and HTTP/3 protocols is largely complete and we are now undertaking the careful process of merging the new code in the nginx-quic branch to the NGINX mainline branch (where we release new features). The changes to the NGINX core are relatively small (~3000 lines of code) and merging is well underway. The larger task is merging the transport‑protocol code (~27,000 lines of code). Based on our experience merging our HTTP/2 implementation into NGINX, we expect this work to take several months.

Our current target for completing the code merge into the NGINX mainline branch is the end of 2021, after which it will be a full part of the NGINX mainline branch and NGINX Plus in subsequent releases.

Our next focus will be to optimize performance. Some of the benefits of QUIC will be immediately apparent, but others will take a little longer. Whereas TCP and UDP have the benefit of decades of development in the kernel and network drivers of numerous operating systems, QUIC is largely implemented in user space. This means that some of the things we take for granted, like congestion notifications and preferred packet size detection, must now be implemented individually by each client and server software provider.

Integrating QUIC’s TLS Layer

QUIC makes encrypted transport a mandatory, non‑negotiable part of the network stack. QUIC supports the TLS 1.3 standard, but the TLS software implementation must also support QUIC’s interfaces. NGINX typically uses OpenSSL as an external dependency which means we can take advantage of the SSL/TLS libraries that ship with the operating system in our official NGINX Open Source packages and in NGINX Plus. Decoupling the cryptographic implementation from NGINX enables bugs and security vulnerabilities in the SSL/TLS library to be addressed without upgrading NGINX itself.

At the time of writing, OpenSSL does not support QUIC, and it might take many months before any operating system distributions ship with a QUIC‑enabled SSL/TLS library. Our QUIC implementation currently supports BoringSSL and the quictls fork of OpenSSL, which must be compiled with NGINX.

We expect the state of QUIC‑enabled SSL/TLS libraries to become clearer during the period when we are merging the nginx-quic development branch into NGINX mainline.

How You Can Help

You can help us by testing our QUIC+HTTP/3 implementation. The more bugs and edge cases that are discovered, the quicker we can complete the merge process and bring QUIC+HTTP/3 support to NGINX.

• If you are familiar with QUIC and/or HTTP/3, do a code review on the parts you know well.
• If you run large‑scale Internet services, experiment with A/B testing by deploying nginx-quic for some users or some services.
• If you manage web sites and/or applications, try nginx-quic instead of the current version in your automated tests.

• Share your findings with us on the NGINX developers mailing list.

  If you encounter a bug or unexpected behavior, or need help with troubleshooting, please also write to the NGINX developers mailing list. The list is actively monitored by NGINX Engineering, which the blog is not. If you have a general comment or suggestion, please add it to the comments section below.

To try the nginx-quic implementation, simply point your browser at https://quic.nginx.org/ and check the README to learn more. For a quick proof of concept in your own environment, use this Dockerfile to build your own nginx-quic image and then follow the configuration instructions in the aforementioned README.

[Editor – The following Dockerfile was updated in November 2022 to correct problems with previous versions.]