This blog post centers on a vulnerability that was recently discovered related to the HTTP/2 protocol. Under certain conditions, this vulnerability can be exploited to execute a denial-of-service attack on NGINX Open Source, NGINX Plus, and related products that implement the server-side portion of the HTTP/2 specification. To protect your systems from this attack, we’re recommending an immediate update to your NGINX configuration.
The Problem with HTTP/2 Stream Resets
After establishing a connection with a server, the HTTP/2 protocol allows clients to initiate concurrent streams for data exchange. Unlike previous iterations of the protocol, if an end user decides to navigate away from the page or halt data exchange for any other reason, HTTP/2 provides a method for canceling the stream. It does this by issuing an RST_STREAM frame to the server, saving it from executing work needlessly.
The vulnerability is exploited by initiating and rapidly canceling a large number of HTTP/2 streams over an established connection, thereby circumventing the server’s concurrent stream maximum. This happens because incoming streams are reset faster than subsequent streams arrive, allowing the client to overload the server without ever reaching its configured threshold.
Impact on NGINX
For performance and resource consumption reasons, NGINX limits the number of concurrent streams to a default of 128 (see http2_max_concurrent_streams). In addition, to optimally balance network and server performance, NGINX allows the client to persist HTTP connections for up to 1000 requests by default using an HTTP keepalive (see keepalive_requests).
By relying on the default keepalive limit, NGINX prevents this type of attack. Creating additional connections to circumvent this limit exposes bad actors via standard layer 4 monitoring and alerting tools.
However, if NGINX is configured with a keepalive that is substantially higher than the default and recommended setting, the attack may deplete system resources. When a stream reset occurs, the HTTP/2 protocol requires that no subsequent data is returned to the client on that stream. Typically, the reset results in negligible server overhead in the form of tasks that gracefully handle the cancellation. However, circumventing NGINX’s stream threshold enables a client to take advantage of this overhead and amplify it by rapidly initiating thousands of streams. This forces the server CPU to spike, denying service to legitimate clients.
Denial-of-service by establishing HTTP/2 streams, followed by stream cancellations under abnormally high keepalive limits.
Steps for Mitigating Attack Exposure
As a fully featured server and proxy, NGINX provides administrators with powerful tools for mitigating denial-of-service attacks. To take advantage of these features, it is essential that the following updates are made to NGINX configuration files, minimizing the server’s attack surface:
- keepalive_requests should be kept at the default setting of 1000 requests
- http2_max_concurrent_streams should be kept at the default setting of 128 streams
We also recommend that these safety measures are added as a best practice:
- limit_conn enforces a limit on the number of connections allowed from a single client. This directive should be added with a reasonable setting balancing application performance and security.
- limit_req enforces a limit on the number of requests that will be processed within a given amount of time from a single client. This directive should be added with a reasonable setting balancing application performance and security.
How We’re Responding
We experimented with multiple mitigation strategies that helped us gain an understanding into how this attack could impact our wide range of customers and users. While this research confirmed that NGINX is already equipped with all the necessary tools to avoid the attack, we wanted to take additional steps to ensure that users who do need to configure NGINX beyond recommended specifications are able to do so.
Our investigation yielded a method for improving server resiliency under various forms of flood attacks that are theoretically possible over the HTTP/2 protocol. As a result, we’ve issued a patch that increases system stability under these conditions. To protect against such threats, we recommend that NGINX Open Source users rebuild binaries from the latest codebase and NGINX Plus customers update to the latest packages (R29p1 or R30p1) immediately.
How the Patch Works
To ensure the early detection of flood attacks on NGINX, the patch imposes a limit on the number of new streams that can be introduced within one event loop. This limit is set to twice the value configured using the http2_max_concurrent_streams directive. The limit will be applied even if the maximum threshold is never reached, like when streams are reset right after sending the request (as in the case of this attack).
Affected Products
This vulnerability impacts the NGINX HTTP/2 module (ngx_http_v2_module). For information about your specific NGINX or F5 product that might be affected, please visit: https://my.f5.com/manage/s/article/K000137106.
For more information on CVE-2023-44487 – HTTP/2 Rapid Reset Attack, please see: https://www.cve.org/CVERecord?id=CVE-2023-44487
Acknowledgements
We would like to recognize Cloudflare, Amazon, and Google for their part in the discovery and collaboration in identifying and mitigating this vulnerability.
