NGINX.COM
Web Server Load Balancing with NGINX Plus

We’re happy to announce the availability of NGINX Plus Release 27 (R27). Based on NGINX Open Source, NGINX Plus is the only all-in-one software web server, load balancer, reverse proxy, content cache, and API gateway.

New and enhanced features in NGINX Plus R27 include:

  • Keepalive connections for health checks – In previous releases, NGINX Plus established a new connection for each health check. The new keepalive_time parameter to the HTTP health_check directive enables keepalive connections for health checks and sets their lifetime. Establishing new connections is computationally expensive, so reusing connections can reduce CPU usage considerably when there are large numbers of upstream servers.
  • Support for Kernel TLS – NGINX Plus now supports Kernel TLS (kTLS) on three operating systems which meet the requirements for supporting it: FreeBSD 13, RHEL 9.0+, and Ubuntu 22.04 LTS.
  • TLS metrics for upstream and virtual servers – NGINX Plus now generates TLS statistics for individual upstreams and virtual servers when proxying over HTTPS is enabled, in addition to the global statistics it generated in previous releases. Having metrics from both the client side and the server side is helpful when troubleshooting problems with TLS handshakes.
  • Customizing the error code from JWT validation failures – In NGINX Plus R25 and R26, you can define custom JWT validation checks, but the error code returned for a validation failure is always 401 (Unauthorized). NGINX Plus R27 introduces the error parameter to the auth_jwt_require directive, enabling you to set the code to 401 or 403 (Forbidden) for each check to better distinguish between authentication and authorization errors.
  • Enhancements to the NGINX JavaScript and Prometheus-njs modules – Enhancements to the NGINX JavaScript module include new directives for fine‑tuning the behavior of the ngx.fetch function and a new object that reports the njs version as a hexadecimal number. The Prometheus-njs module can now convert additional NGINX Plus metrics to a Prometheus‑compliant format: the codes field (which reports counts of individual HTTP status codes for upstream and virtual servers) and metrics generated by the Limit Connections and Limit Requests modules.

Rounding out the release are a new version number (8) for the NGINX Plus API and more even distribution of connections when EPOLLEXCLUSIVE mode is used in Linux kernels.

Important Changes in Behavior

Note: If you are upgrading from a release other than NGINX Plus R26, be sure to check the Important Changes in Behavior section in the announcement blogs for all releases between your current one and this one.

New operating systems supported:

  • Alpine Linux 3.16
  • RHEL 9.0+ (added to NGINX Plus R26 after initial release)
  • Ubuntu 22.04 LTS (added to NGINX Plus R26 after initial release)

Older operating systems and architectures removed:

  • Alpine 3.12, which reached end of life (EOL) on May 1, 2022
  • CentOS 8.1+, which reached EOL on December 31, 2021 (RHEL 8.1+ is still supported because its EOL date is different)
  • Power 8 architecture (ppc64le; affects CentOS and RHEL)

Older operating systems and architectures deprecated and scheduled for removal in NGINX Plus R28:

  • Debian 10, which will reach EOL in August, 2022

New Features in Detail

Keepalive Connections for Health Checks

Keepalive connections between NGINX Plus and upstream servers significantly improve performance for reverse proxy and load balancing use cases. For proxy over HTTPS in particular, the computational expense of the full TLS handshake required for each new connection can put a strain on your computing resources, especially in production environments with large numbers of upstream servers.

In previous releases, NGINX Plus didn’t use keepalive connections for health checks, instead establishing a new connection for every health probe. NGINX Plus R27 introduces keepalive connections for HTTP health checks. The new keepalive_time parameter to the health_check directive sets the lifetime of each keepalive connection, after which a new connection is established. Our testing indicates that for proxy over HTTPS, keepalive connections reduce CPU usage for health checks by a factor of 10 to 20 on NGINX Plus servers. They also save CPU resources on upstream servers.

The following example enables keepalive connections with a lifetime of 60 seconds. Given the health‑check frequency of 1 second set by the interval parameter, NGINX Plus incurs the expense of the TLS handshake and connection establishment just once for every 60 health checks.

Support for Kernel TLS

Transport Layer Security (TLS) is the de facto standard protocol for data encryption on the Internet. Unfortunately, protecting data also adds latency. Implementing TLS in the kernel (kTLS) improves performance when serving static content by significantly reducing the need for copying operations between user space and the kernel.

Combining kTLS and the sendfile() system call means data is encrypted directly in kernel space before being passed to the network stack for transmission, instead of being copied to user space for encryption using TLS libraries and then copied back into kernel space before transmission. kTLS also enables offload of TLS processing to hardware, including offload of TLS symmetric crypto processing to network devices.

There are three requirements for supporting kTLS:

  1. The operating system kernel supports it
  2. The operating system distribution includes the OpenSSL 3.0+ crypto library compiled with kTLS support
  3. NGINX Plus (or NGINX Open Source) is built with the OpenSSL 3.0+ crypto library

In October 2021 we added support for kTLS to NGINX Open Source 1.21.4 on platforms that meet requirements 1 and 2. Now we’re adding support for kTLS in NGINX Plus on these platforms:

  • FreeBSD 13 (in NGINX Plus R26 and later)
  • RHEL 9.0+
  • Ubuntu 22.04 LTS

TLS Metrics for Upstream and Virtual Servers

When NGINX Plus is deployed as a reverse proxy or load balancer, the NGINX Plus API provides a rich set of metrics about traffic, response codes, and latency for each upstream and virtual server, enabling customers to observe and monitor server performance and availability.

In previous releases NGINX Plus generated metrics about TLS activity and errors at a system‑wide level when HTTPS connections were used between NGINX Plus and upstream servers, in both the http and stream contexts (established with the proxy_pass https://path-to-upstream and proxy_ssl on directives respectively). The statistics cover handshakes, failures, and session reuses (as configured with the proxy_ssl_session_reuse directive).

NGINX Plus R27 generates those TLS metrics for individual upstream servers whose configuration includes the zone directive and for virtual servers whose configuration includes the status_zone directive. Having metrics from both the client side and the server side is helpful when troubleshooting problems with TLS handshakes.

The following example enables statistics collection on the upstream server upstream1 and the virtual server that proxies traffic to it.

This output indicates that the first upstream server participated in four TLS handshakes and two reused sessions (for brevity, we’re showing partial output for the first server and omitting the output for the other upstream servers):

$ curl http://127.0.0.1:8000/api/8/http/upstreams/upstream1 | jq
{
  "peers": [
    {
      "id": 0,
      "server": "127.0.0.1:8081",
      "name": "127.0.0.1:8081",
      "backup": false,
      "weight": 1,
      "state": "up",
      "active": 0,
      "ssl": {
        "handshakes": 4,
        "handshakes_failed": 0,
        "session_reuses": 2
      },
      "requests": 4,
      "header_time": 4,
      "response_time": 4,
      ...
    }
    ...
  ]
}

This output indicates that NGINX Plus participated in seven TLS handshakes:

$ curl http://127.0.0.1:8000/api/8/http/server_zones/srv | jq
{
  "processing": 0,
  "requests": 7,
  "responses": {
    "1xx": 0,
    "2xx": 7,
    "3xx": 0,
    "4xx": 0,
    "5xx": 0,
    "codes": {
      "200": 7
    },
    "total": 7
  },
  "discarded": 0,
  "received": 546,
  "sent": 1134,
  "ssl": {
    "handshakes": 7,
    "handshakes_failed": 0,
    "session_reuses": 0
  }
  ...
}

Note that the NGINX Plus API still gathers the global TLS metrics available in previous NGINX Plus releases:

$ curl http://127.0.0.1:8000/api/8/ssl | jq
{
  "handshakes": 21,
  "handshakes_failed": 0,
  "session_reuses": 6
}

Customizing the Error Code for JWT Validation Failures

NGINX Plus R25 introduced the auth_jwt_require directive for defining custom checks during the JWT validation process. In NGINX Plus R25 and R26, the error code returned for a validation failure is always 401 (Unauthorized).

NGINX Plus R27 introduces the error parameter to the auth_jwt_require directive, which you can use to set the code to 401 or 403 (Forbidden) for each directive independently. When a user’s access request fails, the choice of codes enables you to convey the distinction between an authentication failure (JWT is invalid) and an authorization failure (required claim is missing). You can also create customized handlers for the error codes, for example to return a special page explaining the error (with the error_page directive) or to redirect the request to a named internal location for further processing.

The default status code remains 401 for failure of the following types of JWT checks:

  • The (non‑custom) checks that are always performed
  • Custom checks configured with auth_jwt_require but not the error parameter

If there are multiple auth_jwt_require directives in a location block, they are evaluated in the order they appear. Processing stops at the first failure and the specified error code is returned.

In this example, the auth_jwt_require directive returns 403 if either $req1 or $req2 evaluates to an empty value or 0 (zero).

Enhancements to the NGINX JavaScript and Prometheus-njs Modules

NGINX Plus R27 incorporates versions 0.7.3 and 0.7.4 of the NGINX JavaScript module, and includes the following enhancements.

Additional Directives for the njs Fetch API

NGINX Javascript 0.5.3, incorporated into NGINX Plus R24, introduced the ngx.fetch function (also referred to as the Fetch API) for instantiating a simple HTTP client from within the context of a TCP/UDP connection. (For a discussion of use cases for the function, see Leveraging HTTP Services for TCP/UDP with a Simple HTTP Client on our blog.)

NGINX Plus R27 introduces the following directives for fine‑tuning the behavior of the Fetch API:

  • js_fetch_buffer_size[HTTP][Stream] – Sets the size of the buffer used for reading and writing.
  • js_fetch_max_response_buffer_size[HTTP][Stream] – Sets the maximum size of the response received with the Fetch API.
  • js_fetch_timeout[HTTP][Stream] – Defines the timeout for reading and writing between two successive read or write operations (not for the whole response). If no data is transmitted within the timeout period, the connection is closed.
  • js_fetch_verify[HTTP][Stream] – Enables or disables verification of the HTTPS server certificate.

Version Number Reported as a Hexadecimal Number

The new njs.version_number object reports the njs module version as a hexadecimal number. (The existing njs.version object returns the version as a character string.)

Prometheus-njs Module Converts Additional Metrics

The Prometheus-njs module, which converts NGINX Plus metrics to a Prometheus‑compliant format, can now convert metrics exposed at these additional endpoints:

Other Enhancements in NGINX Plus R27

NGINX Plus API Version Change

The version number of the NGINX Plus API is updated from 7 to 8 to reflect the addition of the ssl field to the metrics reported for upstream and virtual servers as described in TLS Metrics for Upstream and Virtual Servers. Previous version numbers still work, but the output doesn’t include metrics added in later API versions.

Better Distribution of Connections in Linux EPOLLEXCLUSIVE Mode

NGINX Plus R27 distributes connections more evenly among the NGINX worker processes when EPOLLEXCLUSIVE mode is used in Linux kernels. Normally, in this mode the kernel notifies only the process that was the first to add the listening socket to the EPOLL instance. As a result, most of the connections are handled by the first worker process. NGINX now re‑adds the socket periodically to give other worker processes a chance to accept connections.

Upgrade or Try NGINX Plus

If you’re running NGINX Plus, we strongly encourage you to upgrade to NGINX Plus R27 as soon as possible. You’ll also pick up several additional fixes and improvements, and it will help NGINX to help you when you need to raise a support ticket.

If you haven’t tried NGINX Plus, we encourage you to try it out – for security, load balancing, and API gateway, or as a fully supported web server with enhanced monitoring and management APIs. You can get started today with a free 30-day trial.

Hero image

Learn how to deploy, configure, manage, secure, and monitor your Kubernetes Ingress controller with NGINX to deliver apps and APIs on-premises and in the cloud.



About The Author

Prabhat Dixit

Prabhat Dixit

Principal Product Manager

About F5 NGINX

F5, Inc. is the company behind NGINX, the popular open source project. We offer a suite of technologies for developing and delivering modern applications. Together with F5, our combined solution bridges the gap between NetOps and DevOps, with multi-cloud application services that span from code to customer.

Learn more at nginx.com or join the conversation by following @nginx on Twitter.