Page 77 – NGINX

High Availability in NGINX Plus R6

May 7, 2015

It’s no secret that in today’s always‑on, always‑connected society, users expect your sites and apps to be accessible 24×7. This is where NGINX and NGINX Plus have always helped you shine. One of the key benefits of deploying NGINX Plus as a reverse proxy and load balancer for your application servers is increased application availability and reliability, and you can improve app and website performance even further by adding high availability (HA) to NGINX Plus itself.

In NGINX Plus Release 6 (R6), we’ve introduced a new solution that makes it fast and easy to deploy NGINX Plus instances in an active‑passive HA configuration. The solution is based on the open source keepalived project, which combines three components to provide an HA solution for Linux systems: the keepalived daemon, an implementation of the Virtual Router Redundancy Protocol (VRRP) that manages assignment of the virtual IP address for the HA pair, and a health‑checking facility that determines whether a service (for example, a web server, PHP backend, or NGINX Plus) is up and operational.

Editor –

For an overview of all the new features in NGINX Plus R6, see Announcing NGINX Plus R6 with Enhanced Load Balancing, High Availability, and Monitoring Features on our blog.

For detailed discussions of other new features in NGINX Plus R6, see these related blog posts:

TCP Load Balancing with NGINX 1.9.0 and NGINX Plus R6

Keeping Tabs on System Health with NGINX Plus Live Activity Monitoring

How Does High Availability Work?

Based on an initial configuration (see Configuring NGINX Plus for HA below), keepalived designates one NGINX Plus node as the (active) primary node and the other as the (passive) backup. Each node has its own IP address, and the primary node is also assigned a virtual IP address, which is the one advertised to clients in the Domain Name System (DNS). During operation, VRRP and the health‑checking facility monitor the operation of the nodes and NGINX Plus so that the virtual IP address can be reassigned to the backup if the primary fails:

The VRRP instance on the backup node listens for advertisement packets from its peer on the primary node. If it does not receive an advertisement packet for a period longer than three times the configured advertisement interval, it takes over as primary and assigns the virtual IP address to itself.
If health checks of NGINX Plus on the primary node fail a configured number of times, keepalived reassigns the virtual IP address from the primary node to the backup node.

Configuring NGINX Plus for HA

To configure an active‑passive pair of NGINX Plus nodes for HA, perform the following steps on both nodes:

Install or upgrade to the NGINX Plus R6 base package (nginx-plus) by following the instructions for your operating system at the NGINX Plus repository.
Using the installation command specified at the NGINX Plus repository for your operating system, install the nginx-ha-keepalived package by replacing nginx-plus with nginx-ha-keepalived, as in this example for Ubuntu:
```
root# apt-get install nginx-ha-keepalived
```
Collect the following information to supply when prompted by the nginx-ha-setup script that you’ll run in the next step:
- The physical IP addresses of both nodes.
- The virtual IP address, which is assigned initially to the primary node. The script refers to it as the “cluster IP address (or endpoint)”. It must not be the actual IP address of either node.
Run the nginx-ha-setup script (it’s installed from the nginx-ha-keepalived package into /usr/bin).
```
root# nginx-ha-setup
```
We recommend that you run the script on both nodes at the same time in side‑by‑side terminal windows, performing each step on both nodes before proceeding to the next step. We also recommend that you use the Linux screen(1) command to start the terminal windows, so that installation can continue if a session gets disconnected.

The `keepalived` Configuration Script

On each node, the nginx-ha-setup script creates the local keepalived configuration file, /etc/keepalived/keepalived.conf, incorporating the values you provided in Step 3 above. The following is an example created on a node with IP address 192.168.100.100 (the other node’s address is 192.168.100.101 and the virtual IP address is 192.168.100.150). Note that although this looks a lot like NGINX Plus configuration, it isn’t – for one thing, semicolons are not used to delimit directives.

vrrp_script chk_nginx_service {
    script "/usr/libexec/keepalived/nginx-ha-check"
    interval 3
    weight 50
}
 
vrrp_instance VI_1 {
    interface eth0
    state BACKUP
    priority 101
    virtual_router_id 51
    advert_int 1
    unicast_src_ip 192.168.100.100
    unicast_peer {
        192.168.100.101
    }
    authentication {
        auth_type PASS
        auth_pass f8f0e5114cbe031a3e1e622daf18f82a
    }
    virtual_ipaddress {
        192.168.100.150
    }
    track_script {
        chk_nginx_service
    }
    notify "/usr/libexec/keepalived/nginx-ha-notify"
}

Explaining the purpose of every directive is beyond the scope of this post, but there are a few things to note.

There are two main configuration blocks in the file:
- The vrrp_script block configures the health‑checking facility to run a script that checks whether NGINX Plus is operational.
  
  The nginx-ha-check script is installed automatically from the nginx-ha-keepalived package into the indicated directory, which varies by operating system.
  
  The interval directive sets how often the script runs, in seconds.
- The vrrp_instance block configures the VRRP instance on this node.
  
  The IP addresses you provide are recorded as the values of three directives: unicast_src_ip (the local node), unicast_peer (the other node), and virtual_ipaddress (the virtual IP address assigned to the primary node).
  
  The advert_int directive controls how often (in seconds) the VRRP instance on the primary node sends advertisements to its peer on the backup node.
  
  The value 51 for the virtual_router_id directive is a sample value; change it as necessary to be unique in your environment (this becomes relevant if you are using VRRP to provide HA for multiple services).
Together, the weight directive in the vrrp_script block and the priority directive in the vrrp_instance block are used in determining which node becomes primary. For more information, see Using a Health‑Checking Script to Control Which Node Is Primary in the NGINX Plus Admin Guide article.
If you have multiple pairs of keepalived instances (or other VRRP instances) running in your local network, create a vrrp_instance block for each one, with a unique name (like VI_1 in the example) and virtual_router_id number.

Similarly, if you are using keepalived to provide HA for other services, for each one you need to create a separate health‑check script and vrrp_script block.

Why Would a Web Server Need TCP Load Balancing?

NGINX is much more than a web server. It began life as a reverse proxy for HTTP traffic to accelerate other web servers. Over time it gained more and more features that offloaded work from the web servers it was accelerating, and NGINX is now a very complete and efficient open source combination of web server and caching/accelerating reverse proxy and application gateway.

NGINX is commonly used as the front end for busy websites – 44.9% of the 10,000 busiest sites use NGINX as their front end as of this writing. If NGINX is managing web traffic, it’s logical for it to act as a front end for other application traffic types as well, and that’s what the TCP load balancing feature provides for.

TCP load balancing can also benefit you when you use NGINX or NGINX Plus internally to proxy and load balance an application. Modern applications use a range of protocols, not all of which are HTTP‑based. NGINX and NGINX Plus can be used to handle all of the internal traffic – MySQL load balancing is a good example.

How Do I Configure TCP Load Balancing?

TCP load balancing is configured in the stream configuration context. This context, like the http and mail contexts, enables you to configure one or more listen servers for TCP connections. Connections are forwarded to the upstream server group named by the proxy_pass directive. A load‑balancing algorithm is used to select one of the upstream servers (in this example, db1, db2, or db3):

stream {
    server {
        listen 3306;
        proxy_pass db;
    }
 
    upstream db {
        server db1:3306;
        server db2:3306;
        server db3:3306;
    }
}

This simple configuration employs the features in NGINX Plus R5, so the TCP Load Balancing in NGINX Plus R5 blog post is a good introduction. To learn how to configure all the R6 features, see TCP Load Balancing in the NGINX Plus Admin Guide and the reference documentation for the four Stream modules:

Stream Core module
Stream Proxy module
Stream SSL module
Stream Upstream module

What Does the Future Hold for TCP Load Balancing in NGINX and NGINX Plus?

Our plan for TCP load‑balancing features in NGINX and NGINX Plus follows the pattern we’ve established for HTTP proxying and load balancing:

Open source NGINX covers reverse proxying (accepting and forwarding traffic to an upstream server) and load balancing (choosing from a group of upstream servers based on a selection algorithm). If you’re prepared to build and support your own NGINX instances, you can pull in various third‑party modifications that implement advanced features not included in the standard NGINX distribution.

NGINX Plus covers reverse proxying and load balancing, and adds the features you see listed in the table for commercial application delivery controller (ADC) products, such as health checks, connection management, session persistence, remote configuration, and monitoring:

	NGINX 1.9.0	NGINX Plus
Core features
Reverse proxy for HTTP and TCP	✅	✅
Load balancing (simple algorithms)	✅	✅
Connection retry on failure (passive monitoring)	✅	✅
SSL termination and encryption	✅	✅
Advanced ADC features
	Some features might be available in third‑party modules	All features are provided and supported by NGINX, Inc.
Advanced load balancing algorithms		✅ (Least Time algorithm)
Health checks		✅ (HTTP and TCP)
Connection management		✅ (HTTP and TCP)
Session persistence		✅ (HTTP)
Dynamic reconfiguration		✅ (HTTP and TCP)
Live activity monitoring		✅ (HTTP and TCP)

As we develop features in NGINX and NGINX Plus for HTTP load balancing, we’ll seek to develop equivalent features for TCP where possible, resulting in a consistent and predictable roadmap for both.

Editor –

To try out TCP load balancing with NGINX Plus for yourself, start your free 30-day trial today or contact us to discuss your use cases.

NGINX 1.8 and 1.9 Released

Today we’re happy to share the release of NGINX 1.8, the new stable branch, and NGINX 1.9, the new mainline branch where we’ll be working on development of new features. These branches replace 1.6 and 1.7 respectively. Over the past year, we’ve pushed 12 updates to 1.7, and all the features we added are now available in 1.8, including these:

Backend SSL/TLS certificate verification – Protects against “man in the middle” (MITM) attacks by checking and validating certificates presented by upstream servers
Logging to syslog – Enables log aggregation over the network from multiple instances using UDP messages
Thread pools support for offloading I/O requests – Improves the performance of NGINX when blocking operations are required
Hash load balancing method – Allows for a server group where the client‑server mapping is based on the hashed variable
Unbuffered upload – Optionally disables the buffering of a request body

For a list of all the features added during the last year, check out the release notes for 1.8.

How Branch Renumbering Works

The branch renumbering is part of our annual checkpoint process where we fork off a “stable” (no new features) branch from the “mainline” (feature development) branch. NGINX 1.8 was forked as the new stable branch from the 1.7 mainline branch, and the 1.7 branch is renumbered to 1.9. We are no longer supporting 1.6 (the former stable branch) or doing development in 1.7.

In NGINX nomenclature, stable (now version 1.8) means that the feature set is fixed; only critical bug fixes are committed to that branch. Stable does not necessarily mean more reliable or more bug‑free. In fact, the mainline branch (now 1.9) is generally regarded as more reliable because we commit a wider range of bug fixes to it, not just the critical fixes merged into the stable branch. On the other hand, any changes to the stable branch are very unlikely to affect the operation of third‑party modules, whereas in the mainline branch the addition of new features might do so.

For more information about how we do our branch renumbering, check out last year’s post about the 1.6 and 1.7 branches.

What to Expect in 1.9

The first new features we’re adding to 1.9 are TCP load balancing, which will allow our users to load balance TCP‑based applications, and support for upstreams in shared memory zones, which improves failover consistency among worker processes. We’ve begun writing deployment guides about load balancing some of the more common services that use TCP. Looking forward to later this year, you can expect an implementation of HTTP/2, and the addition of a dynamic pluggable module interfaces. There’s great work being done and lots to look forward to in 1.9.

Where Does NGINX Plus Fit in All of This?

NGINX Plus is the commercially supported version of NGINX, with a number of additional features. NGINX Plus tracks the mainline version of NGINX, and is typically released on a quarterly cycle. New features from the mainline branch are merged into NGINX Plus and released once they have passed full integration testing and have been proven in the field in the mainline branch.

Keeping Tabs on System Health with NGINX Plus Live Activity Monitoring

April 23, 2015

NGINX Plus Release 6 (R6) introduces a great update to the live activity monitoring dashboard that gives you visibility into the real‑time performance of NGINX Plus. In this article, we’ll dissect the dashboard and look at some of the statistics behind it.

Editor – In NGINX Plus Release 14 (R14) and later, the live activity monitoring dashboard uses the NGINX Plus API to collect the statistics it displays, instead of the separate Extended Status handler described in this post. Further, the organization of the dashboard and the reported metrics have been updated. For a current description and configuration instructions, see the NGINX Plus Admin Guide. For information about upgrading to the NGINX Plus API, see Transitioning to the New NGINX Plus API for Configuration and Monitoring on our blog.

The live activity monitoring dashboard in NGINX Plus displays a wealth of real‑time information:

Key software information and high‑level alerts relating to the performance and operation of your load‑balanced cluster
Real‑time and historical (average) performance data – requests and bandwidth – based on server zones and applications that you define, for the HTTP and TCP services that you configure
Detailed performance and health information for each upstream load‑balanced group
Instrumentation and diagnostics on the operation of each content caches

Like many modern applications, the NGINX Plus dashboard is a static web page (/var/www/html/status.html) that is driven and updated by a RESTful API. The API polls the Extended Status handler each second, retrieving a JSON object containing a large number of counters and gauges, and then charts them in a range of displays:

To see the dashboard in action, check out the demonstration at demo.nginx.com.

Getting Started with the Live Activity Monitoring Dashboard

Our recent post, Live Activity Monitoring of NGINX Plus in 3 Simple Steps, discusses a sample script that we provide to make it easy to configure the dashboard. For reference, the configuration is duplicated here:

server {
    listen 8080;
    root /usr/share/nginx/html;

    # Redirect requests for / to /status.html
    location = / {                # DEPRECATED -- see Editor's note above
        return 301 /status.html;  # DEPRECATED
    }

    location = /status.html { }  # DEPRECATED

    # Everything beginning /status (except for /status.html) is
    # processed by the status handler
    location /status {           # DEPRECATED
        status;                  # DEPRECATED
    }
}

Because this configuration represents a standard NGINX virtual server, you can easily extend it to add access control rules, deliver the content over SSL/TLS, and change the IP address and port for the service.

The Components of the Dashboard

The dashboard provides several different types of information.

Editor – The dashboard has been updated since publication of this post. For current information about the organization of the dashboard, see the NGINX Plus Admin Guide.

Global Traffic Counts

The top section of the dashboard summarizes status and traffic at a high level:

The Connections table lists the number of connections that NGINX Plus has received since startup or configuration reload. The Active and Idle fields report the instantaneous number of connections; it’s normal to see some active connections (meaning there’s a request in flight) and some idle (the client is holding the connection alive for an HTTP keepalive).

Similarly, the Requests table reports the number of HTTP requests NGINX Plus has handled since startup or configuration reload This count is commonly larger than the Connections count, because one connection can transport several requests. The Req/s (requests/second) field reports the single most useful measure of activity; even if NGINX Plus is idle, you’ll see this count increment once per second because the dashboard handler issues one request for the /status URL each second.

Service Status

The service status section gives you an immediate perspective on the health of your NGINX Plus deployment. Details about each element appear in tables further down on the dashboard.

The green check means all is okay, the red X indicates a problem you should look into, and the amber exclamation point warns you that things are not working optimally.

Server Zones

To view a wide range of statistics about a virtual server in the Server zones table, you need to make the server a member of a “status zone” by including the status_zone directive in its configuration block. You can configure more than one virtual server to belong in the same zone (for example, to combine statistics for HTTP and HTTPS servers), in which case the values for the zone are aggregated across all zone members.

The line for zone one in the table above results from this configuration block:

server {
    status_zone one;
}

This is where the deeper analytics begin. When errors occur, the dashboard immediately flags them in red, as shown for the Responses – 4xx field in the table above, and then they fade away. The Traffic section in the table reports both current and average traffic levels so you can determine if current traffic volumes differ significantly from the norm.

You can also configure TCP virtual servers into zones, and view connection rates and the amount of data transferred in the TCP zones table:

Upstreams

Many organizations use NGINX Plus as a load balancer and application delivery controller, and the tables in the Upstreams section of the dashboard monitor the health and performance of the nodes that NGINX Plus is load balancing traffic across.

Each upstream group gets its own table in the Upstreams section (the table above is for the demoupstreams group). The fields under Requests, Responses, Connections, and Traffic enable you to drill down into the performance of each upstream server. The fields under Server checks and Health monitors measure the number of failures when sending requests to each upstream server or running health monitors against each upstream server, respectively.

The dashboard displays some of the most commonly used statistics, and there are many more in the full JSON data stream. For a list of all the statistics, see the documentation for status_zone directive, and you can get a flavor of how to drill down into the JSON data here (the data is for a single server in an upstream group in the demo called demo-backend).

Caches

Finally, the dashboard gives you insight into the performance of the disk‑based content caches that you configure in NGINX Plus, with one report for each defined cache zone:

Caches begin in the Cold state (a snowflake icon), as the cache loader process initializes each cache from the files on disk. Once a cache is fully initialized, its state moves to Warm (a sunshine icon).

Caches grow continually until they reach 100% utilization. The cache manager process periodically scans the cache and prunes old (least recently used) entries to keep the size within the configured maximum limit.

The fields under Traffic and the Hit ratio meters report the effectiveness of the cache – requests served from the cache, requests that were stored in the cache (Written) and requests that bypassed the cache (because the cache content was missing or out of date, or because of user configuration). Hit ratio is a moving average over the past 5 minutes (by default).

Configuring the Dashboard

The settings icon in the dashboard allows you to fine‑tune how frequently the dashboard page pools NGINX Plus, and the thresholds for various responsive components of the display.

The JSON Activity Data

Behind the pretty face of the dashboard is a comprehensive JSON data feed.

extended-status — Drill down to the data you need with NGINX’s Extended Status data

You can explore this data using your web browser – a JSON pretty‑printer extension such as JSONview (for Chrome or Firefox) helps immensely. URLs begin with /status, and you can drill down and see a subset of the data.

Many of the values operate like SNMP counters. For example, the requests counter (URL http://demo.nginx.com/status/requests/total) lists the total number of requests processed since startup or reconfiguration. To derive the number of requests per second, sample this counter twice, and divide the difference by the number of seconds between samples.

Using Third‑Party Monitoring Tools for NGINX Plus

A number of third‑party tools have developed integration packs for the NGINX Plus statistics data stream:

… and with the RESTful interface, it’s easy to add NGINX Plus monitoring to your own custom dashboards and tools, too!

Try out the live activity monitoring dashboard in your own environment – start your free 30-day trial today or contact us to discuss your use cases.

Editor –

Network vs. DevOps: How to Manage Your Control Issues

April 20, 2015

There’s no disputing that technology and infrastructure have changed dramatically over the years. Web sites have long since evolved from “just-a-bunch-of-files-and-scripts” to an intricate mix of modular applications made up of reusable code components – all using HTTP, which has become the dominant protocol for Internet communication. And these web applications now run people’s lives. Users expect immediate response, flawless behavior, and apps that work smoothly on every device.

While the engineer in me finds this all fascinating, the technical manager in me is keenly aware of the challenges brought about by such rapid advancements and their corresponding expectations. Chief among these challenges is the issue of control. We’ve made great progress in transforming our legacy infrastructures to more fluid web architectures, but we’re still using legacy rules to delegate who controls what. That’s why, in many companies today, network teams still control application delivery – not the DevOps engineers who are building these applications and are most responsible for accelerating their performance. Needless to say, this can be problematic.

Competing Priorities and Complexity Hamper Team Performance and Efficiency

Do the following struggles sound familiar? Perhaps you’ve witnessed these situations within your own organization. If so, it’s time to consider how to remedy the tension and conflict caused by this imbalance of control and accountability.

Perhaps you’re a developer and all you want to do is add or change an application‑level access‑control list (ACL) for added protection. Or maybe you’re redirecting traffic temporarily to a standby application server. But instead of just taking care of these tasks yourself, you are required to carefully construct a request to your network operations team and (as they are likely overwhelmed with requests from all over your organization) wait for them to find a free moment to process your request.

Eventually, if you bug them enough, they might be willing to delegate some of these simple controls to you, as they deal with their other (many times more important) assignments. If things go wrong, however, then you’re the one to blame when outages result. All because you forgot to explicitly allow certain traffic where the vendor’s ACL has a default of “implicit deny,” or maybe you messed up the “reverse wildcard masks,” or you just applied the ACL to the wrong interface. You can’t win, and it’s eating up time on both sides either way you look at it.

The network team retains control over application delivery in many instances because companies use hardware‑based application delivery controllers. These centralized application delivery solutions are already vulnerable as single points of failure. When you combine that risk with management by an overburdened network infrastructure team in an organization trying to move quickly in building and deploying applications, you’re taking a serious gamble.

Application developers often outnumber network infrastructure personnel by a wide margin. It’s not uncommon to hear of companies with thousands of virtualized instances, dozens of applications, a hundred people in DevOps – and only two network engineers managing a couple of big application delivery controller boxes in front of a private cloud. Obviously, the network team is uneasy about making many concurrent changes to the web acceleration layer, but they’re equally disconcerted about the application logic moving into the network.

Remember, however, that a fast network doesn’t always mean fast applications. Neither a potentially conflicting configuration line nor a kludge script on the networking box will change that. In fact, all you achieve by sending a never‑ending flow of configuration changes to the networking team is mounting resentment.

Another source of discord centers on the modern, agile development processes used by today’s application system engineers. They can no longer just build a product and then hand it off to someone else to deploy and operate; rather, they must build an iteration, deploy, and repeat the cycle. And this cycle requires full, end‑to‑end control over performance and scalability issues associated with the “heavy lifting” of HTTP. Having to manage this while negotiating who controls what slows the engineers down and can make it difficult for them to make changes that bolster performance.

Software‑Based Solutions: A New Hope

So what’s a DevOps engineer to do? You need to find a viable way to gain control over the complex functions that are critical to your work in web acceleration and application performance, while still respecting the performance and security requirements of your organization.

Well, the bad news is you can’t turn to any tools that were introduced to the market 10 or 15 years ago. They were built for big old monolithic applications (both consumer and enterprise), so they don’t lend themselves to today’s modern, distributed, loosely coupled, and composeable applications. And network engineers can’t do much these days to help application engineers overcome the complexities of HTTP. They’re still primarily in charge of hardware networking appliances that are focused on packets, not applications.

Wait, you might say. What about sending Network Function Virtualization (NVF) to the rescue? That’s a possible – but only partial – solution. To begin with, it offers predominantly Layer 2 and sometimes Layer 3/4 functions, while functions related to Layer 7 aren’t really well developed yet. The other problem with NFV and Software Defined Networking (SDN) is that, so far, they really only address the problems of commoditized network infrastructure, not the challenges of managing integrated Layer 7 functions in a DevOps environment. You need a solution that puts control of the appropriate layers in the right hands. The network team needs Layer 3/4 control, but DevOps engineers need Layer 7 control.

Last but not least, most application frameworks also do not provide any good way to quickly and effortlessly deal not only with the complexities inherent to HTTP – like concurrency handling, security, and access control, and so on – but also with Layer 7 traffic management in general.

What’s the answer then? When neither the “network” nor application frameworks are really helpful in end-to-end web acceleration, are there any software tools to resolve the tension, and fully enable that much‑needed empathy in DevOps?

Fortunately, the answer is yes. There are a few very useful tools around – HAProxy, NGINX Plus, Varnish, and Apache Traffic Server, among others – all of which have been successful in many DevOps environments. These tools combine the elegance and convenience of a compact, software‑only package, suitable for any generic operating system, with the capabilities of a web acceleration tool. And they can be deployed in a matter of seconds on any physical or virtual server instance.

Some of these tools specialize on just a subset of web acceleration techniques – such as server load balancing or SSL termination – while others (like our own NGINX Plus) cover all main functions, such as Layer 7 load balancing, request routing, application and content delivery, web caching, SSL termination, on‑the‑fly compression, protocol optimization, and more.

These DevOps tools follow “application‑style” configuration, and they are most suitable for automation with other tools like Puppet or Docker. Likewise, they run very efficiently in virtualized environments, are easily deployable across cloud environments, are easily co‑located with applications, and lastly are applications themselves. At the same time, these products now equal or surpass the performance of hardware‑based networking appliances, with no limits on throughput at a fraction of the cost.

Finally, and perhaps most important, they also give DevOps teams long‑awaited end‑to‑end control over HTTP performance and scalability.

DevOps Tools Put Control in the Right Hands

Adopting DevOps tools, then, is how you can realize peaceful empathy. DevOps and network infrastructure teams can co‑exist happily and cooperate more effectively when Layer 7 functions are implemented in a software‑based, scaled‑out layer – built and managed entirely by DevOps teams – while the task of moving packets, controlling QoS, and other traditional network‑bound functions remains with the network infrastructure team.

Who benefits? Everyone. The engineering organization benefits from smoother operations, faster deployment and configuration, and streamlined application development. The network team has less of a burden placed on them and retains control where necessary, while being freed to focus on their own work. Users benefit from a better application experience. And, ultimately, the company benefits from the ability to efficiently deliver a high‑performance application that addresses market expectations.

We encourage you to try this software‑based approach for yourself, and see just how much faster, more efficient, and happier your teams can be. Start your free 30-day trial of NGINX Plus today as an alternative to your hardware‑based application delivery controller, or contact us to learn more about how NGINX can help you deliver your applications with performance, security, reliability, and scale in a DevOps‑friendly way.

Using NGINX and NGINX Plus to Protect Against CVE-2015-1635

April 16, 2015

On April 14, Microsoft issued a vulnerability alert – now tracked as CVE-2015-1635 – about an issue that might permit remote code execution if an attacker sends a specially crafted HTTP request to an affected Windows system. If patching your production Windows servers immediately is not an option, then NGINX and NGINX Plus can help protect you from attacks.

The specific details of the vulnerability have not been released as of this writing, but attackers are reportedly trying to find vulnerable systems by sending HTTP requests with very large Range requests, which can trigger a buffer overflow and cause a crash in the Windows system.

Users are strongly advised to apply Microsoft’s patch to address this vulnerability. However, if you are not able to apply the patch to all of your production systems and are using NGINX or NGINX Plus to load balance or proxy traffic to them, a simple configuration change is enough to intercept and fix the special requests sent to vulnerable systems.

Identifying and Handling Reconnaissance Traffic

Mattias Geniar has analysed the attack traffic and reports that HTTP requests with a large byte range in the Range header trigger the crash:

GET / HTTP/1.1rn
Host: stuffrn
Range: bytes=0-18446744073709551615rn
rn

The simplest fix is to use the proxy_set_header directive to set the Range header to "" (the empty string), which effectively deletes the header before the HTTP request is forwarded to the Windows server named by the proxy_pass directive:

server {
    listen 80;
 
    location / {
        proxy_set_header Range "";
        proxy_pass http://windowsserver:80;
    }
}

If your application requires byte‑range support, you can use the map directive to replace any string that resembles a large integer with the empty string, before using the proxy_set_header directive to set the Range header:

map $http_range $saferange {
    "~d{10,}" "";  # if it matches a string of 10 or more integers, remove it
    default $http_range;
}
 
server {
    listen 80;
 
    location / {
        proxy_set_header Range $saferange;
        proxy_pass http://windowsserver:80;
    }
}

Alternatively, you can return HTTP code 444 when the value in the Range header resembles a large integer. Code 444 instructs NGINX and NGINX Plus to close the client connection immediately without returning anything.

server {
    listen 80;
 
    if ($http_range ~ "d{9,}") {
        return 444;
    }
 
    location / {
        proxy_pass http://windowsserver:80;
    }
}

Keep safe and apply the patch, but if you can’t do that immediately, NGINX and NGINX Plus can help close the potential hole over.

To try NGINX Plus, start your free 30-day trial today or contact us for a demo.

NGINX Open Source: Reflecting Back and Looking Ahead

April 14, 2015

Today we announced the availability of NGINX Plus Release 6 (R6). With this milestone event in our commercial business, I thought it would be a good time to reflect back on what we have accomplished as a community and to address where we are taking NGINX from here.

Igor Sysoev, NGINX, Inc. Co‑founder & CTO

The last 12 months have been a hectic time at NGINX, Inc. We have had exceptional growth as a project, as a development team, and as a company. The number of sites using our software has grown enormously, and NGINX is now the most commonly used web frontend for the 10,000 busiest sites on the Internet. We now count some of the most innovative developers in the world as users and as contributors – from disruptive new businesses like Airbnb and Uber to Netflix. NGINX Open Source has benefited from well over 100 new features and updates, and NGINX Plus has matured into a highly capable application delivery platform.

Looking Ahead

What does the future hold for NGINX and the applications we power? It’s often said that history repeats itself, but not in a circular fashion. Each time round, things get a little better. We are currently seeing a resurgence of the concept of service oriented architectures, but in a more modern, loosely coupled, more easily developed way that’s referred to as microservices. In the fluid, turbulent, containerized environments in which many of us are building applications, NGINX is a stable and reliable foundation, both for hosting the services and for routing the traffic between them. This is shaping much of our thinking as we consider future use cases for NGINX.

The next 12 months herald some major new features for NGINX Open Source. The stories about NGINX and JavaScript will be realized – I have a working prototype of a JavaScript VM that is highly optimized for NGINX’s unique requirements and we’ve begun the task of embedding it within NGINX.

Our community of module developers is vital to the success of NGINX in the open source world. We know it’s not as easy as it could be to develop for NGINX, and to address that situation, we’re beginning the implementation of a pluggable module API in the next couple of months. Our goal is to make it simpler for our developer community to create and distribute modules for NGINX, giving users more choice and flexibility to extend the NGINX Open Source core. We’re also establishing a developer relations team to help support the community in this transition.

You may already have read our plan to support HTTP/2 in NGINX. We appreciate how important it is to our users that we continue to support the innovations that others are making in our space, and our HTTP/2 support will of course build on the successful SPDY implementation in use at a number of sites.

The Role of NGINX Plus

People sometimes ask about NGINX Plus and how it relates to NGINX Open Source. NGINX Plus was born from the desire to create a commercial offering that would extend the software’s capabilities and help fund the continued development of NGINX.

The two products have a lot of overlap; in fact, it’s possible to use various open source third‑party modules to implement much of the additional functionality in NGINX Plus. We’re completely comfortable with that. If users have the expertise, patience, and time required to maintain their own custom build of NGINX for comprehensive application delivery, then NGINX Plus is clearly not for them. However, if maintaining and supporting an application delivery platform is not your core competency or if you would prefer your technical resources directed at applications that more directly further your business, we’re here with NGINX Plus and a range of services to do that for you.

Thank you to everyone who purchased subscriptions to NGINX Plus. Not only have you received an application delivery platform that blows every commercial alternative out of the water on price/performance, you’ve also helped to support the growing engineering team in Moscow who maintain the very high standards of both NGINX and NGINX Plus.

We greatly appreciate your support, whether as a commercial user, a third‑party developer, a supporter of NGINX, or just an end user of NGINX. Together, we’re making the web a better place for developers, admins, and end users. We hope that you continue to use NGINX and NGINX Plus, and in return, we remain committed to providing the most powerful, lightweight, and high‑performance software to make your applications as great as they can be.

To try NGINX Plus, start your free 30-day trial today or contact us for a demo.

Announcing NGINX Plus R6 with Enhanced Load Balancing, High Availability, and Monitoring Features

We’re really pleased to announce the availability of NGINX Plus Release 6 (R6). This latest release of our application delivery platform gives NGINX Plus users even more to love, including:

A new “Least Time” load‑balancing algorithm
Full‑featured TCP load balancing
High availability and failover between NGINX Plus instances
An improved live activity monitoring dashboard
Unbuffered upload to support event‑driven backends like Node.js
Support for SSL/TLS authentication of email traffic (IMAP, POP3, and SMTP)
Enhancements to caching

Editor – For more details about key new features in NGINX Plus R6, see these related blog posts:

TCP Load Balancing with NGINX 1.9.0 and NGINX Plus R6

High Availability in NGINX Plus R6

Keeping Tabs on System Health with NGINX Plus Live Activity Monitoring

Our customers have expressed overwhelming interest in using NGINX Plus to replace legacy hardware and to further support the adoption of public and private clouds. With the release of R6, NGINX Plus exceeds the capability of traditional hardware load balancers and ADCs, while providing unlimited throughput at a lower cost than our competitors. We believe it is now the ideal choice for application delivery and load balancing, whether for modern web applications or for enterprise applications like relational databases and mail servers.

New “Least Time” Load‑Balancing Algorithm

The new Least Time load‑balancing algorithm monitors both the number of concurrent connections and the average response time from each node in the load‑balanced pool. It uses this information to select the most appropriate node for each request, with the goal of selecting faster and less‑loaded nodes in preference to slower and more heavily loaded ones.

Least Time outperforms other load‑balancing methods when nodes differ significantly in latency. One common use case is load balancing across nodes located in two separate data centers; local nodes tend to have very little latency compared to nodes in a remote data center. Least Time prefers the low‑latency nodes, but NGINX Plus’ health checks ensure failover to the slower nodes if the faster ones fail or go offline.

Least Time can base its load‑balancing decisions on either the time to receive the response headers from the upstream, or the time to receive the entire response. Two counters being added to the set of extended status statistics, header_time and response_time, present the rolling‑average measurements used as the basis for decisions.

Full‑Featured TCP Load Balancing

The TCP load balancing feature introduced in NGINX Plus R5 has been significantly extended to include TCP health checks, dynamic configuration of upstream server groups, full access logs, and SSL/TLS termination and encryption. Many new extended status counters have been added for TCP load balancing, providing the same level of reporting and visibility that you already enjoy for HTTP load balancing.

TCP load balancing has already been proven in a number of use cases, including load balancing and high availability of MySQL and load balancing and high availability of Microsoft Exchange.

High‑traffic TCP‑based services are not the only ones to benefit from TCP load balancing. Even low‑traffic services can benefit from high availability (using health checks and dynamic reconfiguration), improved security (using SSL/TLS wrapping) and improved visibility (using extended status counters and access logging).

Editor –

For a detailed discussion of this feature, see TCP Load Balancing with NGINX 1.9.0 and NGINX Plus R6

NGINX Plus Release 9 and later support the features discussed in this section for UDP traffic as well as TCP.

High Availability

NGINX Plus supports high‑availability clusters using a solution based on the Linux keepalived utility. You can easily create high‑availability pairs of NGINX Plus instances, using the Virtual Router Redundancy Protocol (VRRP) to assign traffic IP addresses to the primary NGINX Plus instance and transfer them automatically to the backup instance if the primary fails.

To enable and configure this feature, install the optional nginx‑ha‑keepalived package. After initial configuration, you can extend the configuration to implement more complex scenarios, including larger clusters of NGINX Plus instances and use of multiple virtual IP addresses.

For more details about the high‑availability package and its installation process, see the NGINX Plus Admin Guide and High Availability in NGINX Plus R6 on our blog.

Updated Dashboard for Live Activity Monitoring

NGINX Plus R6 includes a new, richer status dashboard that charts the health and activity of your NGINX Plus instance using a wealth of live activity monitoring information:

Key software information and high‑level alerts relating to the performance and operation of your load‑balanced cluster
Real‑time and historical (average) performance data – requests and bandwidth – based on server zones and applications that you define, for the HTTP and TCP services that you configure
Detailed performance and health information for each upstream load‑balanced group
Instrumentation and diagnostics on the operation of each content cache

As in earlier releases, the live activity monitoring data is provided in JSON format via a RESTful interface so that you can incorporate NGINX statistics directly into your own dashboards and other monitoring tools.

For a live demonstration, check out demo.nginx.com. For a more detailed exploration of the dashboard, see Keeping Tabs on System Health with NGINX Plus Live Activity Monitoring on our blog.

Support for Unbuffered Upload

You can now configure NGINX Plus for unbuffered upload, meaning that it streams large HTTP requests (such as file uploads) to the server as they arrive, rather than buffering and forwarding them only after the entire request is received.

This modification improves the responsiveness of web applications that handle large file uploads, because the applications can react to data as it is received, enabling them, for example, to update progress bars in real time. It also reduces disk I/O and can improve the performance of uploads in some situations. By default, NGINX buffers uploaded data to avoid tying up resources in worker‑based backends while the data arrives, but buffering is less necessary for event‑driven backends like Node.js.

SSL/TLS Enhancements

NGINX Plus R6 can provide a client certificate to authenticate itself when communicating with an upstream HTTPS or uwSGI server. This improves security, particularly when communicating with secure services over an unprotected network.

NGINX Plus R6 supports SSL/TLS client authentication for IMAP, POP3, and SMTP traffic.

Caching Enhancements

The proxy_cache directive now supports variables. This simple change means you can define multiple disk‑based caches and select a cache based on request data.

This feature is most useful when you need to create a very large content cache and use multiple disks to cache content. By creating one cache per disk, you can ensure that temporary files are written to the same disk as their final location and thus eliminate disk‑to‑disk copies.

Upgrade or Try NGINX Plus

If you’re running NGINX Plus, we strongly encourage you to update to Release 6 as soon as possible. You’ll pick up a number of fixes and improvements, and it will help us to help you if you need to raise a support ticket. Installation and upgrade instructions can be found at the customer portal.

If you’ve not tried NGINX Plus, we encourage you to try it out for web acceleration, load balancing, and application delivery, or as a fully supported web server with an API for enhanced monitoring and management. You can get started for free today with a 30‑day trial and see for yourself how NGINX Plus can help you scale out and deliver your applications.

Editor – For more details about key new features in NGINX Plus R6, see these related blog posts:

TCP Load Balancing with NGINX 1.9.0 and NGINX Plus R6

High Availability in NGINX Plus R6

Keeping Tabs on System Health with NGINX Plus Live Activity Monitoring

Solution Quick Links

Product Quick Links

Resources Quick Links

Resources Quick Links

Blog Quick Links

Blog Quick Links

NGINX Quick Links

Partners Quick Links

NGINX Quick Links

How Does High Availability Work?

Configuring NGINX Plus for HA

The keepalived Configuration Script

Related Reading

Why Would a Web Server Need TCP Load Balancing?

How Do I Configure TCP Load Balancing?

What Does the Future Hold for TCP Load Balancing in NGINX and NGINX Plus?

How Branch Renumbering Works

What to Expect in 1.9

Where Does NGINX Plus Fit in All of This?

Getting Started with the Live Activity Monitoring Dashboard

The Components of the Dashboard

Global Traffic Counts

Service Status

Server Zones

Upstreams

Caches

Configuring the Dashboard

The JSON Activity Data

Using Third‑Party Monitoring Tools for NGINX Plus

Competing Priorities and Complexity Hamper Team Performance and Efficiency

Software‑Based Solutions: A New Hope

DevOps Tools Put Control in the Right Hands

Identifying and Handling Reconnaissance Traffic

Looking Ahead

The Role of NGINX Plus

New “Least Time” Load‑Balancing Algorithm

Full‑Featured TCP Load Balancing

High Availability

Updated Dashboard for Live Activity Monitoring

Support for Unbuffered Upload

SSL/TLS Enhancements

Caching Enhancements

Upgrade or Try NGINX Plus

Get Started

Try Out NGINX Plus on Your Turf

Ask Us a Question

The `keepalived` Configuration Script