The way companies deploy new features in their applications has changed dramatically over the past few years. Whereas companies used to wait and deliver a large set of new features all at once, the focus now is on delivering features to customers as soon as the code is ready. Using automation and a strong tool set, some companies are able to do this over 50 times per day.
Deploying changes this frequently, however, puts additional requirements on your application delivery infrastructure, because each deployment can require a change in configuration. For example, if you delete and replace application servers during a deployment rather than updating them, you must also update the load balancer configuration. Or, if the newly deployed functionality requires access to additional resources, such as a different database, you also need to update the security and access control configuration. On the negative side, if you discover a bug in the new functionality, you might need to roll back the configuration.
In a traditional development environment where an IT team controls deployment, making changes to the application delivery infrastructure typically requires filing a ticket and waiting for a change window, which can sometimes take weeks. If you’re deploying 50 times per day, that model clearly doesn’t work. Making configuration changes should be flexible, automatable, and easily reversible. In addition, the infrastructure must be able to stay online and process traffic while undergoing constant configuration changes.
NGINX and NGINX Plus are application delivery platforms designed to work in the most demanding environments. NGINX can be easily reconfigured without any downtime or loss of traffic, and NGINX Plus adds additional features to help further automate and ease the process of modifying configuration. This blog post covers techniques for dynamically modifying and automating configuration for both NGINX and NGINX Plus.
Modifying NGINX Configuration
NGINX configuration is captured in configuration files. When you change a configuration file and restart NGINX to pick up the new configuration, it implements a “graceful restart”. Both the old and new copies of NGINX run side by side for a short period of time. The old processes don’t accept any new connections and terminate once all their existing connections terminate.
One of the most common reasons to change NGINX configuration is to add and remove backend servers. As discussed earlier, if servers are deleted and replaced rather than updated, this type of change might happen often. Another use case is scaling: adding servers to handle a traffic spike and then removing them when the traffic has died down.
It may not always be convenient to modify configuration files and restart NGINX or NGINX Plus. For example, if you are experiencing large amounts of traffic and high load, restarting NGINX and reloading the configuration at that point further increases load on the system and can temporarily degrade performance.
For the common case of adding and removing backend servers, we have added an HTTP‑based API as well as DNS‑based reconfiguration in NGINX Plus to perform these operations without restarting NGINX Plus.
DNS-Based Reconfiguration with NGINX Plus
With NGINX Plus, your DNS server can keep the definitive list of the servers that belong to an upstream group. Update the list of available servers in DNS and NGINX Plus automatically picks up the changes.
To use DNS‑based reconfiguration, create multiple A records with the same hostname but different IP addresses, as in this example from Digital Ocean. The result is similar to what happens with Round Robin DNS, except that in this case NGINX Plus doesn’t just use the first address in the list from DNS as browser clients tend to do. Instead it applies the configured load‑balancing algorithm to all the servers in the list, which can be Least Connections, Round Robin, or any of its other load‑balancing algorithms. NGINX Plus is effectively using the DNS server as a database for upstream servers.
The NGINX Plus configuration looks like this:
http {
upstream backend {
zone backend 64k;
server backend.example.com resolve;
}
resolver 192.168.1.50;
}Putting resolve after the hostname of the server instructs NGINX to refresh the list of IP addresses for that hostname periodically. (NGINX Plus’ default behavior is to fetch DNS information only once, as it initializes.)
The resolver directive specifies the DNS server to use. By default, NGINX Plus refreshes a DNS entry according to the time‑to‑live (TTL) that DNS sets for it. To override the default, include the valid parameter to the resolver directive:
resolver 192.168.1.50 valid=30s;In this example, NGINX Plus caches DNS entries for 30 seconds before refreshing them.
Dynamic Configuration of Upstream Groups Using the NGINX Plus API
[Editor – This section has been updated to use the NGINX Plus API, which replaces and deprecates the separate module for dynamic configuration that was originally discussed here.]
The NGINX Plus API is an HTTP‑based interface for adding, removing, and modifying the servers in an upstream (backend) group dynamically and without having to reload the configuration. This is useful for autoscaling and when you want to take a server down temporarily for maintenance. It’s especially great for transient changes, as changes made using this interface are not preserved by default when you restart NGINX Plus or reload its configuration.
Configuring the NGINX Plus API
To enable the NGINX Plus API, create a separate location block in the configuration block for a virtual server and include the api directive. We’re using the conventional name for this location, also /api. We want to restrict use of the interface to administrators on the local LAN, so we also include allow and deny directives:
server {
listen 8080; # Listen on a local port, which can be protected by a firewall
location /api {
api write=on;
allow 10.0.0.0/8; # Allow access only from LAN
deny all; # Deny everyone else
}
}The API requires a shared memory zone to store information about an upstream group of servers, so we include the zone directive in the configuration, as in this example for an upstream group called backend:
upstream backend {
zone backend 64k;
server 10.2.2.90:8000;
server 10.2.2.91:8000;
server 10.2.2.92:8000;
}Using the NGINX Plus API
Let’s say we want to remove a faulty server from the backend upstream group. First we can send a list request with any HTTP tool to see the IDs assigned to the servers. Here we use curl, and pipe the output to the jq utility to filter for just the server name or address, and ID, of each server:
# curl -s http://localhost:8080/api/version/http/upstreams/backend/servers | jq -c '.peers[] | {server, id}'
{"server":"10.2.2.90:8000","id":0}
{"server":"10.2.2.91:8000","id":1}
{"server":"10.2.2.92:8000","id":2}Once we have the IDs, we can specify the one we want to remove by its ID:
# curl -X DELETE localhost:8080/api/version/http/upstreams/backend/servers/2Let’s now add more servers to the upstream group. We can do this easily with an add request:
# curl -X POST -d '{"server":"10.2.2.93:8000"}' http://localhost:8080/api/version/http/upstreams/backend/serversThis is just a sample of what you can do with the NGINX Plus API. You can also put servers into draining mode, alter their weights, and reconfigure them in a variety of other ways.
Conclusion
With NGINX, you can make configuration changes with no downtime and no interruption to traffic processing. NGINX Plus adds additional methods for modifying configuration dynamically. By using DNS as the definitive repository for upstream servers, you can make changes without touching NGINX Plus configuration files. An HTTP‑based API provides a familiar interface to make transient changes to the NGINX Plus configuration, again without having to touch any configuration files.
[Editor – In NGINX Plus R8 and later, you can have changes to upstream groups made with the NGINX Plus API persist across reloads of the NGINX Plus configuration; see the NGINX Plus Admin Guide.]
To try NGINX Plus, start your free 30‑day trial today or contact us to discuss your use cases.
