Monitoring Microservices in Docker Containers with NGINX Amplify

Most microservices deployments use container technologies in one way or another. When deploying NGINX in containers, thousands of users choose Docker for its ease of use and rapidly growing community. The NGINX image is one the most frequently downloaded in the Docker Registry.

Recently we announced the beta version of NGINX Amplify, a new service for monitoring NGINX and recommending configuration improvements, and we have received a lot of questions regarding how to monitor NGINX and NGINX Plus using NGINX Amplify and Docker in the same deployment. In this post we discuss how these new free technologies can work together to improve the performance and manageability of your microservices deployment.

Editor – To learn more about NGINX Amplify and see a demo, watch our on‑demand webinar, Improve App Performance & Reliability with NGINX Amplify. A transcript of the first part of the webinar, an overview of NGINX Amplify features, is available on our blog.

Also check out the many other posts about NGINX Amplify on the blog.

How NGINX Amplify Components Work Together to Monitor NGINX

NGINX Amplify consists of multiple components. The ones that are visible to the user are:

  • The NGINX Amplify Agent, which runs on the same system as NGINX
  • The NGINX Amplify cloud receiver and backend
  • The NGINX Amplify web interface with dashboard

The NGINX Amplify agent performs the following operations:

  • Checks system processes to find all running NGINX instances
  • Monitors NGINX configuration files
  • Monitors NGINX logs
  • Collects metrics for NGINX from the stub_status module, and for NGINX Plus from the Status (live activity monitoring) module
  • Aggregates data
  • Sends data to the cloud backend

Monitoring NGINX in a Traditional Deployment

First let’s take a quick look at using NGINX Amplify to monitor NGINX in a traditional, noncontainerized environment.

There are two methods for using multiple instances of NGINX. One method is to run a single NGINX server per machine. In this case the NGINX Amplify dashboard lists every machine as a separate host:

In a traditional, non-containerized environment, the NGINX Amplify dashboard lists each machine as a separate host. This changes when configuring how to monitor NGINX in a microservices environment.

The other method involves running multiple NGINX instances on the same machine. In this case, you can have multiple NGINX configuration files and indicate which file to use for each instance with the -c option on the nginx command line:

# nginx -c /path/to/nginx.conf

Here is an example of multiple instances on the same host:

20385 ?    S  591:44 amplify-agent
11921 ? Ss 0:00 /lib/systemd/systemd --user
11922 ? S 0:00 \_ (sd-pam)
12204 ? Ss 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
12211 ? R 0:11 \_ nginx: worker process
12344 ? Ss 0:00 nginx: master process nginx -c /opt/conf/nginx2.conf
12345 ? S 0:00 \_ nginx: worker process
12366 ? Ss 0:00 nginx: master process nginx -c /opt/conf/nginx3.conf
12367 ? S 0:00 \_ nginx: worker process
12374 ? Ss 0:00 nginx: master process nginx -c /opt/conf/nginx4.conf
12375 ? S 0:00 \_ nginx: worker process

The NGINX Amplify Agent automatically detects the multiple NGINX instances and provides detailed statistics for each of them. To see per‑instance statistics on the dashboard, click the Show/Hide button on the line for NGINX in the host’s listing.

Click the Show button for a host on the NGINX Amplify dashboard to see separate statistics for each NGINX instance running on the host - how to monitor NGINX

Configuring NGINX Amplify to Monitor NGINX in a Docker Deployment

Now let’s take a look at how to monitor NGINX with NGINX Amplify in a Docker deployment.

Configuring the NGINX Amplify Agent

The NGINX Amplify Agent includes configuration for Docker deployments.

To try the NGINX Amplify agent in your Docker environment, follow this procedure:

  1. Get the latest source for the NGINX Amplify Agent for Docker:

    # git clone https://github.com/nginxinc/docker-nginx-amplify

    The repository includes samples you can adapt for your deployment, including a sample Dockerfile, a launch script, and a configuration file for the stub_status module.

    # cd docker-nginx-amplify
  2. Define your NGINX Amplify API key for Docker. To obtain the key, click the New System button on the NGINX Amplify dashboard.

    You define the API key for your Docker environment by setting the API_KEY variable in your choice of three places:

    • In the launch.sh script
    • With ENV API_KEY in the Dockerfile
    • As an environment variable set during container startup

    The environment variable takes precedence over the other methods.

  3. Edit the Dockerfile to fit your needs, for example adding your NGINX configuration files, mounting data volumes, and so on.

For more information on how to run NGINX in Docker, refer to the following guides:

Each time we run a new container in a Docker environment, it is assigned a hostname based on its MAC address. A unique name makes it possible to track metrics on a per‑container basis, which is useful, but it’s hard to recognize the service name or its purpose from this kind of name, as shown in this screenshot:

By default, the name for a new container displayed on the NGINX Amplify dashboard is a long alphanumeric string that Docker derives from the MAC address when configuring how to monitor NGINX with Amplify in a docker deployment

Also, if you often scale up or down this style of naming can quickly overflow your NGINX Amplify dashboard.

To achieve more visibility in your deployment you have a couple options. You can set a more readable hostname for your Docker container using the ‑h option on the docker command line. In the following example we start three containers using the same image (we have defined the API_KEY variable in the Dockerfile and so don’t include it on the command line):

# docker run -h myservice-01 -d -p 8097:80 nginx-with-amplify
# docker run -h myservice-02 -d -p 8098:80 nginx-with-amplify
# docker run -h myservice-03 -d -p 8099:80 nginx-with-amplify

You now see each container as a separate item in the NGINX Amplify dashboard:

You can assign a human-readable name to a Docker container, to make it easier to identify on the NGINX Amplify dashboard when configuring how to monitor NGINX with Amplify in a Docker deployment

As an alternative to defining the hostname as a container setting, you can set a special variable that only the NGINX Amplify Agent uses, AMPLIFY_HOSTNAME. Like API_KEY, you can set AMPLIFY_HOSTNAME in your choice of three places:

  • In the launch.sh script
  • With ENV AMPLIFY_HOSTNAME in the Dockerfile
  • As an environment variable set during container startup

The environment variable takes precedence over the other methods.

Here’s an example of a full command line that uses the environment variables:

# docker run --name myservice-01 -e API_KEY=ecfdee2e...effc7 AMPLIFY_HOSTNAME=myservice-01 -d nginx-with-amplify

Configuring Aggregate Mode for NGINX Amplify Agent in a Docker Container

Now this brings us to the question “What happens if I set the same hostname for multiple containers?”

In order to fully answer this question, let’s have a quick look at the internals of the NGINX Amplify Agent.

The Agent’s configuration file is located in /etc/amplify‑agent/agent.conf and it has an interesting variable that helps to uniquely identify the NGINX instance in NGINX Amplify: UUID. The Agent generates the UUID variable as it first starts up, which is usually the last step of its installation. If you start multiple containers from the same Docker image with the same agent.conf file, the same hostname, and the same API key, the Agent generates the same UUID for all of them. As a result, for each metric the Agent aggregates the values from the containers and reports them as a single number.

Aggregate mode is useful for monitoring microservices as it shows the total value of each metric for the multiple instances of NGINX related to the same microservice. For gauge metric types, the reported value is the average across multiple instances. Gauge metrics include CPU, memory, disk usage, etc. For counters, the reported value is the total sum of the counters across all instances of a microservice. Counters include requests per second, HTTP version (HTTP/1.1 and HTTP/2), error code counters, etc.

Summary

Please bear in mind that support for Docker with NGINX Amplify is still experimental, and quite a few things will be fixed and amended in the future. Please let us know of your experiences with monitoring NGINX inside Docker containers – we would really appreciate your feedback.

Try monitoring NGINX with NGINX Amplify in your own traditional or microservices environment:

Cover image
Microservices: From Design to Deployment
The complete guide to microservices development