Web Server Load Balancing with NGINX Plus

This post is part of our series about container technology:

Containers are everywhere in the modern application landscape. Developers are using them in multiple ways – building them, pushing them to registries, and generally making applications work within containers.

In this blog I explore the world of container images – specifically, how to make your container images smaller and why that’s important. Along the way, I show the code and commands for building a very small container image that can be used for testing.

What Are Container Images?

The best definition that I’ve seen of a container image is this one:

Images can exist without containers, but a container needs to run an image to exist.

This is somewhat circular, but still accurate. A container image is a computing object that contains application code and is “run” by a container runtime (such as Docker, rkt, and podman). Kubernetes is the most popular container orchestration system, but the three other tools mentioned can be used for orchestration if you’re doing local development.

The image defines how the application is deployed – for example, which ports to expose, the application runtime startup (or entry point), and so on.

Why Is a Smaller Image Better?

A smaller container image has three main advantages:

  • Reduced build time for the application. The build time includes the container build, but also the time to push the container into a registry.
  • A much smaller memory footprint. The smaller the image, the less memory you end up using. This may not be a concern when operating in a public cloud provider, but definitely is when working on a laptop for development purposes.
  • A much smaller attack surface and fewer dependencies (especially if the container doesn’t use a base image). This makes for a reduced security surface and a smaller footprint in terms of extraneous libraries, dependencies, and other things inside your image.

A smaller container image typically has fewer components inside it. This means there is a reduced amount of non‑application code inside the image. The largest amount of “non application” code that you usually find inside a container image consists of shared libraries. Shared libraries are pieces of software that implement functionality you know is used (or likely to be used) by multiple applications. Using a shared library means the same functionality doesn’t have to be coded over and over in each new application. Under normal circumstances, shared libraries are a good thing because they externalize shared code and so enable application binaries to be smaller.

When running a single application in a container, shared libraries aren’t needed. After all, there’s no other application to share the code with!

Shared libraries take up space, and their life cycles need to be managed independently as part of the build process. Shared libraries are usually shipped with an operating system, and maintained quite separately from the applications that use them.

When operating within a container, there is no need to have this usual separation. Not separating out shared libraries means that, as a developer, you include only the components that your application needs to run – and nothing more. In a container context, this means no shared libraries.

Tools for Reducing Image Size

The traditional approach to building container images is to include and use a prebuilt operating system. If we use one of the more commonly used base images, we see something like this listing for Ubuntu (here divided across multiple lines for readability):

# podman images
REPOSITORY                                     TAG     IMAGE ID     ...                       latest  9873176a8ff5 ...                       latest  055b2e5ebc94 ...  latest  af1f279fed20 ...
     ... CREATED       SIZE
     ... 3 weeks ago   75.1 MB
     ... 7 weeks ago   184 MB
     ... 6 months ago  351 MB

As you can see, there’s quite a wide variation in image size: from just 75 MB for Ubuntu to a whopping 351 MB for the complete fedora-toolbox image. Each time one of these images is run, it takes time to start and load it, not counting the build time if you recompile an application and push the image into a registry.

There are two common choices for reducing image size: Alpine Linux and the Red Hat Universal Base Image (UBI).

Alpine Linux is based on the musl implementation of the C standard library (libc) and on BusyBox, a minimal kernel that nonetheless includes a plethora of tools. The use of musl libc means that you must recompile the application code to work with Alpine Linux, which is problematic if you don’t have access to the application’s source code.

UBI from Red Hat is another angle here. UBI is a standardized set of container images that have a set of runtimes for developers to use.

Using Alpine Linux usually results in a much smaller image, just 5.87 MB in this listing:

# podman images
REPOSITORY                                    TAG     IMAGE ID     ...           latest  8215cb84fa58 ...   latest  3f32499d4f3a ...                      latest  d4ff818577bc ...
     ... CREATED      SIZE
     ... 2 weeks ago  234 MB
     ... 2 weeks ago  105 MB
     ... 3 weeks ago  5.87 MB

Alpine and UBI address the same question (how do I build a smaller image?) but approach it from different starting points. Alpine starts with a very small codebase, and only adds the tools that are required. UBI, on the other hand, starts with a larger operating system and strips it down to bare essentials.

Building Using a Minimal Image

Before you can build a minimal image, you of course need an application. For the purposes of this blog, I’ve written the following very simple application in C. (Actually, I first wrote this app for a colleague who wanted to spin up thousands of containers for Istio testing and so needed a tiny container image.)

Perhaps the most notable thing about it is that it doesn’t actually do anything! It just calls the pause() function and waits for a signal.

# more pausle.c
#include <unistd.h>
int main(void) {return pause(); }

This may seem like an odd application to use as an example, but it’s good for illustrating my point about image size. Being so small, the application has very little effect on the container size.

Under normal circumstances, I run this gcc command to compile the app with several optimizations:

# gcc -Os -fdata-sections -ffunction-sections -fipa-pta -W1,--gc-sections -W1,-O1 -W1,--as-needed -W1,--strip-all pausle.c -o pausle-dynamic

The result is a very small binary (just 15 KB):

# ls -lh pausle-dynamic
-rwxr-xr-x. 1 root root 15K Jul 22 22:00 pausle-dynamic

This application is dynamically linked. That is, it requires shared libraries on the operating system in order for it to run. I can check this by running the ldd command.

# ldd pausle-dynamic (0x00007fffafbe3000) => /lib64/ (0x00007fb193983000)
        /lib64/ (0x00007fb193b5d000)

In order to build this into a container, I need to use a base image that contains the shared libraries. I’m using podman, and Dockerfile for builds as it’s somewhat ubiquitous.

# more Dockerfile

ADD pausle-dynamic /
CMD ["/pausle-dynamic"]

I use the RedHat UBI minimal image (in STEP 1) and add my precompiled application with the command to run it (in STEP 3) when the container starts.

# podman build --tag=pausle-dynamic .
STEP 2: ADD pausle-dynamic /
--> 344589591c7
STEP 3: CMD ["/pausle-dynamic"]
STEP 4: COMMIT pausle-dynamic
--> 1f72538cf84

The podman images command shows that the new image is no bigger (105 MB) than the ubi-minimal base image, because my application only adds 15 KB.

# podman images
REPOSITORY                                   TAG     IMAGE ID     ...
localhost/pausle-dynamic                     latest  1f72538cf84c ...  latest  3f32499d4f3a ...                     latest  d4ff818577bc ...
     ... CREATED             SIZE
     ... About a minute ago  105 MB
     ... 4 weeks ago         105 MB
     ... 5 weeks ago         5.87 MB

When I run this image, I can see that it is available and running.

# podman run -d pausle-dynamic
# podman ps -a
CONTAINER ID  IMAGE                              COMMAND       ...
5905d1ae4dc0  localhost/pausle-dynamic:latest  /pausle-dynamic ...
     ... CREATED        STATUS
     ... 4 seconds ago  Up 4 seconds ago

I can execute a shell inside the new container and look at my binary:

# podman exec -it 5905d1ae4dc0 /bin/bash
# ldd pausle-dynamic (0x00007ffd89762000) => /lib64/ (0x00007fa65a658000)
        /lib64/ (0x00007fa65aa1d000)

The application is the same one that I compiled and built into the container. Because it’s dynamically linked, it requires shared libraries in the operating system to run. This means that my application needs a base image containing these libraries in order to run. This increases the size of my container, so it’s still not as minimal as I want.

Building Without an Image

I can do two things to make this container image a lot smaller. I can statically link my code, meaning that my application “bundles in” the shared libraries into the binary, for want of a better description.

I run this command to statically link:

# gcc -Os -s static -ffunction-sections -fipa-pta -W1,--gc-sections pausle.c -o pausle-dynamic
# strip pausle-static
# ls-lh pausle-static
-rwxr-xr-x. 1 root root 697K Jul 22 22:31 pausle-static

The ls command shows that the resulting application binary is 697 KB – considerably larger than the dynamically linked application – because the libraries are bundled into the application.

Now when I run the ldd command to display the shared libraries, I get a message telling me that the executable is not dynamic.

# ldd pausle-static
        not a dynamic executable

In the Dockerfile, I’m using the special no‑op keyword scratch to indicate I’m not using a base image.

# more Dockerfile
FROM scratch

ADD pausle-static /
CMD ["/pausle-static"]

I now build an image in the same way as before by running the podman build command:

# podman build --tag=pausle-static .
STEP 1: FROM scratch
STEP 2: ADD pausle-static /
--> 7fb16e85314
STEP 3: CMD ["/pausle-static"]
STEP 4: COMMIT pausle-static
--> f7f7c833975

The container builds in exactly the same way as the dynamic executable, but the resulting image is much smaller than the dynamic image – only 716 KB, which is barely bigger than the statically compiled binary itself (697 KB).

# podman images
REPOSITORY                                     TAG     IMAGE ID     ...
localhost/pausle-static                        latest  f7f7c8339754 ...
localhost/pausle-dynamic                       latest  1f72538cf84c ...
     ... CREATED         SIZE
     ... 2 minutes ago   716 kB
     ... 20 minutes ago  105 MB

I initialize the container and confirm it’s running:

# podman run -d pausle-static
# podman ps -a
CONTAINER ID   IMAGE                           COMMAND        ...
1748d59be199  localhost/pausle-static:latest  /pausle-static ...
     ... CREATED        STATUS
     ... 5 seconds ago  Up 6 seconds ago

If I try to exec a shell, or run the ls command inside the container, I get error messages indicating that there are no other applications in the container. This is because the container doesn’t include an operating system or base image.

# podman exec -it 1748d59be199 /bin/bash
Error: executable file `/bin/bash` not found in $PATH: No such file or directory: OCI not found
# podman exec -it 1748d59be199 /bin/ls
Error: executable file `/bin/ls` not found in $PATH: No such file or directory: OCI not found


Building small container images is useful in all sorts of scenarios – development and testing, for example. It significantly cuts down the build time of new images, including the time that it takes to push an image to a remote registry.

As mentioned before, smaller container images (especially if they do not use a base image) also have a much smaller attack surface and fewer dependencies, reducing their footprint in terms of extraneous libraries, dependencies, and other things inside your image. Mostly though, creating small images carries a sense of neatness and symmetry, making it an uber cool thing to do!

Hero image
Managing Kubernetes Traffic with F5 NGINX: A Practical Guide

Learn how to manage Kubernetes traffic with F5 NGINX Ingress Controller and F5 NGINX Service Mesh and solve the complex challenges of running Kubernetes in production.

About The Author

Scott van Kalken

Solution Architect

About F5 NGINX

F5, Inc. is the company behind NGINX, the popular open source project. We offer a suite of technologies for developing and delivering modern applications. Together with F5, our combined solution bridges the gap between NetOps and DevOps, with multi-cloud application services that span from code to customer.

Learn more at or join the conversation by following @nginx on Twitter.