This deployment guide explains how to create and configure the initial components for an all‑active, high‑availability deployment of NGINX Plus on Google Compute Engine (GCE), the Google Cloud Platform (GCP) product for running workloads on virtual machines (VMs). Multiple instances of NGINX Plus in active pairs load balance incoming connections across multiple application server environments.

Notes:

  • GCE is a highly dynamic environment where the names and arrangement of GUI elements (such as menu items, checkboxes, and configuration navigation) are subject to change. We have made every effort to accurately represent the GCE GUI at the time of publication, but options and locations might change over time. Use this guide as a reference and adapt to the current GCE working environment as necessary.
  • The configuration described in this guide allows anyone from any public IP address to access the NGINX Plus application instances directly without restriction. While this accommodates the most flexible testing scenarios in a test environment, we do not recommend it for production use. Before deploying the architecture in production, we strongly recommend that you disable HTTP and HTTPS access to the app-1 and app-2 instances over the GCE external IP addresses, or remove the external IP addresses for all application instances so that they’re accessible only on the internal GCE network.

Design and Topology

The deployment combines the following technologies:

  • NGINX Plus – Load balances HTTP connections across multiple instances of two different applications. Instructions are provided both for manual installation on a standard GCE VM image and for setting up the prebuilt NGINX Plus VM image available in the Google Marketplace.
  • PHP-FPM – Supports the two sample applications.
  • GCE network load balancer – Provides TCP connectivity between clients and the NGINX Plus load‑balancing (LB) instances in a GCP region, as well as maintaining session persistence for each NGINX Plus instance.
  • GCE instance groups – Provide a mechanism for managing groups of VM instances as a unit.
  • GCE health checks – Maintain high availability of the NGINX Plus LB instances by controlling when GCE creates a new instance in the instance group.

Topology of the all-active deployment of NGINX Plus as the Google Cloud Platform load balancer.

Session persistence is managed at the network layer by GCE network load balancer (based on client IP address) and at the application layer by the NGINX Plus LB instance (via a session cookie). When a new client connection enters the GCE network environment, GCE network load balancer assigns it to a specific frontend NGINX Plus LB instance, and the association persists as long as the LB instance is up and functional. The NGINX Plus LB instance forwards the request to a specific application instance in one of two groups of application instances, selected using its default Round Robin algorithm. It also issues a cookie to the client to represent the session with that application instance, so that subsequent requests from the client are forwarded to that application instance as long as it is up and running.

This deployment in this guide utilizes two groups of application instances – app-1 and app-2 – to demonstrate load balancing between different application types, but the application configurations are the same in both groups. The deployment can be very easily adapted to distribute unique connections to different groups of application instances by creating discrete upstream blocks and doing content routing based on URI. Please refer to the NGINX documentation for details on configuring multiple upstream server groups.

Prerequisites

This guide assumes that you:

  • Have a Google account (a separate GCP or GCE account is not needed).
  • Have enrolled in a free trial with available credit or have an established payment account with GCP.
  • Have a basic working knowledge of GCE and its GUI control panel:
    • Navigation
    • Creating instances
    • Managing IAM policies
  • Understand basic networking.
  • Have an NGINX Plus subscription. You can start a free 30‑day trial if you don’t already have a paid subscription.
  • Know how to install NGINX Plus, have a basic understanding of how it performs in load balancing and application delivery modes, and are familiar with its configuration syntax.
  • Are familiar with GitHub and know how to clone a repository.

All component names – projects, instances, templates, instance groups, and so on – are examples only. You can change them as suits your needs.

Task 1 – Creating a Project and Firewall Rules

Create a new GCE project to host the all‑active NGINX Plus deployment.

  1. Log into the GCP Console at console.cloud.google.com.

  2. The GCP Home > Dashboard tab opens. Its contents depend on whether you have any existing projects.

    • If there are no existing projects, click the  Create a project  button.

      Screenshot of the Google Cloud Platform dashboard that appears when there are no existing projects (creating a project is the first step in configuring NGINX Plus as the Google Cloud load balancer)

    • If there are existing projects, the name of one of them appears in the upper left of the blue header bar (in the screenshot, it’s  My Test Project ). Click the project name and select Create project from the menu that opens.

      Screenshot of the Google Cloud Platform page that appears when other projects already exist (creating a project is the first step in configuring NGINX Plus as the Google Cloud load balancer)

  3. Type your project name in the New Project window that pops up, then click CREATE. We’re naming the project NGINX Plus All-Active-LB.

    Screenshot of the New Project pop-up window for naming a new project on the Google Cloud Platform, which is the first step in configuring NGINX Plus as the Google load balancer

Next create firewall rules that allow access to the HTTP and HTTPS ports on your GCE instances. You’ll attach the rules to all the instances you create for the deployment.

  1. Navigate to the Networking > Firewall rules tab and click  +  CREATE FIREWALL RULE. (The screenshot shows the default rules provided by GCE.)

    Screenshot of the Google Cloud Platform page for defining new firewall rules; when configuring NGINX Plus as the Google Cloud load balancer, we open ports 80, 443, and 8080 for it.

  2. Fill in the fields on the Create a firewall rule screen that opens:

    • Name – nginx-plus-http-fw-rule
    • Description – Allow access to ports 80, 8080, and 443 on all NGINX Plus instances.
    • Source filter – On the drop‑down menu, select either Allow from any source (0.0.0.0/0) or IP range if you want to restrict access to users on your private network. In the second case, fill in the Source IP ranges field that opens. We are allowing unrestricted access.
    • Allowed protocols and ports – tcp:80; tcp:8080; tcp:443

      Note: As mentioned in the introduction to this guide, opening these ports for your application instances is appropriate only in a test environment. We strongly recommend that before deploying the architecture in production you create a new firewall rule for your application instances that blocks all port access to the external IP address, or disable external IP addresses for the instances to make them accessible only on the internal GCE network.

    • Target tags – nginx-plus-http-fw-rule

     

    Screenshot of the interface for creating a Google Compute Engine (GCE) firewall rule, used during deployment of NGINX Plus as the Google load balancer.

  3. Click the  Create  button. The new rule is added to the table on the Firewall rules tab.

Task 2 – Creating Source Instances

Create three GCE source instances that will serve as templates for the instance groups you will create later on: one instance for the NGINX Plus load balancer and two instances for NGINX Plus PHP application servers.

You can create source instances in either of two ways:

The instructions for the two methods are significantly different, but after you create the source instances all subsequent instructions are the same.

Creating Source Instances from Ubuntu VM Images

Create three source VM instances based on a GCE VM image. We’re basing our instances on the Ubuntu 16.04 LTS image.

  1. Verify that the NGINX Plus All-Active-LB project is still selected in the Google Cloud Platform header bar.

  2. Navigate to the Compute Engine > VM instances tab.

  3. Click the  Create instance  button. The Create an instance page opens.

Creating the First Application Server Instance

  1. On the Create an instance page, modify or verify the fields and checkboxes as indicated (a screenshot of the completed page appears in Step 5 below):

    • Name – nginx-plus-app-1
    • Zone –The GCP zone that makes sense for your location. We’re using us-west1-a.
    • Machine type – The appropriate size for the level of traffic you anticipate. We’re selecting micro, which is ideal for testing purposes.
    • Boot disk – Click Change. The Boot disk page opens to the OS images subtab. Perform the following steps:

      1. Click the Ubuntu 16.04 LTS radio button (or the Unix or Linux image of your choice).
      2. Accept the default values in the Boot disk type and Size (GB) fields (Standard persistent disk and 10 respectively).
      3. Click the  Select  button.

       

      Screenshot of the 'Boot disk' page in Google Cloud Platform for selecting the OS on which a VM runs. In the deployment of NGINX Plus as the Google load balancer, we select Ubuntu 16.04 LTS.

    • Identity and API access – Unless you want more granular control over access, keep the defaults for the Service account field (Compute Engine default service account) and Access scopes radio button (Allow default access).
    • Firewall – Verify that neither check box is checked (the default). The firewall rule invoked in the Tags field on the Management subtab (see Step 6 below) controls this type of access.
  2.  

  3. Click Management, disk, networking, SSH keys to open that set of subtabs. (The screenshot shows the values entered in Step 4.)

    Screen shot of the 'Create an instance' page for an application server in the deployment of NGINX Plus as the Google Cloud Platform load balancer.

  4. On the Management subtab, modify or verify the fields as indicated:

    • Description – NGINX Plus app-1 Image
    • Tags – nginx-plus-http-fw-rule
    • Preemptibility – Off (recommended) (the default)
    • Automatic restart – On (recommended) (the default)
    • On host maintenance – Migrate VM instance (recommended) (the default)

     

    Screenshot of the Management subtab used during creation of a new VM instance, part of deploying NGINX Plus as the Google load balancer.

  5. On the Disks subtab, uncheck the checkbox labeled Delete boot disk when instance is deleted.

    Screenshot of the Disks subtab used during creation of a new VM instance, part of deploying NGINX Plus as the Google Cloud load balancer.

  6. On the Networking subtab, verify the default settings, in particular Ephemeral for External IP and Off for IP Forwarding.

    Screenshot of the Networking subtab used during creation of a new VM instance, part of deploying NGINX Plus as the Google Cloud load balancer.

  7. If you are using your own SSH public key instead of the default keys associated with your GCE identity, on the SSH Keys subtab paste the hexadecimal key string into the box that reads Enter entire key data.

    Screenshot of the SSH Keys subtab used during creation of a new VM instance, part of deploying NGINX Plus as the Google Cloud Platform load balancer.

  8. Click the  Create  button at the bottom of the Create an instance page.

    The VM instances summary page opens. It can take several minutes for the instance to be created. Wait to continue until the green check mark appears.

    Screenshot of the summary page that verifies the creation of a new VM instance, part of deploying NGINX Plus as the load balancer for Google Cloud.

Creating the Second Application Server Instance

  1. On the VM instances summary page, click CREATE INSTANCE.

  2. Repeat Steps 4 through 10 to create the second application server instance, specifying the same values as for the first application server instance, except as noted:

    • In Step 4, Name – nginx-plus-app-2
    • In Step 6, Description – NGINX Plus app-2 Image

Creating the Load Balancing Instance

  1. On the VM instances summary page, click CREATE INSTANCE.

  2. Repeat Steps 4 through 10 to create the load balancing instance, specifying the same values as for the first application server instance, except as noted:

    • In Step 4, Name – nginx-plus-lb
    • In Step 6, Description – NGINX Plus Load Balancing Image

Configuring PHP and FastCGI on the Instances

Install and configure PHP and FastCGI on the instances.

Repeat these instructions for all three source instances (nginx-plus-app-1, nginx-plus-app-2, and nginx-plus-lb).

  1. Connect to the instance over SSH using the method of your choice. GCE provides a built‑in mechanism:

    1. Navigate to the Compute Engine > VM instances tab.
    2. In the instance’s row in the table, click the triangle icon in the Connect column at the far right and select a method (for example, Open in browser window).

     

    Screenshot showing how to connect via SSH to a VM instance, part of deploying NGINX Plus as the Google load balancer.

  2. Working in the SSH terminal, install PHP 7 (the default PHP version for Ubuntu 16.04 LTS) and FastCGI.

    $ sudo apt-get install php7.0-fpm
  3. Edit the PHP 7 configuration using sudo so that it binds to a local network port instead of a Unix socket. Using your preferred text editor, remove the following line from /etc/php/7.0/fpm/pool.d:

    listen = /run/php/php7.0-fpm.sock

    and replace it with these two lines:

    listen = 127.0.0.1:9000
    listen.allowed_clients = 127.0.0.1
  4. Restart PHP:

    $ sudo service php7.0-fpm restart
  5. Leave the SSH connection open for reuse in the next section.

Installing and Configuring NGINX Plus on the Instances

Now install NGINX Plus and download files that are specific to the all‑active deployment:

  • An NGINX Plus configuration file customized for the function the instance performs (application server or load balancer)
  • A set of content files (HTML, images, and so on) served by the application servers in the deployment

Both the configuration and content files are available at the NGINX, Inc. GitHub repository.

Repeat these instructions for all three source instances (nginx-plus-app-1, nginx-plus-app-2, and nginx-plus-lb).

  1. Install NGINX Plus. For instructions, see the NGINX Plus Admin Guide.

  2. Clone the GitHub repository for the all-active load balancing deployment. (Instructions for downloading the files directly from the GitHub repository are provided below, in case you prefer not to clone it.)

  3. Copy the contents of the usr_share_nginx subdirectory from the cloned repository to the local /usr/share/nginx directory, creating the local directory if necessary. (If you choose not to clone the repository, you need to download each file from the GitHub repository individually.)

  4. Copy the appropriate configuration file from the etc_nginx_conf.d subdirectory of the cloned repository to /etc/nginx/conf.d:

    • On nginx-plus-app-1 and nginx-plus-app-2, copy gce-all-active-app.conf.

      You can also run the following commands to download the configuration file directly from the GitHub repository:

      $ cd /etc/nginx/conf.d/
      $ sudo curl -o gce-all-active-app.conf https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/gce-all-active-app.conf

      or

      $ cd /etc/nginx/conf.d/
      $ sudo wget https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/gce-all-active-app.conf
    • On nginx-plus-lb, copy gce-all-active-lb.conf.

      You can also run the following commands to download the configuration file directly from the GitHub repository:

      $ cd /etc/nginx/conf.d/
      $ sudo curl -o gce-all-active-lb.conf https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/gce-all-active-lb.conf

      or

      $ cd /etc/nginx/conf.d/
      $ sudo wget https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/gce-all-active-lb.conf
  5. On the LB instance (nginx-plus-lb), use your preferred text editor to open gce-all-active-lb.conf and change the server directives in the upstream block to reference the internal IP addresses of the nginx-plus-app-1 and nginx-plus-app-2 instances. (No action is required on the two application server instances themselves.)

    You can look up internal IP addresses in the Internal IP column of the table on the Compute Engine > VM instances summary page.

    upstream upstream_app_pool {
    server Internal-IP-address-of-nginx-plus-app-1;
    server Internal-IP-address-of-nginx-plus-app-2;
    zone upstream-apps 64k;
    sticky cookie GCPPersist expires=300;
    }
  6. To avoid potential configuration conflicts, disable the default.conf file distributed with NGINX Plus. The configuration files provided for the all‑active deployment include equivalent instructions plus additional function‑specific directives.

    $ sudo mv default.conf default.conf.orig
  7. Enable the NGINX Plus live activity monitoring dashboard for the instance by copying status.html from the etc_nginx_conf.d subdirectory of the cloned repository to /etc/nginx/conf.d.

    You can also run the following commands to download the configuration file directly from the GitHub repository:

    $ cd /etc/nginx/conf.d/
    $ sudo curl -o status.conf https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/status.conf

    or

    $ cd /etc/nginx/conf.d/
    $ sudo wget https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/status.conf
  8. Validate the NGINX Plus configuration and restart NGINX Plus:

    $ sudo nginx -t
    $ sudo nginx -s reload
  9. Verify the instance is working by accessing it at its external IP address. (As previously noted, we recommend blocking access to the external IP addresses of the application instances in a production environment.) The external IP address for the instance appears on the Compute Engine > VM instances summary page, in the External IP column of the table.

    • Access the index.html page either in a browser or by running this curl command.

      $ curl http://external-IP-address
    • Access its NGINX Plus live activity monitoring dashboard in a browser, at:

      https//:external-IP-address:8080/status.html

  10. Proceed to Task 3 – Creating “Gold” Images.

Creating Source Instances from Prebuilt NGINX Plus Images

Create three source instances based on a prebuilt NGINX Plus image running on Ubuntu 14.04 LTS, available in the Google Marketplace. Google requires that you use the Cloud Launcher utility to provision the first instance. Then you can clone the additional two instances from the first one.

Creating the First Application Server Instance

  1. Verify that the NGINX Plus All-Active-LB project is still selected in the Google Cloud Platform header bar.

  2. Navigate to the Cloud Launcher tab and search for nginx plus.

  3. Click the NGINX Plus box in the results area.

    Screenshot of the Cloud Launcher home page on the Google Cloud Platform, which can be used to create a prebuilt NGINX Plus VM instance when deploying NGINX Plus as the load balancer for Google Cloud.

  4. On the NGINX Plus page that opens, click the  Launch on Compute Engine  button.

  5. Fill in the fields on the New NGINX Plus deployment page as indicated.

    • Deployment name – nginx-plus-app-1
    • Zone – The GCP zone that makes sense for your location. We’re using us-west1-a.
    • Machine type – The appropriate size for the level of traffic you anticipate. We’re selecting f1-micro, which is ideal for testing purposes.
    • Disk type – Standard Persistent Disk (the default)
    • Disk size in GB – 10 (the default and minimum allowed)
    • Network name – default
    • Subnetwork name – default
    • Firewall – Verify that the Allow HTTP traffic checkbox is checked.

     

    Screenshot of the Cloud Launcher page for creating a prebuilt NGINX Plus VM instance when deploying NGINX Plus as the Google Cloud Platform load balancer.

  6. Click the  Deploy  button.

    It can take several minutes for the instance to deploy. Wait to continue until the green check mark appears, confirming that your instance has been deployed.

    Screenshot of the Cloud Launcher page that confirms the creation of a prebuilt NGINX Plus VM instance when deploying NGINX Plus as the Google load balancer.

  7. Navigate to the Compute Engine > VM instances tab and click on nginx-plus-app-1-vm in the Name column in the table. (Cloud Launcher adds the suffix -vm automatically as it creates an instance.)

    Screenshot showing how to access the page where configuration details for a VM instance can be modified during deployment of NGINX Plus as the Google Cloud load balancer.

  8. On the VM instances page that opens, click EDIT at the top of the page. In fields that can be edited, the value changes from static text to text boxes, drop‑down menus, and checkboxes.

  9. Modify or verify the indicated editable fields (non‑editable fields are not listed):

    • Tags – If a default tag appears in the field (for example, nginx-plus-app-1-tcp-80), click on the X after its name to remove it, and type in nginx-plus-http-fw-rule.
    • External IP – Ephemeral (the default)
    • Boot disk and local disks – Uncheck the checkbox labeled Delete boot disk when when instance is deleted.
    • Additional disks – No changes
    • Network – We recommend keeping the default settings, but if you need to change them (if, for example, you’re configuring a production environment), click default and then EDIT on the Network details page that opens. After making your changes click the  Save  button.
    • Firewall – Verify that neither check box is checked (the default). The firewall rule invoked in the Tags field on the Management subtab (see Step 6 below) controls this type of access.
    • Automatic restart – On (recommended) (the default)
    • On host maintenance – Migrate VM instance (recommended) (the default)
    • Custom metadata – No changes
    • SSH Keys – If you are using your own SSH public key instead of the default keys associated with your GCE identity, paste the hexadecimal key string into the box that reads Enter entire key data.
    • Serial port – Verify that the check box labeled Enable connecting to serial ports is not checked (the default).

     

    The screenshot shows the results of your changes, omitting some fields than cannot be edited or for which we recommend retaining the defaults.

    Screenshot showing the configuration modifications for a VM instance being deployed as part of setting up NGINX Plus as the Google load balancer.

  10. Click the  Save  button.

Creating the Second Application Server Instance

Create the second application server instance by cloning the first one.

  1. Navigate back to the summary page on the Compute Engine > VM instances tab (click the arrow that is circled in the following figure).

    Screenshot showing how to return to the VM instance summary page during deployment of NGINX Plus as the Google Cloud Platform load balancer.

  2. Click nginx-plus-app-1-vm in the Name column of the table (shown in the screenshot in Step 7).

  3. On the VM instances page that opens, click CLONE at the top of the page.

  4. On the Create an instance page that opens, modify or verify the fields and checkboxes as indicated:

    • Name – nginx-plus-app-2-vm (add the -vm suffix to make the name consistent with the first instance; GCE does not add it automatically when you clone an instance)
    • Zone – The GCP zone that makes sense for your location. We’re using us-west1-a.
    • Machine type – The appropriate size for the level of traffic you anticipate. We’re selecting f1-micro, which is ideal for testing purposes.
    • Boot disk type – New 10 GB standard persistent disk (the value inherited from nginx-plus-app-1-vm)
    • Identity and API access – Set the Access scopes radio button to Allow default access and accept the default values in all other fields. If you want more granular control over access than is provided by these settings, modify the fields in this section as appropriate.
    • Firewall – Verify that neither check box is checked (the default).
  5.  

  6. Click Management, disk, networking, SSH keys to open that set of subtabs.

  7. Verify the following settings on the subtabs, modifying them as necessary:

    • Management – In the Tags field: nginx-plus-http-fw-rule
    • Disks – The Deletion rule checkbox (labeled Delete boot disk when instance is deleted) is not checked
  8.  

  9. Click the  Create  button.

Creating the Load Balancing Instance

Create the source load balancing instance by cloning the first instance again.

  1. Repeat Steps 12 through 17. In Step 14, specify nginx-plus-lb-vm as the name.

Configuring PHP and FastCGI on the Instances

Install and configure PHP and FastCGI on the instances.

Repeat these instructions for all three source instances (nginx-plus-app-1-vm, nginx-plus-app-2-vm, and nginx-plus-lb-vm).

  1. Connect to the instance over SSH using the method of your choice. GCE provides a built‑in mechanism:

    1. Navigate to the Compute Engine > VM instances tab.
    2. In the row for the instance in the table, click the triangle icon in the Connect column at the far right and select a method (for example, Open in browser window).

     

    The screenshot shows instances based on the prebuilt NGINX Plus images.

    Screenshot showing how to connect via SSH to a VM instance, part of deploying NGINX Plus as the Google load balancer.

  2. Working in the SSH terminal, install PHP 5 (the default PHP version for Ubuntu 14.04 LTS) and FastCGI.

    $ sudo apt-get install php5-fpm
  3. Edit the PHP 5 configuration using sudo so that it binds to a local network port instead of a Unix socket. Using your preferred text editor, remove the following line from /etc/php5/fpm/pool.d:

    Listen = /run/php/php5-fpm.sock

    and replace it with these two lines:

    Listen = 127.0.0.1:9000
    Listen.allowed_clients = 127.0.0.1
  4. Restart PHP:

    $ sudo service php5-fpm restart
  5. Leave the SSH connection open for reuse in the next section.

Configuring NGINX Plus on the Instances

Now download files that are specific to the all‑active deployment:

  • An NGINX Plus configuration file customized for the function the instance performs (application server or load balancer)
  • A set of content files (HTML, images, and so on) served by the application servers in the deployment

Both the configuration and content files are available at the NGINX, Inc. GitHub repository.

Repeat these instructions for all three source instances (nginx-plus-app-1-vm, nginx-plus-app-2-vm, and nginx-plus-lb-vm).

  1. Clone the GitHub repository for the all-active load balancing deployment. (Instructions for downloading the files directly from the GitHub repository are provided below, in case you prefer not to clone it.)

  2. Copy the contents of the usr_share_nginx subdirectory from the cloned repository to the local /usr/share/nginx directory, creating the local directory if necessary. (If you choose not to clone the repository, you need to download each file from the GitHub repository individually.)

  3. Copy the appropriate configuration file from the etc_nginx_conf.d subdirectory of the cloned repository to /etc/nginx/conf.d:

    • On nginx-plus-app-1-vm and nginx-plus-app-2-vm, copy gce-all-active-app.conf.

      You can also run the following commands to download the configuration file directly from the GitHub repository:

      $ cd /etc/nginx/conf.d/
      $ sudo curl -o gce-all-active-app.conf https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/gce-all-active-app.conf

      or

      $ cd /etc/nginx/conf.d/
      $ sudo wget https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/gce-all-active-app.conf
    • On nginx-plus-lb-vm, copy gce-all-active-lb.conf.

      You can also run the following commands to download the configuration file directly from the GitHub repository:

      $ cd /etc/nginx/conf.d/
      $ sudo curl -o gce-all-active-lb.conf https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/gce-all-active-lb.conf

      or

      $ cd /etc/nginx/conf.d/
      $ sudo wget https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/gce-all-active-lb.conf
  4. On the LB instance (nginx-plus-lb-vm), use your preferred text editor to open gce-all-active-lb.conf and change the server directives in the upstream block to reference the internal IP addresses of the nginx-plus-app-1-vm and nginx-plus-app-2-vm instances. (No action is required on the two application server instances themselves.)

    You can look up internal IP addresses in the Internal IP column of the table on the Compute Engine > VM instances summary page.

    upstream upstream_app_pool {
    server Internal-IP-address-of-nginx-plus-app-1-vm;
    server Internal-IP-address-of-nginx-plus-app-2-vm;
    zone upstream-apps 64k;
    sticky cookie GCPPersist expires=300;
    }
  5. To avoid potential configuration conflicts, disable the default.conf file distributed with NGINX Plus. The configuration files provided for the all‑active deployment include equivalent instructions plus additional function‑specific directives.

    $ sudo mv default.conf default.conf.orig
  6. Enable the NGINX Plus live activity monitoring dashboard for the instance by copying status.html from the etc_nginx_conf.d subdirectory of the cloned repository to /etc/nginx/conf.d.

    You can also run the following commands to download the configuration file directly from the GitHub repository:

    $ cd /etc/nginx/conf.d/
    $ sudo curl -o status.conf https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/status.conf

    or

    $ cd /etc/nginx/conf.d/
    $ sudo wget https://github.com/nginxinc/NGINX-Demos/blob/master/gce-nginx-plus-deployment-guide-files/etc_nginx_conf.d/status.conf
  7. Validate the NGINX Plus configuration and restart NGINX Plus:

    $ sudo nginx -t
    $ sudo nginx -s reload
  8. Verify the instance is working by accessing it at its external IP address. (As previously noted, we recommend blocking access to the external IP addresses of the application instances in a production environment.) The external IP address for the instance appears on the Compute Engine > VM instances summary page, in the External IP column of the table.

    • Access the index.html page either in a browser or by running this curl command.

      $ curl http://external-IP-address
    • Access the NGINX Plus live activity monitoring dashboard in a browser, at:

      https//:external-IP-address:8080/status.html

  9. Proceed to Task 3 – Creating “Gold” Images.

Task 3 – Creating “Gold” Images

Create gold images, which are base images that GCE clones automatically when it needs to scale up the number of instances. They are derived from the instances you created in Creating Source Instances. Before creating the images, you must delete the source instances to break the attachment between them and the disk (you can’t create an image from a disk that’s attached to a VM instance).

  1. Verify that the NGINX Plus All-Active-LB project is still selected in the Google Cloud Platform header bar.

  2. Navigate to the Compute Engine > VM instances tab.

  3. In the table, select all three instances:

    • If you created source instances from Ubuntu images: nginx-plus-app-1, nginx-plus-app-2, and nginx-plus-lb
    • If you created source instances from prebuilt NGINX Plus images: nginx-plus-app-1-vm, nginx-plus-app-2-vm, and nginx-plus-lb-vm
  4.  

  5. Click STOP in the top toolbar to stop the instances.

    Screenshot of the toolbar on the Google Compute Engine page that lists VM instances, used when deploying NGINX Plus as the Google Cloud load balancer.

  6. Click DELETE in the top toolbar to delete the instances.

    Note: If the pop‑up confirmation window warns that the boot disk for any instance will also be deleted, cancel the deletion, and perform these steps for each affected instance:

    1. Navigate to the Compute Engine > VM instances tab and click on the instance in the Name column in the table. (The screenshot shows nginx-plus-app-1-vm.)

      Screenshot showing how to access the page where configuration details for a VM instance can be modified during deployment of NGINX Plus as the Google Cloud load balancer.

    2. On the VM instances page that opens, click EDIT at the top of the page. In fields that can be edited, the value changes from static text to text boxes, drop‑down menus, and checkboxes.
    3. In the Boot disk and local disks field, uncheck the checkbox labeled Delete boot disk when when instance is deleted.
    4. Click the  Save  button.
    5. On the VM instances summary page, select the instance in the table and click DELETE in the top toolbar to delete it.
  7.  

  8. Navigate to the Compute Engine > Images tab.

  9. Click [+] CREATE IMAGE.

  10. On the Create an image page that opens, modify or verify the fields as indicated:

    • Name – nginx-plus-app-1-image
    • Family – Leave the field empty
    • Description – NGINX Plus Application 1 Gold Image
    • Encryption – Automatic (recommended) (the default)
    • Source – Disk (the default)
    • Source disk – nginx-plus-app-1 or nginx-plus-app-1-vm, depending on the method you used to create source instances (select the source instance from the drop‑down menu)
  11.  

  12. Click the  Create  button.

  13. Repeat Steps 7 through 9 to create a second image with the following values (retain the default values in all other fields):

    • Name – nginx-plus-app-2-image
    • Description – NGINX Plus Application 2 Gold Image
    • Source disk – nginx-plus-app-2 or nginx-plus-app-2-vm, depending on the method you used to create source instances (select the source instance from the drop‑down menu)
  14.  

  15. Repeat Steps 7 through 9 to create a third image with the following values (retain the default values in all other fields):

    • Name – nginx-plus-lb-image
    • Description – NGINX Plus LB Gold Image
    • Source disk – nginx-plus-lb or nginx-plus-lb-vm, depending on the method you used to create source instances (select the source instance from the drop‑down menu)
  16.  

  17. Verify that the three images appear at the top of the table on the Compute Engine > Images tab.

Task 4 – Creating Instance Templates

Create instance templates, which are the compute workloads that are created in instance groups, either manually or automatically when GCE detects a failure.

Creating the First Application Server Instance Template

  1. Verify that the NGINX Plus All-Active-LB project is still selected in the Google Cloud Platform header bar.

  2. Navigate to the Compute Engine > Instance templates tab.

  3. Click the  Create instance template  button.

  4. On the Create an instance template page that opens, modify or verify the fields as indicated:

    • Name – nginx-plus-app-1-instance-template
    • Machine type – The appropriate size for the level of traffic you anticipate. We’re selecting micro, which is ideal for testing purposes.
    • Boot disk – Click Change. The Boot disk page opens. Perform the following steps:

      1. Open the Custom Images subtab.

        Screenshot of the 'Boot disk' page in Google Cloud Platform for selecting the source instance of a new instance template, part of deploying NGINX Plus as the Google load balancer.

      2. Select NGINX Plus All-Active-LB from the drop‑down menu labeled Show images from.

      3. Click the nginx-plus-app-1-image radio button.

      4. Accept the default values in the Boot disk type and Size (GB) fields (Standard persistent disk and 10 respectively).

      5. Click the  Select  button.

    • Identity and API access – Unless you want more granular control over access, keep the defaults in the Service account field (Compute Engine default service account) and Access scopes field (Allow default access).
    • Firewall – Verify that neither check box is checked (the default). The firewall rule invoked in the Tags field on the Management subtab (see Step 6 below) controls this type of access.
  5.  

  6. Click Management, disk, networking, SSH keys to open that set of subtabs.

    Screenshot of the interface for creating a Google Compute Engine (GCE) instance template, used during deployment of NGINX Plus as the Google load balancer.

  7. On the Management subtab, modify or verify the fields as indicated:

    • Description – NGINX Plus app-1 Instance Template
    • Tags – nginx-plus-http-fw-rule
    • Preemptibility – Off (recommended) (the default)
    • Automatic restart – On (recommended) (the default)
    • On host maintenance – Migrate VM instance (recommended) (the default)

     

    Screenshot of the Management subtab used during creation of a new VM instance template, part of deploying NGINX Plus as the Google load balancer.

  8. On the Disks subtab, verify that the checkbox labeled Delete boot disk when instance is deleted is checked.

    Instances created from this template are ephemeral instantiations of the gold image, so we want GCE to reclaim the disk when the instance is terminated. New instances are always based on the gold image, so there is no reason to have the instantiations persist on disk when the instance is deleted.

    Screenshot of the Disks subtab used during creation of a new VM instance template, part of deploying NGINX Plus as the Google Cloud load balancer.

  9. On the Networking subtab, verify the default settings of Ephemeral for External IP and Off for IP Forwarding.

    Screenshot of the Networking subtab used during creation of a new VM instance template, part of deploying NGINX Plus as the Google load balancer.

  10. If you are using your own SSH public key instead of the default keys associated with your GCE identity, on the SSH Keys subtab paste the hexadecimal key string into the box that reads Enter entire key data.

    Screenshot of the SSH Keys subtab used during creation of a new VM instance, part of deploying NGINX Plus as the Google Cloud Platform load balancer.

  11. Click the  Create  button.

Creating the Second Application Server Instance Template

  1. On the Instance templates summary page, click CREATE INSTANCE TEMPLATE.

  2. Repeat Steps 4 through 10 to create a second application instance template, specifying the same values as for the first instance template, except as noted:

    • In Step 4:
      • Name – nginx-plus-app-2-instance-template
      • Boot disk – Click the nginx-plus-app-2-image radio button
    • In Step 6, Description – NGINX Plus app-2 Instance Template

Creating the Load Balancing Instance Template

  1. On the Instance templates summary page, click CREATE INSTANCE TEMPLATE.

  2. Repeat Steps 4 through 10 to create the load balancing instance template, specifying the same values as for the first instance template, except as noted:

    • In Step 4:
      • Name – nginx-plus-lb-instance-template.
      • Boot disk – Click the nginx-plus-lb-image radio button

    • In Step 6, Description – NGINX Plus Load Balancing Instance Template

Task 5 – Creating Image Health Checks

Create a simple HTTP health check that GCE uses to verify that each NGINX Plus LB image is running correctly, and re‑create it if not.

  1. Verify that the NGINX Plus All-Active-LB project is still selected in the Google Cloud Platform header bar.

  2. Navigate to the Compute Engine > Health checks tab.

  3. Click the  Create a health check  button.

  4. On the Create a health check page that opens, modify or verify the fields as indicated:

    • Name – nginx-plus-http-health-check
    • Description – Basic HTTP health check to monitor NGINX Plus instances
    • Protocol – HTTP (the default)
    • Port – 80 (the default)
    • Request path – /status-old.html
  5.  

  6. If the Health criteria section is not already open, click More.

  7. Modify or verify the fields as indicated:

    • Check interval – 10 seconds
    • Timeout – 10 seconds
    • Healthy threshold – 2 consecutive successes (the default)
    • Unhealthy threshold – 10 consecutive failures
  8.  

  9. Click the  Create  button.

Screenshot of the interface for creating a health check in Google Compute Engine (GCE), which Google network load balancer uses to monitor NGINX Plus as the Google cloud load balancer.

Task 6 – Creating Instance Groups

Create three independent instance groups, one for each type of function‑specific instance.

  1. Verify that the NGINX Plus All-Active-LB project is still selected in the Google Cloud Platform header bar.

  2. Navigate to the Compute Engine > Instance groups tab.

  3. Click the  Create instance group  button.

Creating the First Application Server Instance Group

  1. On the Create a new instance group page that opens, modify or verify the fields as indicated. Ignore fields that are not mentioned:

    • Name – nginx-plus-app-1-instance-group
    • Description – Instance group to host NGINX Plus app-1 instances
    • Location –
    • Creation method – Use instance template radio button (the default)
    • Instance template – nginx-plus-app-1-instance-template (select from the drop‑down menu)
    • Autoscaling – Off (the default)
    • Number of instances – 2
    • Health check – nginx-plus-http-health-check (select from the drop‑down menu)
    • Initial delay – 300 seconds (the default)
  2.  

  3. Click the  Create  button.

    Screenshot of the interface for creating a Google Compute Engine (GCE) instance group, used during deployment of NGINX Plus as the load balancer for Google Cloud.

Creating the Second Application Server Instance Group

  1. On the Instance groups summary page, click CREATE INSTANCE GROUP.

  2. Repeat Steps 4 and 5 to create a second application instance template, specifying the same values in Step 4 as for the first instance template, except as noted:

    • Name – nginx-plus-app-2-instance-group
    • Description – Instance group to host NGINX Plus app-2 instances
    • Instance template – nginx-plus-app-2-instance-template (select from the drop‑down menu)

Creating the Load Balancing Instance Group

  1. On the Instance groups summary page, click CREATE INSTANCE GROUP.

  2. Repeat Steps 4 and 5 to create the load balancing instance template, specifying the same values in Step 4 as for the first instance template, except as noted:

    • Name – nginx-plus-lb-instance-group
    • Description – Instance group to host NGINX Plus load balancing instances
    • Instance template – nginx-plus-lb-instance-template (select from the drop‑down menu)

Updating and Testing the NGINX Plus Configuration

Update the NGINX Plus configuration on the two LB instances (nginx-plus-lb-instance-group-[a. . .z]) to list the internal IP addresses of the four application servers (two instances each of nginx-plus-app-1-instance-group-[a. . .z] and nginx-plus-app-2-instance-group-[a. . .z]).

Repeat these instructions for both LB instances.

  1. Connect to the LB instance over SSH using the method of your choice. GCE provides a built‑in mechanism:

    1. Navigate to the Compute Engine > VM instances tab.
    2. In the row for the instance in the table, click the triangle icon in the Connect column at the far right and select a method (for example, Open in browser window).
  2.  

  3. Working in the SSH terminal, use your preferred text editor to modify gce-all-active-lb.conf, changing the server directives in the upstream block to reference the internal IP addresses of the two nginx-plus-app-1-instance-group-[a. . .z] and the two nginx-plus-app-2-instance-group-[a. . .z] instances (the addresses appear in the Internal IP column of the table on the Compute Engine > VM instances summary page). For example:

    upstream upstream_app_pool {
    server 10.10.10.1;
    server 10.10.10.2;
    server 10.10.10.3;
    server 10.10.10.4;
    zone upstream-apps 64k;
    sticky cookie GCPPersist expires=300;
    }
  4. Validate the NGINX Plus configuration and restart NGINX Plus:

    $ sudo nginx -t
    $ sudo nginx -s reload
  5. Verify that the four application instances are receiving traffic and responding by accessing the external IP address of the nginx-plus-lb-instance-group-[a…z] instance in a browser. (The external IP address for an instance appears on the Compute Engine > VM instances summary page, in the External IP column of the table.)

    https//:LB-external-IP-address:8080/status.html

  6. Working on a separate client machine, verify that NGINX Plus is load balancing traffic among the four application instance groups:

    $ while true; do curl -s LB-external-IP-address | grep Server: ;done

    If load balancing is working properly, the unique Server field from the index page for each application instance appears in turn.

Task 7 – Configuring GCE Network Load Balancer

Set up GCE network load balancer to distribute incoming client traffic to the NGINX Plus LB instances. The first step is to reserve a static IP address for GCE network load balancer to be advertised to clients.

  1. Verify that the NGINX Plus All-Active-LB project is still selected in the Google Cloud Platform header bar.

  2. Navigate to the Networking > External IP addresses tab.

  3. Click the  Reserve static address  button.

  4. On the Reserve a static address page that opens, modify or verify the fields as indicated:

  5.  

  6. Click the  Reserve  button.

    Screenshot of the interface for reserving a static IP address for Google Compute Engine network load balancer.

  7. Navigate to the Networking > Load balancing tab.

  8. Click the  Create load balancer  button.

  9. On the Load balancing page that opens, click Start configuration in the TCP Load Balancing box.

  10. On the page that opens, click the From Internet to my VMs and No (TCP) radio buttons (the defaults).

  11. Click the  Continue  button. The New TCP load balancer page opens.

  12. In the Name field, type nginx-plus-network-lb-frontend.

  13. Click Backend configuration in the left column to open the Backend configuration interface in the right column. Fill in the fields as indicated:

    • Region – The GCP region you specified in Step 4. We’re using us-west1.
    • Backends – With Select existing instance groups selected, select nginx-plus-lb-instance-group from the drop‑down menu
    • Backup pool – None (the default)
    • Failover ratio – 10 (the default)
    • Health check – nginx-plus-http-health-check
    • Session affinity – Client IP

     

    Screenshot of the interface for backend configuration of GCE network load balancer, used during deployment of NGINX Plus as the Google Cloud Platform load balancer.

  14. Click Frontend configuration in the left column to open the Frontend configuration interface in the right column.

  15. Create three Protocol-IP-Port tuples, with:

    • Protocol – TCP
    • IP – Select the address you reserved in Step 5 from the drop‑down menu (where it is labeled nginx-plus-network-lb-static-ip)
    • Port – 80, 8080, and 443 in the three tuples respectively
  16.  

  17. Click the  Create  button.

    Screenshot of the interface for frontend configuration of GCE network load balancer, used during deployment of NGINX Plus as the Google Cloud load balancer.

Task 8 – Testing the All-Active Load Balancing Deployment

Verify that GCE network load balancer is properly routing traffic to both NGINX Plus LB instances:

  1. Working on a separate client machine, run this command, using the static IP address you set in the previous section for GCE network load balancer:

    $ while true; do curl -s GCE-Network-LB-external-static-IP-address | grep Server: ;done

    Alternatively, you can use a web browser to access this URL:

    http://GCE-Network-LB-external-static-IP-address

    If load balancing is working properly, the unique Server field from the index page for each application instance appears in turn.

Verify that high availability is working.

  1. Connect to one of the images in the nginx-plus-lb-instance-group over SSH and run this command to force it offline:

    $ sudo iptables -A INPUT -p tcp --destination-port 80 -j DROP
  2. Verify that with one LB instance offline, the other LB instance still forwards traffic to the application instances (there might be a delay before GCE network load balancer detects that the first instance is offline). Continue monitoring and verify that GCE network load balancer re‑creates the first LB instance and brings it online.

  3. When the LB instance is back online, run this command to return it to its working state:

    $ sudo iptables -F

Revision History

  • Version 1 (November 2016) – Initial version (NGINX Plus R11, NGINX 1.11.5)