nginxinc / docker-nginx-controller

Table of Contents generated with DocToc

• 1. Overview
  • 1.1. Current Scenarios
  • 1.2 Before You Begin
• 2. How to Build and Run an NGINX Controller-Enabled NGINX Plus Image
  • 2.1. Building an NGINX Controller-Enabled Image with NGINX Plus
  • 2.2. Building a NAP-Enabled NGINX Docker Container
  • 2.3. Running an NGINX Controller-Enabled NGINX Docker Container
• 3.0 Adding a Controller Agent Layer to an Existing Container or Image
  • 3.1 At Run Time
  • 3.2 As an Image Layer
• 4.0 Build Time and Run Time Options
  • 4.1 Default Naming Behavior
  • 4.2 Persisting an Instance Identity through Stops and Starts
  • 4.3 Applying a Unique Location to a Container at Run Time
  • 4.4 Overriding Agent NGINX Controller configuration
  • 4.5 Supported run-time variables
  • 4.6 Overriding NGINX Plus version
• 5.0 Build Unprivileged Docker Image
  • 5.1 Required changes in Dockerfile
  • 5.2 Required changes in entrypoint.sh
  • 5.3 New build arguments
• 6.0 Support

We are actively working on improving support for containers with NGINX Controller.

The following is a set of guidelines that you can use today as we enhance the experience.

1. Overview

NGINX Controller is a centralized monitoring and management control-plane solution for the NGINX Plus data plane. NGINX Controller is developed and maintained by NGINX -- the people behind NGINX software.

With NGINX Controller, it is possible to collect and aggregate metrics across NGINX Plus instances, your applications, environments, and locations however they run -- presenting a coherent set of visualizations of the critical NGINX Plus performance data, such as active connections or requests per second. It is also easy to quickly check for any performance degradations and traffic anomalies and to get a more in-depth insight into the NGINX configuration in general.

A small agent (NGINX Controller Agent) is necessary inside the container alongside NGINX Plus to use NGINX Controller to monitor and/or manage your fleet of NGINX Plus instances.

For security protection of your web application, a web application firewall NGINX App Protect could be installed alongside NGINX Plus. The official documentation for NGINX App Protect is available here.

The official documentation for NGINX Controller is available here.

Guidance around NGINX Plus is available here.

The Dockerfiles in this repository are supported by and tested against NGINX Controller v3.10 and later.

1.1. Current Scenarios

The following list summarizes known container scenarios with NGINX Controller:

• The NGINX Controller Agent manages/monitors NGINX Plus from inside the container. It is not possible to run the Controller Agent in a separate container and monitor the neighboring containers running NGINX Plus, nor is it possible to install the Controller Agent on the container host and monitor/manage NGINX Plus running in containers.

1.2 Before You Begin

Before proceeding, first complete the following required steps:

1. Install NGINX Controller.
2. Download your NGINX Plus certificate and key (that is, nginx-repo.crt and nginx-repo.key).
3. Obtain the API key for your NGINX Controller instance.

2. How to Build and Run an NGINX Controller-Enabled NGINX Plus Image

2.1. Building an NGINX Controller-Enabled Image with NGINX Plus

Note: If you are new to Docker or the Dockerfile-based image building process, refer to the Install Docker Engine guide for installation instructions. See also the Get started guide for instructions on obtaining and building images.

Here's how to build the container image with the Controller Agent inside, based on the official NGINX image:

1. Clone this repository:

   git clone https://github.com/nginxinc/docker-nginx-controller.git

   cd docker-nginx-controller/<distribution>/no-nap

2. Copy your NGINX Plus repository certificate and key to the folder of the Dockerfile you will be using for your Linux distribution.

3. Edit the Dockerfile with your API_KEY and CONTROLLER_URL.

   Note: Use the CONTROLLER_URL format that's supported by your version of NGINX Controller:

   NGINX Controller v3.10 or older: CONTROLLER_URL=https://<fqdn>:8443/1.4/install/controller

   NGINX Controller v3.11 or newer: CONTROLLER_URL=https://<fqdn>/install/controller-agent

   For example:

   sudo DOCKER_BUILDKIT=1 docker build --build-arg CONTROLLER_URL=https://<fqdn>/install/controller-agent --build-arg API_KEY='abcdefxxxxxx' --secret id=nginx-crt,src=nginx-repo.crt --secret id=nginx-key,src=nginx-repo.key -t nginx-agent .

   The DOCKER_BUILDKIT=1 enables docker build to recognize the --secret flag which allows the user to pass secret information to be used in the Dockerfile for building docker images in a safe way that will not end up stored in the final image. This is a recommended practice for the handling of the certificate and private key for NGINX repository access (nginx-repo.crt and nginx-repo.key files). More information here.

   Note: If you are using an older version of Docker (less than version 20), you may need to insert the following line at the top of your Dockerfile before building your image: syntax=docker/dockerfile:experimental

4. After the image is built, view the list of Docker images:

   sudo docker images

   The output looks similar to the following example:

   REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
   nginx-agent       latest              d039b39d2987        3 minutes ago       241.6 MB

2.2. Building a NAP-Enabled NGINX Docker Container

If you want your Docker image to include a web application firewall, in addition to the Controller Agent, use a Dockerfile with NGINX App Protect included. The file is located at docker-nginx-controller/<distribution>/nap:

cd docker-nginx-controller/<distribution>/nap

2.3. Running an NGINX Controller-Enabled NGINX Docker Container

Take the following steps to run an NGINX Controller-enabled NGINX Docker Container:

1. To start a container from the new image, run the following command:

   docker run --name mynginx1 -d nginx-agent

2. After the container has started, check its status with docker ps:

   docker ps

   The output looks similar to the following example:

   CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
   7d7b47ba4c72        nginx-agent       "/entrypoint.sh"    3 seconds ago       Up 2 seconds        80/tcp, 443/tcp     mynginx1

3. You can also check the docker logs for the container status:

   docker logs 7d7b47ba4c72

   The output looks similar to the following example:

   starting nginx ...
   updating /etc/controller-agent/agent.conf ...
   ---> using api_key = 1234567890
   starting controller-agent ...

4. Check which processes have started:

   docker exec 7d7b47ba4c72 ps axu

   If you see the controller-agent process, the setup went smoothly, and you should see the new container in the NGINX Controller interface after approximately one minute.

   The output looks similar to the following example:

   USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
   root         1  0.0  0.1   4328   676 ?        Ss   19:33   0:00 /bin/sh /entrypoint.sh
   root         5  0.0  0.5  31596  2832 ?        S    19:33   0:00 nginx: master process nginx -g daemon off;
   nginx       11  0.0  0.3  31988  1968 ?        S    19:33   0:00 nginx: worker process
   nginx       65  0.6  9.1 111584 45884 ?        S    19:33   0:06 controller-agent

   If your container includes NAP (NGINX App Protect) then you should also see NAP-specific processes:

   nginx       10  0.0  2.5 129684 52320 ?        S    11:14   0:05 /usr/bin/perl /opt/app_protect/bin/bd_agent
   nginx       14  2.4 12.7 1057612 260260 ?      Sl   11:14   5:54 /usr/share/ts/bin/bd-socket-plugin tmm_count 4 proc_cpuinfo_cpu_m

5. To view the Controller Agent log, run the following command:

   docker exec 7d7b47ba4c72 tail /var/log/nginx-controller/agent.log

   The output looks similar to the following example:

   2016-08-05 19:49:39,001 [65] supervisor agent started, version=0.37-1 pid=65 uuid=<..>
   2016-08-05 19:49:39,047 [65] nginx_config running nginx -t -c /etc/nginx/nginx.conf
   2016-08-05 19:49:40,047 [65] supervisor post https://<controller url>:8443/<..>/ffeedd0102030405060708/agent/ 200 85 4 0.096
   2016-08-05 19:50:24,674 [65] bridge_manager post https://<controller url>:8443/<..>/ffeedd0102030405060708/update/ 202 2370 0 0.084

When you're done with the container, run the following command to stop it:

docker stop 7d7b47ba4c72

To check the status of all the containers that are running and stopped, run the following command:

docker ps -a

The output looks similar to the following example:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                        PORTS               NAMES
7d7b47ba4c72        nginx-agent       "/entrypoint.sh"         22 minutes ago      Exited (137) 19 seconds ago                       mynginx1

3.0 Adding a Controller Agent Layer to an Existing Container or Image

An alternate way to handle the Controller Agent within containers is to include the necessary Controller Agent commands in the run command for an existing image. This way, you don't have to build the Controller Agent into your container before running, and you might find it handy for a Proof of Concept.

3.1 At Run Time

To add a Controller Agent layer to an existing container or image at run time, take the following steps:

# nginx-plus is an example base image located in debian/examples/nginx-plus
FROM nginx-plus

# Start a container with environment variables for CONTROLLER_URL and API_KEY
# docker run --name api-gw --hostname api-gw -e CONTROLLER_URL=https://<fqdn>/install/controller-agent -e API_KEY=deadbeef -d -P nginx-ctrl

# Install everything needed to install the Controller Agent so that the container can start quickly
RUN apt-get update &&\
        apt install -y \
        curl procps sudo 

EXPOSE 80 443 8080
STOPSIGNAL SIGTERM

WORKDIR /controller

RUN printf "curl -skSL \$CONTROLLER_URL | bash -s - -y\n exec nginx -g 'daemon off;'" > start

CMD ["sh", "/controller/start"]

It takes 1-2 minutes to start the container. After docker run , use docker logs --follow CONTAINER to watch the installation and startup progress.

3.2 As an Image Layer

For your convenience, Dockerfiles that define a Controller Agent image layer are provided for you. These Dockerfiles are built following the pattern for running multiple services in a Docker image and can be found under each distribution in the following location:

cd docker-nginx-controller/<distribution>/examples/agent-layer

The build process is the same as above, referencing your custom image as the source.

4.0 Build Time and Run Time Options

4.1 Default Naming Behavior

By default, the Controller Agent determines the OS hostname during installation using the hostname -f command. The hostname value is then assigned to the instance_name key in the Controller Agent configuration file (agent.conf) and is used to generate a UUID. The UUID and instance name together provide a means of uniquely identifying the NGINX instance in NGINX Controller. When the Agent is run inside a container, the default hostname is a shortened Docker Container ID on the host. You can override the automatically assigned instance_name at run time by setting the ENV_CONTROLLER_INSTANCE_NAME environment variable to the desired value.

Providing the ENV_CONTROLLER_INSTANCE_NAME variable for the container sets the container's name that is displayed in NGINX Controller for the displayName of the instance. This also sets the instance object name, which is used in configuration references.

If you do not override the default instance name, the containerID is registered as the instance name and displayName in NGINX Controller.

4.2 Persisting an Instance Identity through Stops and Starts

Using the optional build-time setting of STORE_UUID=True ensures that the dynamically generated UUID persists in the Controller Agent configuration. This, together with the instance_name, allows the container instance to be stopped and started or persist if the container host is rebooted.

Each new container started from an NGINX Controller-enabled Docker image is reported as a unique system in the NGINX Controller console. This is the recommended configuration. NGINX Controller aggregates metrics across your instances based on the application, application component, location, environment, and so forth.

VAR STORE_UUID=True can be set during the image build process and applies to all containers derived from the image.

sudo DOCKER_BUILDKIT=1 docker build --secret id=nginx-crt,src=nginx-repo.crt --secret id=nginx-key,src=nginx-repo.key --build-arg CONTROLLER_URL=https://<fqdn>/install/controller-agent --build-arg API_KEY='abcdefxxxxxx' --build-arg STORE_UUID=True -t nginx-agent .

4.3 Applying a Unique Location to a Container at Run Time

By default, new instances are placed in the NGINX Controller location named unspecified. There are situations where instances should be associated with specific locations. This can be defined at build time to apply to all containers derived from the image or during run time and apply to a subset of containers.

Using the optional run time setting of ENV_CONTROLLER_LOCATION, when your container instance reports to NGINX Controller, the new instance will automatically register itself with a specific location already present in NGINX Controller.

docker run --name mynginx-east-1 -e ENV_CONTROLLER_LOCATION=east -d nginx-agent

The location will not be automatically created in NGINX Controller and needs to be defined separately.

4.4 Overriding Agent NGINX Controller configuration

Container could be build using configuration of one instance of NGINX Controller and run using another instance of NGINX Controller. ENV_CONTROLLER_API_KEY and ENV_CONTROLLER_API_URL variable could be used to override during run-time NGINX Controller configuration.

docker run --name mynginx1 -d -e ENV_CONTROLLER_API_URL=<fqdn_or_ip>:443 -e ENV_CONTROLLER_API_KEY=deadbeef nginx-agent

4.5 Supported run-time variables

You can use the following run-time variables to override the default NGINX configuration values:

4.6 Overriding NGINX Plus version

Version of NGINX Plus installed inside docker image could changed using NGINX_PLUS_VERSION build time argument. NGINX_PLUS_VERSION should be set to release number of NGINX Plus e.g. 25

DOCKER_BUILDKIT=1 docker build --secret id=nginx-crt,src=nginx-repo.crt --secret id=nginx-key,src=nginx-repo.key --build-arg CONTROLLER_URL=https://<fqdn>/install/controller-agent --build-arg API_KEY='abcdefxxxxxx' --build-arg NGINX_PLUS_VERSION=22 -t nginx-agent .

For NAP-Enabled NGINX Docker Containers, information about App Security requirements can be found here

4.7 Configuring and updating agent

When manually editing the agent.conf file or upgrading the NGINX Agent service running inside the container, use --restart always with docker run, as restarting the NGINX Agent service will trigger the container to exit.

docker run --restart always --name mynginx -d nginx-agent

5.0 Build Unprivileged Docker Image

5.1 Required changes in Dockerfile

Add the following snippet under the ARG NGINX_PLUS_VERSION=nn line:

ARG EXPOSE_PORT=8080
ENV NGINX_EXPOSE_PORT=$EXPOSE_PORT

ARG NON_ROOT_USER=nginx
ENV CONTROLLER_USER=$NON_ROOT_USER

ARG NON_ROOT_GROUP=nginx
ENV CONTROLLER_GROUP=$NON_ROOT_GROUP

ARG NAP_SYSLOG_PORT=5114
ENV LISTENERS_NAP_SYSLOG_PORT=$NAP_SYSLOG_PORT

Replace the EXPOSE 80 line with the following snippet:

# Update ownership for the necessary filesystem objects for running under non-root user
RUN chown -R $CONTROLLER_USER:$CONTROLLER_GROUP /var/run/ \
  && chown -R $CONTROLLER_USER:$CONTROLLER_GROUP /var/log/nginx/ \
  && chown -R $CONTROLLER_USER:$CONTROLLER_GROUP /var/cache/nginx/ \
  && chown -R $CONTROLLER_USER:$CONTROLLER_GROUP /etc/controller-agent/ \
  && chown -R $CONTROLLER_USER:$CONTROLLER_GROUP /etc/nginx/ \
  && chown -R $CONTROLLER_USER:$CONTROLLER_GROUP /var/log/app_protect/ || true \
  && chown -R $CONTROLLER_USER:$CONTROLLER_GROUP /opt/app_protect/ || true \  
  && sed -i "s,listen       80 default_server;,listen       $NGINX_EXPOSE_PORT default_server;," /etc/nginx/conf.d/default.conf \
  && sed -i '/user  nginx;/d' /etc/nginx/nginx.conf

USER $CONTROLLER_USER

EXPOSE $NGINX_EXPOSE_PORT

Examples of unprivileged docker files can be found in unprivileged/examples directory.

5.2 Required changes in entrypoint.sh

Only applicable to images running NGINX App Protect - remove prefixes /bin/su -s under # Launch NAP (two occurrences).

The result should look similar to the following snippet:

# Launch NAP
/bin/bash -c '/opt/app_protect/bin/bd_agent &' nginx
bd_agent_pid=$(pgrep bd_agent)
/bin/bash -c "/usr/share/ts/bin/bd-socket-plugin tmm_count 4 proc_cpuinfo_cpu_mhz 2000000 total_xml_memory 307200000 total_umu_max_size 3129344 sys_max_account_id 1024 no_static_config 2>&1 > /var/log/app_protect/bd-socket-plugin.log &" nginx
bd_socket_pid=$(pgrep bd-socket)

5.3 New build arguments

Here is the list of new build arguments introduced by the changes above:

The build and run instructions are the same. Please refer to 2. How to Build and Run an NGINX Controller-Enabled NGINX Plus Image.

6.0 Support

This project is supported and has been validated with Controller Agent v3.10 and later.