Set Up Docker with TLS: /Documentation/Archive/23.3

Set Up Docker with TLS

Premium Feature — Docker supports RStudio, which is part of the Professional and Enterprise Editions of LabKey Server. Learn more or contact LabKey for details.

This topic outlines an example TLS and certificate configuration used to set up Docker for use with RStudio in LabKey Server. The main instructions for configuring Docker and RStudio are in this topic: Connect to RStudio

Installation Instructions for Docker Daemon
Install the Docker Daemon
Create TLS Certificates
Change Docker Daemon Configuration
Changes to RServe Instance to Support RStudio Usage
Install AppArmor Configuration to be used by Docker Containers
Install New Firewall Rules
Changes to Web Server Instance to Support RStudio Usage

On a development machine, you do not have to set up TLS if both of the following are true:

Use port 2375 instead of 2376
Run with devmode=true

Otherwise a secure connection is required, whether the docker server is local or remote.

Installation Instructions for Docker Daemon

First, identify where the server will run the Docker Daemon on the Test Environment.

The Docker documentation includes the following recommendation: "...if you run Docker on a server, it is recommended to run exclusively Docker in the server, and move all other services within containers controlled by Docker. Of course, it is fine to keep your favorite admin tools (probably at least an SSH server), as well as existing monitoring/supervision processes (e.g., NRPE, collectd, etc)."

The options available include:

Create a New Instance in the Test Environment for the Docker Daemon. This option is preferred in an ideal world, as it is the most secure and follows all Docker best practices. If a runaway RStudio process were to overwhelm the instance's resources it would not affect Rserve and the Web Server processes and thus the other applications would continue to function properly.
Install and Run the Docker Daemon on the Rserve instance. This allows quick installation and configuration of Docker in the Test Environment.
Install and Run the Docker Daemon on the Web Server.

For the Test Environment, option #2 is recommended as this allows us to test the RStudio features using a configuration similar to what will be used in Production.

If during the testing, we begin to see resource contention between RStudio and Rserve processes, we can change the Docker Daemon configuration to limit resources used by RStudio or request a new instance to run Docker.
If a runaway RStudio process overwhelms the instance's resources it will not affect the Web Server processes and thus the other applications will continue to function properly.

Install the Docker Daemon

Install the Docker Daemon on the Rserve instance by running the following commands:

sudo apt-get update 
sudo apt-get install apt-transport-https ca-certificates
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

sudo vi /etc/apt/sources.list.d/docker.list
[Added]
deb https://apt.dockerproject.org/repo ubuntu-trusty main

sudo apt-get update
sudo apt-get purge lxc-docker
sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
sudo apt-get install docker-engine

IMPORTANT: Do not add any users to the docker group (members of the docker group can access dockerd Linux socket).

Create TLS Certificates

The TLS certificates are used by the LabKey Server to authenticate to the Docker Daemon process.

Create the directory that will hold the CA certificate/key and the Client certificate/key. You can use a different directory if you want than the one shown below. This is the value of "DOCKER_CERT_PATH":

sudo su - 
mkdir -p /labkey/apps/ssl 
chmod 700 /labkey/apps/ssl

Create the Certificate Authority private key and certificate

Create the CA key CA certificate. Configure the certificate to expire in 10 years.

openssl genrsa -out /labkey/apps/ssl/ca-key.pem 4096
openssl req -x509 -new -nodes -key /labkey/apps/ssl/ca-key.pem -days 3650 -out /labkey/apps/ssl/ca.pem -subj '/CN=docker-CA'

Create an openssl.cnf configuration file to be used by the CA.

vi /labkey/apps/ssl/openssl.cnf 
[added]

    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    extendedKeyUsage = serverAuth, clientAuth

Create the Client TLS certificate

Create the client certificates and key. The client certificate will be good for 10 years.

openssl genrsa -out /labkey/apps/ssl/client-key.pem 4096
openssl req -new -key /labkey/apps/ssl/client-key.pem -out /labkey/apps/ssl/client-cert.csr -subj '/CN=docker-client' -config /labkey/apps/ssl/openssl.cnf
openssl x509 -req -in /labkey/apps/ssl/client-cert.csr -CA /labkey/apps/ssl/ca.pem -CAkey /labkey/apps/ssl/ca-key.pem -CAcreateserial -out /labkey/apps/ssl/client-cert.pem -days 3650 -extensions v3_req -extfile /labkey/apps/ssl/openssl.cnf

Create the TLS certificates to be used by the Docker Daemon

Create the directory from which the docker process will read the TLS certificate and key files.

mkdir /etc/docker/ssl 
chmod 700 /etc/docker/ssl/

Copy over the CA certificate.

cp /labkey/apps/ssl/ca.pem /etc/docker/ssl

Create the openssl.cnf configuration file to be used by the Docker Daemon. Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.

vi /etc/docker/ssl/openssl.cnf 
[added]

    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    extendedKeyUsage = serverAuth, clientAuth
    subjectAltName = @alt_names

    [alt_names]
    DNS.1 = yourtestweb | yourprodweb
    DNS.2 = yourtestrserve | yourprodrserve
    IP.1 = 127.0.0.1
    IP.2 = 10.0.0.87 | 10.10.0.37

Create the key and certificate to be used by the Docker Daemon.

openssl genrsa -out /etc/docker/ssl/daemon-key.pem 4096
openssl req -new -key /etc/docker/ssl/daemon-key.pem -out /etc/docker/ssl/daemon-cert.csr -subj '/CN=docker-daemon' -config /etc/docker/ssl/openssl.cnf
openssl x509 -req -in /etc/docker/ssl/daemon-cert.csr -CA /etc/docker/ssl/ca.pem -CAkey /labkey/apps/ssl/ca-key.pem -CAcreateserial -out /etc/docker/ssl/daemon-cert.pem -days 3650 -extensions v3_req -extfile /etc/docker/ssl/openssl.cnf

Set the correct permission on the certificates.

chmod 600 /etc/docker/ssl/*

Change Docker Daemon Configuration

Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.

Change the Docker Daemon to use your preferred configuration. The changes are:

Daemon will listen Linux socket at "/var/run/docker.sock" and "tcp://10.0.1.204 | 10.10.1.74:2376"
Use TLS certificates on the TCP socket for encryption and authentication
Turn off inter-container communication
Set default limits on container usage and enable userland-proxy

These options were set creating the daemon.json configuration file at:

/etc/docker/daemon.json

vi /etc/docker/daemon.json 
[added]
  {
    "icc": false,
    "tls": true,
    "tlsverify": true,
    "tlscacert": "/etc/docker/ssl/ca.pem",
    "tlscert": "/etc/docker/ssl/daemon-cert.pem",
    "tlskey": "/etc/docker/ssl/daemon-key.pem",
    "userland-proxy": false,
    "default-ulimit": "nofile=50:100",
    "hosts": ["unix:///var/run/docker.sock", "tcp://10.0.1.204 | 10.10.1.74:2376"]
  }

Start the Docker daemon by running:

service docker start

We can test if docker is running:

docker info

If it is running, you will see something similar to:

Containers: 0
     Running: 0
     Paused: 0
     Stopped: 0
    Images: 0
    Server Version: 1.12.1
    Storage Driver: aufs
     Root Dir: /var/lib/docker/aufs
     Backing Filesystem: extfs
     Dirs: 0
     Dirperm1 Supported: false
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Plugins:
     Volume: local
     Network: host overlay bridge null
    Swarm: inactive
    Runtimes: runc
    Default Runtime: runc
    Security Options: apparmor
    Kernel Version: 3.13.0-92-generic
    Operating System: Ubuntu 14.04.4 LTS
    OSType: linux
    Architecture: x86_64
    CPUs: 1
    Total Memory: 489.9 MiB
    Name: myubuntu
    ID: WK6X:HLMO:K5IQ:MENK:ALKP:JN4Q:ALYL:32UC:Q2OD:ZNFG:XLZJ:4KPA
    Docker Root Dir: /var/lib/docker
    Debug Mode (client): false
    Debug Mode (server): false
    Registry: https://index.docker.io/v1/
    WARNING: No swap limit support
    Insecure Registries:
     127.0.0.0/8

We can test if the TLS configuration is working by running:

docker -H 10.0.1.204 | 10.10.1.74:2376 --tls --tlscert=/labkey/apps/ssl/client-cert.pem --tlskey=/labkey/apps/ssl/client-key.pem  ps -a

This should output:

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Install AppArmor Configuration to be used by Docker Containers

Create new custom profile named "docker-labkey-myserver".

vi /etc/apparmor.d/docker-labkey-myserver
    [added]
    
    #include <tunables/global>
    
    
    profile docker-labkey-myserver flags=(attach_disconnected,mediate_deleted) {

        #include <abstractions/base>

        network inet tcp,
        network inet udp,
        deny network inet icmp,
        deny network raw,
        deny network packet,
        capability,
        file,
        umount,

        deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
        # deny write to files not in /proc/<number>/** or /proc/sys/**
        deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
        deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
        deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
        deny @{PROC}/sysrq-trigger rwklx,
        deny @{PROC}/mem rwklx,
        deny @{PROC}/kmem rwklx,
        deny @{PROC}/kcore rwklx,

        deny mount,

        deny /sys/[^f]*/** wklx,
        deny /sys/f[^s]*/** wklx,
        deny /sys/fs/[^c]*/** wklx,
        deny /sys/fs/c[^g]*/** wklx,
        deny /sys/fs/cg[^r]*/** wklx,
        deny /sys/firmware/efi/efivars/** rwklx,
        deny /sys/kernel/security/** rwklx,


        # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
        ptrace (trace,read) peer=docker-labkey-myserver,


        # Rules added by LabKey to deny running executables and accessing files
        deny /bin/dash mrwklx,
        deny /bin/bash mrwklx,
        deny /bin/sh mrwklx,
        deny /usr/bin/top mrwklx,
        deny /usr/bin/apt* mrwklx,
        deny /usr/bin/dpkg mrwklx,
    
        deny /bin/** wl,
        deny /boot/** wl,
        deny /dev/[^null]** wl,
        deny /lib/** wl,
        deny /lib64/** wl,
        deny /media/** wl,
        deny /mnt/** wl,
        deny /opt/** wl,
        deny /proc/** wl,
        deny /root/** wl,
        deny /sbin/** wl,
        deny /srv/** wl,
        deny /sys/** wl,
        deny /usr/[^local]** wl,
        deny /teamcity/** rwklx,
        deny /labkey/** rwklx,
        deny /share/files/** rwklx,
    
    }

Load the new profile.

apparmor_parser -r -W /etc/apparmor.d/docker-labkey-myserver

Now that the profile is loaded, we can force the container to use the profile by specifying it at run time, for example:

docker run --security-opt "apparmor=docker-labkey-myserver" -i -t  ubuntu /bin/bash

When starting a new Docker container manually, you should always use the "docker-labkey-myserver" profile. This is done by specifying the following option whenever a container is started:

--security-opt "apparmor=docker-labkey-myserver"

Install New Firewall Rules

Install file containing new firewall rules to block certain connections from running containers Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.

mkdir /etc/iptables.d
    vi /etc/iptables.d/docker-containers
    [added]
    
        *filter
        :INPUT ACCEPT
        :FORWARD ACCEPT
        :OUTPUT ACCEPT

        -A INPUT -i eth0 -p tcp --destination-port 2376 -s 10.0.0.87 | 10.10.0.37/32 -j ACCEPT
        -A INPUT -i docker0 -p tcp --destination-port 2376 -s 172.17.0.0/16 -j REJECT
        -A INPUT -p tcp --destination-port 2376 -j REJECT
        -A INPUT -i docker0 -p tcp --destination-port 6312 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p tcp --destination-port 22 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p tcp --destination-port 111 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p udp --destination-port 111 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p tcp --destination-port 1110 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p udp --destination-port 1110 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p tcp --destination-port 2049 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p udp --destination-port 2049 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p tcp --destination-port 4045 -s 172.17.0.0/16 -j REJECT
        -A INPUT -i docker0 -p udp --destination-port 4045 -s 172.17.0.0/16 -j REJECT
        -I DOCKER 1 -i eth0 ! -s 10.0.0.87/32 -j DROP
        -I FORWARD 1 -i docker0 -p tcp --destination-port 80 -j ACCEPT
        -I FORWARD 2 -i docker0 -p tcp --destination-port 443 -j ACCEPT
        -I FORWARD 3 -i docker0 -p tcp --destination-port 53 -j ACCEPT
        -I FORWARD 4 -i docker0 -p upd --destination-port 53 -j ACCEPT
        -I FORWARD 5 -i docker0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
        -I FORWARD 6 -i docker0 -j REJECT
        COMMIT

Create the Systemd Configuration

Install 'systemd', then create the systemd configuration file to ensure iptables rules are applied at startup. For more details see: https://docs.docker.com/config/daemon/systemd/

# Create a systemd drop-in directory for the docker service 
mkdir /etc/systemd/system/docker.service.d 

# create a service file at 
# vi /etc/systemd/system/docker.service.d/iptables.conf 
# cp below 
[Unit] 
Description = Load iptables firewall changes 

[Service] 
ExecStartPost=/sbin/iptables-restore --noflush /etc/iptables.d/docker-containers 

# flush changes 
systemctl daemon-reload 

# restart docker 
systemctl restart docker 

# check that iptables.conf was run 
systemctl status docker.service 

# check iptables 
iptables -L

Start the new service.

initctl reload-configuration
    initctl start iptables.service

Changes to RServe Instance to Support RStudio Usage

In order to support file sharing between the RStudio process and the LabKey Server we will need to:

Create new user accounts on Rserve instance
Create new groups on Rserve instance
Modify permission on the /share/users directory

Add new groups and user accounts Rserve instance

Create the new group named "labkey-docker" this group is created to facilate the file sharing.

sudo groupadd -g 6000 labkey-docker

Create the new RStudio user. This will be the OS user which runs the RStudio session

sudo useradd -u 6005 -m -G labkey-docker rstudio

Create the directory which will hold the Home Directory Fileroots used by each server

We will use acls to ensure that newly created files/directories have the correct permissions. The ACLs only need to be set on the Rserve operating system. They do not need to specified or changed on the container.

These ACLs should only be specified on the LabKey Server Home Directory FileRoot. Never set these ACLs on any other directory on the Rserve. Sharing files between the docker host and container should only be done in the LabKey Server Home Directory FileRoot.

Install required package:

sudo apt-get install acl

Create the home dir fileroot and set the correct permissions using the newly created accounts. These instructions assume:

Tomcat server is being run by the user "myserver"
LabKey Server Home Directory FileRoot is located at "/share/users"

Run the following commands:

setfacl -R -m u:myserver:rwx /share/users
    setfacl -Rd -m u:myserver:rwx /share/users
    setfacl -Rd -m u:rstudio:rwx /share/users
    setfacl -Rd -m g:labkey-docker:rwx /share/users

Changes to Web Server Instance to Support RStudio Usage

In order to support file sharing between the RStudio process and the LabKey Server we will need to:

Create new user accounts on Rserve instance
Create new groups on Rserve instance
Ensure the NFS share "/share" is mounted with "acl" option.

Add new groups and user accounts Rserve instance

Create the new group named "labkey-docker" to facilitate file sharing.

sudo groupadd -g 6000 labkey-docker

Create the new RStudio user. This will be the OS user which runs the RStudio session.

sudo useradd -u 6005 -m -G labkey-docker rstudio

LabKey Support

LabKey Support