Dockerising ALL THE THINGS

June 1, 2015

In my post A Break I mentioned that I’d converted my web VPS to being fully Dockerised and that I might describe in a future post how I’d done that. This is that post.

Right upfront however let me point out that unlike most of my technical posts this one will not be a complete and exhaustive guide. The main reason for this is that the volume of configuration here is larger – to the extent that I’m maintaining several (private) Git repos with the configs and scripts I’ve developed to support deploying each container that now runs on my web VPS. Unless I were to make those Git repos public and reference those, this post either needs to be deliberately brief, or it would need to duplicate example configs and scripts from my repos. I won’t do the latter because they would only get out of date as and when my containers evolve. And I won’t make all the Git repos public because some of them contain private keys for X509 (TLS/SSL) certificates.

Instead what I will do is simply show what images I use/have developed, and what containers run on my web VPS.

The images:

david@maple:~? sudo docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
dhpcs/#########.dhpcs.com           latest              f4da845b8347        14 hours ago        367.6 MB
dhpiggott/www.dhpcs.com             latest              bc990a9be0a9        2 days ago          369.2 MB
dhpiggott/maple.dhpiggott.net-php   latest              e09c7cb46469        9 days ago          457.5 MB
dhpiggott/www.piggott.me.uk-php     latest              5c3b523e7249        9 days ago          452.4 MB
java                                jre                 d262a00470ad        12 days ago         334.6 MB
php                                 apache              29cd0d5fdb04        2 weeks ago         436 MB
dhpiggott/www.dhpiggott.net         latest              99be6c1cca26        3 weeks ago         134.9 MB
dhpiggott/503                       latest              cdfaf596be41        3 weeks ago         132.8 MB
nginx                               1                   42a3cf88f3f0        4 weeks ago         132.8 MB
mysql                               5                   56f320bd6adc        5 weeks ago         282.9 MB
jwilder/docker-gen                  0.3.7               4db95024ba4a        10 weeks ago        95.2 MB

The containers:

david@maple:~? sudo docker ps -a
CONTAINER ID        IMAGE                                      COMMAND                CREATED             STATUS              PORTS                                      NAMES
864aeebc81d2        dhpcs/#########.dhpcs.com:latest           "bin/#########-serve   14 hours ago        Up 14 hours         80/tcp                                     serene_leakey               
eb22b3775458        dhpiggott/www.dhpcs.com:latest             "bin/www-dhpcs-com"    2 days ago          Up 2 days           80/tcp                                     hopeful_galileo             
167e8fb95f73        dhpiggott/maple.dhpiggott.net-php:latest   "apache2-foreground"   9 days ago          Up 9 days           80/tcp                                     naughty_einstein            
58f5871b1c3a        dhpiggott/www.piggott.me.uk-php:latest     "apache2-foreground"   9 days ago          Up 9 days           80/tcp                                     pensive_bardeen             
544f20c15b1d        dhpiggott/www.dhpiggott.net:latest         "nginx -g 'daemon of   3 weeks ago         Up 2 weeks          80/tcp, 443/tcp                            pensive_elion               
6abfab9ac94b        nginx:1                                    "nginx -g 'daemon of   3 weeks ago         Up 2 weeks          0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   proxy-nginx                 
64c9b798c28b        dhpiggott/503:latest                       "nginx -g 'daemon of   3 weeks ago         Up 2 weeks          80/tcp, 443/tcp                            adoring_archimedes          
9f55c421fa50        mysql:5                                    "/entrypoint.sh mysq   4 weeks ago         Up 2 weeks          3306/tcp                                   maple.dhpiggott.net-mysql   
3170e20bd844        mysql:5                                    "/entrypoint.sh mysq   4 weeks ago         Up 2 weeks          3306/tcp                                   www.piggott.me.uk-mysql     
d7ae6513ddf4        jwilder/docker-gen:0.3.7                   "/usr/local/bin/dock   9 weeks ago         Up 2 weeks                                                     proxy-docker-gen

I’ll describe these ordered by the level of functionality they provide.

jwilder/docker-gen

This is one part of the two that make up my deployment of the brilliant Automated Nginx Reverse Proxy for Docker system described at http://jasonwilder.com/blog/2014/03/25/automated-nginx-reverse-proxy-for-docker/.

I went with the two container approach because:

  1. My deployment does not use the nginx.tmpl provided in the single container image. I’ve modified the template to improve the TLS/SSL config, to add support for a few custom features, and to remove one or two things I didn’t need (e.g. digest authentication). Having this custom template means there’s really nothing in the single-container image that it’s worth me extending from.
  2. It was hardly any extra effort and it may be more secure than the one container approach. It’s certainly should be no less secure.

nginx

This is just part two of the above. I store the simple deployment script for this in the same Git repo as my docker-gen deployment script and Nginx template, custom config files & cryptographic materials.

dhpiggott/503

This is a very basic Nginx image that serves a custom static HTTP 503 error page. My deployment script sets its VIRTUAL_HOST environment variable to a list of every domain that I have the Nginx proxy serving. My deployment script also sets a BACKUP environment variable to true. This enables one of the ”features” I said I had modified my nginx.tmpl for. It marks all containers that have this flag set as backup hosts in the nginx upstream blocks that are output when the template is processed (or as normal hosts if the  container is the only one marked as a host for a given domain).

Put together this ensures that:

  1. If the “real” container for a domain fails, the Nginx proxy will instead have the 503 container serve up a clear and friendly error.
  2. If/when I am doing maintenance on a domain’s containers, such that there are no “real” containers to serve a request, the Nginx proxy will instead have the 503 container serve up a clear and friendly error. (This case is the more common one).

Needless to say, the Dockerfile isn’t anything special, though in addition to the static content it does copy in some custom Nginx config to make the HTTP status code for results be a 503.

mysql

I have two of these. One hosts the database for Piwik which runs on maple.dhpiggott.net and the other hosts the database for WordPress which runs on www.piggott.me.uk. The deployment script for each lives in the same repo as the scripts and configs for their counterpart PHP containers. In addition to deployment scripts I have a pair of shell scripts named load-data and save-data that import/export the application data from/to a .sql file at a path given as argument.

dhpiggott/maple.dhpiggott.net

This is based on the official Docker PHP image. To support Piwik my Dockerfile installs the pdo_mysql and mbstring PHP extensions and copies in a custom php.ini.

The Dockerfile:

FROM php:apache

RUN apt-get update \
 && apt-get install -y \
 libjpeg-dev \
 libfreetype6-dev \
 libpng12-dev \
 && apt-get clean -y \
 && rm -rf /var/lib/apt/lists/* \
 && docker-php-ext-configure gd \
 --with-jpeg-dir=/usr \
 --with-freetype-dir=/usr \
 --with-png-dir=/usr \
 && docker-php-ext-install gd

RUN docker-php-ext-install pdo_mysql

RUN docker-php-ext-install mbstring

COPY php.ini $PHP_INI_DIR/

The custom php.ini:

always_populate_raw_post_data=-1

Like the MySQL counterpart container I have a pair of shell scripts named load-data and save-data that import/export the application files from/to a file tree at a path given as argument.

dhpiggott/www.dhpiggott.net

This isn’t anything special, it’s just an extension of the official Nginx image where all my Dockerfile does is copy in the static content for www.dhpiggott.net.

dhpiggott/www.piggott.me.uk

This is based on the official Docker PHP image. To support WordPress my Dockerfile installs the mysqli PHP extension and enables Apache’s rewrite module.

The Dockerfile:

FROM php:apache

RUN apt-get update \
 && apt-get install -y \
 libjpeg-dev \
 libfreetype6-dev \
 libpng12-dev \
 && apt-get clean -y \
 && rm -rf /var/lib/apt/lists/* \
 && docker-php-ext-configure gd \
 --with-jpeg-dir=/usr \
 --with-freetype-dir=/usr \
 --with-png-dir=/usr \
 && docker-php-ext-install gd

RUN docker-php-ext-install mysqli

RUN a2enmod rewrite

Again, like the MySQL counterpart container I have a pair of shell scripts named load-data and save-data that import/export the application files from/to a file tree at a path given as argument.

dhpiggott/www.dhpcs.com

This is prepared by sbt-native-packager’s docker:stage task and built by calling Docker build on that output. So the whole deploy script looks like this:

#!/bin/bash

set -e

DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

(cd $DIR && ./activator docker:stage)

sudo docker build -t dhpiggott/www.dhpcs.com $DIR/target/docker/docker/stage/
sudo docker run -d \
 --restart always \
 -e VIRTUAL_HOST=dhpcs.com,www.dhpcs.com \
 dhpiggott/www.dhpcs.com

One thing to note is that a workaround is necessary to prevent the container’s process continually terminating when restarted after an ungraceful shutdown.

dhpcs/#########.dhpcs.com

This isn’t anything worth describing yet*.

With the images and containers described, I have a few notes on the Docker host.

Before this change all of the above domains had been hosted by a single hand-crafted Apache configuration. Piwik and WordPress shared their PHP and MySQL environments, and www.dhpcs.com was installed as an Upstart job that Apache proxied to. This all ran on a 1GB Linode – so $10/month.

But what that all meant was that operations such as transferring the stack to a different VPS or restoring after a failure were events I lived in fear of, in the sense that experience had taught to me ensure I had at least several hours available to execute them in but also that such operations become necessary at the least convenient of times. And even at the best of times it was simply repetitive and tedious.

As part of this change I did have to upgrade the host to be a 2GB Linode – so $20/month at current prices. But I am now running two MySQL processes and two PHP processes where before there was only one of each, so that’s not really surprising. And for me an extra $10/month is a price worth paying for the peace of mind I now have; the level of automation and configuration control I now have is such that I should be able to deploy all my web presences on a fresh host with comparatively negligible effort beyond installing Docker, cloning my repos,  running the deploy scripts, and importing the data from a Duplicity backup or the previous host (I configure a Cron job on the host that runs the save-data scripts for containers that have them and then incorporates the output in a Duplicity backup).

One other change I had to make was in the way the web applications that send email actually do so. When everything ran directly on the host I had a Postfix installation that was configured as a relay to my Mail-in-a-Box (initially using the method proposed at https://github.com/mail-in-a-box/mailinabox/pull/212 and later using the method described at https://mailinabox.email/advanced-configuration.html#relaying). The web applications that send email (Piwik, WordPress and www.dhpcs.com) were able to do so without any additional configuration, as PHP’s default mail function used the localhost SMTP server and I’d set www.dhpcs.com’s play-mailer config to use the localhost SMTP server too.

Of course once each application was running in its own container, isolated from the host, their use of the “local” SMTP server meant that they were trying to connect to non-existent SMTP servers within the application containers. Initially I tried finding a way to configure PHP’s default mail function to use a different host, but my brief research yielded no obvious way. Fortunately another option presented itself: setting a specific SMTP server in the configuration for each application. This was simple enough for Piwik (it includes built in support) and www.dhpcs.com (I was already familiar with the play-mailer config). For WordPress I had to install the WP Mail SMTP plugin.

One downside to this approach is that if my Mail-in-a-Box is down when an application attempts to submit mail, the application will present an error to the user (certainly that’s what happens with www.dhpcs.com/contact). With the previous setup the local Postfix relay installation provided the benefit of a queue, so mail would be accepted locally and users would not get errors. But as consolation, one upside to this approach is that each application now uses its own Mail-in-a-Box credentials, so access control is a bit more granular than before.

I should point out that although none of my web applications now use the host’s Postfix installation, the host does still have one and is still setup to relay to my Mail-in-a-Box. The sole purpose of it now is to forward the output of the daily backup Cron job so I can spot any problems.

In conclusion, and having been running this setup for over two months, I do highly recommend Docker and its ecosystem as a means of converting brittle tightly-coupled manually configured stacks into flexible loosely-coupled automatically configured ones. But I will also acknowledge that there are three aspects where there is room for improvement:

  1. Logging. I’m really quite ignorant here. All I know is that when catting/tailing the logs of some of my containers (e.g. the proxy Nginx) it can take a long time to get to the end, and that that might be something to do with Docker JSON encoding container output. I’ve really not investigated logging at all because the setup as a whole has met the goal I set it up for and I’m focused on other things now.
  2. (Security) updates. Prior to Dockerising everything I configured Ubuntu’s unattended-upgrades package and took some comfort in having done so. Now obviously such upgrades won’t cover any vulnerabilities in software running in my containers. So far I’ve just been periodically checking for new versions of the base images I use and rebuilding and redeploying my derived containers when there are. It would be nice to automate this somehow. On the other hand, even without automatic updates, I can at least hope that the setup is more secure in the sense that things are more compartmentalised now.
  3. (Data) volumes. Though Docker does support data volumes and data volume containers, after trying both I actually went with mounting directories from /var/www on the host. At the time that I developed the setup (and at the time of writing this), Docker does not have any built in support for managing data volumes, so there is no way to delete unused volumes. So they were out to avoid accumulating redundant data on disk. I did use data volume containers briefly but switched to host mounts simply to minimise the number of containers managed by my deployment scripts and output by docker ps. If/when Docker gains support for managing data volumes I will likely switch to using them.

*It’s just the server-side for the Android application I said I had an idea for in A Break and which I’ve now begun developing. The idea was originally for a device-side-only application using Wi-Fi Direct, but on the first day of prototyping I ruled out Wi-Fi direct as unreliable. Hence the need for a server-side. There are two consequences of this:

  1. I now have the ideal vehicle by which to solidify the Scala knowledge I’ve picked up from using it in smaller side-projects over the last couple of years: a larger project. I’ll probably say more about the project and Scala in a future post.
  2. The project is now larger in scope than it was when in ”A Break” I mentioned an interest in OCaml & MirageOS, but also much more interesting than it would otherwise have been (now I get to teach myself Scala/Akka/Reactive Programming as part of the project!). Consequently I no longer plan to pursue the OCaml & MirageOS interest.