Supervise docker containers with systemd

Poor mans container orchestration.

Published on Mar 16, 2019


For my work on OpenAppStack I’m dealing with Kubernetes a lot, and it’s fun ! However, it’s a super complex monster which can overwhelm you pretty easily. You need to learn a lot of new concepts, and I’m far from fully understanding every component it contains.

Running applications in containers is not only good for isolation but also because containers ship all dependencies bundled, so you don’t need to clutter your host system with dependent (debian) packages. Also, packaging complex applications can be a tiresome work. Saying that I’d like to say Kudos for every packager/ debian maintainer out there who takes on this burden!

So for my personal projects I decided to host my applications the container way. Mainting a kubernetes cluster was a bit of an overkill for those few apps I’m hosting, so I decided to take the simple approach: Using docker containers that are supervised as systemd service.

The main advantage is that it comes very close to the administration experience you with legacy systems (forgive me that term): You can control your services on a host with one frontend (systemd), as you’re used to, regardless if the application runs as regular host application or inside a docker container.

This approach is based on two parts:

Below is an example how to use this role for deploying nextcloud and mariadb as docker containers, and using traefik as ingress controller together with automatic fetching of letsencrypt certs (yeah!). You might need to adopt this to your needs, this snippet is mostly to get the picture how it works. There’s still room for improvements though, but it’s a good start moving to the containerized applications without going full in with Kubernetes. float might be an interesting simple solution as well, and it’s also using systemd-docker. I couldn’t get it stripped down to the bare minimum I wanted though but I hope this will be possible in the future.

- hosts: all

  # Configure and start traefik
  pre_tasks:
    - name: Create traefik config directories
      file:
	path: "{{ item }}"
	state: directory
	mode: '0750'
      with_items:
	- '/etc/traefik/letsencrypt'

    - name: Ensure /etc/traefik/acme.json is present
      copy:
	content: ""
	dest: '/etc/traefik/acme.json'
	force: no
	mode: '0600'

    - name: Deploy traefik config
      template:
	dest: /etc/traefik/traefik.toml
	src: ../templates/traefik/traefik.toml
	mode: '0600'

  tasks:
    - name: Install systemd-docker and python-docker
      package:
	name: '{{ item }}'
      with_items:
	- python-docker
	- systemd-docker

    - name: Add autistici apt gpg key (for i.e. systemd-docker)
      copy:
	src: "apt/deb_autistici_org.gpg"
	dest: "/etc/apt/trusted.gpg.d/deb_autistici_org.gpg"

    - name: Install autistici sources.list
      apt_repository:
	repo: "deb http://deb.autistici.org/urepo ai3/"
	state: present
	update_cache: yes

    - name: Start traefik container
      include_role:
	name: varac.docker-systemd-service
      vars:
	template_unit_path: '../../../templates/docker-systemd-service/unit.j2'
	container_name: traefik
	container_image: traefik:1.7.9-alpine
	container_env:
	  # Set this !
	  SECRET_KEY_BASE: '...'
	container_volumes:
	  - '/var/run/docker.sock:/var/run/docker.sock'
	  - '/etc/traefik/traefik.toml:/traefik.toml'
	  - '/etc/traefik/acme.json:/acme.json'
	container_ports:
	  - '80:80'
	  - '443:443'
	container_args: '--network web'
	container_labels:
	  - 'traefik.frontend.rule=Host:monitor.example.org'
	  - 'traefik.port=80'
	docker_path: '/usr/bin/systemd-docker'

  # Nextcloud mariadb
  - name: Start mariadb container for nextcloud
    include_role:
      name: varac.docker-systemd-service
    vars:
      template_unit_path: '../../../templates/docker-systemd-service/unit.j2'
      container_name: mariadb-nextcloud
      container_image: mariadb
      container_args: '--network nextcloud'
      container_volumes:
	- 'mariadb-nextcloud:/var/lib/mysql'
      container_env:
	MYSQL_ROOT_PASSWORD: "{{ mariadb_nextcloud_root_pw }}"
	MYSQL_PASSWORD: '{{ mariadb_nextcloud_db_pw }}'
	MYSQL_DATABASE: 'nextcloud'
	MYSQL_USER: 'nextcloud'
      container_labels:
	- "traefik.enable=false"
      docker_path: '/usr/bin/systemd-docker'


  # Nextcloud
  #
  - name: Start nextcloud container at https://cloud.example.org
    include_role:
      name: varac.docker-systemd-service
    vars:
      template_unit_path: '../../../templates/docker-systemd-service/unit.j2'
      container_name: nextcloud
      container_image: nextcloud:14
      container_docker_pull: false
      container_args: '--network web'
      container_links:
	- mariadb-nextcloud
      container_volumes:
	- 'nextcloud-html:/var/www/html'
	- 'nextcloud-apps:/var/www/html/custom_apps'
	- 'nextcloud-config:/var/www/html/config'
	- 'nextcloud-data:/var/www/html/data'
      container_labels:
	- 'traefik.backend=nextcloud'
	- 'traefik.frontend.rule=Host:cloud.example.org'
	- 'traefik.docker.network=web'
	- 'traefik.port=80'
      docker_path: '/usr/bin/systemd-docker'
      service_execstartpost: '/bin/sh -c "sleep 2; /usr/bin/docker network connect nextcloud nextcloud"'

And here’s the traefik.toml template I’m using:

defaultEntryPoints = ['http', 'https']
logLevel = 'INFO'

[entryPoints]
  [entryPoints.dashboard]
    address = ':8080'
    [entryPoints.dashboard.auth]
      [entryPoints.dashboard.auth.basic]
        users = ['admin:{{ traefik_admin_pw }}']

  [entryPoints.http]
    address = ':80'
      [entryPoints.http.redirect]
        entryPoint = 'https'
  [entryPoints.https]
    address = ':443'
      [entryPoints.https.tls]
[api]
entrypoint='dashboard'

[acme]
email = '{{ traefik_admin_email }}'
storage = 'acme.json'
entryPoint = 'https'
onHostRule = true
  [acme.httpChallenge]
  entryPoint = 'http'

[docker]
domain = '{{ traefik_docker_domain }}'
watch = true
network = 'web'

And the custom templates/docker-systemd-service/unit.j2:

{% macro params(name, vals) %}
{% for v in vals %}-{{ name }} {{ v }} {% endfor %}
{% endmacro %}
[Unit]
After=docker.service
PartOf=docker.service
Requires=docker.service

[Service]
{% if container_env is defined %}
EnvironmentFile={{ sysconf_dir }}/{{ container_name }}
{% endif %}
ExecStartPre=-/usr/bin/docker rm -f {{ container_name }}
ExecStart=/usr/bin/systemd-docker run --name {{ container_name }} --rm {% if container_env is defined %}--env-file {{ sysconf_dir }}/{{ container_name }} {% endif %}{{ params('v', container_volumes) }}{{ params('p', container_ports) }}{{ params('-link', container_links) }}{{ params('l', container_labels) }}{{ container_args | default('') |trim }} {{ container_image }} {{ container_cmd | default('') | trim }}
{% if service_execstartpost is defined %}ExecStartPost={{ service_execstartpost }}
{% endif %}
ExecStop=/usr/bin/docker stop {{ container_name }}

SyslogIdentifier={{ container_name }}
Restart=always
RestartSec=10s

[Install]
WantedBy=docker.service

And finally the output of the according systemd service status:

# systemctl status nextcloud_container.service
* nextcloud_container.service
   Loaded: loaded (/etc/systemd/system/nextcloud_container.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2019-02-23 23:50:20 CET; 2 weeks 6 days ago
 Main PID: 1491 (systemd-docker)
    Tasks: 19 (limit: 4915)
   Memory: 135.1M
      CPU: 1min 16.505s
   CGroup: /system.slice/nextcloud_container.service
           |-1491 /usr/bin/systemd-docker run --name nextcloud --rm -v nextcloud-html:/var/www/html -v nextcloud-apps:/var/www/html/custom_apps -v nextcloud-config:/var/www/html/config -v nextcloud-data:/var/www/html/data --link mariadb-nextcloud -l traefik.backend=nextcloud -l traefik.frontend.rule=Host:cloud.example.org -l traefik.docker.network=web -l traefik.port=80 --network web nextcloud:14
           |-1701 apache2 -DFOREGROUND
           |-2185 apache2 -DFOREGROUND
           |-2187 apache2 -DFOREGROUND
           |-2188 apache2 -DFOREGROUND
           |-2726 apache2 -DFOREGROUND
           |-2729 apache2 -DFOREGROUND
           |-2730 apache2 -DFOREGROUND
           |-2731 apache2 -DFOREGROUND
           |-2732 apache2 -DFOREGROUND
           |-2796 apache2 -DFOREGROUND
           `-3011 apache2 -DFOREGROUND

Mar 15 08:17:59 moewe nextcloud[1491]: 172.18.0.3 - - [15/Mar/2019:07:17:58 +0000] "GET / HTTP/1.1" 302 1443 "-" "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)"