We’ve covered quite a lot on our journey to automate the creation of one single VPS, hosting multiple websites or webapps. As you can see in the below diagram, we’re deploying to a Digital Ocean droplet, using Terraform and cloud-init. All code goes in git of course. Docker and Dockerfiles have been covered in part II of this series, Docker compose and a couple of web apps in part III.

Today we will be wrapping up this adventure with Nextcloud, Traefik reverse proxy and automatically generated TLS certificates. This post is going to be a long one, so hang in there.

Last time

This is the status of our project:

At last, we will fill in the final blanks.

Nextcloud

Where our previous ‘apps’ were self-explanatory, a onepager and WordPress, Nextcloud might need a bit more explanation.

I remember that over a decade ago I discovered Dropbox and thought it was the best invention since sliced bread. A utility that made sure that my files remained synchronized between my Mac and PC and later my mobile devices. The 2 GB for free got you started and soon I synced my entire homefolder for $10 per month. This was in a time when data privacy wasn’t such a huge concern for your average Joe.

When the open source fever got me, privacy was more prominently on everybody’s agenda. Dropbox negatively made the news a couple of times privacy-wise, so for me it was time to move on. I found ownCloud which was probably one of the first projects I setup on any Linux machine.

The ownCloud project was later forked and a new project was started that soon became even more attractive: Nextcloud. In its core it still syncs your files but currently it does a lot more than that. Not a huge fan of this Google Docs/Office 365-like approach, but you can just ignore what you don’t need, I guess. Let’s set it up!

This is what we’ll be creating:

				
					# tree -a
.
├── .env
├── cache
│   └── Dockerfile
├── db
│   └── Dockerfile
├── docker-compose.yaml
└── nc
    └── Dockerfile

The Nextcloud application consists of much needed Redis cache (nc-cache), a database (nc-db), and the main Nextcloud application (nc-app).

Setup your base:

				
					mkdir /opt/data.devops.tf && cd /opt/data.devops.tf
git init
mkdir db nc cache

Then the .env file:

				
					MYSQL_DATABASE=nextcloud-db
MYSQL_HOST=nc-db
MYSQL_PASSWORD=mkFWDFPOlk96GB32SXc7bhGt
MYSQL_ROOT_PASSWORD=dwfghNW7kl983Dvfwd31G7jH
MYSQL_USER=nextcloud-db-user
NEXTCLOUD_ADMIN_USER=nextcloud-admin
NEXTCLOUD_ADMIN_PASSWORD=grwd-gt87-cwDr-43GG-16gH
NEXTCLOUD_TRUSTED_DOMAINS=data.devops.tf
REDIS_HOST=nc-cache

You should already be familiar with the fact that this file holds some sensitive data. We’ll be using this data as variables in our compose file later. The compose file gets committed to the repo, the .env file absolutely not. Put this file in your .gitignore.

The cache/Dockerfile:

				
					FROM redis:6.2.6-alpine

The db/Dockerfile:

				
					FROM mariadb:10.3.32

The nc/Dockerfile:

				
					FROM nextcloud:22.2.3-apache

Seems no big deal, right? Let’s see how we put it all together in the docker-compose.yaml.

				
					version: "3"

services:
  cache:
    container_name: nc-cache
    build:
      context: ./cache
      dockerfile: ./Dockerfile
    depends_on:
      - nc-db
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - nc_cache_vol:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 10s

  db:
    container_name: nc-db
    build:
      context: ./db
      dockerfile: ./Dockerfile
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - nc_db_vol:/var/lib/mysql
    env_file: .env
    environment:
      MYSQL_DATABASE: $MYSQL_DATABASE
      MYSQL_HOST: $MYSQL_HOST
      MYSQL_PASSWORD: $MYSQL_PASSWORD
      MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD
      MYSQL_USER: $MYSQL_USER
    healthcheck:
      test: ["CMD", "mysql", "--host=$MYSQL_HOST", "--user=$MYSQL_USER", "--password=$MYSQL_PASSWORD", "--silent", "--execute", "SELECT 1;"]
      interval: 10s
      timeout: 2s
      retries: 2
      start_period: 10s

  nc:
    container_name: nc-app
    build:
      context: ./nc
      dockerfile: ./Dockerfile
    depends_on:
      - nc-db
      - nc-cache
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - nc_app_vol:/var/www/html
    ports:
      - 8383:80
    environment:
      env_file: .env
    environment:
      MYSQL_DATABASE: $MYSQL_DATABASE
      MYSQL_HOST: $MYSQL_HOST
      MYSQL_PASSWORD: $MYSQL_PASSWORD
      MYSQL_USER: $MYSQL_USER
      NEXTCLOUD_ADMIN_PASSWORD: $NEXTCLOUD_ADMIN_PASSWORD
      NEXTCLOUD_ADMIN_USER: $NEXTCLOUD_ADMIN_USER
      NEXTCLOUD_TRUSTED_DOMAINS: $NEXTCLOUD_TRUSTED_DOMAINS
      REDIS_HOST: $REDIS_HOST
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

volumes:
  nc_cache_vol: {}
  nc_db_vol: {}
  nc_app_vol: {}

A lot of stuff in there that we’ve already covered in the previous post and therefor already know. So, what’s new or can we expand on a bit?

cache - general. I like to order my services in alphabetical order, but for compose it doesn’t matter. You can control the order with the depends_on As for the service itself: Nextcloud needs fast caching to prevent file locking problems
cache - healthcheck. This is a default healthcheck provided with the redis-cli. Has nothing to do with Docker or compose
db - command. These are actually extra arguments given to the containers entrypoint. You can configure this for MariaDB to work with Nextcloud properly
db - environment. In contrast to our WordPress installation last time, here I’m explicitly stating all variables and their values from the .env It’s not necessary, but personally I like this more. Also, I opted this time to make everything variable, not only sensitive information
nc - environment. Same as above, and I also defined some extra Nextcloud variables which I got straight from the Nexcloud Dockerfile readme. You can also omit them (with the exception of the Redis host) and then you have to configure them the first time you login. But why would you, right?

Fire it up and do some checks:

				
					docker compose up -d
watch docker compose ps
docker compose logs -f

When all seems ready, browse to data.yourdomain.xyz:8383 and you can login with the Nextcloud credentials from the .env file. When needed, bring your stack down with:

				
					docker compose down

When you make edits to one of your Dockerfiles, be it version updates or customizations, you can rebuild them with:

				
					docker compose build
docker compose restart

If some settings don’t seem to stick (adding a new port is a good example) just bring your stack down and up again:

				
					docker compose down
docker compose up -d

Proxies

Old school that I am, the first reverse proxies I encountered were Apache and Nginx in reverse proxy mode, and HAProxy. Except for Apache, I still manage them daily. But the cloud native era brought us many new cool stuff, and reverse proxy-wise this is no exception.

Let me first explain the difference between a regular, ‘forward proxy’ and the ‘reverse proxy’. Check out this image:

An image really tells more than a thousand words.

Forward proxy on the left. Clients (end users) on the same network use a forward proxy to access the internet, using all kinds of servers and services
Reverse proxy on the right. Clients (end users) on the same or different locations access the internet. Access to different webservices of the same company, can be distributed via a reverse proxy to different backend services

To paint an even clearer picture: our project we are working on the last several posts. We have different backend services (one-page HTML, WordPress and Nextcloud), but our Droplet only has 1 public IP by default. We still want to access all our services and don’t want to enter different ports to access them, like we did until this point. We’re going to fix this with our reverse proxy!

In our example we have 1 host with 3 different services. We’re just working in a lab with a test setup. In the real world this can be 10 hosts with 30 different services. Or more. At larger scale we’ll physically move the reverse proxy in front of these hosts and services and make it high available. You get my point; I just want to make sure you get an idea about the practical implementation options.

Traefik

So, forget these old school reverse proxies for a moment, let’s talk Traefik. It does exactly what the former reverse proxies do, but it has the tagline “The Cloud Native Application Proxy – Simplify networking complexity while designing, deploying, and operating applications”. That’s a nice promise. Let me quote a small bit from their site:

Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them.

What sets Traefik apart, besides its many features, is that it automatically discovers the right configuration for your services. The magic happens when Traefik inspects your infrastructure, where it finds relevant information and discovers which service serves which request.

Personally, starting out with Traefik 1 or 2 years ago, I thought it was hell. I just couldn’t understand what was happening. I found the learning curve steep and the documentation didn’t make any sense to me. Some time and experience later, I’ve now used it with Kubernetes and Docker compose, and can testify that in its core it’s actually quite easy.

We’re going to setup the Traefik application first. This is what we’ll be creating:

				
					# tree
.
├── docker-compose.yaml
└── traefik
    ├── acme.json
    └── traefik.yaml

Yep, that’s all. Setup the base:

				
					mkdir /opt/traefik.devops.tf && cd /opt/traefik.devops.tf
git ini
mkdir traefik && cd traefik
touch acme.json && chmod 600 acme.json
vim traefik.yaml

A lot of known steps in there. The acme.json file will be automatically populated by Traefik with our fully valid TLS certificates (more on that later). The privileges to this file need to be limited (the chmod), else Traefik will error out. Our traefik.yaml holds the main configuration:

				
					providers:
  docker:
    exposedByDefault: false
    endpoint: unix:///var/run/docker.sock
    network: traefik

ping: {}

api:
  dashboard: true
  insecure: false
  debug: true

log:
  level: INFO

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      email: support@devops.tf
      storage: /etc/traefik/acme.json
      tlsChallenge: true
      caServer: https://acme-v02.api.letsencrypt.org/directory

I think many of these settings are not hard to figure out what they do. I’ll give a short summary on them and some useful links where necessary:

providers. Of course, the type of provider we are using, Docker in our example. Read more on it here
ping. Can be used for troubleshooting purposes i.e.
api. The functionality we are using from the Traefik API
log. Log settings, level, rotation, e.g.
entrypoints. What are we going to expose that Traefik is going to handle for us? Quite some options here
Automatic TLS generation sweetness. One of the many benefits of using Traefik. Read up on it here

Let’s bring this config together in the docker-compose.yaml:

				
					version: "3"

services:
  traefik:
    image: traefik:2.6.0
    container_name: traefik-main
    restart: unless-stopped
    volumes:
      - ./traefik/acme.json:/etc/traefik/acme.json
      - ./traefik/traefik.yaml:/etc/traefik/traefik.yaml
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - traefik
    ports:
      - 80:80
      - 443:443
    healthcheck:
      test: ["CMD", "traefik", "healthcheck"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
    labels:
      - traefik.enable=true
      - traefik.http.routers.dashboard.rule=Host(`traefik.devops.tf`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      - traefik.http.routers.dashboard.service=api@internal
      - traefik.http.routers.dashboard.tls=true
      - traefik.http.routers.dashboard.tls.certresolver=letsencrypt
      - traefik.http.routers.dashboard.middlewares=auth
      - traefik.http.middlewares.auth.basicauth.users=henkbatelaan:$$apr1$$ceyth2l3$$hg54fcbjhg.bn43df

networks:
  traefik:
    external: true

This needs some explaining:

volumes. At volume we mount our config file and json file. The location and name are important here. Don’t change it
networks. We’ll create a Traefik dedicated Docker network. Here we’re telling Traefik to use this network
ports. As always, the ports we’re exposing. These are the ports Traefik will be listening on and will forward requests to our apps
labels. This is where most of the Traefik magic happens. I can’t explain absolutely every detail without making this post twice as long, but will expand on some of it below

Create the Traefik Docker network:

				
					docker network create traefik

We can’t start Traefik just yet. We need some important edits to our One-page HTML, WordPress and Nextcloud docker-compose.yaml files. But first: labels.

Labels

As you can probably deduce from the content above, Traefik uses labels to determine how it handles and routes incoming traffic flows. First the most important labels from the Traefik Docker compose:

http.routers.dashboard.rule. We’ve enabled the dashboard and API in the traefik.yaml. Here we’re telling Traefik where we can access them
http.routers.dashboard.middlewares=auth. We don’t want unauthenticated access to our dashboard
http.middlewares.auth.basicauth.users. At this label we are determining the user and password to access the dashboard

I think the rest of these labels are self-explanatory. Leaves me with a quick instruction how to generate the user and password (on Debian based distros) that you see in the compose file:

				
					sudo apt search htpasswd
sudo apt install apache2-utils
echo $(htpasswd -nB henkbatelaan) | sed -e s/\\$/\\$\\$/g

How do we connect our apps to Traefik? Yes, that takes a couple of edits to our compose files and, you might have guessed it, a lot of labels! For our example I’m grabbing the WordPress site we setup last time. In a nutshell we need to:

Remove the old ports configuration
Add the Traefik network to each service and at the bottom
Add our Traefik labels to our frontend, the ‘web’ service

And that is exactly what we will be doing. See below the edited and final WordPress docker-compose.yaml file:

				
					version: "3"

services:
  db:
    container_name: wp-db
    build:
      context: ./db
      dockerfile: ./Dockerfile
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - wp_db_vol:/var/lib/mysql
    env_file: .env
    environment:
      MYSQL_DATABASE: wordpress-db
    healthcheck:
      test: ["CMD", "mysql", "--host=wp-db", "--user=$MYSQL_USER", "--password=$MYSQL_PASSWORD", "--silent", "--execute", "SELECT 1;"]
      interval: 10s
      timeout: 2s
      retries: 2
      start_period: 10s
    networks:
      - traefik
  php:
    container_name: wp-php
    build:
      context: ./php
      dockerfile: ./Dockerfile
    depends_on:
      - wp-db
    restart: unless-stopped
    volumes:
      - ./php/php-uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
      - /etc/localtime:/etc/localtime:ro
      - wp_web_vol:/var/www/html
    env_file: .env
    environment:
      WORDPRESS_DB_HOST: wp-db
      WORDPRESS_DB_NAME: wordpress-db
      WORDPRESS_DB_USER: $MYSQL_USER
      WORDPRESS_DB_PASSWORD: $MYSQL_PASSWORD
    healthcheck:
      test: ["CMD", "healthcheck"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
    networks:
      - traefik

  web:
    container_name: wp-web
    build:
      context: ./web
      dockerfile: ./Dockerfile
    depends_on:
      - wp-php
    restart: unless-stopped
    volumes:
      - ./web/nginx.conf:/etc/nginx/conf.d/default.conf
      - /etc/localtime:/etc/localtime:ro
      - wp_web_vol:/var/www/html
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
    networks:
      - traefik
    labels:
      # 1 - Enable because disabled by default in traefik.yml
      - traefik.enable=true
      # 2 - Always redirect to https
      - traefik.http.middlewares.redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.redirect.redirectscheme.permanent=true
      # 3 - Http section
      - traefik.http.routers.wordpress.rule=Host(`wordpress.devops.tf`)
      - traefik.http.routers.wordpress.entrypoints=web
      - traefik.http.routers.wordpress.middlewares=redirect
      # 4 - Https section
      - traefik.http.routers.wordpress-secure.rule=Host(`wordpress.devops.tf`)
      - traefik.http.routers.wordpress-secure.entrypoints=websecure
      - traefik.http.routers.wordpress-secure.service=wordpress-secure
      - traefik.http.services.wordpress-secure.loadbalancer.server.port=80
      # 5 - Connect middlewares to router
      - traefik.http.routers.wordpress-secure.middlewares=compression
      # 6 - All secure middlewares go here
      - traefik.http.middlewares.compression.compress=true
      # 7 - TLS configuration to be used
      - traefik.http.routers.wordpress-secure.tls=true
      - traefik.http.routers.wordpress-secure.tls.certresolver=letsencrypt

volumes:
  wp_db_vol: {}
  wp_web_vol: {}

networks:
  traefik:
    external: true

Everything should be familiar here, except for the label configuration. I’ve commented and numbered them for you:

As the comment says, here we’re enabling Traefik for this service, since we did not enable this by default in the yaml file
We want to redirect all incoming requests directly to TLS
In the http section we’re laying the groundwork to redirect to https
1. We’re specifying the host. Multiple hosts can be defined using commas, for example (`devops.tf`, `wordpress2.devops.tf`)
2. We’re specifying the web entrypoint as defined in yaml
3. We’re using a Traefik middleware to redirect the traffic that comes in. More awesome stuff on middlewares here
In the https section we’re doing more of the same as we did in the http section. Added is our endpoint port, which is port 80
Another middleware. We can use multiple, see the above link
TLS magic. This is so awesome, see the next section

TLS

When I started out in IT and people discovered that sending all your data plain text over the internet is not a wise choice, you could purchase an SSL (Secure Sockets Layer) certificate for a few euros a year and make your site/app secure. You had to jump through some hoops to get one though and it often was not the most user-friendly experience to get it up and running. Not unimportantly, you also must do it all over every 1 or 2 years.

Fortunately, times have changed. Although most people are still calling it ‘SSL Certificates’, it’s actually ‘TLS Certificates’, as in TLS (Transport Layer Security), the successor of SSL. You can still buy your certificates if you want, but you can also automatically generate them, for free, with Let’s Encrypt Certificate Authority. We can bake this in in our Traefik implementation and everything we have been setting up until now has prepared us to do so:

Creation of the json file
The certificatesResolvers settings in the yaml file
The http.routers.<name>.tls.certresolver labels in the compose files

Start it all up and see it in action. Important note up front: your DNS needs to resolve to your droplets IP. Check it with:

				
					host wordpress.yourdomain.xyz

Make sure your WordPress uses your latest config (also mind the removal of volumes we covered last post):

				
					cd /opt/wordpress.devops.tf
docker compose down
docker compose up -d
watch docker compose ps

Fire up the Traefik config

				
					cd /opt/traefik.devops.tf
docker compose up -d
docker compose logs -f

Keeping an eye on the logs is extra important in this case because when you see it fail, you should immediately stop the log output with a CTRL-C and shut it down with a docker compose down. The Let’s Encrypt API is rate limited, and if you hit the limit while troubleshooting, you won’t be able to obtain a valid certificate for a week. So, when you see it going wrong, stop your containers and change the APIs URL from production:

				
					caServer: https://acme-v02.api.letsencrypt.org/directory

To staging:

				
					 caServer: https://acme-staging-v02.api.letsencrypt.org/directory

In the traefik.yaml and try again.

But if everything goes as planned, no errors show up in the log and within a minute or so your valid certificate shows up in the acme.json. This will automatically be renewed every 3 months. Check it out by browsing to wordpress.yourdomain.xyz.

This wraps it up for our config. We’ve got the working WordPress site going with automatic TLS and the full Traefik config. I’ll let you do the one-page HTML and Nextcloud instance on your own, which I’m sure you can manage. Just follow the three above explained steps:

Remove the old ports configuration
Add the Traefik network to each service and at the bottom
Add our Traefik labels to our frontend

And don’t forget to check out the cool Traefik dashboard, that we setup with all this configuring. It’s at https://traefik.yourdomain.com/dashboard/. The full path including the trailing slash is important. Use the name and password you setup before with htpasswd.

Wrapping up

Oh my god this was quite the journey. I’ve set up configurations like this a couple of times, the last couple of years. So, for me this didn’t have a lot of surprises, but to make a few solid blogs about it; that’s some hard work.

This is the result, all the technologies we’ve covered in the last 4 posts:

And now we’re finally done. Before I’m off though, here are a couple more final whims I would like to share:

We started the series off with automating stuff with the DO API, Terraform and cloud-init, but we ended up with doing a lot of manual stuff on our droplet. So, you might ask yourself, what the deal is. Well in all honesty, the answer is time. The idea of one post grew to be 4 posts and I simply had to draw the line somewhere. I’ve automated setups like this more with more Terraform and a lot of Ansible, but it would probably double the post count
This immediately brings me to stuff I missed and hopefully expand on in later blogs. The first things that come to mind is Ansible, way more on Terraform, more building your own images, alternatives to Docker and also a lot more on Traefik, TLS and PKI (Public Key Infrastructure)
I think it’s safe to say that in Enterprise production environments, you won’t come across any Docker compose setups. It’s probably all Kubernetes out there. I decided to create these posts anyway, because on smaller scale this is surely being used, these are still awesome technologies, and it gives you a great introduction into many of the building blocks Enterprise environments are built on

Webservices part IV – Nextcloud, Traefik and TLS

What's on your mind?

What's on your mind?

Greater BV

Open positions

Other links

Search

Follow us!

KubeCon + CloudNativeCon Europe 2022

Run Native Windows Apps in UNIX-like Systems with WINE

What's on your mind?

What's on your mind?

Greater BV

Open positions

Other links

Search

Follow us!