We’ve covered quite a lot on our journey to automate the creation of one single VPS, hosting multiple websites or webapps. As you can see in the below diagram, we’re deploying to a Digital Ocean droplet, using Terraform and cloud-init. All code goes in git of course. Docker and Dockerfiles have been covered in part II of this series, Docker compose and a couple of web apps in part III.
Today we will be wrapping up this adventure with Nextcloud, Traefik reverse proxy and automatically generated TLS certificates. This post is going to be a long one, so hang in there.
Last time
This is the status of our project:
At last, we will fill in the final blanks.
Nextcloud
Where our previous ‘apps’ were self-explanatory, a onepager and WordPress, Nextcloud might need a bit more explanation.
I remember that over a decade ago I discovered Dropbox and thought it was the best invention since sliced bread. A utility that made sure that my files remained synchronized between my Mac and PC and later my mobile devices. The 2 GB for free got you started and soon I synced my entire homefolder for $10 per month. This was in a time when data privacy wasn’t such a huge concern for your average Joe.
When the open source fever got me, privacy was more prominently on everybody’s agenda. Dropbox negatively made the news a couple of times privacy-wise, so for me it was time to move on. I found ownCloud which was probably one of the first projects I setup on any Linux machine.
The ownCloud project was later forked and a new project was started that soon became even more attractive: Nextcloud. In its core it still syncs your files but currently it does a lot more than that. Not a huge fan of this Google Docs/Office 365-like approach, but you can just ignore what you don’t need, I guess. Let’s set it up!
This is what we’ll be creating:
# tree -a
.
├── .env
├── cache
│ └── Dockerfile
├── db
│ └── Dockerfile
├── docker-compose.yaml
└── nc
└── Dockerfile
The Nextcloud application consists of much needed Redis cache (nc-cache), a database (nc-db), and the main Nextcloud application (nc-app).
Setup your base:
mkdir /opt/data.devops.tf && cd /opt/data.devops.tf
git init
mkdir db nc cache
.env file:
MYSQL_DATABASE=nextcloud-db
MYSQL_HOST=nc-db
MYSQL_PASSWORD=mkFWDFPOlk96GB32SXc7bhGt
MYSQL_ROOT_PASSWORD=dwfghNW7kl983Dvfwd31G7jH
MYSQL_USER=nextcloud-db-user
NEXTCLOUD_ADMIN_USER=nextcloud-admin
NEXTCLOUD_ADMIN_PASSWORD=grwd-gt87-cwDr-43GG-16gH
NEXTCLOUD_TRUSTED_DOMAINS=data.devops.tf
REDIS_HOST=nc-cache
.env file absolutely not. Put this file in your .gitignore. The cache/Dockerfile:
FROM redis:6.2.6-alpine
The db/Dockerfile:
FROM mariadb:10.3.32
The nc/Dockerfile:
FROM nextcloud:22.2.3-apache
docker-compose.yaml.
version: "3"
services:
cache:
container_name: nc-cache
build:
context: ./cache
dockerfile: ./Dockerfile
depends_on:
- nc-db
restart: unless-stopped
volumes:
- /etc/localtime:/etc/localtime:ro
- nc_cache_vol:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 3
start_period: 10s
db:
container_name: nc-db
build:
context: ./db
dockerfile: ./Dockerfile
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
restart: unless-stopped
volumes:
- /etc/localtime:/etc/localtime:ro
- nc_db_vol:/var/lib/mysql
env_file: .env
environment:
MYSQL_DATABASE: $MYSQL_DATABASE
MYSQL_HOST: $MYSQL_HOST
MYSQL_PASSWORD: $MYSQL_PASSWORD
MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD
MYSQL_USER: $MYSQL_USER
healthcheck:
test: ["CMD", "mysql", "--host=$MYSQL_HOST", "--user=$MYSQL_USER", "--password=$MYSQL_PASSWORD", "--silent", "--execute", "SELECT 1;"]
interval: 10s
timeout: 2s
retries: 2
start_period: 10s
nc:
container_name: nc-app
build:
context: ./nc
dockerfile: ./Dockerfile
depends_on:
- nc-db
- nc-cache
restart: unless-stopped
volumes:
- /etc/localtime:/etc/localtime:ro
- nc_app_vol:/var/www/html
ports:
- 8383:80
environment:
env_file: .env
environment:
MYSQL_DATABASE: $MYSQL_DATABASE
MYSQL_HOST: $MYSQL_HOST
MYSQL_PASSWORD: $MYSQL_PASSWORD
MYSQL_USER: $MYSQL_USER
NEXTCLOUD_ADMIN_PASSWORD: $NEXTCLOUD_ADMIN_PASSWORD
NEXTCLOUD_ADMIN_USER: $NEXTCLOUD_ADMIN_USER
NEXTCLOUD_TRUSTED_DOMAINS: $NEXTCLOUD_TRUSTED_DOMAINS
REDIS_HOST: $REDIS_HOST
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
volumes:
nc_cache_vol: {}
nc_db_vol: {}
nc_app_vol: {}
A lot of stuff in there that we’ve already covered in the previous post and therefor already know. So, what’s new or can we expand on a bit?
cache - general. I like to order my services in alphabetical order, but for compose it doesn’t matter. You can control the order with thedepends_onAs for the service itself: Nextcloud needs fast caching to prevent file locking problemscache - healthcheck. This is a default healthcheck provided with the redis-cli. Has nothing to do with Docker or composedb - command. These are actually extra arguments given to the containers entrypoint. You can configure this for MariaDB to work with Nextcloud properlydb - environment. In contrast to our WordPress installation last time, here I’m explicitly stating all variables and their values from the.envIt’s not necessary, but personally I like this more. Also, I opted this time to make everything variable, not only sensitive informationnc - environment. Same as above, and I also defined some extra Nextcloud variables which I got straight from the Nexcloud Dockerfile readme. You can also omit them (with the exception of the Redis host) and then you have to configure them the first time you login. But why would you, right?
Fire it up and do some checks:
docker compose up -d
watch docker compose ps
docker compose logs -f
.env file.
When needed, bring your stack down with:
docker compose down
When you make edits to one of your Dockerfiles, be it version updates or customizations, you can rebuild them with:
docker compose build
docker compose restart
If some settings don’t seem to stick (adding a new port is a good example) just bring your stack down and up again:
docker compose down
docker compose up -d
Proxies
Old school that I am, the first reverse proxies I encountered were Apache and Nginx in reverse proxy mode, and HAProxy. Except for Apache, I still manage them daily. But the cloud native era brought us many new cool stuff, and reverse proxy-wise this is no exception.
Let me first explain the difference between a regular, ‘forward proxy’ and the ‘reverse proxy’. Check out this image:
An image really tells more than a thousand words.
- Forward proxy on the left. Clients (end users) on the same network use a forward proxy to access the internet, using all kinds of servers and services
- Reverse proxy on the right. Clients (end users) on the same or different locations access the internet. Access to different webservices of the same company, can be distributed via a reverse proxy to different backend services
To paint an even clearer picture: our project we are working on the last several posts. We have different backend services (one-page HTML, WordPress and Nextcloud), but our Droplet only has 1 public IP by default. We still want to access all our services and don’t want to enter different ports to access them, like we did until this point. We’re going to fix this with our reverse proxy!
In our example we have 1 host with 3 different services. We’re just working in a lab with a test setup. In the real world this can be 10 hosts with 30 different services. Or more. At larger scale we’ll physically move the reverse proxy in front of these hosts and services and make it high available. You get my point; I just want to make sure you get an idea about the practical implementation options.
Traefik
So, forget these old school reverse proxies for a moment, let’s talk Traefik. It does exactly what the former reverse proxies do, but it has the tagline “The Cloud Native Application Proxy – Simplify networking complexity while designing, deploying, and operating applications”. That’s a nice promise. Let me quote a small bit from their site:
Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them.
What sets Traefik apart, besides its many features, is that it automatically discovers the right configuration for your services. The magic happens when Traefik inspects your infrastructure, where it finds relevant information and discovers which service serves which request.
Personally, starting out with Traefik 1 or 2 years ago, I thought it was hell. I just couldn’t understand what was happening. I found the learning curve steep and the documentation didn’t make any sense to me. Some time and experience later, I’ve now used it with Kubernetes and Docker compose, and can testify that in its core it’s actually quite easy.
We’re going to setup the Traefik application first. This is what we’ll be creating:
# tree
.
├── docker-compose.yaml
└── traefik
├── acme.json
└── traefik.yaml
Yep, that’s all. Setup the base:
mkdir /opt/traefik.devops.tf && cd /opt/traefik.devops.tf
git ini
mkdir traefik && cd traefik
touch acme.json && chmod 600 acme.json
vim traefik.yaml
acme.json file will be automatically populated by Traefik with our fully valid TLS certificates (more on that later). The privileges to this file need to be limited (the chmod), else Traefik will error out.
Our traefik.yaml holds the main configuration: 
providers:
docker:
exposedByDefault: false
endpoint: unix:///var/run/docker.sock
network: traefik
ping: {}
api:
dashboard: true
insecure: false
debug: true
log:
level: INFO
entryPoints:
web:
address: ":80"
websecure:
address: ":443"
certificatesResolvers:
letsencrypt:
acme:
email: support@devops.tf
storage: /etc/traefik/acme.json
tlsChallenge: true
caServer: https://acme-v02.api.letsencrypt.org/directory
providers. Of course, the type of provider we are using, Docker in our example. Read more on it hereping. Can be used for troubleshooting purposes i.e.api. The functionality we are using from the Traefik APIlog. Log settings, level, rotation, e.g.entrypoints. What are we going to expose that Traefik is going to handle for us? Quite some options here- Automatic TLS generation sweetness. One of the many benefits of using Traefik. Read up on it here
docker-compose.yaml:
version: "3"
services:
traefik:
image: traefik:2.6.0
container_name: traefik-main
restart: unless-stopped
volumes:
- ./traefik/acme.json:/etc/traefik/acme.json
- ./traefik/traefik.yaml:/etc/traefik/traefik.yaml
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock
networks:
- traefik
ports:
- 80:80
- 443:443
healthcheck:
test: ["CMD", "traefik", "healthcheck"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
labels:
- traefik.enable=true
- traefik.http.routers.dashboard.rule=Host(`traefik.devops.tf`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
- traefik.http.routers.dashboard.service=api@internal
- traefik.http.routers.dashboard.tls=true
- traefik.http.routers.dashboard.tls.certresolver=letsencrypt
- traefik.http.routers.dashboard.middlewares=auth
- traefik.http.middlewares.auth.basicauth.users=henkbatelaan:$$apr1$$ceyth2l3$$hg54fcbjhg.bn43df
networks:
traefik:
external: true
volumes. At volume we mount our config file andjsonfile. The location and name are important here. Don’t change itnetworks. We’ll create a Traefik dedicated Docker network. Here we’re telling Traefik to use this networkports. As always, the ports we’re exposing. These are the ports Traefik will be listening on and will forward requests to our appslabels. This is where most of the Traefik magic happens. I can’t explain absolutely every detail without making this post twice as long, but will expand on some of it below
docker network create traefik
We can’t start Traefik just yet. We need some important edits to our One-page HTML, WordPress and Nextcloud docker-compose.yaml files. But first: labels.
Labels
As you can probably deduce from the content above, Traefik uses labels to determine how it handles and routes incoming traffic flows. First the most important labels from the Traefik Docker compose:
http.routers.dashboard.rule. We’ve enabled the dashboard and API in thetraefik.yaml. Here we’re telling Traefik where we can access themhttp.routers.dashboard.middlewares=auth. We don’t want unauthenticated access to our dashboardhttp.middlewares.auth.basicauth.users. At this label we are determining the user and password to access the dashboard
I think the rest of these labels are self-explanatory. Leaves me with a quick instruction how to generate the user and password (on Debian based distros) that you see in the compose file:
sudo apt search htpasswd
sudo apt install apache2-utils
echo $(htpasswd -nB henkbatelaan) | sed -e s/\\$/\\$\\$/g
- Remove the old ports configuration
- Add the Traefik network to each service and at the bottom
- Add our Traefik labels to our frontend, the ‘web’ service
docker-compose.yaml file:
version: "3"
services:
db:
container_name: wp-db
build:
context: ./db
dockerfile: ./Dockerfile
restart: unless-stopped
volumes:
- /etc/localtime:/etc/localtime:ro
- wp_db_vol:/var/lib/mysql
env_file: .env
environment:
MYSQL_DATABASE: wordpress-db
healthcheck:
test: ["CMD", "mysql", "--host=wp-db", "--user=$MYSQL_USER", "--password=$MYSQL_PASSWORD", "--silent", "--execute", "SELECT 1;"]
interval: 10s
timeout: 2s
retries: 2
start_period: 10s
networks:
- traefik
php:
container_name: wp-php
build:
context: ./php
dockerfile: ./Dockerfile
depends_on:
- wp-db
restart: unless-stopped
volumes:
- ./php/php-uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
- /etc/localtime:/etc/localtime:ro
- wp_web_vol:/var/www/html
env_file: .env
environment:
WORDPRESS_DB_HOST: wp-db
WORDPRESS_DB_NAME: wordpress-db
WORDPRESS_DB_USER: $MYSQL_USER
WORDPRESS_DB_PASSWORD: $MYSQL_PASSWORD
healthcheck:
test: ["CMD", "healthcheck"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
networks:
- traefik
web:
container_name: wp-web
build:
context: ./web
dockerfile: ./Dockerfile
depends_on:
- wp-php
restart: unless-stopped
volumes:
- ./web/nginx.conf:/etc/nginx/conf.d/default.conf
- /etc/localtime:/etc/localtime:ro
- wp_web_vol:/var/www/html
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
networks:
- traefik
labels:
# 1 - Enable because disabled by default in traefik.yml
- traefik.enable=true
# 2 - Always redirect to https
- traefik.http.middlewares.redirect.redirectscheme.scheme=https
- traefik.http.middlewares.redirect.redirectscheme.permanent=true
# 3 - Http section
- traefik.http.routers.wordpress.rule=Host(`wordpress.devops.tf`)
- traefik.http.routers.wordpress.entrypoints=web
- traefik.http.routers.wordpress.middlewares=redirect
# 4 - Https section
- traefik.http.routers.wordpress-secure.rule=Host(`wordpress.devops.tf`)
- traefik.http.routers.wordpress-secure.entrypoints=websecure
- traefik.http.routers.wordpress-secure.service=wordpress-secure
- traefik.http.services.wordpress-secure.loadbalancer.server.port=80
# 5 - Connect middlewares to router
- traefik.http.routers.wordpress-secure.middlewares=compression
# 6 - All secure middlewares go here
- traefik.http.middlewares.compression.compress=true
# 7 - TLS configuration to be used
- traefik.http.routers.wordpress-secure.tls=true
- traefik.http.routers.wordpress-secure.tls.certresolver=letsencrypt
volumes:
wp_db_vol: {}
wp_web_vol: {}
networks:
traefik:
external: true
- As the comment says, here we’re enabling Traefik for this service, since we did not enable this by default in the
yamlfile - We want to redirect all incoming requests directly to TLS
- In the http section we’re laying the groundwork to redirect to https
- We’re specifying the host. Multiple hosts can be defined using commas, for example (`
devops.tf`, `wordpress2.devops.tf`) - We’re specifying the
webentrypoint as defined inyaml - We’re using a Traefik middleware to redirect the traffic that comes in. More awesome stuff on middlewares here
- We’re specifying the host. Multiple hosts can be defined using commas, for example (`
- In the https section we’re doing more of the same as we did in the http section. Added is our endpoint port, which is port 80
- Another middleware. We can use multiple, see the above link
- TLS magic. This is so awesome, see the next section
When I started out in IT and people discovered that sending all your data plain text over the internet is not a wise choice, you could purchase an SSL (Secure Sockets Layer) certificate for a few euros a year and make your site/app secure. You had to jump through some hoops to get one though and it often was not the most user-friendly experience to get it up and running. Not unimportantly, you also must do it all over every 1 or 2 years.
- Creation of the
jsonfile - The
certificatesResolverssettings in theyamlfile - The
http.routers.<name>.tls.certresolverlabels in the compose files
host wordpress.yourdomain.xyz
Make sure your WordPress uses your latest config (also mind the removal of volumes we covered last post):
cd /opt/wordpress.devops.tf
docker compose down
docker compose up -d
watch docker compose ps
Fire up the Traefik config
cd /opt/traefik.devops.tf
docker compose up -d
docker compose logs -f
docker compose down. The Let’s Encrypt API is rate limited, and if you hit the limit while troubleshooting, you won’t be able to obtain a valid certificate for a week. So, when you see it going wrong, stop your containers and change the APIs URL from production:
caServer: https://acme-v02.api.letsencrypt.org/directory
To staging:
caServer: https://acme-staging-v02.api.letsencrypt.org/directory
In the traefik.yaml and try again.
But if everything goes as planned, no errors show up in the log and within a minute or so your valid certificate shows up in the acme.json. This will automatically be renewed every 3 months. Check it out by browsing to wordpress.yourdomain.xyz.
This wraps it up for our config. We’ve got the working WordPress site going with automatic TLS and the full Traefik config. I’ll let you do the one-page HTML and Nextcloud instance on your own, which I’m sure you can manage. Just follow the three above explained steps:
- Remove the old ports configuration
- Add the Traefik network to each service and at the bottom
- Add our Traefik labels to our frontend
And don’t forget to check out the cool Traefik dashboard, that we setup with all this configuring. It’s at https://traefik.yourdomain.com/dashboard/. The full path including the trailing slash is important. Use the name and password you setup before with htpasswd.
Wrapping up
Oh my god this was quite the journey. I’ve set up configurations like this a couple of times, the last couple of years. So, for me this didn’t have a lot of surprises, but to make a few solid blogs about it; that’s some hard work.
This is the result, all the technologies we’ve covered in the last 4 posts:
And now we’re finally done. Before I’m off though, here are a couple more final whims I would like to share:
- We started the series off with automating stuff with the DO API, Terraform and cloud-init, but we ended up with doing a lot of manual stuff on our droplet. So, you might ask yourself, what the deal is. Well in all honesty, the answer is time. The idea of one post grew to be 4 posts and I simply had to draw the line somewhere. I’ve automated setups like this more with more Terraform and a lot of Ansible, but it would probably double the post count
- This immediately brings me to stuff I missed and hopefully expand on in later blogs. The first things that come to mind is Ansible, way more on Terraform, more building your own images, alternatives to Docker and also a lot more on Traefik, TLS and PKI (Public Key Infrastructure)
- I think it’s safe to say that in Enterprise production environments, you won’t come across any Docker compose setups. It’s probably all Kubernetes out there. I decided to create these posts anyway, because on smaller scale this is surely being used, these are still awesome technologies, and it gives you a great introduction into many of the building blocks Enterprise environments are built on
I’m a passionate, communicative go-getter and highly motivated to build, maintain and improve a stable and effective IT infrastructure at different sized companies. My hearth is with open source, Linux, DevOps, Kubernetes and everything that is cloud native.
