Giving you a high overview of what we’ll be doing, the plan is to automate the creation of one single VPS, hosting multiple websites or webapps. We’re putting these sites or apps in containers and distribute the incoming requests with a cloud native reversed proxy. Since I’ll be introducing quite a lot of new technologies in this Greater blog series, I’m dividing this into four posts.

Although we can setup the multi webservices on about any platform out there (including AWS, Azure and GCP) I would like to do it on a Digital Ocean (DO) droplet. A Droplet is just a VPS with a non-standard name. Check out my ‘Cloud technology‘ post for some talk about VPS’s, types of cloud, the shared responsibility model and other useful stuff.

The Plan

We’ll be using quite a lot of technologies; some we’ve already discussed but a lot are new to this series. Let’s name them:

The plan is to use Terraform and cloud-init to get a single droplet up and running in a minute or two. We’ll automatically install needed tools like Docker and Docker compose. With the tools we’ll configure and spin up as much containers as your droplet can handle, every container serving one of your websites. Traefik will be our main entry point and will take care for routing to the proper sites and TLS termination.

Digital Ocean steps

If you have an API you can talk to, you can automate stuff. Let’s create an account if you haven’t already and get this API ready to go.

Sign up at the DO signup page https://cloud.digitalocean.com/registrations/new
In the left bar click API and create a new token with ‘Generate New Token’ (not a ‘new Spaces Access Key’)
Put the token in a safe place like a password manager
Now in the left bar go to ‘Settings’ – ‘Security’ and add a SSH key
Paste in your public key as created in the first git post, name it and save
With Terraform we can reference this key, or create one from a local file

Terraform

Our first stop is Terraform. Now for me, starting out with Terraform not too long ago, really felt like the next level. As a non-developer it really felt like developing your infrastructure. And that is what it is: you’re putting your entire infrastructure into code and with a couple of commands you can spin it up or destroy it. Very cool and powerful. Depending on your operating system, find out how to get Terraform here. In a nutshell:

You declare what you want your infrastructure to look like
You plan and apply your configuration
Terraform keeps the configuration it knows to be operational in its state file
At a next plan and apply, Terraform compares your declaration (the .tf files) with the state file (.tfstate) and makes adjustments accordingly

You keep the majority of your code in git (with a few exceptions) and your state file normally in a central location (S3, Azure blob, GitLab, etc.) where it can be shared and locked. For our setup today we keep it locally, but outside version control. Use the above linked exceptions with one small addition: also exclude all tfplan files from git (line 30).

First steps

Getting started with Terraform and git, first setup your project locally.

				
					 cd && mkdir multiweb-do && cd multiweb-do
 git config --global init.defaultBranch main && git init
 vim README.md
 vim .gitignore

You can put your notes in the README and fill the .gitignore with the earlier provided example. Create the providers.tf file:

				
					terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "2.16.0"
    }
  }
}

provider "digitalocean" {
  token = var.do_token
}

Provider config like this can always be found in the Terraform Registry.

Initialize the .terraform directory, providers, and lock file. Let’s get this code online at GitLab. Create a new project by following the steps at ‘Remotes’ in the first git post, commit and push your code to GitLab:

				
					 git add .
 git commit -am "First commit to get us started"
 git remote add origin git@gitlab.com:iohenkies/multiweb-do.git
 git push -u origin main

Out of scope for this post, GitLab detects your .tf file and will ask you from the web interface to store your Terraform state in GitLab. Totally optional but you could give it a try.

Create a droplet

OK let’s create a droplet without a single mouse click. There are dozens of ways to name and organize your .tf files. I like to split them up quite a bit but maintain readability and a logical approach. For a single droplet not too important but know that Terraform simply reads all .tf files and doesn’t care what you put where.

First, I’m creating a security.tf. Here we will be using and adding our SSH key to be created at DO and added to our droplet later. If needed check out SSH key creation at my first git post.

				
					# This will create the SSH key so we can login
resource "digitalocean_ssh_key" "henk_batelaan" {
  name       = "SSH Key Henk Batelaan"
  public_key = file("/Users/iohenkies/.ssh/id_rsa.pub")
}

We could also reference the SSH key we already created at the DO control panel, but I’m now adding it like so. Know that there are multiple ways. Second, a simple main.tf where we are creating a resource of type digitalocean_droplet called web01 with a couple of key/value pairs and putting it in a newly created project called ‘multisite’:

				
					# We are creating a separate project for our web01 droplet
resource "digitalocean_project" "multisite" {
  name        = "multisite"
  description = "The is our multisite project with one droplet"
  purpose     = "Web Application"
  environment = "Development"
  resources   = [digitalocean_droplet.web01.urn]
}

# Create a new Web droplet for our multisite project in the Amsterdam region
resource "digitalocean_droplet" "web01" {
  image    = "ubuntu-20-04-x64"
  name     = "web01"
  region   = var.region
  size     = var.size
  tags     = var.tags
  backups  = var.backup
  ssh_keys = [digitalocean_ssh_key.henk_batelaan.fingerprint]
}

I’m not a magician creating all this stuff out of thin air. No, we’ve got some excellent documentation. For the structure and keys check out the provider’s docs and for the possible values the API reference guide.

Variables

As you might have noticed, I’ve been using a couple of var.keyname. We need to declare these values. Create a variables.tf file:

				
					# Let’s enable backups by default
variable "backup" {
  type        = string
  default     = true
  description = "The Digital Ocean default is false; we want it to be true"
}

# Declaration of the Digital Ocean API token
variable "do_token" {
  type        = string
  description = "This is our token to authorize against the Digital Ocean API"
}

# Declaration of the region variable
variable "region" {
  type        = string
  default     = "ams3"
  description = "This is our region variable. The default is Amsterdam 3"
}


# Declaration of the size variable with a default size specified
variable "size" {
  type        = string
  default     = "s-1vcpu-1gb"
  description = "This is our size variable. The default is the smallest size possible"
}

# Default droplet tags
variable "tags" {
  type        = list(any)
  default     = ["ubuntu", "docker", "terraform"]
  description = "Just some default tags going with our droplet"
}

A bit verbose for this example, but you catch my drift. In the main.tf we’re referencing the region, size, tags and backups. Since we’ve specified a default, these get filled in automatically.

The ‘do_token’ is a special one though. Of course, we’re not hardcoding this in any of our code that we commit to git (this is a security hazard). But we need Terraform to authenticate, so how can we access the token? Well, there are many ways, and it depends on what provider you’re using, if you need to collaborate, if you are using pipelines, etc. For our simple example you can use one of two ways:

Use export TF_VAR_do_token="your token here" from the command line
Create a file terraform.tfvars with one line do_token="your token here"

Both without the single quotes. Check your .gitignore, you’ll see that terraform.tfvars will be ignored by git. I also want to point out that when you want to run everything that we will be creating, at the same time, you’ll need a slightly bigger droplet than the default. Check your options out with (you’ll need to install ‘jq’ first):

				
					 DIGITALOCEAN_TOKEN='insert your token'
 curl -X GET \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  "https://api.digitalocean.com/v2/droplets?page=1&per_page=1" | jq

Outputs

Last file to be created is a outputs.tf. Here we’re putting stuff Terraform needs to spit out after every run or after running the terraform output command. In this case we’ll immediately get the IP address we can use to SSH to.

				
					output "web01_ip" {
  value = digitalocean_droplet.web01.ipv4_address
}

This gets everything ready to deploy your droplet!
Format and check your code:

				
					
 terraform init
 terraform fmt
 terraform validate

Plan and apply your code:

				
					 terraform plan --out firstrun.tfplan
 terraform apply firstrun.tfplan

Although you can omit the plan file, it’s generally good practice to use it. This run will create our resources and spit out the IP address we can connect to. A FQDN (Fully Qualified Domain Name) would be better, but I can’t be sure you have a valid domain name yet.

SSH into your shiny new droplet:

				
					 ssh root@ipaddress

You can also check the web interface for the project, droplet, backup, tags, SSH key, etc. Next, we will make a couple of important additions to our script. First destroy everything:

				
					 terraform destroy

User data

Going from such a default installation to something more to your liking is easier than you might think. I do want to point out that there are multiple strategies to setup your default server, too many to sum up, really.

In this context I would like to differentiate two strategies:

Develop – Deploy – Configure. This is the more traditional deployment model using mutable infrastructure
1. Mutable as in infrastructure you deploy and can and will change afterwards
2. Starting with a distro’s base image and configuring stuff with cloud-init and/or Ansible is an example here
Develop – Configure – Deploy. This is the deployment model that has gotten more traction in the last several years, using immutable infrastructure
1. Immutable as in you deploy something and never change it after deployment
2. If you do need a configuration change, you edit your base image, deploy a new server, and kill off the old server
3. Creation and deployment of your own images using Packer is an example here

I will probably do a post on the second option soon (specifically Packer), but for this post I’m using the first option with the more traditional Develop – Deploy – Configure model.

So now that we have the theory out of the way, let’s get practical. We started out with the default DO Ubuntu image, and I would like to add some customization. In DO and many other providers, we can use the ‘user data’, where you can use cloud-init syntax to setup your instance. We can add our config to the already created ‘digitalocean_droplet’ resource as an extra configuration parameter (pasting the full code for completeness).

				
					# Create a new Web droplet for our multisite project in the Amsterdam region
resource "digitalocean_droplet" "web01" {
  image     = "ubuntu-20-04-x64"
  name      = "web01"
  region    = var.region
  size      = var.size
  tags      = var.tags
  backups   = var.backup
  ssh_keys  = [digitalocean_ssh_key.henk_batelaan.fingerprint]
  user_data = <<EOF
#cloud-config
apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
groups:
  - docker
users:
  - name: cloud_deploy
    ssh-authorized-keys:
      - ssh-rsa <removed actual key, insert your own public key>
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: [sudo, docker]
    shell: /bin/bash
packages:
  - apt-transport-https
  - ca-certificates
  - containerd.io
  - curl
  - gnupg-agent
  - dnsutils
  - docker-ce
  - docker-ce-cli
  - git
  - jq
  - nmap
  - psmisc
  - screen
  - ufw
  - vim
package_update: true
package_upgrade: true
write_files:
  - path: /etc/ssh/sshd_config
    content: |
      PrintMotd no
      Banner none
      HostKey /etc/ssh/ssh_host_rsa_key
      HostKey /etc/ssh/ssh_host_ecdsa_key
      HostKey /etc/ssh/ssh_host_ed25519_key
      SyslogFacility AUTHPRIV
      LogLevel VERBOSE
      PermitRootLogin no
      MaxAuthTries 3
      MaxSessions 2
      AuthorizedKeysFile .ssh/authorized_keys
      PasswordAuthentication no
      ChallengeResponseAuthentication no
      GSSAPIAuthentication no
      GSSAPICleanupCredentials no
      UsePAM yes
      AllowAgentForwarding no
      AllowTcpForwarding no
      X11Forwarding no
      TCPKeepAlive no
      Compression no
      ClientAliveCountMax 2
      UseDNS no
      AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
      AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
      AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
      AcceptEnv XMODIFIERS
      Subsystem sftp /usr/lib/openssh/sftp-server
runcmd:
  - apt-get update
  - timedatectl set-timezone Europe/Amsterdam
  - timedatectl set-ntp on
  - ufw allow 22/tcp
  - ufw enable
  - sed -i -e '/pam_motd.so/s/^/# /g' /etc/pam.d/sshd
  - sed -i -e '/#startup_message/s/^#//' /etc/screenrc
  - reboot
EOF
}

This does all kinds of simple, yet awesome stuff. Just do a new terraform plan and apply as explained earlier and after a couple of minutes login to your new droplet:

				
					 ssh cloud_deploy@ip_address

The droplet is done under a minute, but remember that we’re adding all kinds of stuff with cloud-init. If you login too soon, you will end up on a system that is half configured or not reachable at all. So for this reason, just give it about 10 minutes to do all the magic.

I will explain what actually happened here, next time when we continue setting up our webserver. You can probably guess most options on your own, but I will elaborate on a couple.

For now, delete your instance when you’re ready:

				
					 terraform destroy

See you next time.

Webservices part I – Provider, Terraform and cloud-init

The Plan

Digital Ocean steps

Terraform

First steps

Create a droplet

Variables

Outputs

User data

What's on your mind?

What's on your mind?

Greater BV

Open positions

Other links

Search

Follow us!

The Plan

Digital Ocean steps

Terraform

First steps

Create a droplet

Variables

Outputs

User data

Kubernetes, Kubernetes Certs and Getting Your Feet Wet

Gran Canaria, here we come!

What's on your mind?

What's on your mind?

Greater BV

Open positions

Other links

Search

Follow us!