Docker on Google Compute Engine: the easiest way to deploy a Gitlab runner

6 min readAug 24, 2019

Gitlab CE is one of the best options available today for self-hosting Git, and one of its best features is Gitlab CI, its continuous integration service.

Gitlab CI works off decentralized runners. Your pipelines don’t run on your Gitlab instance itself, but on Gitlab runners that can be hosted in the cloud or even on laptops.

On top of that, Gitlab runners can have different execution strategies, the most popular being:

shell: The simplest kind, which runs jobs directly on the machine.
docker: Runs jobs in containers.
docker-machine: Like docker runners, but can spin up runner instances on-the-fly (this requires certain permissions on your cloud provider).
kubernetes: Like the docker executor, but runs in a k8s cluster.

You really want the container-based executors

Security issues aside, the shell executor makes it really easy to fall into situations where:

Software gets installed into (or removed from) the runner to get pipelines working.
That specific runner gets decommissioned — perhaps you want to change where your runners are hosted, or you want to try a different execution method.
No one knows what went into the runner (if you have enough runners across your org, you may have no idea where it even is). Panic ensues and pipelines stop working.

Managing docker runners, and building images

For the rest of this post I’ll talk about the docker executor:

docker-machine is conceptually similar but because it spins up runner instances dynamically (usually taking a couple minutes minimum), the resulting build times don’t fit many workloads.
The kubernetes executor also works similarly, but a k8s cluster itself requires some expertise to set up.

The docker executor makes your jobs to be reproducible (a good thing) and forces you to keep all dependencies within your gitlab-ci.yml file (also a good thing).

There are three main concerns that crop up when using the docker executor:

When using the runner this way to build a docker image (called a docker-in-docker build), there specific things you need to do to get it working — either use a docker:dind service, or mount the docker socket into the container. I usually use the latter method.
Caching is more complex. If you are building docker images, use cache layers.
Gitlab runners can and do run out of space or need maintenance, and installing Docker on an instance and setting up Gitlab runner does take a bit of work. How can we solve this?

Introducing Containers on Google Compute Engine

A widely underappreciated feature of Google Compute is its container support — if you don’t have a Kubernetes cluster or just want to try out a container quickly, it’s very useful.

Specifying a container image to deploy into the VM

When creating your compute VM, simply check “Deploy a container image to this VM instance”. Then put in the image name, in my case gitlab/gitlab-runner:v11.11.2. Note: you probably want to increase the size of the boot disk as well.

Next, expand the “Advanced container options” link.

Check the “Run as privileged” option.

We need to mount the VM’s docker socket into the gitlab-runner container so runner can in turn pass it to the jobs it creates.

In the “Volume mounts” section, add a volume of type “Directory” and mount /var/run.docker.sock from the Host (the VM we are creating) into the Mount path (the gitlab-runner container that the VM will automatically run).

We are all set, but remember an important concept: Docker containers are ephemeral — all data will be reset if you restart your VM. This is actually useful if your VM runs out of space, but we want to ensure that the gitlab-runner configuration will not reset if that happens.

To solve this, we need to create a separate GCP disk that will be mounted into the VM, which the VM then mounts into the container’s /etc/gitlab-runner, where configuration is kept.

Navigate to the bottom of the VM setup screen (past the “Advanced container options” section) to the “Additional disks” section.

Click on “Add new disk”.

You really don’t need much here, even a 1GB disk would do. You’ll notice a message at the bottom of this section saying that you can now mount additional disks to the container.

Click done, and scroll back up to the “Advanced container options->Volume Mounts” section. Add a NEW volume, but with the “Disk” type this time, and choose the disk you created. Mount it into /etc/gitlab-runner in the container you are about to create.

Click Done, and we are nearly there. Create the VM instance.

Setting up the Runner

Remember — we are running a gitlab-runner container in the VM, so we need to get into the VM, then into the container.

After the VM starts, SSH into it from the Cloud Console, or if you have gcloud set up in your terminal, type this:

gcloud compute ssh [instance-name]

Run docker ps to get the name or container ID of the gitlab-runner container — note that it may take a few minutes for the container to be created in the VM. Then start a Bash session in the container:

#get the container ID or instance name of the gitlab-runner container. It may take a few minutes to start.
$ docker ps
$ docker exec -it [container ID or name] bash

In the example above, the command would be “docker exec -it klt-shared-gitlab-runner-xqtm bash”

You are now in the Gitlab runner proper. Set up the runner, providing the coordinator URL and token as per the standard setup.

# In the CONTAINER
root@gitlab-runner $ gitlab-runner register# Edit the config file
root@gitlab-runner $ vi /etc/gitlab-runner/config.toml

As a last step, to get Docker builds running you will need to configure the /etc/gitlab-runner/config.toml file as follows (covered in gitlab.com here):

Set runners.docker.image to “docker:stable”
Add /var/run/docker.sock:/var/run/docker.sock to runners.docker.volumes. Essentially, you are mounting the Docker socket (which, if you remember, you mounted from the VM into the runner) from the runner into the containers it creates.

All done!

All done! You can restart the runner in two ways:

“gitlab-runner stop/start” in the container
Type exit to go back to the VM, then “docker restart” to restart the runner.

You now have a gitlab-runner which lives in a VM that you can:

Reset to solve disk space issues or any outages.
Reproduce, upgrade easily. Just attach the config disk where needed.

To recap:

A persistent disk is created to store the gitlab-runner configuration. This is added to the VM, which then mounts it into the gitlab-runner container.
Mounting the Docker socket from the VM into the gitlab-runner container allows it to use the docker executor, which creates a container for every job. If the job containers themselves need to work with Docker, (e.g. create Docker images), then mounting the Docker socket into the job containers as well (by modifying config.toml) allows this.

Fun fact: Since we are creating a gitlab-runner container in a VM, which in turn creates containerized jobs, this is technically a Docker-in-Docker-in-Docker strategy!

P.S. A very common issue with Gitlab runners is running out of space because of dangling images. To solve this, we simply need to SSH into the VM and execute docker system prune -a , which will remove unused and dangling images.