Working on Multiple Web Projects with Docker Compose and Traefik

A single project with Docker Compose

I use docker compose to manage local development instances of these projects. A typical compose file for a web project might look like this:

 1# proj/compose.yaml
 2services:
 3  db:
 4    image: "postgres"
 5    environment:
 6      POSTGRES_DB: "proj"
 7      POSTGRES_USER: "user"
 8      POSTGRES_PASSWORD: "pass"
 9
10  web:
11    build: .
12    depends_on:
13      - "db"
14    environment:
15      DATABASE_URL: "postgres://user:pass@db/proj"
16    ports:
17      - "8000:80"

Note the very last line. This is where we map port 8000 from the host to port 80 of the container such that the service can be accessed via http://127.0.0.1:8000.

This works quite well for a single project, but it suffers from a couple of problems if you work on multiple projects:

1. It doesn’t scale. If I want to run another project at the same time, I’ll have to use a different port number, maybe 8001, then 8002 etc.,

2. What if that compose.yaml file is checked in as part of the project? Does the whole team have to agree on a set of port numbers to use for each project?

Using overrides for multiple projects

Fortunately Docker Compose does have a solution for (2) in the form of the compose.override.yaml file. This file will be automatically be merged into the compose.yaml without any extra configuration.

Unlike some other guides (including the official docs) concerning this file, I prefer to not check compose.override.yaml into version control and instead add it to the .gitignore file. Adding it to version control completely defeats the purpose of it: to allow individual developers to override the standard compose file.

So, with this in mind, I no longer expose any ports by default in compose.yaml because I don’t know what will be convenient for each developer. This set up might look like this:

 1# compose.yaml
 2services:
 3  db:
 4    image: "postgres"
 5    environment:
 6      POSTGRES_DB: "proj"
 7      POSTGRES_USER: "user"
 8      POSTGRES_PASSWORD: "pass"
 9
10  web:
11    build: .
12    depends_on:
13      - "db"
14    environment:
15      DATABASE_URL: "postgres://user:pass@db/proj"

1# compose.override.yaml (to be created by each developer)
2services:
3  web:
4    ports:
5      - "8000:80"

Using Traefik

So now each developer can pick their own port numbers for each project, but we can still do better than this. People aren’t good at remembering numbers. We are much better at remembering names. Traefik is a free software edge router that can be used as a simple and super easy to configure reverse-proxy in container-based set ups.

Using Docker, Traefik can automatically discover services to create routes to. It uses container labels to further configure these routes. The following tiny example from the docs is illustrative:

 1# traefik/compose.yaml
 2services:
 3  reverse-proxy:
 4    image: traefik:v2.10
 5    ports:
 6      - "80:80"
 7    volumes:
 8      - /var/run/docker.sock:/var/run/docker.sock
 9  whoami:
10    image: traefik/whoami
11    labels:
12      - "traefik.http.routers.whoami.rule=Host(`whoami.docker.localhost`)"

This starts two containers on the same docker network. The reverse proxy listens on port 80 and forwards traffic with a host header of “whoami.docker.localhost” to the whoami service. Traefik guesses which port to send it to whoami based on the ports exposed by the container.

If you haven’t played with Traefik before it’s worth going through the quick-start properly now then coming back to see how we can make this work for multiple projects.

Traefik with multiple projects

This doesn’t quite solve our problem yet. We don’t want all of our various projects inside one compose file. Luckily Traefik communicates with the Docker daemon directly and doesn’t really care about the compose file, but you do need to make sure a few things are in order for this to work.

Firstly, make a docker network especially for Traefik to communicate with other services that you want to expose, for example:

 1# traefik/compose.yaml
 2services:
 3  reverse-proxy:
 4    image: traefik:v2.10
 5    restart: unless-stopped
 6    command: --api.insecure=true --providers.docker
 7    ports:
 8      - "80:80"
 9      - "8080:8080"
10    volumes:
11      - "/var/run/docker.sock:/var/run/docker.sock"
12    networks:
13      - traefik
14
15networks:
16  traefik:
17    attachable: true
18    name: traefik

We create the network traefik and give it the name “traefik” (otherwise docker compose would scope it by project, e.g. “traefik_traefik”). We also allow other containers to attach to this network.

Then in our compose.override.yaml file from above, instead of mapping ports, we do the following:

 1# proj/compose.override.yaml
 2services:
 3  web:
 4    labels:
 5      - "traefik.http.routers.proj.rule=Host(`proj.traefik.me`)"
 6      - "traefik.http.services.proj.loadbalancer.server.port=8000"
 7      - "traefik.docker.network=traefik"
 8    networks:
 9      - default
10      - traefik
11
12networks:
13  traefik:
14    external: true

Now, after bringing up first the traefik project then your web project, you should be able to browse to http://proj.traefik.me/ in your web browser.

There’s a few things going on here. First, we have declared the traefik network as an external network. This means compose won’t manage it, but expects it to exist (so you must start your traefik composition first). Next we override the networks setting of web to make it part of the traefik network too. Note we also have to add the default network, otherwise it wouldn’t be able to communicate with db and other services on its own default network.

The traefik.http.routers.proj.rule label configures Traefik to route HTTP traffic with the “proj.traefik.me” hostname to the container. The traffic.docker.network label is necessary because web is on two networks. Finally, we set traefik.http.services.proj.loadbalancer.server.port for completeness, just in case your container needs a different port mapping than the port it is set to expose, or if it exposes multiple ports.

There is one final piece of magic: the “traefik.me” hostname. What is that? You can read about it at http://traefik.me/. Essentially it is a DNS service that resolves to any IP address that you want, but by default it resolves <xxx>.traefik.me to 127.0.0.1. There are other services like this including https://sslip.io/ and https://nip.io/.

Now, because we don’t need to define any ports at all, it is possible to take advantage of a newish compose feature and reinstate the ports in the original compose.yaml file for those developers who don’t want to set up Traefik for themselves. So our final configuration looks like this:

 1# compose.yaml
 2services:
 3  db:
 4    image: "postgres"
 5    environment:
 6      POSTGRES_DB: "proj"
 7      POSTGRES_USER: "user"
 8      POSTGRES_PASSWORD: "pass"
 9
10  web:
11    build: .
12    depends_on:
13      - "db"
14    environment:
15      DATABASE_URL: "postgres://user:pass@db/proj"
16    ports:
17      - "8000:80"

 1# compose.override.yaml (to be created by each developer)
 2services:
 3  web:
 4    labels:
 5      - "traefik.http.routers.proj.rule=Host(`proj.traefik.me`)"
 6      - "traefik.http.services.proj.loadbalancer.server.port=8000"
 7      - "traefik.docker.network=traefik"
 8    networks:
 9      - default
10      - traefik
11    ports: !reset []
12
13networks:
14  traefik:
15    external: true

The !reset [] tag sets the ports back to empty; you can read about it here. Note that unfortunately it can’t be used to set new ports, only reset them to default (you would have to use two layers of override file to set new ports). The !reset tag requires a fairly recent version of docker compose, at least greater than 2.18.0.

A final note: you can check that these overrides are working correctly by running docker compose config.

Conclusion

By leveraging both the compose.override.yaml file and Traefik it’s easy to run multiple web projects on your development system at the same time and have easy to remember names to access them all. Each developer is free to run as many as they want and create their own easily-manageable configurations. Traefik and traefik.me can also be used to allow other developers on your network to easily access your local development instances with no DNS configuration required.

It’s a shame that the docs instruct people to use the override file for a distributed developer config rather than let individual developers use it, but hopefully it’s not too hard to remove this file from repos if already present.