Building docker images for multiple architectures with docker buildx

Introduction

In this previous post we have been exploring how to build docker images for multiple architectures.

In this post we will look into streamlining this approach using docker buildx, both locally and in gitlab-ci.

Step-by-step

Enable buildx

In order to use docker buildx you will need:

• A recent docker version; I am running 19.03.11 on linux
• enable the experimental features: export DOCKER_CLI_EXPERIMENTAL=enabled

Running docker buildx should show you:

Usage:	docker buildx COMMAND

Build with BuildKit

Management Commands:
  imagetools  Commands to work on images in registry

Commands:
  bake        Build from a file
  build       Start a build
  create      Create a new builder instance
  inspect     Inspect current builder instance
  ls          List builder instances
  rm          Remove a builder instance
  stop        Stop builder instance
  use         Set the current builder instance
  version     Show buildx version information 

Run 'docker buildx COMMAND --help' for more information on a command.

Create a builder

Now that we have a CLI we need a builder, for the platforms we target:

docker buildx create --platform "linux/amd64,linux/arm64,linux/arm/v7" --name container-builder --use

Note: builders are persistent. You can check the existing builders with docker buildx ls, and delete a builder with docker buildx rm <name>.

You can see what builders are active and what architectures they support by running docker buildx inspect --bootstrap. After creating the builder this shows me:

Name:   container-builder
Driver: docker-container

Nodes:
Name:      container-builder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/arm/v7, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v6

Install an emulator

If you want to do cross-platform builds you need to run qemu to emulate the different architectures:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Build

Now we can build (note that docker buildx build also pushed to the registry; you will have to be logged-on for that to succeed):

export REPOSITORY=your-registry/your-repo/your-image
export VERSION=your-tag
docker buildx build --platform "linux/amd64,linux/arm64,linux/arm/v7" -t $REPOSITORY:$VERSION . --push

As a Makefile

# Build docker image with buildx
# Experimental docker feature to build cross platform multi-architecture docker images
# https://docs.docker.com/buildx/working-with-buildx/
docker-buildx:
  export DOCKER_CLI_EXPERIMENTAL=enabled
  @if ! docker buildx ls | grep -q container-builder; then\
    docker buildx create --platform "linux/amd64,linux/arm64,linux/arm/v7" --name container-builder --use;\
  fi
  docker buildx build --platform "linux/amd64,linux/arm64,linux/arm/v7" \ 
    -t $(REPOSITORY):$(VERSION) . --push

CI/CD builds

I will focus on the tricky parts here, but you can find the whole .gitlab-ci.yml here (this is how this blog is built).

Changes to newer docker images

We are going to use docker:19.03.11 and docker:19.03.11-dind in our pipeline and there are a few things to be noted about these images:

• even if very recent docker:19.03.11 does not come with the buildx plugin enabled, so we will have to install it
• since 18.9+ the dind version of the image has changed its default behavior to require TLS by default (see also here unter “TLS”). It took me half of the afternoon figuring-out this change - completely unrelated to multi-arch builds -, and I expect that this change will break many gotlab-ci pipelines that rely on latest and docker-dind

Architecture of the pipeline

Our build job looks like this, where the tricky details are in bold, and marked with a number, e.g. (1) that are not part of the job’s source code, but rather markers for easier reading:

docker buildx:
  stage: build
  image: docker:19.03.11
  tags:
    - asksven-homelab-prd-public
  services:
    - name: docker:19.03.11-dind
      (1)command: ["--experimental"] 
  variables:
    (2)DOCKER_TLS_CERTDIR: "" # set this to disable TLS (default on docker 19.03+)
    PLATFORMS: "linux/amd64,linux/arm64,linux/arm/v7"  
    (3)DOCKER_HOST: tcp://localhost:2375/
    DOCKER_DRIVER: overlay2
    (4)DOCKER_CLI_EXPERIMENTAL: enabled
  script:
    # install depends
    - apk add curl jq
    # enable experimental buildx features
    - docker version
    # Download latest buildx bin from github
    - mkdir -p ~/.docker/cli-plugins/
    - (5)BUILDX_LATEST_BIN_URI=$(curl -s -L https://github.com/docker/buildx/releases/latest | grep 'linux-amd64' | grep 'href' | sed 's/.*href="/https:\/\/github.com/g; s/amd64".*/amd64/g')
    - curl -s -L ${BUILDX_LATEST_BIN_URI} -o ~/.docker/cli-plugins/docker-buildx
    - chmod a+x ~/.docker/cli-plugins/docker-buildx
    # Get and run the latest docker/binfmt tag to use its qemu parts
    - (6)BINFMT_IMAGE_TAG=$(curl -s https://registry.hub.docker.com/v2/repositories/docker/binfmt/tags | jq '.results | sort_by(.last_updated)[-1].name' -r)
    - docker run --rm --privileged docker/binfmt:${BINFMT_IMAGE_TAG}
    # create the multibuilder
    - docker buildx create --name multibuilder
    - docker buildx use multibuilder    
    # login
    - docker login -u "${DOCKER_REGISTRY_USER}" -p "$DOCKER_REGISTRY_PASSWORD" ${DOCKER_REGISTRY}
    # build and push
    - docker buildx build --cache-from ${DOCKER_REGISTRY}/${DOCKER_IMAGE_URL}:${CI_BUILD_REF_SLUG} --platform "${PLATFORMS}" -t ${DOCKER_REGISTRY}/${DOCKER_IMAGE_URL}:${CI_BUILD_REF} . --push

1. We need to enable the service (dind) to have the experimental features enabled
2. We must set DOCKER_TLS_CERTDIR to empty in order to disable dind to go for the new TLS-default
3. As I run the gitlab-runner on Kubernetes - as privileged pod - DOCKER_HOST must be set to point to the underlying dind
4. DOCKER_CLI_EXPERIMENTAL: enabled enabled the experimental features in the docker image that will run the job
5. We must install the buildy plugin as it does not come pre-installed with docker:19.03.11
6. We must download and run qemu if we want to build for multiple architectures on the amd64 node of the runner