Building docker images for multiple architectures

Introduction

Since Kubernetes runs on the Raspberry PI I have been investigating ways to build my blog so that it can run on my x86 (Proxmox) as well as ARM Kubernetes cluster, composed of Raspberry PIs and an Nvidia Jetson Nano.

This post will take you through my learnings of the taxonomy of architectures and platforms, as well as building docker images for multiple architectures.

Architectures

Well, I already knew that rpi has a different architecture than my Intel-based hardware, so let’s get into how these are named. Executing uname -a and arch/uname -m on the different nodes of my cluster should give us some clues, right?

rpi 4 running raspbian

• uname -a: Linux armk8snode1 4.19.97-v7l+ #1294 SMP Thu Jan 30 13:21:14 GMT 2020 armv7l GNU/Linux
• arch: armv7l

Nvidia Jeson Nano running Ubuntu

• uname -a: Linux armk8sgpunode1 4.9.140-tegra #1 SMP PREEMPT Wed Apr 8 18:10:49 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux
• arch: aarch64

Proxmox VM

• uname -a: Linux k8s-rancher-lab-master 4.14.122-rancher #1 SMP Tue May 28 01:50:21 UTC 2019 x86_64 GNU/Linux
• uname -m: x86_64

Docker architectures

Modern Docker images support multiple architectures. So let’s look at what we find there:

export DOCKER_CLI_EXPERIMENTAL=enabled
docker manifest inspect nginx:1.15-alpine | grep architecture

returns

"architecture": "amd64",
"architecture": "arm",
"architecture": "arm64",
"architecture": "386",
"architecture": "ppc64le",
"architecture": "s390x"

Note: It is interesting to notice that docker manifest is still experimental in Docker 19.03.8.

Wrap-up

This is all you need to know when you want to build a docker image for a given architecture:

Using the right base-image

Using the mapping it is easy to refer to an image for a specific architecture the syntax is:

So for a Raspberry PI running Raspbian Lite 4.19.97-v7l+, the Dockerfile of my blog would look like:

FROM arm32v6/nginx:1.15-alpine

COPY public/ /usr/share/nginx/html/

Building for multiple architectures

Note: if your Dockerfile does not have any RUN commands, using binfmt_misc is not required. In my example Dockerfile, qemu is not required: I can build docker images for the right architecture by using the right base-image. This would also be true if you use multi-staged builds where the final stage only copies what was created in the build stage, as long as the results of the build stage is not architecture-dependent.

In order to build for multiple architectures we are going to use binfmt_misc:

1. Register handler (once per build-host): docker run --rm --privileged multiarch/qemu-user-static:register
2. Getting handlers, e.g. using the script below
3. Building with different base-images

RELEASE="v4.2.0-7"

for target_arch in aarch64 arm x86_64; do
  wget -N https://github.com/multiarch/qemu-user-static/releases/download/${RELEASE}/x86_64_qemu-${target_arch}-static.tar.gz
  tar -xvf x86_64_qemu-${target_arch}-static.tar.gz
done

Creating your Dockerfile

The Dockerfile will depend on the architecture in two aspects:

1. The base-image
2. The qemu binary that we copy into the image (qemu-aarch64-static, qemu-arm-static or qemu-x86_64-static)

but fortunately using ARGuments, we can keep the Dockerfile generic.

The Dockerfile

Since the Dockerfile of my blog only copies files, we will add a RUN command so that we can demonstrate how to use binfm_misc. We will add two ARGuments to set the base-image as well as the qemu binary, depending on the platform we build for:

ARG BASE_IMAGE=nginx:1.15-alpine
ARG QEMU_BINARY=qemu-x86_64-static

FROM $BASE_IMAGE

COPY $QEMU_BINARY /usr/bin

COPY public/ /usr/share/nginx/html/

RUN ls -a /usr/share/nginx/html/*

Notice the default values of BASE_IMAGE and QEMU_BINARY. These are set so that docker build . on an X86_64 machine works.

The docker build command

Since we have added ARGs, we can now pass the right arguments for each platform, adding the architecture to the tag.

# Build the container for amd64
docker build --build-arg BASE_IMAGE=amd64/nginx:1.15-alpine --build-arg QEMU_BINARY=qemu-x86_64-static . -t asksven/blog.asksven.io:amd64-new

# Build the container for arm64v8
docker build --build-arg BASE_IMAGE=arm64v8/nginx:1.15-alpine --build-arg QEMU_BINARY=qemu-aarch64-static . -t asksven/blog.asksven.io:arm64v8-new

# Build the container for arm32v7
docker build --build-arg BASE_IMAGE=arm32v6/nginx:1.15-alpine --build-arg QEMU_BINARY=qemu-arm-static . -t asksven/blog.asksven.io:arm32v6-new

Manifest

Until now we have achieved building and pushing three docker images, for each architecture:

1. asksven/blog.asksven.io:amd64-new
2. asksven/blog.asksven.io:arm64v8-new
3. asksven/blog.asksven.io:arm32v6-new

If you know the architecture you want to run these images on, you can refer to these images (e.g. in a Kubernetes deployment). This has limitations though. Imagine a Kubernetes cluster with nodes of different architectures: in that case you would find yourself writing different deployments - for each architecture -, and using taints and tolerations to run the right image on each node.

Fortunately Docker has the concept of multi-arch manifests: a way to bundle images for different architecture as one Docker image. Annotations help the docker daemon determining what image it should run.

First we create the manifest, bundling all the images we have built, and giving it an architecture-independent name (asksven/blog.asksven.io:new):

docker manifest create asksven/blog.asksven.io:new --amend asksven/blog.asksven.io:arm32v6-new --amend asksven/blog.asksven.io:arm64v8-new --amend asksven/blog.asksven.io:amd64-new

then we annotate the images (note that amd64 does not require an annotation):

docker manifest annotate asksven/blog.asksven.io:new asksven/blog.asksven.io:arm32v6-new --os linux --arch arm
docker manifest annotate asksven/blog.asksven.io:new asksven/blog.asksven.io:arm64v8-new --os linux --arch arm64 --variant armv8

finally, we push the manifest:

docker manifest push asksven/blog.asksven.io:new

This table summarizes the annotation needed for hinting the docker daemon to pick the right image from the manifest: