Backend / Docker / 11_multi_arch_builds.md

Multi-arch builds

Updated 4 interview angles 5 min read source
On this page7
  1. The image index
  2. Building: three strategies
  3. Python specifics
  4. Where it breaks in production
  5. The trade not taken
  6. Related
  7. Interview angle

Multi-arch builds

Article.

One image tag, several CPU architectures: python:3.14-slim runs on an Intel laptop, an Apple-silicon Mac, an AWS Graviton instance and a Raspberry Pi, because the tag points to an index of per-platform images and each machine pulls the one that matches it. Building such images is routine now that arm64 servers (Graviton) are cheaper than x86, and the way you build them decides whether the build takes twenty seconds or twenty minutes.

An image built on a laptop crashes on the server with exec format error. What happened, and how do you build for both? Measured with Docker Engine 29.8 and BuildKit on an x86_64 machine.

The image index

text
$ docker buildx imagetools inspect python:3.14-slim
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:a2b82f3c4855...      (the index)
  Platform: linux/amd64     sha256:cfe2e24a7530...
  Platform: linux/arm64/v8  sha256:7f47c8ffd4e7...
  Platform: linux/arm/v5, arm/v7, 386, ppc64le,
            riscv64, s390x
  Platform: unknown/unknown  (attestations)

The tag points to an index (a manifest list), which lists one manifest per platform, each with its own digest. docker pull reads the index and fetches the manifest for the machine’s os/architecture/variant. The unknown/unknown entries are build attestations, not runnable images.

Two consequences:

  • Pin the index digest, python@sha256:a2b82f3c4855..., not one platform’s digest. Pinning the amd64 manifest digest makes arm64 nodes pull an x86 image.
  • A single-arch push to an existing tag replaces the index, and the other architecture disappears from that tag.

Building: three strategies

Emulation (QEMU)

Docker Desktop and docker buildx can run foreign-architecture binaries through QEMU user-mode emulation:

bash
docker buildx build --platform linux/amd64,linux/arm64 -t app:1.2 --push .

It works for any Dockerfile and is very slow for anything CPU-bound:

On an x86_64 host amd64 (native) arm64 (QEMU)
pure-Python CPU loop 0.60 s 19.62 s
pip install numpy pandas 15.1 s 196.4 s

Thirty-three times slower for Python bytecode, thirteen times for an install that only unpacks wheels. A package without an arm64 wheel that compiles from source under emulation can turn into an hour-long CI job.

Cross-platform: do the work natively, copy the result

dockerfile
FROM --platform=$BUILDPLATFORM python:3.14-slim AS deps
COPY --from=ghcr.io/astral-sh/uv:0.9.0 /uv /bin/uv
ARG TARGETARCH
RUN case "$TARGETARCH" in \
      arm64) P=aarch64-manylinux_2_28 ;; amd64) P=x86_64-manylinux_2_28 ;; esac && \
    uv pip install --no-cache --python-platform "$P" --python-version 3.14 \
      --target /deps numpy==2.3.3 pandas==2.3.3

FROM python:3.14-slim
COPY --from=deps /deps /usr/local/lib/python3.14/site-packages
text
arm64 image, pip install under QEMU:            204.6 s
arm64 image, arm64 wheels fetched natively:      19.5 s
docker run --platform linux/arm64 ... -> works on aarch64, numpy 2.3.3

The deps stage runs on the build machine’s own architecture ($BUILDPLATFORM) and downloads wheels for the target architecture ($TARGETARCH); the final stage, for the target platform, only copies files, so almost nothing runs under emulation. Ten times faster here. It requires wheels for the target to exist; a source build still needs the target CPU.

Native builders

The fastest and simplest at scale: build each architecture on a machine of that architecture (CI has arm64 runners now), push each image by digest, then join them into one index:

bash
docker buildx imagetools create -t registry/app:1.2 \
  registry/app@sha256:<amd64-digest> registry/app@sha256:<arm64-digest>

docker buildx create --append can also combine a remote arm64 machine and a local x86 one into one builder.

In GitHub Actions, a matrix over native runners does this without emulation:

yaml
jobs:
  build:
    strategy:
      matrix:
        include:
          - {platform: linux/amd64, runner: ubuntu-24.04}
          - {platform: linux/arm64, runner: ubuntu-24.04-arm}
    runs-on: ${{ matrix.runner }}
    steps:
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v7
        with:
          platforms: ${{ matrix.platform }}
          outputs: type=image,name=registry/app,push-by-digest=true,push=true
          cache-from: type=gha,scope=${{ matrix.platform }}
          cache-to: type=gha,mode=max,scope=${{ matrix.platform }}
  # a final job collects the two digests and runs `imagetools create`

Each architecture keeps its own cache scope, so an arm64 build never invalidates the amd64 cache.

Python specifics

Most popular packages publish manylinux aarch64 wheels, so installs on arm64 are as fast as on x86. The problems are the long tail: a dependency with no arm64 wheel compiles from source (slow, needs compilers, and under QEMU very slow), and Alpine images use musl, which needs musllinux wheels or compiles everything. Check the wheel coverage of every compiled dependency before moving a service to Graviton.

Where it breaks in production

exec format error

An image built on an Apple-silicon Mac is arm64-only; the x86 cluster pulls it and the process cannot start (exec format error, a CrashLoopBackOff, an ECS task stopping at once). Build both platforms, or pass --platform linux/amd64 explicitly when building for x86 servers from an arm laptop. docker image inspect --format '{{.Architecture}}' and uname -m in the container show what actually runs.

A hard-coded FROM --platform

FROM --platform=linux/amd64 python:3.14-slim forces amd64 even on arm64 builders and laptops, so everything there runs under emulation. Docker’s --check build checks flag a constant platform in FROM; use $BUILDPLATFORM or $TARGETPLATFORM instead.

Attestations confusing older consumers

Build attestations appear as extra unknown/unknown entries in the index. Some consumers that expect a plain image (AWS Lambda container images, older registries) reject the index; build those with --provenance=false --sbom=false.

Registry cleanup deleting platform images

Per-platform manifests are untagged children of a tagged index. A lifecycle rule that expires “untagged images” can delete them and leave a tag whose index points at missing manifests. Check how the registry treats index children before enabling cleanup.

The trade not taken

Separate tags per architecture (app:1.2-arm64, app:1.2-amd64) avoid indexes altogether and make every deployment state its architecture explicitly, at the cost of every manifest and deploy script choosing the right tag. A single-architecture fleet avoids the question entirely. The index is worth its complexity once the fleet is mixed.

That is typically when arm64 (Graviton costs about 20% less per vCPU-hour on Fargate and per GB-second on Lambda) joins an x86 estate, or when developers on Apple silicon must run the same images as production.

Interview angle 4

  • “An image built on a Mac crashes with exec format error on the server. Why?” - it is arm64-only and the server is x86_64. Build a multi-platform image (an index with both), or build --platform linux/amd64 for x86.
    • Follow-up: “How does one tag serve both?” - the tag points to an image index listing one manifest per platform; docker pull picks the one matching the machine.
  • “Why is the arm64 build so slow in CI?” - it runs under QEMU emulation: 33 times slower for a CPU loop, 13 times for installing wheels here.
    • Follow-up: “How do you fix it?” - native arm64 runners, or a cross-platform stage on $BUILDPLATFORM that fetches arm64 wheels and copies them: 204.6 s down to 19.5 s.
  • “Which digest do you pin?” - the index digest; a platform manifest digest pins one architecture and breaks the others.
  • “Is moving to Graviton free?” - about 20% cheaper compute, once every compiled dependency has an arm64 wheel and the image is multi-arch.