Backend / Docker / 04_docker_image_layers.md

Docker image layers and build caching

Updated 4 interview angles 6 min read source
On this page9
  1. How each instruction is keyed
  2. Order: least to most frequently changed
  3. Cache mounts: when dependencies do change
  4. Size: layers only ever add
  5. A Dockerfile parsing trap
  6. Where it breaks in production
  7. The trade not taken
  8. Related
  9. Interview angle

Docker image layers and build caching

Article.

An image is a stack of read-only layers. RUN, COPY and ADD each add one layer of filesystem changes; ENV, CMD, LABEL, USER and EXPOSE only change the image config. When rebuilding, BuildKit reuses a step’s cached result if its cache key is unchanged, and the first changed step invalidates every step after it.

Requirements did not change, but CI reinstalls every dependency on every commit. Why? The answer depends on how each instruction’s cache key is computed. Measured with BuildKit (Docker Engine 29.8) on python:3.14-slim.

How each instruction is keyed

dockerfile 
FROM python:3.14-slim
RUN date +%s%N > /built_at
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
ENV MODE=prod
CMD ["python", "app.py"]
Change What was rebuilt
nothing nothing: 4 of 4 steps cached
touch requirements.txt nothing: 4 of 4 cached
edit requirements.txt everything from that COPY on
  • COPY and ADD are keyed on file contents. touch changed only the modification time, and the step stayed cached; editing the file invalidated it and every step after it, including the pip install.
  • RUN is keyed on the command text, not on what it does. The date step stayed cached across all of these builds: /built_at still held the first build’s timestamp. A RUN apt-get update or RUN curl .../latest never notices that upstream changed; only editing the line, --no-cache, or an earlier invalidation reruns it.
  • Config instructions are free.
text 
docker history:
0B       CMD ["python" "app.py"]
0B       ENV MODE=prod
8.19kB   COPY app.py .
14.6MB   RUN pip install --no-cache-dir -r requirements.txt
8.19kB   COPY requirements.txt .

Order: least to most frequently changed

The first changed step invalidates everything below it, so the order of the Dockerfile is the order of cache invalidation. Dependencies before source:

dockerfile 
COPY pyproject.toml uv.lock ./
RUN uv sync --locked --no-install-project      # cached until the lockfile changes
COPY src/ src/
RUN uv sync --locked                           # installs the project itself

A code change then rebuilds only the last two steps. Copying the whole directory before installing (COPY . . then install) reinstalls every dependency on every commit, which is the usual answer to the opening question.

A .dockerignore (.git, .venv, __pycache__, test data) keeps irrelevant files from invalidating a broad COPY and keeps the build context small.

Cache mounts: when dependencies do change

A dependency change invalidates the install step no matter how it is ordered. A cache mount keeps the package manager’s download cache between builds without putting it in the image:

dockerfile 
RUN --mount=type=cache,target=/root/.cache/uv \
    uv pip install --system -r requirements.txt
text 
add one small package to numpy + pandas, rebuild:
  without a cache mount:   Prepared 8 packages in 2.75s   (downloads all)
  with a cache mount:      Prepared 1 package  in 0.148s  (only the new one)

With large wheels (PyTorch is gigabytes) that is the difference between minutes and seconds. The mount lives on the builder, so in CI it helps only if the builder persists between runs; ephemeral runners need the build cache exported (--cache-to type=registry or type=gha), where mode=max caches the intermediate stages of a multi-stage build too, at the cost of pushing more data.

Size: layers only ever add

dockerfile 
RUN head -c 50000000 /dev/urandom > /big
RUN rm /big
text 
docker history:
4.1kB   RUN rm /big
50MB    RUN head -c 50000000 /dev/urandom > /big

Deleting the file in a later layer hides it, but its 50 MB stay in the earlier layer and in every pull. Creating and deleting in the same RUN adds nothing.

The same applies to secrets: a token copied in and deleted later is still in the image history, extractable by anyone who can pull it. Use RUN --mount=type=secret for build-time secrets, and multi-stage builds so compilers and build tools never reach the final image (Multi-stage builds).

The number of layers barely matters for size; what matters is what each layer leaves behind. Chaining every command with && to “minimise layers” is advice from before multi-stage builds and cache mounts.

A tested uv multi-stage pattern

dockerfile 
FROM python:3.14-slim AS build
COPY --from=ghcr.io/astral-sh/uv:0.9.0 /uv /bin/uv
WORKDIR /app
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked --no-dev --no-install-project
COPY src/ src/
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked --no-dev --no-editable

FROM python:3.14-slim
WORKDIR /app
COPY --from=build /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH"
USER 1000
CMD ["python", "-m", "demo"]

Three details were found by running it. uv is not in the Python image, so it is copied in from uv’s own image, pinned. Without --no-editable, uv sync installs the project as an editable link back to /app/src, and the runtime stage, which copies only .venv, failed with No module named demo.

And --locked, not --frozen: with a stale lockfile --frozen exited 0 and silently installed the old set, while --locked failed with “The lockfile at uv.lock needs to be updated”.

A Dockerfile parsing trap

dockerfile 
COPY requirements.txt app.py /app/   # copy sources
text 
ERROR: failed to build: failed to solve: failed to compute cache key ... not found

Dockerfile comments must start a line. Mid-line, #, copy and sources became extra arguments and the build failed; USER 1000 # not root sets the user to that whole string and the container fails to start. RUN lines escape this only because the shell treats # as a comment.

Where it breaks in production

Stale security fixes from a cached apt-get update

The RUN apt-get update && apt-get install ... layer was cached months ago, so every build ships the same old packages. Rebuild base layers on a schedule with --no-cache (or --pull plus a cache bust), and let an image scanner flag what is old.

A broad COPY before the install

COPY . . first: any edit to any file, a README included, reinstalls all dependencies. Copy the manifests, install, then copy the source.

A secret in a layer

A .npmrc or pip.conf with a token is copied in, used, deleted. It is still in the earlier layer. Build secrets go through --mount=type=secret; check existing images with docker history and a secret scanner.

Cache that never reaches CI

Builds are fast on the laptop and slow in CI, because each CI job starts with an empty builder. Export and import the cache (--cache-from/--cache-to with a registry or the CI’s cache backend).

The trade not taken

Reproducible builds argue the other way: a cached layer means a build today may not match a clean build of the same commit. Pinning base images by digest and dependencies by lockfile keeps the cache and reproducibility together; the remaining trade is between cached speed and picking up upstream security fixes, settled by scheduled clean rebuilds.

Interview angle 4

  • “Requirements did not change, but CI reinstalls every dependency. Why?” - either a broad COPY . . before the install, or a CI builder with no cache. COPY is keyed on file contents: touch kept the cache, an edit broke it.
    • Follow-up: “You added a cache mount and CI is still slow.” - the mount lives on the builder; ephemeral CI runners start empty. Export the build cache to a registry or the CI cache.
  • “How does BuildKit decide a step is cached?” - COPY/ADD: checksums of the copied files. RUN: the command text and the previous layer, not what the command fetches; RUN date kept its first value for every rebuild.
  • “Why doesn’t deleting a file in a later layer shrink the image?” - layers only add: the 50 MB layer stayed, the rm layer was 4.1 kB. Create and delete in the same RUN, or use multi-stage builds.
    • Follow-up: “So a deleted secret is still there?” - yes, in the earlier layer. Use build secret mounts.
  • “Which instructions create layers?” - RUN, COPY, ADD (and WORKDIR when it creates the directory); ENV, CMD, USER, EXPOSE are 0-byte config entries.
Previous
Purpose of Containerization