Worked Design — File and image processing pipeline

Article.

“Users upload images and documents; we scan them for malware, make thumbnails and previews, and show them in the app. Design it so uploads never block the user.”

The traps are all on the boundary with untrusted input: bytes flowing through your API servers, a file served before it was scanned, an image that decompresses to 50 GB of pixels, an SVG that runs JavaScript on your domain, and a worker that processes the same message twice.

The shape of the answer: the client uploads directly to object storage, a file lands in quarantine, scanning and processing run asynchronously from events, a state machine tells the user where their file is, and nothing is served from the bucket until it is marked clean. AWS names are used; GCS and Azure Blob have the same pieces. The method is in System Design Interview — The Framework.

1. Requirements

Functional

• Upload images (JPEG, PNG, WebP, HEIC) and documents (PDF), up to 50 MB.
• Scan every file for malware; reject infected ones.
• Images: thumbnails at 3 sizes, metadata (EXIF) stripped. PDFs: a first-page preview.
• The user sees progress and the result; the app keeps working meanwhile.

Non-functional

• No file is downloadable by anyone before it is scanned and processed.
• Upload throughput does not depend on API server capacity.
• Processing is retried on failure and never produces duplicates.
• Uploaded content cannot run script in our web origin.

2. Estimate

uploads     200 k/day x 4 MB = 800 GB/day (~24 TB/month)
            ~2.3/s average, ~10/s peak
thumbnails  3 sizes, ~300 KB together = 60 GB/day
processing  ~0.5 s per image: 10/s x 0.5 s = 5 in flight
scanning    24,000 GB x $0.09 + 6 M objects x $0.215/1,000
            ≈ $3,450/month (GuardDuty for S3)
storage     +24 TB/month: ~290 TB after a year,
            ~$6,600/month in S3 Standard

What the numbers decide:

• 800 GB/day through API servers would mean 10 s of a worker’s time per 4 MB upload on a slow mobile link, multiplied by every concurrent upload. Bytes go straight to S3.
• Five images in flight is a handful of workers; compute is not the problem.
• Storage growth is the cost line: lifecycle rules (move originals to infrequent-access after 30 days, delete rejected files) matter more than instance sizes.

3. Upload: straight to storage

The API never touches the bytes. It creates a file record and returns a pre-signed POST that only allows this key, this content type and this size range:

post = s3.generate_presigned_post(
    Bucket="uploads-quarantine",
    Key=f"incoming/{file_id}",
    Fields={"Content-Type": declared_type},
    Conditions=[
        ["content-length-range", 1, 50 * 1024 * 1024],   # enforced by S3
        {"Content-Type": declared_type},
    ],
    ExpiresIn=600,
)
# → client POSTs the file to post["url"] with post["fields"]

• The size limit is enforced by S3, not by trusting the client. A pre-signed PUT URL can at most pin an exact signed Content-Length; a pre-signed POST policy enforces a range.
• Large files (video, archives) use multipart upload with a pre-signed URL per part, so a dropped connection resumes from the last part.
• Abandoned uploads cost money: a lifecycle rule with AbortIncompleteMultipartUpload (a few days) deletes orphaned parts, and a sweeper marks records stuck in pending as expired.
• The quarantine bucket blocks public access and has no CDN in front of it.

4. The state machine the user sees

pending → uploaded → scanning → processing → ready
              │          │            │
              └──────────┴─▶ rejected └─▶ failed (retryable)

CREATE TABLE files (
  id            uuid PRIMARY KEY,
  owner_id      bigint NOT NULL,
  status        text   NOT NULL,      -- the states above
  declared_type text   NOT NULL,
  detected_type text,                 -- from magic bytes, after upload
  sha256        bytea,
  size_bytes    bigint,
  reject_reason text,
  updated_at    timestamptz NOT NULL DEFAULT now()
);

-- every transition is conditional on the expected previous state
UPDATE files SET status = 'processing', updated_at = now()
 WHERE id = :id AND status = 'scanning';      -- 0 rows → someone else did it

The conditional transition is what makes duplicate events harmless: two workers receiving the same message race on this UPDATE, one gets a row, the other stops. The user’s app polls GET /files/{id} (or listens on a server-sent event stream) and shows a local preview meanwhile; nothing in the UI waits on the pipeline.

5. Scanning before anything else

A managed scanner

GuardDuty Malware Protection for S3 scans every object created in a protected bucket (or prefix), without running the rest of GuardDuty.

It publishes a result event to EventBridge and, if enabled at setup, tags the object GuardDutyMalwareScanStatus with NO_THREATS_FOUND, THREATS_FOUND, UNSUPPORTED, ACCESS_DENIED or FAILED. Objects up to 100 GB are scanned; archives are unpacked up to 100,000 files or 100 levels deep.

Two details to say:

• Results are delivered at least once, so the handler must be idempotent (the conditional state transition above).
• Only NO_THREATS_FOUND means clean. UNSUPPORTED (an encrypted archive, say) and FAILED are not “safe”; route them to rejection or manual review, never to ready.

Making “not scanned” unreadable

Even with the pipeline correct, a bug elsewhere could hand out a link to a quarantined object. A bucket policy can deny reads unless the tag says clean, for every principal except the scanner:

{
  "Effect": "Deny",
  "Principal": "*",
  "Action": ["s3:GetObject", "s3:GetObjectVersion"],
  "Resource": "arn:aws:s3:::uploads-quarantine/*",
  "Condition": {
    "StringNotEquals": {
      "s3:ExistingObjectTag/GuardDutyMalwareScanStatus": "NO_THREATS_FOUND"},
    "ArnNotEquals": {
      "aws:PrincipalArn": "arn:aws:iam::123456789012:role/GuardDutyS3Scan"}}
}

The self-hosted alternative is ClamAV (clamd) in a worker: free, fully under your control, but a gigabyte or more of RAM per instance for the signature database, signatures to keep updated, and signature-only detection.

6. Processing: treat every file as hostile

Malware scanning finds known malware. It does not stop a valid file built to hurt the code that parses it. The processing worker:

1. Detects the type from the bytes (magic numbers, for example with python-magic), not from the filename or the declared Content-Type, and rejects anything not on the allow-list.
2. Caps decoded size. A 50 KB PNG can declare 100,000 x 100,000 pixels. Pillow raises DecompressionBombError above twice MAX_IMAGE_PIXELS (default about 89 M pixels) and only warns between; lower the limit and turn the warning into an error.
3. Re-encodes the image instead of copying it. A fresh encode drops whatever was appended or hidden in the original container, and removes EXIF, including GPS coordinates of the user’s home.
4. Runs with limits: a memory and CPU cap per worker, a timeout per file, no network access except to S3. Libraries such as ImageMagick and PDF renderers have long CVE histories; isolate them, and use ImageMagick’s policy.xml to disable coders you do not need.

The worker, in code

import io, warnings
from PIL import Image, ImageOps

Image.MAX_IMAGE_PIXELS = 40_000_000                 # ~40 MP is plenty
warnings.simplefilter("error", Image.DecompressionBombWarning)

def make_thumbnails(src_path: str, file_id: str, sha: str) -> None:
    with Image.open(src_path) as im:
        im.draft("RGB", (2048, 2048))               # JPEG: decode at reduced size
        im = ImageOps.exif_transpose(im)            # apply rotation, then drop EXIF
        for width in (160, 480, 1200):
            out = im.copy()
            out.thumbnail((width, width))
            key = f"derived/{sha}/{width}.webp"     # deterministic: retries overwrite
            buf = io.BytesIO()
            out.save(buf, "WEBP", quality=80)       # no exif= → metadata not copied
            s3.put_object(Bucket="media-clean", Key=key, Body=buf.getvalue(),
                          ContentType="image/webp")

Idempotent by construction

Idempotency by construction: output keys derive from the content hash and the variant, so a retried or duplicated job overwrites the same objects instead of creating new ones. Identical uploads share derivatives for free.

The queue between the steps

The scan-result event goes through EventBridge to an SQS queue that the workers consume:

• Visibility timeout above the worst processing time (say 5 minutes), or a slow job is redelivered to a second worker while the first is still running.
• A dead-letter queue after a few receives (maxReceiveCount 5): a file that crashes the decoder every time must not loop forever. Alert on DLQ depth; mark the file failed.
• Back-pressure is free: if uploads spike, the queue grows and the workers catch up; scale workers on queue backlog per worker, not CPU.

Heavier work (video transcoding, OCR of long PDFs) goes to separate queues and worker pools, so a 2-hour video never delays a profile photo.

7. Serving

• Clean derivatives and originals live in a separate bucket behind a CDN (CloudFront with origin access control, so the bucket itself is private). Private files use short-lived signed URLs.
• Serve user content from a separate domain (usercontent.example.net, not app.example.com), with X-Content-Type-Options: nosniff, the stored Content-Type, and Content-Disposition: attachment for anything that is not an image you produced. An uploaded SVG or HTML file served inline from your own origin runs its script with access to your cookies; this is the same reason Google serves user content from googleusercontent.com.
• On-the-fly resizing at the edge (an image CDN or a resize Lambda) is the alternative to pre-generating sizes; it saves storage for rarely viewed sizes, at the price of cold misses and an image decoder exposed to request parameters, which need an allow-list of sizes.

8. Failure walk

Alert on the age of the oldest file in each non-final state, not only on errors: a file stuck in scanning for an hour is an outage the error rate does not show.

9. Where it breaks in production

• API servers fall over on upload day: files streamed through the application instead of straight to S3.
• Malware served for a few seconds: the app showed the original as soon as it was uploaded, before the scan.
• One image takes down a worker pool: a decompression bomb without a pixel limit, OOM-killing every worker that picks up the retried message.
• Stored XSS: an SVG avatar served inline from the main domain.
• Duplicate thumbnails and a growing bucket: output keys with random names, so every retry creates new objects.
• A doubled storage bill: no lifecycle rules for abandoned multipart parts and rejected files.
• Users’ home addresses published: EXIF GPS data copied into the public derivative.

10. The trade not taken

Synchronous processing in the upload request (receive the file, scan it, resize it, respond). It is simpler and fine for a back-office tool with ten uploads a day.

At scale it ties an API worker to every slow upload and every scan, makes one bad file a failed request with nothing retryable, and couples upload availability to the scanner’s. The asynchronous design costs a state machine and a status endpoint, and in return every step can fail, retry and scale on its own.

Related

• System Design Interview — The Framework - the method, Little’s law, the failure walk
• Worked Design — Distributed Job Scheduler - at-least-once jobs and idempotency
• Async I/O and background work - background work in Python services
• XSS and CSRF - why user content needs its own origin
• SSRF and Command Injection - hostile input reaching parsers and tools