5 October, version 3 — core knowledge for the ad

One self-contained file: for every technology and practice the ad names — what it is, what it is for, how it works, and the facts worth remembering. No links, nothing to open elsewhere. Facts that change over time are as of October 2026.

Every entry follows the same shape: What it is → Purpose → How it works → Key concepts → Remember.

Sections: 1 Python · 2 Web frameworks · 3 APIs · 4 Databases · 5 Testing · 6 Git and workflow · 7 Regulated environments · 8 Event-driven architecture · 9 Containers · 10 Infrastructure as code · 11 Observability · 12 AI-assisted development · 13 Healthcare and data-sensitive domains · 14 FHIR and HL7 · 15 Performance · 16 Cloud and CI/CD · then a one-line glossary of everything.

1. Python — type hints, async patterns, project layout

Modern type hints

What it is. Optional annotations on variables, parameters and return values (PEP 484 onwards), checked by a separate static type checker — not by the interpreter.

Purpose. Catch whole classes of bugs before the code runs (wrong types, None where a value is expected, unhandled cases), document intent, and power editor completion and safe refactoring. In a team codebase, types are the contract between modules.

How it works. Python stores annotations but does not enforce them at runtime. A checker reads the code and reports mismatches, usually in CI: mypy (the established default), pyright (faster, stricter), and the newer Rust-based checkers ty (Astral) and Pyrefly (Meta). Some libraries read annotations at runtime on purpose — Pydantic validates data against them, which is how FastAPI turns type hints into request validation.

The modern vocabulary, by version.

• 3.9 — built-in generics: list[int], dict[str, float] instead of List, Dict.
• 3.10 — unions with the bar operator: int | None instead of Optional[int].
• 3.11 — Self, Never, assert_never, Required / NotRequired in TypedDict.
• 3.12 — PEP 695 generics: def first[T](xs: list[T]) -> T, class Box[T], type UserId = int; and @override.
• 3.13 — TypeIs for precise narrowing, ReadOnly TypedDict keys, defaults for type parameters.
• 3.14 — annotations are evaluated lazily (PEP 649), so forward references work without quotes.

Key constructs.

• Protocol — structural typing: “anything with these methods fits”. The typed form of duck typing, and the usual choice for seams such as repositories and API clients, because a fake in tests fits without inheriting anything. An ABC is nominal: you must inherit from it.
• dataclass — a plain typed record. TypedDict — the shape of a dict, usually JSON you do not own. A Pydantic BaseModel — a typed record validated at runtime, for untrusted input.
• Literal["draft", "final"] and Enum — closed sets of values. Final — a name that must not be reassigned.
• Annotated[int, Field(gt=0)] — a type plus metadata that frameworks read; FastAPI and Pydantic rely on it heavily.
• ParamSpec — types decorators so the wrapped function keeps its signature.
• TypeGuard / TypeIs — your own narrowing functions.
• Any switches checking off; object means “anything, used carefully”.

from dataclasses import dataclass
from typing import Literal, Protocol, assert_never

Status = Literal["active", "inactive", "deceased"]


@dataclass(frozen=True, slots=True)
class Patient:
    id: str
    status: Status


class PatientRepo(Protocol):  # any class with this method fits — no inheritance
    def get(self, patient_id: str) -> Patient | None: ...


def label(status: Status) -> str:
    match status:
        case "active":
            return "Active"
        case "inactive":
            return "Inactive"
        case "deceased":
            return "Deceased"
        case _:
            assert_never(status)  # a new status that is not handled fails type checking

Remember.

• Hints are not enforced at runtime: validate untrusted data at the boundary (Pydantic), and trust the types inside because the checker proved them.
• Adopt strictness gradually on existing code — strict for new packages, then module by module — and require a reason and an error code for every # type: ignore[...].
• “Make illegal states unrepresentable”: model states as distinct types and let the checker enforce exhaustive handling.

Async patterns

What it is. asyncio is Python’s standard library for cooperative concurrency: many tasks interleave on one thread, switching only at await points.

Purpose. Handle many I/O-bound operations at once — thousands of open connections, parallel calls to APIs and databases — without a thread per operation. It does not make CPU-bound work faster.

How it works. An event loop runs ready tasks one at a time. When a task awaits something that is not ready yet (a socket read, a timer), it yields control and the loop runs another task; the operating system tells the loop when the I/O is ready. async def defines a coroutine function; calling it creates a coroutine object that does nothing until it is awaited or wrapped in a task.

Core patterns.

import asyncio

import httpx


async def fetch_all(urls: list[str], limit: int = 10) -> list[int]:
    sem = asyncio.Semaphore(limit)
    async with httpx.AsyncClient(timeout=10) as client:

        async def fetch(url: str) -> int:
            async with sem:  # at most `limit` requests in flight
                response = await client.get(url)
                return response.status_code

        async with asyncio.timeout(30):  # one deadline for the whole batch
            async with asyncio.TaskGroup() as tg:
                tasks = [tg.create_task(fetch(url)) for url in urls]
    return [task.result() for task in tasks]

Pitfalls.

• A blocking call — a synchronous database driver, requests, time.sleep, heavy CPU work — inside async def freezes every task on that loop.
• asyncio.create_task(...) without keeping a reference: the task can be garbage-collected mid-flight. Keep a reference, or use a TaskGroup.
• Catching CancelledError without re-raising it breaks cancellation.
• Async is contagious: async code has to be awaited all the way up the call chain.

Remember.

• asyncio is concurrency, not parallelism. For CPU parallelism: processes; free-threaded Python (officially supported since 3.14 as an opt-in build); or subinterpreters (in the standard library since 3.14).
• Related tools: anyio (structured concurrency on top of asyncio or trio), uvloop (a faster event loop), contextvars (per-task context such as a request id).

Idiomatic project layout

What it is. The conventional structure and tooling of a modern Python project.

Purpose. Reproducible installs, one place for configuration, imports that behave the same in tests and in production, and quality gates that run the same way on every machine.

The pieces.

• pyproject.toml — the single configuration file: project metadata (PEP 621), the build backend (PEP 517/518), dependency groups (PEP 735), and tool settings ([tool.ruff], [tool.mypy], [tool.pytest.ini_options]). setup.py, setup.cfg and a hand-maintained requirements.txt are legacy.
• uv (Astral) — the current default package and project manager. It creates the virtual environment, resolves and locks dependencies into uv.lock, installs Python versions and runs commands (uv run pytest). It replaces pip, pip-tools, virtualenv, pyenv and much of Poetry, and is 10 to 100 times faster.
• A lockfile — the exact version of every dependency. Applications commit it so every machine and CI install the same thing; libraries publish loose version ranges instead.
• The src layout — code lives in src/<package>/, so tests import the installed package rather than whatever happens to be in the working directory, which surfaces packaging mistakes early.
• Quality tools — Ruff (linter and formatter in one, replacing flake8, isort and black), mypy or pyright (types), pytest (tests), pre-commit (runs the fast checks before each commit). CI runs all of them as the real gate.
• Configuration — environment variables read once at startup into one typed settings object (pydantic-settings), following the twelve-factor rule.

my-service/
├── pyproject.toml        metadata, deps, tool config
├── uv.lock               committed
├── Dockerfile
├── src/
│   └── my_service/
│       ├── main.py       app entry point
│       ├── config.py     typed settings
│       └── patients/     one package per feature
│           ├── router.py
│           ├── schemas.py
│           ├── service.py
│           └── repository.py
└── tests/
    ├── unit/
    └── integration/

Remember.

• Organise by feature (patients, appointments), not by technical layer — code that changes together lives together.
• Keep the framework at the edge: thin routes, business logic in services, data access in repositories.
• A py.typed file marks a library as typed for its users (PEP 561).

2. Web frameworks — FastAPI, Django, Flask

FastAPI

What it is. A modern, async-first Python framework for HTTP APIs, built on Starlette (an ASGI toolkit) and Pydantic (data validation).

Purpose. Build typed APIs quickly: type hints become request validation, response serialisation and an OpenAPI specification automatically.

How it works. Each endpoint is a function decorated with @app.get(...), @app.post(...) and so on. FastAPI inspects the function signature: path, query and body parameters are parsed and validated with Pydantic, and Depends(...) declares dependencies that FastAPI resolves for each request (caching each one within that request). Responses are validated against the declared response model. It runs on any ASGI server — Uvicorn, Hypercorn, Granian.

Key concepts.

• async def endpoints run on the event loop; plain def endpoints run in a thread pool (40 threads by default). So endpoints that call blocking libraries should be plain def.
• Dependency injection with Depends: database sessions, authentication, settings. yield dependencies give setup and guaranteed teardown; app.dependency_overrides swaps them for fakes in tests.
• Automatic documentation: Swagger UI at /docs, ReDoc at /redoc, the raw specification at /openapi.json (OpenAPI 3.1).
• lifespan for startup and shutdown resources (connection pools, HTTP clients); APIRouter to split the app by feature; exception handlers for one error format; middleware; BackgroundTasks for small work after the response (not durable — use a real queue for anything important).
• The FastAPI CLI: fastapi dev (auto-reload) and fastapi run (production).

from typing import Annotated

from fastapi import Depends, FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()


class Patient(BaseModel):
    id: str
    name: str


class PatientRepo:
    def __init__(self) -> None:
        self._rows = {"p1": Patient(id="p1", name="Ada")}

    async def get(self, patient_id: str) -> Patient | None:
        return self._rows.get(patient_id)


def get_repo() -> PatientRepo:  # in real code: a repository bound to a DB session
    return PatientRepo()


@app.get("/patients/{patient_id}", response_model=Patient)
async def read_patient(
    patient_id: str, repo: Annotated[PatientRepo, Depends(get_repo)]
) -> Patient:
    patient = await repo.get(patient_id)
    if patient is None:
        raise HTTPException(status_code=404, detail="Patient not found")
    return patient

Remember. FastAPI is code-first OpenAPI. One worker process has one event loop, so production runs several workers to use several CPU cores.

Django

What it is. A “batteries-included” web framework: ORM, migrations, an admin interface, authentication, forms, templates and security middleware, all integrated. Django REST Framework (DRF) adds the API layer.

Purpose. Build data-heavy products quickly with strong conventions and a mature ecosystem; the admin alone is a back office for free.

How it works. A request goes through middleware to the URL router, then to a view (a function or class), which uses the ORM and returns a response — the model–view–template pattern. Models are Python classes; makemigrations generates migration files from model changes and migrate applies them.

Key concepts.

• QuerySets are lazy: built with .filter(), executed when iterated. The N+1 problem is fixed with select_related (an SQL join, for foreign keys) and prefetch_related (a second query, for many-to-many and reverse relations).
• transaction.atomic() for transactions, F() expressions for atomic updates, Q() objects for complex conditions.
• DRF: serializers (validation and representation), viewsets and routers, permissions, throttling, pagination.
• Async: views can be async under ASGI, and the ORM has async methods (aget, acreate) — but the database driver underneath is still synchronous, so database-heavy views gain little concurrency.
• Versions: Django 6.1 is current (August 2026) and 5.2 is the long-term support release. Django 6.0 added a built-in background tasks API; the worker that runs the tasks comes from a backend you choose.

Remember. Choose Django when the product benefits from the batteries (admin, auth, ORM, migrations); choose FastAPI for focused, typed, I/O-heavy APIs.

Flask

What it is. A minimal WSGI micro-framework built on Werkzeug (the HTTP and WSGI toolkit) and Jinja2 (templates).

Purpose. Provide routing, request and response handling and templating — and leave every other choice (ORM, validation, authentication) to you, through extensions.

How it works. @app.route("/path") maps URLs to view functions. The current request is reached through context-local proxies (request, g, current_app) that point at the active request. Larger applications use the application-factory pattern and Blueprints (modular groups of routes). Common extensions: Flask-SQLAlchemy, Flask-Migrate (Alembic), Flask-Login, and Marshmallow or Pydantic for validation.

Key facts. Flask is synchronous (WSGI). It accepts async def views, but each request still occupies a worker, so it gains little for high concurrency — Quart is the ASGI version of the same API. In production it runs behind Gunicorn or Waitress with several workers.

Remember. Flask suits small synchronous services and existing codebases. You assemble the stack yourself: flexibility for a small service, extra work for a large one.

Choosing between them

3. APIs — REST, OpenAPI, versioning, pagination, errors

REST

What it is. An architectural style for networked APIs (Roy Fielding, 2000) built on HTTP: resources identified by URLs, manipulated with standard methods, exchanged as representations (usually JSON), statelessly.

Purpose. Uniform, cacheable, loosely coupled APIs that any HTTP client can use and that intermediaries — proxies, caches, gateways — understand.

Core constraints. Client–server separation; statelessness (every request carries everything needed, including credentials); cacheable responses; a uniform interface (resources, standard methods, self-descriptive messages, hypermedia); a layered system.

Methods.

Design rules.

• Plural nouns and nesting for ownership: /patients/{id}/appointments. Verbs come from the method, not the URL; actions become sub-resources or state changes.
• Status codes that mean something: 201 with a Location header on create, 204 for no content, 400 for a malformed request, 401 not authenticated, 403 not allowed, 404 not found, 409 conflict, 422 validation failure, 429 rate limited (with Retry-After), 5xx for server-side failures.
• Idempotency keys: the client sends a unique key with a POST; the server stores the result and replays it for a retry, so a timeout never creates a duplicate.
• Conditional requests with ETag: If-None-Match for caching (304 Not Modified), If-Match to prevent lost updates (412 Precondition Failed).

Remember. Most production APIs sit at level 2 of the Richardson maturity model — resources, methods and status codes. Full hypermedia (level 3, HATEOAS) is rare and mainly pays off for public APIs with unknown clients.

OpenAPI and Swagger

What it is. The OpenAPI Specification (OAS) is a standard, machine-readable description of an HTTP API in YAML or JSON: its paths, operations, parameters, request and response bodies, data schemas and authentication. “Swagger” was the specification’s original name; today it is SmartBear’s tooling around it — Swagger UI, Swagger Editor, Swagger Codegen.

Purpose. One source of truth for the API contract, from which you get interactive documentation, generated clients and server stubs, request validation, mock servers, contract tests and automated detection of breaking changes.

How it works. The document lists paths, each with operations (get, post, …) that have parameters, a requestBody and responses. Reusable data models live under components/schemas, written in JSON Schema; securitySchemes describe authentication (OAuth 2.0, bearer tokens, API keys). Version 3.1 (2021) aligned fully with JSON Schema and is what most tools — FastAPI included — generate today; 3.2 followed in 2025.

openapi: 3.1.0
info: {title: Patients API, version: 1.4.0}
paths:
  /patients/{id}:
    get:
      parameters:
        - {name: id, in: path, required: true, schema: {type: string}}
      responses:
        "200":
          description: The patient
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Patient"}
        "404": {description: Not found}
components:
  schemas:
    Patient:
      type: object
      required: [id, name]
      properties:
        id: {type: string}
        name: {type: string}

Code-first vs design-first. Code-first generates the specification from code (FastAPI, drf-spectacular): always accurate and fast to work with. Design-first writes and reviews the specification, then implements it: better when other teams build against the API in parallel. A common middle path: commit the generated specification and diff it in CI.

Remember. Run a breaking-change check (oasdiff, for example) against the committed specification on every pull request. Consumers can only trust a specification that cannot drift from the code.

Versioning and evolving an API

What it is. The rules for changing an API without breaking its clients.

Purpose. Clients deploy on their own schedule. A breaking change shipped without warning is an outage for somebody else.

Breaking vs non-breaking.

• Non-breaking (additive): new endpoints; new optional fields or parameters; new enum values — only if clients were told to tolerate unknown values.
• Breaking: removing or renaming a field; changing a type or format; making an optional input required; tightening validation; changing error shapes, status codes or pagination.

Strategies. A version in the URL path (/v1/...) — the most common and most visible; a version in a header or media type (Accept: application/vnd.example.v2+json); a query parameter. Whatever the mechanism, clients should be tolerant readers that ignore unknown fields.

Deprecation. Announce the change; signal it on the old endpoint with the Deprecation (RFC 9745) and Sunset (RFC 8594) response headers; measure who still calls it; remove it when the traffic is gone, not merely when the date passes.

Remember. The best versioning strategy is rarely needing a new version: design for additive change from the first day.

Pagination

What it is. Returning large collections in pages.

Purpose. Bounded response sizes, predictable latency, and protection of the database from unbounded queries.

-- keyset pagination: the cursor encodes the sort key of the last row seen
SELECT id, created_at, name
FROM patients
WHERE (created_at, id) > ($1, $2)
ORDER BY created_at, id
LIMIT 50;

Remember. Keep the cursor opaque (for example, base64 of the sort key) so its contents can change; cap the page size; return next_cursor (or a Link header) instead of a total count, which is expensive on large tables.

Error-handling conventions

What it is. One consistent, documented shape for errors across the whole API.

Purpose. Clients handle errors programmatically, support can correlate a report with logs, and nothing internal leaks out.

The standard. Problem Details for HTTP APIs — RFC 9457 (2023, replacing RFC 7807) — with the media type application/problem+json. Standard members: type (an identifier for the kind of problem), title, status, detail, instance, plus any extension members you need.

{
  "type": "/problems/validation-error",
  "title": "Request validation failed",
  "status": 422,
  "detail": "birth_date must be in the past",
  "instance": "/patients",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "errors": [{"field": "birth_date", "message": "must be in the past"}]
}

Rules. The right status code first; the same shape everywhere, validation errors included; a stable, machine-readable type or code clients can switch on; a trace or request id in every error; no stack traces, SQL or internal host names in responses; a Retry-After header on 429 and 503.

4. Databases — PostgreSQL, MySQL, migrations, query tuning

Relational databases and SQL

What it is. A relational database stores data in tables with typed columns, protects integrity with keys and constraints, and answers declarative SQL queries. PostgreSQL and MySQL are the two dominant open-source ones.

Purpose. Durable, consistent storage with transactions, joins across related data, and a query planner that turns declarative SQL into an efficient execution plan.

SQL to be fluent in. Inner and left JOIN; GROUP BY with HAVING; aggregates; subqueries and EXISTS; CTEs (WITH); window functions (ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...), running totals, ranking); upserts (INSERT ... ON CONFLICT in PostgreSQL, ON DUPLICATE KEY UPDATE in MySQL).

-- the latest observation per patient, with a window function
SELECT patient_id, code, value, observed_at
FROM (
    SELECT o.*,
           ROW_NUMBER() OVER (PARTITION BY patient_id ORDER BY observed_at DESC) AS rn
    FROM observations AS o
) AS ranked
WHERE rn = 1;

Transactions and isolation. ACID: atomicity, consistency, isolation, durability. Both databases use MVCC, so readers do not block writers. Isolation levels:

• Read Committed — the PostgreSQL default: each statement sees data committed before it began.
• Repeatable Read — the MySQL (InnoDB) default: one snapshot for the whole transaction.
• Serializable — prevents every anomaly; PostgreSQL implements it with serializable snapshot isolation, and the application must retry on serialization failures.

The classic bug is the lost update in read-modify-write code. Fix it with an atomic statement (UPDATE ... SET count = count + 1), optimistic locking (a version column checked in the WHERE), pessimistic locking (SELECT ... FOR UPDATE), or a constraint that makes the bad state impossible.

PostgreSQL vs MySQL

Migrations

What it is. Versioned, ordered scripts that change the database schema (and sometimes data), stored in the repository with the code. Tools: Alembic (SQLAlchemy), Django migrations, Flyway, Liquibase.

Purpose. Every environment’s schema is reproducible and matches the code; schema changes are reviewed, tested and applied automatically by the deploy pipeline.

How it works. Each migration has an id and a parent; the tool records the applied versions in a table (alembic_version, django_migrations) and applies the missing ones in order. Autogenerate features draft a migration from model changes — always read the draft before committing it.

"""add a nullable email column to patients"""
import sqlalchemy as sa
from alembic import op

revision = "a1b2c3"
down_revision = "f9e8d7"


def upgrade() -> None:
    op.add_column("patients", sa.Column("email", sa.String(255), nullable=True))


def downgrade() -> None:
    op.drop_column("patients", "email")

Zero-downtime rules — expand and contract. During a deploy the old and new code run at the same time, so every migration must work with both.

• Add columns as nullable (or with a constant default), backfill in small batches, and enforce NOT NULL afterwards.
• Never rename in place: add the new column, write to both, backfill, switch reads, stop writing the old column, and drop it in a later release.
• Build indexes without blocking writes: CREATE INDEX CONCURRENTLY in PostgreSQL (it cannot run inside a transaction); online DDL in MySQL, or gh-ost and pt-online-schema-change for what online DDL cannot do.
• In PostgreSQL, add constraints as NOT VALID and VALIDATE them separately, which checks existing rows without blocking writes.
• Set lock_timeout: a migration stuck behind a long query should fail fast instead of making every later query queue behind it.

Query tuning

What it is. Finding and fixing slow queries.

Purpose. The database is usually the bottleneck of a backend, and one bad query can saturate it for everyone.

The method.

1. Find what costs the most in total: pg_stat_statements in PostgreSQL, the slow query log or Performance Schema in MySQL. A fast query called a million times often matters more than one slow query.
2. Read the plan: EXPLAIN (ANALYZE, BUFFERS) in PostgreSQL (since 18, ANALYZE includes buffer counts by default), EXPLAIN ANALYZE in MySQL. Look for sequential scans of large tables, row estimates far from the actual counts (stale statistics — run ANALYZE), nested loops over many rows, and sorts spilling to disk. Note that ANALYZE executes the query.
3. Change one thing and measure again.

Indexes.

• B-tree is the default: equality, ranges and sorting.
• Composite indexes follow the left-prefix rule: equality columns first, the range column last.
• Partial indexes (WHERE status = 'pending') for selective subsets; covering indexes (INCLUDE) for index-only scans; expression indexes for conditions such as lower(email).
• GIN for JSONB and full-text search; BRIN for very large, append-only time-series tables.
• Every index slows down writes — drop the unused ones.

Why an index is ignored. A function or cast applied to the column (WHERE date(created_at) = ...); a leading wildcard (LIKE '%x'); a type mismatch; the wrong composite column order; or low selectivity — when a query matches a large share of the table, a sequential scan is genuinely cheaper.

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM appointments
WHERE clinic_id = 42 AND starts_at >= now()
ORDER BY starts_at
LIMIT 20;

-- if it scans, add an index that matches the filter and the sort:
-- equality column first, range column last
CREATE INDEX CONCURRENTLY idx_appointments_clinic_start
    ON appointments (clinic_id, starts_at);

Beyond the query. N+1 queries from the ORM (fix with eager loading); connection pooling (PgBouncer); short transactions; caching hot reads; read replicas for read-heavy load; partitioning for very large tables.

5. Testing — pytest, unit, integration, contract

pytest

What it is. The standard Python testing framework.

Purpose. Tests as plain functions with plain assert statements, setup composed from reusable fixtures, and a large plugin ecosystem.

How it works. pytest discovers test_*.py files and test_* functions, rewrites assert statements to show detailed failure messages, and injects fixtures into tests by parameter name.

Key concepts.

• Fixtures — reusable setup and teardown, requested by name. yield separates setup from teardown. Scope: function (the default), class, module, package or session. Shared fixtures live in conftest.py.
• @pytest.mark.parametrize — one test, many inputs.
• Markers — @pytest.mark.integration, then pytest -m "not integration" to select or skip groups.
• Built-in fixtures: tmp_path, monkeypatch, capsys, caplog.
• Plugins: pytest-asyncio or anyio (async tests), pytest-cov (coverage), pytest-xdist (parallel runs), Hypothesis (property-based testing), pytest-mock.

import pytest


def label(status: str) -> str:
    return status.capitalize()


@pytest.fixture
def patients():
    rows = {"p1": {"id": "p1", "status": "active"}}  # setup
    yield rows
    rows.clear()  # teardown


@pytest.mark.parametrize(
    ("status", "expected"),
    [("active", "Active"), ("inactive", "Inactive")],
)
def test_label(status: str, expected: str) -> None:
    assert label(status) == expected


def test_unknown_patient_is_absent(patients: dict) -> None:
    assert "missing" not in patients

Kinds of test

Purpose of the split. Fast feedback from many unit tests, real confidence from integration tests, and only a few end-to-end tests — the test pyramid. The “testing trophy” variant puts more weight on integration tests, which is usually right for backend services.

Practices.

• Never mock the database in integration tests: run real PostgreSQL with testcontainers and wrap each test in a transaction that is rolled back.
• Stub external HTTP at the transport layer (respx for httpx, responses for requests), using responses recorded from the real API.
• Test doubles: a stub returns canned answers; a mock verifies how it was called; a fake is a working lightweight implementation (an in-memory repository); a spy records calls.
• Test behaviour through public interfaces, not implementation details, so refactoring does not break tests.
• Use coverage to find untested code, not as a target; gate new code on diff coverage.

Contract tests

What it is. Tests that verify two services agree on their interface, without deploying them together.

Purpose. Catch breaking API changes before deploy when services are deployed independently — without slow, flaky shared end-to-end environments.

How it works — consumer-driven contracts with Pact.

1. The consumer’s tests run against a Pact mock provider and record every interaction the consumer relies on: the request, and the parts of the response it actually uses. The result is a contract file (a “pact”).
2. The pact is published to a broker (Pact Broker or PactFlow).
3. The provider’s CI replays every consumer’s pact against the real provider and verifies the responses.
4. Before a deploy, can-i-deploy checks that the version is compatible with what is already running in that environment.

Schema-based alternatives. Validate responses against the OpenAPI specification, or generate property-based tests from it with Schemathesis. These prove the provider matches what it declares — but not which fields consumers actually depend on.

Third-party APIs. You cannot run their CI, so the contract becomes recorded real responses replayed in your tests, plus a scheduled job against their sandbox that alerts when their API drifts.

Remember. Contract tests replace most cross-service end-to-end tests; they pay off when different teams deploy services independently.

6. Git, code review and trunk-based development

Git

What it is. A distributed version-control system: every clone holds the complete history.

Purpose. Record every change with its author and reason, let people work in parallel on branches, merge safely, and recover almost anything.

How it works. History is a directed acyclic graph of immutable commits. Each commit points to its parent commits and to a snapshot of the files, and is identified by a hash of its contents. Branches and tags are just movable pointers to commits; HEAD points to the current branch.

Key operations.

• merge joins two histories with a merge commit; rebase replays your commits on top of another branch for a linear history. Never rebase commits that other people have built on.
• A squash merge turns a whole pull request into one commit on main.
• git push --force-with-lease refuses to overwrite the remote if someone else pushed in the meantime; plain --force silently destroys their work.
• git revert creates a new commit that undoes an old one (safe on shared branches); git reset moves the branch pointer (for local history only).
• git reflog lists every position HEAD has had — the safety net for recovering “lost” commits.
• git bisect binary-searches history for the commit that introduced a bug.
• Also: cherry-pick, stash, and worktree (several branches checked out at once).

Code review

What it is. Another engineer examining a change before it merges, usually as a pull request (a merge request in GitLab).

Purpose. Find defects and design problems early, spread knowledge across the team, keep the codebase consistent — and, in regulated environments, produce evidence that every change was reviewed.

What a good review looks at. Correctness and edge cases first; then design and boundaries; tests (do they test behaviour, and would they fail if the code were wrong?); security and data handling; operability (logging, timeouts, migration safety, N+1 queries); readability. Formatting and style belong to tools — Ruff, mypy, pre-commit — and to CI, not to humans.

Mechanics and etiquette. Small pull requests — a few hundred lines — get real reviews. Label comments as blocking, suggestion or nit; ask questions rather than assume; explain why; review within hours, not days. A CODEOWNERS file routes reviews automatically; branch protection requires approvals and passing CI before merge.

Trunk-based development and short-lived branches

What it is. A branching model in which everyone integrates into one shared branch (main, the “trunk”) at least daily, using branches that live for hours or a day or two.

Purpose. Continuous integration in the literal sense: small changes merge constantly, so conflicts stay small, main is always releasable, and changes reach production quickly. The research behind the DORA metrics associates it with high-performing delivery teams.

How it works.

• Short-lived branch → small pull request → review and CI → merge (usually a squash) → deploy from main.
• Unfinished features are merged but hidden behind feature flags, which also separate deploying code from releasing a feature.
• main is protected: required reviews, required status checks, no direct or force pushes.
• Releases come from the trunk, or every merge deploys continuously; fixes go to the trunk first.

Remember. Long-lived branches are the problem trunk-based development solves: they drift away from main and produce painful merges and enormous reviews.

7. Regulated environments — HIPAA, GDPR, ISO 27001, SOC 2, medical devices

HIPAA

What it is. The US Health Insurance Portability and Accountability Act (1996) and its rules for protected health information (PHI): the Privacy Rule, the Security Rule and the Breach Notification Rule.

Purpose. Protect the privacy and security of health information held by healthcare organisations and the vendors that work for them.

Who it applies to. Covered entities — healthcare providers, health plans, clearinghouses — and their business associates: any vendor that creates, receives, stores or transmits PHI on their behalf, cloud and SaaS providers included. A Business Associate Agreement (BAA) is required with each one.

What it means for engineering. The Security Rule requires administrative, physical and technical safeguards: access control (unique user ids, least privilege), audit controls (records of who accessed which PHI), integrity controls, authentication, and transmission security (encryption in transit). Encryption at rest is technically “addressable” but expected in practice. “Minimum necessary”: use and disclose only the PHI a task needs. Breaches must be reported to the affected individuals without unreasonable delay and within 60 days of discovery.

De-identification. Either Safe Harbor — remove 18 kinds of identifiers (names, geographic units smaller than a state, all date elements except the year, contact details, record numbers and more) — or Expert Determination, in which a qualified statistician certifies that the re-identification risk is very small. De-identified data is no longer PHI.

GDPR

What it is. The EU General Data Protection Regulation, in force since May 2018. It governs the processing of personal data of people in the EU, wherever the processing company is based.

Purpose. Give people control over their personal data and make organisations accountable for how they process it.

Key concepts.

• Roles. The controller decides why and how data is processed; a processor processes it on the controller’s behalf under a data processing agreement (Article 28) — the European counterpart of a BAA.
• Lawful basis. Every processing needs one: consent, contract, legal obligation, vital interests, public task or legitimate interests. Health data is a special category (Article 9) and needs an additional condition, such as explicit consent or the provision of healthcare.
• Principles. Lawfulness, purpose limitation, data minimisation, accuracy, storage limitation, integrity and confidentiality, accountability.
• Rights. Access, rectification, erasure, restriction, portability, objection. Erasure has exceptions — medical records with a legal retention period are one.
• Privacy by design and by default, and a DPIA (data protection impact assessment) for high-risk processing such as health data at scale.
• Breaches must be reported to the supervisory authority within 72 hours.
• Transfers outside the EU need a legal mechanism: an adequacy decision (the EU–US Data Privacy Framework is one) or Standard Contractual Clauses.
• Fines reach €20 million or 4% of worldwide annual turnover.

Engineering consequences. Know where every personal field lives (a data inventory); support export and deletion per person; retention policies with automatic deletion; pseudonymisation; data residency (EU data in EU regions).

ISO/IEC 27001

What it is. The international standard for an information security management system (ISMS); the current edition is from 2022.

Purpose. A certifiable, organisation-wide framework for managing information-security risk that customers and partners can rely on.

How it works. The organisation defines the scope of its ISMS, assesses risks, selects controls from Annex A (93 controls in four themes: organisational, people, physical and technological), records its choices in a Statement of Applicability, and runs a continuous cycle of monitoring, internal audits, management reviews and improvement. An accredited certification body audits it; a certificate lasts three years, with annual surveillance audits.

What engineers feel. Access control and periodic access reviews, a secure development policy, change management (reviewed and tested changes), logging and monitoring, vulnerability management, backups with tested restores, supplier management and incident response.

SOC 2

What it is. An attestation report, defined by the AICPA, in which an independent CPA firm reports on a service organisation’s controls against the Trust Services Criteria: Security (always included), Availability, Processing Integrity, Confidentiality and Privacy.

Purpose. Let customers — mostly in the US — trust a SaaS vendor with their data without auditing it themselves.

Type I vs Type II. Type I says the controls were suitably designed on one date. Type II says they operated effectively over a period, typically 3 to 12 months — the one customers ask for. SOC 2 is a report, not a certification.

What engineers feel. Nearly the same controls as ISO 27001, with the emphasis on evidence. The good pattern is controls that produce their own evidence: branch protection and CI logs prove change management, identity provider exports prove access reviews, scanner output proves vulnerability management. Anything that depends on a person remembering shows up as a gap in a Type II period.

Medical-device software

What it is. Software that is itself a medical device — Software as a Medical Device (SaMD), for example an algorithm that diagnoses or recommends treatment — or software inside a device. It is regulated: by the FDA in the US, and under the Medical Device Regulation (MDR 2017/745) in the EU.

Purpose of the rules. Software that can affect patient safety must be developed, verified and maintained with discipline proportional to the risk.

The standards and rules.

• IEC 62304 — the software lifecycle standard. Each software system gets a safety class by the worst harm a failure could cause: A (no injury possible), B (non-serious injury), C (death or serious injury). The class decides how much is required: planning, requirements, architecture, detailed design (class C), unit verification, integration and system testing, release, maintenance, configuration management and problem resolution.
• ISO 14971 — risk management: identify hazards, estimate and control risks, verify the controls.
• ISO 13485 — the quality management system for device manufacturers. In the US, the FDA’s Quality Management System Regulation, aligned with ISO 13485, took effect in February 2026.
• SOUP — “software of unknown provenance”: every third-party component (a PyPI package, an operating-system library) is listed with its version and assessed for known anomalies.
• Traceability — from every requirement to its design, code and tests; from every risk control to its verification.
• Cybersecurity — US submissions for connected devices must include a software bill of materials (SBOM) and a plan to monitor and patch vulnerabilities.
• Market paths — in the US, 510(k), De Novo or PMA depending on risk; in the EU, conformity assessment by a notified body, with most decision-support software classed IIa or higher. AI in medical devices also counts as high-risk under the EU AI Act, with those obligations phasing in from late 2027 to 2028.

Remember. Agile is compatible with IEC 62304 when documentation and traceability are produced as part of the work. Your lockfile is the start of your SOUP list.

8. Event-driven architecture — Kafka, RabbitMQ, SQS, Pub/Sub

The concepts

What it is. An architecture in which services communicate by producing and consuming events — facts about something that happened, such as PatientRegistered or LabResultReceived — through a broker, instead of calling each other synchronously.

Purpose. Decoupling (producers do not know their consumers), resilience (a slow consumer does not slow the producer), independent scaling, and the ability to add new consumers without touching producers.

Key concepts.

• Event, command, query. An event states a fact (past tense, any number of listeners); a command asks for something to happen (one handler); a query asks for data.
• Point-to-point vs publish/subscribe. A queue gives each message to one consumer (work distribution); a topic gives each message to every subscription (fan-out).
• Delivery guarantees. At-most-once may lose messages; at-least-once may duplicate them and is the practical default everywhere; exactly-once exists only within limited scopes. So consumers must be idempotent — deduplicate by message id, or make the operation naturally idempotent (an upsert).
• Transactional outbox. A service that updates its database and publishes an event writes the event into an outbox table in the same transaction; a relay publishes it afterwards. This solves the dual-write problem — the database updated but the event lost, or the reverse.
• Ordering. Usually guaranteed only within a partition, message group or ordering key — choose that key by entity, such as the patient id.
• Dead-letter queue (DLQ). Where a message goes after repeated failures, so one “poison” message cannot block the rest. Someone has to watch it.
• Schema evolution. Events outlive the code that produced them: evolve them compatibly (add optional fields), often with a schema registry (Avro, Protobuf or JSON Schema).
• Choreography vs orchestration. Services reacting to each other’s events, versus a central coordinator — a saga orchestrator or a workflow engine such as AWS Step Functions or Temporal — driving the steps and running compensating actions when a step fails.
• Eventual consistency. Readers can briefly see stale data; design the user experience for it.

Apache Kafka

What it is. A distributed, replicated, append-only log — an event streaming platform.

Purpose. High-throughput, durable event streams that many consumers read independently and can replay: event backbones between services, change data capture, analytics pipelines and stream processing.

How it works. Producers append records to a topic, which is split into partitions; every record gets an increasing offset within its partition. Records are kept for a configured time or size — or compacted down to the latest value per key — whether or not anyone has read them. Consumer groups divide a topic’s partitions among their members, and each group tracks its own offsets, so many groups read the same data independently. Each partition is replicated across brokers: a leader and in-sync followers.

Key facts.

• Ordering is per partition. The record key decides the partition, so all events for one patient stay in order.
• Parallelism is capped by the partition count: consumers beyond it sit idle.
• Producers with acks=all and idempotence enabled avoid duplicates from their own retries; consumers commit offsets after processing for at-least-once delivery. Exactly-once exists for Kafka-to-Kafka processing through transactions — not for side effects in external systems.
• Kafka 4.0 (2025) removed ZooKeeper — Kafka manages its own metadata (KRaft) — and the 4.x line also brings a faster consumer rebalance protocol and share groups for queue-style consumption.
• Ecosystem: Kafka Connect (connectors to databases and storage), Kafka Streams, Schema Registry. Managed offerings: Amazon MSK, Confluent Cloud.

Use it for many consumers, replay, high volume or stream processing. Avoid it for a simple job queue, where its operational weight buys nothing.

RabbitMQ

What it is. A message broker implementing AMQP 0-9-1 (plus AMQP 1.0, MQTT and STOMP), built around flexible routing of messages into queues.

Purpose. Task queues, work distribution, request/reply and complex routing — messages that represent work to be done once.

How it works. Producers publish to an exchange, and bindings route messages to queues according to the exchange type: direct (exact routing key), topic (wildcard patterns such as lab.*.result), fanout (every bound queue) or headers. The broker pushes messages to consumers, which acknowledge each one; unacknowledged messages are redelivered. prefetch limits how many unacknowledged messages one consumer can hold.

Key facts.

• Quorum queues (replicated with Raft) are the high-availability queue type; classic mirrored queues were removed in RabbitMQ 4.0.
• Dead-letter exchanges catch rejected or expired messages; TTLs can be set per message or per queue.
• Streams add a replayable, append-only log type for Kafka-like needs.
• In ordinary queues a consumed message is gone — there is no replay.
• RabbitMQ is a common broker for Celery.

Amazon SQS

What it is. A fully managed message queue on AWS.

Purpose. Decouple producers from consumers with no operations work: no servers, scaling or replication to manage.

How it works. Producers send messages; consumers poll for them (long polling waits up to 20 seconds). A received message becomes invisible for the visibility timeout; the consumer deletes it after successful processing, otherwise it reappears and is retried. After maxReceiveCount failed attempts a redrive policy moves it to a dead-letter queue. Messages are retained for up to 14 days (4 by default).

Two queue types.

• Standard — nearly unlimited throughput, at-least-once delivery, best-effort ordering.
• FIFO — strict ordering within a message group id, and duplicates removed within a five-minute window using a deduplication id; lower throughput than standard queues.

Key facts. Set the visibility timeout longer than the slowest processing time. Pair SQS with SNS (topics) to fan one message out to several queues, or with EventBridge to route events by their content. Lambda can consume SQS directly through an event source mapping.

Google Cloud Pub/Sub

What it is. Google Cloud’s fully managed, global publish/subscribe messaging service.

Purpose. Fan-out messaging and event ingestion at scale, without managing brokers.

How it works. Publishers send messages to a topic; every subscription on it receives each message. Subscribers pull messages — or Pub/Sub pushes them to an HTTPS endpoint, authenticated with an OIDC token — and must acknowledge them before the ack deadline, or they are delivered again. Export subscriptions write messages straight into BigQuery or Cloud Storage.

Key facts. At-least-once delivery by default, with an optional exactly-once mode for pull subscriptions; ordering through ordering keys; dead-letter topics after a maximum number of delivery attempts; messages retained for up to 7 days per subscription (longer at topic level); “seek” to a timestamp or snapshot to replay.

Comparison

9. Containers — Docker and Kubernetes

Docker

What it is. A platform for building and running containers: isolated processes packaged together with their filesystem, libraries and configuration as an image.

Purpose. “It works on my machine” becomes “it works everywhere”: the same image runs on a laptop, in CI and in production. Containers start in seconds and are far lighter than virtual machines.

How it works. Containers share the host’s kernel. Linux namespaces isolate what a process can see (processes, network, filesystem) and cgroups limit what it can use (CPU, memory). An image is a stack of read-only layers, one per Dockerfile instruction, cached and reused between builds; a running container adds a thin writable layer on top. Images follow the OCI standard and live in registries — Docker Hub, Amazon ECR, Google Artifact Registry, Azure Container Registry.

Key concepts. Dockerfile instructions (FROM, COPY, RUN, ENV, USER, EXPOSE, CMD / ENTRYPOINT, HEALTHCHECK); multi-stage builds (build tools in one stage, a slim runtime in the final one); layer caching (copy dependency files before the source code, so a code change does not reinstall everything); .dockerignore; volumes for persistent data; networks; Docker Compose for multi-container local environments.

# build stage: install the locked dependencies, then the project
FROM python:3.14-slim AS build
RUN pip install --no-cache-dir uv
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev --no-install-project
COPY src ./src
RUN uv sync --frozen --no-dev

# runtime stage: no build tools, non-root user
FROM python:3.14-slim
ENV PYTHONUNBUFFERED=1 PATH="/app/.venv/bin:$PATH"
WORKDIR /app
COPY --from=build /app /app
USER 10001
EXPOSE 8000
CMD ["uvicorn", "my_service.main:app", "--host", "0.0.0.0", "--port", "8000"]

Remember. A slim base image; one process per container; a non-root user; configuration through environment variables; never bake secrets into an image — they stay visible in the layer history.

Kubernetes

What it is. An open-source container orchestrator — originally from Google, now a CNCF project — that runs containers across a cluster of machines.

Purpose. You declare the desired state (“three replicas of this image, reachable at this name”) and Kubernetes keeps reality matching it: scheduling, self-healing, scaling, rolling updates, service discovery and configuration.

How it works. The control plane — the API server, etcd (the cluster’s state store), the scheduler and the controller manager — stores the desired state and runs reconciliation loops that push reality towards it. Every node runs a kubelet (which starts and watches containers through a container runtime such as containerd) and kube-proxy (service networking). You apply YAML manifests with kubectl, Helm charts, or GitOps tools such as Argo CD.

Core objects.

Health and resources.

• A failing liveness probe restarts the container — keep it trivial (“is the process alive”). A failing readiness probe removes the pod from load balancing — it checks the dependencies needed to serve. A startup probe protects slow starters.
• Resource requests drive scheduling and guarantee capacity; limits cap usage — a container over its memory limit is killed, one over its CPU limit is throttled.
• Graceful shutdown: handle SIGTERM and finish in-flight requests within the termination grace period.

apiVersion: apps/v1
kind: Deployment
metadata: {name: patients-api}
spec:
  replicas: 3
  selector: {matchLabels: {app: patients-api}}
  template:
    metadata: {labels: {app: patients-api}}
    spec:
      containers:
        - name: api
          image: patients-api:1.4.0
          ports: [{containerPort: 8000}]
          resources:
            requests: {cpu: 250m, memory: 256Mi}
            limits: {memory: 512Mi}
          readinessProbe: {httpGet: {path: /ready, port: 8000}}
          livenessProbe: {httpGet: {path: /live, port: 8000}}

Remember. Managed Kubernetes (EKS, GKE, AKS) removes the control-plane work but not the complexity. For a handful of services, simpler platforms — ECS with Fargate, Cloud Run, Azure Container Apps — are often the better choice.

10. Infrastructure as code — Terraform, Pulumi, CloudFormation

The concept

What it is. Defining cloud infrastructure — networks, databases, queues, permissions — in version-controlled files instead of clicking through a console.

Purpose. Reproducible environments (staging really matches production), reviewable changes (pull requests for infrastructure), an audit trail, disaster recovery by re-applying the code, and no undocumented hand-made resources.

How it works. Declarative tools compare the desired state in code with the real state and compute a plan — create, update in place, replace, delete. You review the plan, then apply it. Most tools track what they manage in a state file or service.

Practices. Plan on every pull request and apply from the pipeline, never from a laptop; remote, locked and encrypted state; reusable modules; one stack or workspace per environment; secrets from a secret manager, never in code; drift detection (someone changed something by hand); policy as code (OPA, Checkov, Sentinel) to block insecure changes; protection against accidentally replacing stateful resources such as databases.

Terraform

What it is. A declarative IaC tool by HashiCorp (part of IBM since 2025) that uses HCL, the HashiCorp Configuration Language, with thousands of providers for clouds and SaaS products.

How it works. You declare resource blocks (things to manage), data sources (things to read), variables and outputs, grouped into reusable modules. terraform plan refreshes the real state, diffs it against the code and shows the actions; terraform apply executes them. The state file maps the code to real resource ids. Teams keep it in a remote backend (for example S3 with locking) because it must be shared, locked against two applies at once, and protected — it holds secrets in plain text.

resource "aws_s3_bucket" "reports" {
  bucket = "clinic-reports-prod"
}

resource "aws_s3_bucket_server_side_encryption_configuration" "reports" {
  bucket = aws_s3_bucket.reports.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

Key facts. One workflow across every cloud. import blocks bring existing resources under management; moved blocks rename resources without destroying them; “forces replacement” in a plan on a database means data loss. Licence: the Business Source License since 2023; OpenTofu is the open-source fork with the same workflow.

Pulumi

What it is. An IaC tool in which infrastructure is written in general-purpose programming languages — Python, TypeScript, Go, C#, Java (or YAML) — instead of a configuration language. It is open source (Apache 2.0).

How it works. Your program declares resources as objects; running it produces the desired state, which the Pulumi engine diffs against the stack’s recorded state. pulumi preview shows the plan, pulumi up applies it, pulumi destroy removes everything. A stack is one instance of the program — dev, staging or prod — with its own configuration and state. State lives in Pulumi Cloud by default, or in a backend you run (S3, Azure Blob Storage, Google Cloud Storage, a local folder). Secrets in configuration are encrypted. Many providers are bridged from Terraform’s, so cloud coverage is similar.

import pulumi
import pulumi_aws as aws

bucket = aws.s3.Bucket("reports", tags={"stack": pulumi.get_stack()})
pulumi.export("bucket_name", bucket.bucket)

Why choose it. Loops, functions, classes, packages and unit tests come from the language; the team writes infrastructure in the language it already knows; the Automation API lets programs drive deployments. The trade-off: real code can become too clever, where a configuration language forces simplicity.

AWS CloudFormation

What it is. AWS’s native IaC service: templates in YAML or JSON, deployed as stacks.

How it works. A template has Parameters, Resources and Outputs (plus Mappings and Conditions), wired together with intrinsic functions such as !Ref, !GetAtt and !Sub. CloudFormation creates resources in dependency order and rolls back automatically if something fails. Change sets preview changes, drift detection finds manual edits, and StackSets deploy across accounts and regions. AWS keeps the state, so there is no state file to manage.

Resources:
  ReportsBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault: {SSEAlgorithm: aws:kms}
Outputs:
  BucketName:
    Value: !Ref ReportsBucket

Related. AWS SAM extends CloudFormation for serverless applications (Lambda, API Gateway) with sam build and sam deploy; the AWS CDK lets you write infrastructure in TypeScript, Python and other languages and generates CloudFormation from it.

Comparison

11. Observability — OpenTelemetry, Datadog, Grafana, Sentry

The concepts

What it is. The ability to understand what a running system is doing, and why, from the telemetry it emits.

Purpose. Detect problems before users report them, find root causes quickly, and measure reliability against agreed targets.

The signals.

• Logs — timestamped records of discrete events, structured as JSON so they can be queried.
• Metrics — numeric time series: request rate, error rate, latency percentiles, queue depth. Cheap to store and to alert on.
• Traces — the path of one request through every service, as a tree of timed spans linked by a trace id that travels in request headers.
• Profiles — where CPU time and memory go inside a process.

Key ideas.

• Correlate everything by trace id: from a dashboard spike to a slow trace to that trace’s log lines.
• RED metrics for services (Rate, Errors, Duration); USE for resources (Utilisation, Saturation, Errors).
• SLI — what you measure; SLO — the target, for example 99.9% of requests under 300 ms over 30 days; error budget — how much unreliability the SLO allows. Alert on symptoms users feel, not on every possible cause.
• Cardinality: every unique combination of metric labels is a separate time series — never put user or patient ids into metric labels.
• Sampling for traces: head-based (decided when the trace starts) or tail-based (decided after it ends, so errors and slow requests are always kept).

OpenTelemetry

What it is. OpenTelemetry (OTel) is the open, vendor-neutral standard for producing telemetry: APIs, SDKs, semantic conventions, a wire protocol (OTLP) and the Collector. It is a CNCF project, formed from the merger of OpenTracing and OpenCensus.

Purpose. Instrument your code once and send the data to any backend — Datadog, Grafana, Honeycomb, Elastic, a cloud provider — without lock-in.

How it works. Libraries and auto-instrumentation create spans and metrics. Context travels between services in the W3C traceparent header. Exporters send the data over OTLP (gRPC on port 4317, HTTP on 4318) to an OpenTelemetry Collector, whose pipeline of receivers, processors (batching, sampling, redaction) and exporters forwards it to one or more backends. Traces, metrics and logs are stable; profiles are still in development.

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "patients-api"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("validate-submission") as span:
    span.set_attribute("submission.rows", 1200)

Remember. Auto-instrumentation — opentelemetry-instrument python app.py — covers FastAPI, Django, Flask, SQLAlchemy, httpx and more without code changes.

Datadog

What it is. A commercial SaaS observability platform: infrastructure metrics, APM (traces), logs, real-user monitoring, synthetic tests, security monitoring and more, in one product.

Purpose. Buy rather than build — metrics, traces and logs correlated in one place, with hundreds of integrations and no observability infrastructure to operate.

How it works. The Datadog Agent runs on every host (as a DaemonSet on Kubernetes): it collects host metrics, receives custom metrics (DogStatsD) and traces, tails logs, and forwards everything to Datadog. Python services are instrumented with ddtrace (ddtrace-run python app.py) or with OpenTelemetry. Unified service tagging (env, service, version) ties the signals together.

Remember. The cost is driven by hosts, custom-metric cardinality and log indexing — control tags, sample traces, and exclude noisy logs.

Grafana and Prometheus

What it is. Grafana is an open-source (AGPLv3) dashboard and visualisation tool that queries many data sources — Prometheus, Loki, Tempo, Elasticsearch, CloudWatch, SQL databases — and alerts on them. Grafana Labs’ “LGTM” stack is Loki (logs), Grafana, Tempo (traces) and Mimir (metrics); Grafana Cloud hosts it. Prometheus (a graduated CNCF project) is the standard open-source metrics system.

Purpose. Self-hostable, vendor-neutral monitoring and dashboards.

How Prometheus works. It pulls (scrapes) metrics from HTTP /metrics endpoints at intervals, stores them as local time series, and queries them with PromQL; Alertmanager routes the alerts. Metric types: counter (only goes up), gauge (goes up and down), histogram and summary (distributions, for latency percentiles). Prometheus 3 also accepts OpenTelemetry metrics natively.

Remember. A typical open-source stack: OpenTelemetry for instrumentation; Prometheus or Mimir for metrics, Loki for logs, Tempo for traces; Grafana for dashboards and alerts.

Sentry

What it is. An application error-tracking and performance-monitoring platform, used as SaaS or self-hosted.

Purpose. Know about every unhandled exception — with its stack trace, the request context and the release that introduced it — before users report it.

How it works. The Sentry SDK hooks into the framework (FastAPI, Django, Flask and Celery integrations) and captures exceptions with stack traces, local variables, breadcrumbs (the events leading up to the error), tags and user context. It groups similar errors into issues, tracks regressions by release, and records traces for performance monitoring.

Remember. In regulated domains configure data scrubbing — send_default_pii=False, include_local_variables=False, and a before_send hook that removes sensitive fields — so no patient data reaches Sentry; and have a BAA or DPA in place if any could.

12. AI-assisted development tools

What it is. Tools that use large language models to help write, understand, test and review code.

Purpose. Speed up the mechanical work — boilerplate, tests, refactors, unfamiliar APIs, exploring a codebase — so engineers spend more time on design and judgement.

Categories.

• Inline completion in the editor, such as GitHub Copilot.
• Chat assistants for design discussion and explaining code.
• Agentic coding tools that read the codebase, edit many files, run commands and tests, and iterate on the results: Claude Code, Cursor, OpenAI Codex, GitHub Copilot’s agent mode, Windsurf, Cline; spec-driven tools such as Kiro (AWS) and agent-first environments such as Antigravity (Google).
• AI code review on pull requests.
• MCP (Model Context Protocol) — an open protocol for connecting AI tools to external systems such as issue trackers, databases and documentation.

How teams use them well.

• A project context file the tool reads (CLAUDE.md, AGENTS.md, .cursorrules) with conventions, commands and architecture notes.
• Small, well-specified tasks; asking for the plan before the implementation.
• Tests written first, or at least reviewed by a human, so the tests are the specification rather than whatever the model decided.
• Every generated change goes through the same review and CI gates as human code — linting, type checks, tests, security scanning. A prompt asking for quality is a request; a gate is a guarantee.
• Extra care where generated code looks right but is not: security, concurrency, error handling, unfamiliar library contracts.

Risks. Plausible but wrong code; invented APIs; security flaws; licensing questions; review overload; and data leakage — in regulated work, no patient data, secrets or production payloads go into a tool unless an enterprise agreement with zero-retention terms keeps it inside the compliance boundary.

Remember. The honest answer to “has it made you faster?” — faster on mechanical work, not on understanding the problem; reviewing matters more than it did, not less.

13. Healthcare, life sciences and data-sensitive domains

What it is. Domains where the data is personal and sensitive — health records, genetic data, insurance and financial data — and mistakes carry legal and human consequences.

Purpose of the extra discipline. Patients and customers trust the system with information that can harm them if exposed, and regulators require proof that it is protected.

What changes for an engineer.

• Classify the data: PHI or PII, de-identified, or public — and know where every sensitive field lives.
• Minimise: collect, select and keep only what is needed; separate identifiers from clinical data (pseudonymisation).
• Control access: role- and attribute-based, enforced on the server for every request; emergency “break-glass” access that is logged and reviewed.
• Audit: who viewed or changed which record, and when — append-only, tamper-evident, retained for years, and queryable. It is a functional requirement, not debug logging.
• Encrypt: TLS everywhere, including between internal services; encryption at rest with managed keys (KMS) and rotation; field-level encryption for the most sensitive fields.
• Log carefully: never log payloads; allow-list the fields that may be logged; scrub error trackers.
• Respect consent: record what the person agreed to and when, and enforce it in processing.
• Retain and delete deliberately: medical records often must be kept for years by law, while privacy law demands deletion when data is no longer needed — model both explicitly.
• Keep production data out of non-production: use synthetic or properly de-identified data in development and staging.
• Contract with vendors: a BAA (US) or DPA (EU) with every vendor that touches the data — cloud, email, error tracking, LLM APIs.
• Respect residency: keep data in the regions that law and contracts require.

Life sciences additions. GxP — good clinical, manufacturing and laboratory practice; FDA 21 CFR Part 11 for electronic records and signatures (audit trails, validated systems, controlled access); ALCOA+ data integrity — attributable, legible, contemporaneous, original, accurate, plus complete, consistent, enduring and available; computerised system validation, now done with the FDA’s risk-based Computer Software Assurance approach. In practice: never destructively update a regulated record — keep its history with append-only tables or event sourcing.

14. FHIR, HL7 and healthcare interoperability

HL7 version 2

What it is. The original Health Level Seven messaging standard, from the 1980s onwards: pipe-delimited text messages that still carry most hospital interface traffic — admissions, orders, lab results.

Purpose. Let hospital systems — admissions, laboratory, pharmacy, electronic health record — exchange events in a common format.

How it works. A message is a list of segments (lines). Fields are separated by | and components by ^. MSH is the header (sender, receiver, message type, control id), PID the patient, OBR and OBX the observation request and its results. Common message types: ADT^A01 (admission), ORM (order), ORU^R01 (observation result). Messages usually travel over MLLP on TCP, and the receiver replies with an ACK or a NAK. Senders resend when an ACK times out, so receivers must be idempotent on the message control id. Hospitals add custom “Z-segments”. Integration engines (Mirth / NextGen Connect, Rhapsody) route and transform the messages.

MSH|^~\&|LAB|HOSP|EHR|HOSP|20261005083000||ORU^R01|MSG00042|P|2.5.1
PID|1||123456^^^HOSP^MR||DOE^JANE||19800214|F
OBR|1||LAB789|2345-7^Glucose^LN
OBX|1|NM|2345-7^Glucose^LN||95|mg/dL|70-99|N|||F

HL7 FHIR

What it is. Fast Healthcare Interoperability Resources — HL7’s modern standard for exchanging healthcare data through a RESTful API, as JSON or XML resources.

Purpose. Make health-data exchange web-native: any developer who knows REST and JSON can read clinical data from an electronic health record, with consistent structures across systems. It underpins patient-access and interoperability regulation in the US and many national health programmes.

How it works. Data is modelled as about 150 resource types — Patient, Practitioner, Encounter, Observation, Condition, MedicationRequest, AllergyIntolerance, DiagnosticReport — that reference each other (an Observation’s subject points to Patient/123). The REST API is uniform: GET /Patient/123, GET /Observation?patient=123&code=2345-7, POST /Patient, PUT /Patient/123, with _history for earlier versions. Bundles group resources for search results and for atomic transactions. A server describes what it supports in its CapabilityStatement.

{
  "resourceType": "Observation",
  "id": "obs-42",
  "status": "final",
  "code": {
    "coding": [{"system": "http://loinc.org", "code": "2345-7", "display": "Glucose"}]
  },
  "subject": {"reference": "Patient/123"},
  "effectiveDateTime": "2026-10-05T08:30:00Z",
  "valueQuantity": {"value": 95, "unit": "mg/dL"}
}

Key concepts.

• Versions: R4 (2019) is what almost every production system implements; R5 (2023) has limited adoption; R6 is in ballot. Build against R4.
• Profiles and implementation guides (such as US Core) constrain the base resources. The base specification makes almost everything optional, so conformance is always measured against a profile.
• Terminologies give the codes meaning: LOINC (lab tests and observations), SNOMED CT (clinical terms), ICD-10 (diagnoses), RxNorm (medications).
• SMART on FHIR — authorisation built on OAuth 2.0 and OpenID Connect that lets apps launch from an EHR with scoped access, such as patient/Observation.read.
• Bulk Data ($export) — population-level export as NDJSON files.
• Implementations: HAPI FHIR (Java); managed FHIR stores from AWS (HealthLake), Google (Cloud Healthcare API) and Azure (Health Data Services); in Python, the fhir.resources package gives Pydantic models for every resource.

Other interoperability standards

• CDA (Clinical Document Architecture) — XML clinical documents such as discharge summaries; C-CDA in the US.
• DICOM — medical imaging files and their network protocol.
• IHE profiles — integration patterns that combine standards, such as cross-organisation document sharing.
• OMOP CDM — a relational common data model for research analytics. FHIR is for exchange; OMOP is for analysis.
• openEHR — a specification for clinical data repositories, common in Europe.

Remember. A realistic platform speaks HL7 v2 to hospitals and FHIR to modern systems, and normalises both into one internal model behind an adapter layer. The hard part is terminology mapping, not the syntax.

15. Performance — high throughput, low latency

What it is. Making a system handle more work (throughput — requests per second) and respond faster (latency — time per request), especially at the slow tail.

Purpose. User experience, meeting SLOs, and cost: a faster service needs fewer machines.

Core concepts.

• Percentiles. p50 is the typical request; p95 and p99 are the slow tail that many users still hit. Averages hide the tail. With fan-out, tails compound: a page that calls ten services is slow whenever any of them is.
• Little’s Law. Requests in flight = arrival rate × time in the system. It sizes pools, workers and queues.
• Orders of magnitude. Memory access ~100 ns; SSD read ~10–100 µs; a round trip inside a data centre ~0.5 ms; across a continent ~50–100 ms. Network and database calls dominate a backend’s time.
• The trade. Batching raises throughput but adds latency; queues absorb bursts but add waiting time.
• Saturation. Latency climbs steeply as utilisation approaches 100% — leave headroom.

The method. Measure first — traces, metrics, and profilers: cProfile, py-spy (attaches to a running process), Scalene, memray (memory). Find the bottleneck; change one thing; load-test before and after at realistic concurrency (Locust, k6), watching p99. Beware coordinated omission: a load generator that waits for slow responses under-reports latency.

Common fixes in Python backends.

• Database: fix N+1 queries, add the right index, reduce round trips, pool connections, add read replicas.
• Concurrency: async I/O or more workers for I/O-bound work; processes, native extensions or NumPy (or free-threaded builds) for CPU-bound work; never block the event loop.
• Caching: in-process for small hot data, Redis for shared data, HTTP caching and CDNs at the edge; protect against cache stampedes.
• Less work: pagination, batching, compression, faster serialisation (orjson, msgspec), smaller payloads.
• Asynchronous processing: move slow work to queues and background workers, and answer 202 Accepted.
• Behaviour under load: timeouts on every call, bounded queues, backpressure, rate limiting and load shedding — degrade gracefully rather than collapse.

Remember. “Measure, don’t guess” — and report an improvement as p99 before and after, under the same load.

16. Cloud deployment and CI/CD

The three clouds

What it is. AWS, Google Cloud and Azure rent compute, storage, databases, networking and managed services on demand, in regions made of isolated availability zones.

Purpose. No hardware to run, elastic capacity, managed services instead of self-operated ones, and global reach.

Key concepts. The shared responsibility model — the provider secures the cloud, you secure what you put in it; IAM with least privilege and short-lived roles instead of long-lived keys; VPCs, subnets and security groups; deployment across availability zones for resilience; managed services first; tagging and cost monitoring. For health data, use only services covered by the provider’s BAA.

CI/CD

What it is. Continuous integration: every change is merged often and verified automatically — build, lint, type check, tests. Continuous delivery: every change that passes is releasable at the push of a button. Continuous deployment: every change that passes goes to production automatically.

Purpose. Small, frequent, low-risk releases; fast feedback; a repeatable, auditable path to production.

A typical pipeline.

1. On a pull request: format and lint checks, type checks, unit tests, integration tests against a real database in a container, contract tests, security scans (dependencies, secrets, the container image), and an infrastructure plan.
2. On merge to main: build one container image, tagged with the commit, and push it to the registry.
3. Deploy to staging and run smoke tests; promote the same image to production.
4. Release with a strategy: rolling (replace instances gradually), blue-green (switch traffic between two complete environments), canary (send a small share of traffic first and watch the metrics) — with feature flags to separate deploying from releasing.
5. Run database migrations as a pipeline step, backward-compatible (expand before contract).
6. Roll back automatically — redeploy the previous image — when health checks or SLOs fail.

name: ci
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:18
        env: {POSTGRES_PASSWORD: test}
        ports: ["5432:5432"]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv sync --frozen
      - run: uv run ruff check . && uv run mypy src
      - run: uv run pytest

Tools. GitHub Actions, GitLab CI, Bitbucket Pipelines, Jenkins, AWS CodePipeline, Azure DevOps; GitOps tools (Argo CD, Flux) deploy to Kubernetes by keeping the cluster in sync with a Git repository.

Securing the pipeline. Authenticate to the cloud with short-lived OIDC federation instead of stored keys; pin third-party actions; give pipeline roles least privilege; produce an SBOM and sign images (Sigstore cosign) for supply-chain integrity (SLSA).

Measuring delivery. The four DORA metrics: deployment frequency, lead time for changes, change failure rate, and time to restore service.

One line each — the ten-minute refresh

• Type hints — static annotations checked by mypy or pyright in CI; not enforced at runtime.
• Protocol — a structural interface: anything with the right methods fits.
• asyncio — single-threaded cooperative concurrency for I/O; tasks switch at await.
• TaskGroup — structured concurrency: tasks fail and cancel together.
• pyproject.toml — the one file for metadata, dependencies and tool configuration.
• uv — fast package and project manager with a committed lockfile.
• src layout — code under src/, so tests import the installed package.
• FastAPI — async API framework; type hints become validation and OpenAPI.
• Django — batteries-included: ORM, migrations, admin, auth; DRF for APIs.
• Flask — minimal WSGI framework; you choose the rest.
• REST — resources, standard methods, meaningful status codes, statelessness.
• OpenAPI — the machine-readable API contract; Swagger is its tooling.
• API evolution — additive by default; breaking changes get a new version plus Deprecation and Sunset.
• Cursor pagination — WHERE key > last_seen; fast at any depth.
• Problem Details — RFC 9457, the standard error body.
• PostgreSQL — feature-rich; Read Committed by default; transactional DDL.
• MySQL — InnoDB clusters on the primary key; Repeatable Read by default; use utf8mb4.
• Migrations — versioned schema changes; expand and contract for zero downtime.
• EXPLAIN ANALYZE — the real plan with timings; the starting point of tuning.
• pytest fixture — reusable setup and teardown, injected by name.
• Integration test — real dependencies in containers; the database is never mocked.
• Contract test — the consumer’s expectations, verified in the provider’s CI.
• Trunk-based development — integrate into main at least daily; flags hide unfinished work.
• Code review — correctness, design, tests and operability; tools own the style.
• HIPAA — US health privacy and security; BAAs; minimum necessary; 60-day breach notice.
• GDPR — EU personal data; health is a special category; 72-hour breach notice.
• ISO 27001 — a certifiable information security management system.
• SOC 2 — an auditor’s report on controls; Type II covers months of operation.
• IEC 62304 — the medical-device software lifecycle; safety classes A, B, C; SOUP.
• Event-driven — services exchange events through a broker; at-least-once, so idempotent consumers.
• Outbox — the event is written in the same database transaction and published afterwards.
• Kafka — a replicated log: partitions, consumer groups, offsets, replay.
• RabbitMQ — an AMQP broker: exchanges route messages to queues; consumers acknowledge.
• SQS — a managed AWS queue: visibility timeout; standard or FIFO.
• Pub/Sub — managed GCP topics and subscriptions; ordering keys.
• Docker — images and containers; layers; multi-stage builds.
• Kubernetes — reconciles desired state: pods, deployments, services, probes.
• Infrastructure as code — infrastructure in versioned code: plan, review, apply.
• Terraform — HCL, providers and a state file; OpenTofu is the open fork.
• Pulumi — infrastructure in Python or TypeScript: preview, up, stacks.
• CloudFormation — AWS-native templates and stacks with automatic rollback.
• OpenTelemetry — vendor-neutral traces, metrics and logs; OTLP; the Collector.
• Datadog — SaaS observability: the Agent, ddtrace, unified tags.
• Grafana — dashboards and alerts over Prometheus, Loki and Tempo.
• Prometheus — pull-based metrics and PromQL.
• Sentry — error tracking with stack traces; scrub personal data.
• AI-assisted development — completion, chat and agents; human review and CI gates; no PHI in prompts.
• FHIR — REST and JSON health resources; build on R4; profiles; SMART on FHIR.
• HL7 v2 — pipe-delimited hospital messages over MLLP, acknowledged with ACK or NAK.
• p99 — the latency 99% of requests beat; the tail users actually feel.
• CI/CD — verify automatically, build once, promote the same image, roll back fast.
• DORA metrics — deployment frequency, lead time, change failure rate, time to restore.