Application debugging and introspection

The interview question is really “what do you do when you cannot reproduce it”. The tooling below splits by that: interactive tools for a bug you can trigger, introspection and telemetry for one you cannot.

Interactive: when you can trigger it

breakpoint() is the built-in entry point, and it respects PYTHONBREAKPOINT, so you can route it at your IDE’s debugger or disable it in production without touching the code.

def charge(order):
    if order.total < 0:
        breakpoint()      # PYTHONBREAKPOINT=0 disables
    return gateway.charge(order)

The pdb commands worth memorising: n next, s step in, c continue, w where (the stack), p expr print, pp pretty-print, u/d move up and down frames.

Conditional breakpoints beat print statements for a bug that only happens on one input. In pdb that is b app.py:41, order.total < 0; in an IDE it is a field on the breakpoint.

Introspection: when you have an object and no documentation

inspect answers “what is this and where did it come from” at runtime, which is how you debug a framework you did not write:

import inspect

inspect.signature(fn)        # (a: int, b: str = 'x') -> bool
inspect.getsourcefile(fn)    # which file it really came from
inspect.getmro(cls)          # the full resolution order

getmro is the one that settles arguments. When a method does something unexpected in a class with several bases, the MRO tells you which implementation actually wins — see Inheritance in Python.

inspect.stack() gives you the caller when a function is invoked from somewhere you cannot find:

caller = inspect.stack()[1]
log.warning("called from %s:%d", caller.filename, caller.lineno)

Useful for a deprecation warning that needs to name the offender. Expensive — it builds frame objects — so keep it out of hot paths.

Live processes: when you cannot restart it

py-spy attaches to a running process and samples its stack without modifying or pausing it, which is the only option for a production service that is misbehaving now:

py-spy dump --pid 4213        # what is it doing right now
py-spy top --pid 4213         # live profile, like top

That answers “is it stuck or slow”, which is the first fork in the diagnosis and the one a log line rarely settles.

Unreproducible: start from telemetry

For a bug that happened yesterday to one user, the tooling above is useless. The order that works:

1. Metrics — when did it start, and does it correlate with a deploy?
2. Traces — which span holds the time or the error?
3. Logs, filtered by correlation ID — what did that one request do?

Which only works if the correlation ID is on every line:

log.info(
    "charge_failed",
    extra={"trace_id": ctx.trace_id, "order": order.id},
)

Reproduce locally after telemetry tells you the conditions, not before. See Observability.

Memory that grows

import tracemalloc

tracemalloc.start()
snap1 = tracemalloc.take_snapshot()
...
for stat in snap2.compare_to(snap1, "lineno")[:10]:
    print(stat)

Compare two snapshots rather than reading one — the absolute numbers are noise, the delta is the leak. The usual causes are an unbounded cache, module-level state that only accumulates, and objects held alive by a reference cycle.

Gotcha: profile under production-like load and data size. A profile taken against ten rows optimises the wrong function, because the code that dominates at scale is often not the code that dominates at rest.