Bash: triage and scripting

The interview question is never “what does ls do”. It is “the server is misbehaving, what do you type” — and, if you write scripts, whether yours fail safely.

The safety line

bash

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

Without it, a script keeps going after a failed command and does the next thing against a half-finished state.

Flag	Stops
`-e`	on any command returning non-zero
`-u`	on an unset variable
`-o pipefail`	on a failure anywhere in a pipe

pipefail is the one people omit. curl bad-url \| jq .name succeeds without it, because only jq’s exit status counts — so the script proceeds with an empty value.

bash

# fail loudly, at the top
: "${DATABASE_URL:?must be set}"
tmp=$(mktemp)
# cleanup on any exit path
trap 'rm -f "$tmp"' EXIT

And quote every expansion. rm -rf $dir with dir="/var/tmp " unset or containing a space is the classic destructive bug; rm -rf "$dir" is not.

Triage: the disk is full

bash

df -h                    # which filesystem
du -sh /var/* | sort -h  # descend into the biggest
lsof +L1                 # deleted files still held open

lsof +L1 is the one that explains the mystery: a process holding a deleted log file keeps the space allocated until it closes the handle, so du shows plenty of room and df disagrees. Restart the process, or truncate rather than delete:

bash

# frees space with the file still open
: > /var/log/app.log

Triage: something is slow

bash

top -o %CPU              # or htop
ps aux --sort=-%mem | head
ss -tulpn                # which process owns which port

For one suspect process:

bash

lsof -p 4213             # open files and sockets
strace -p 4213 -f -e trace=network
cat /proc/4213/status    # memory, threads, state

The state field is the quick answer to “stuck or busy”: R is running, D is uninterruptible sleep, which almost always means blocked on disk or NFS.

Text processing that earns its place

bash

# Top 10 IPs in an access log.
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# Requests per minute, to spot the spike.
awk '{print substr($4, 2, 17)}' access.log | uniq -c

sort | uniq -c | sort -rn is the single most useful pipeline in operations — count occurrences, rank them. It answers “what is hammering us” in one line, on a box with no observability stack installed.

Gotcha: uniq only collapses adjacent duplicates, which is why sort comes first. uniq -c on unsorted input silently under-counts, and the output looks entirely plausible.

Interview angle 6

“What should every non-trivial script start with?” - set -euo pipefail: exit on error, fail on unset variables, and propagate failure through pipes. Without pipefail a broken curl piped into jq reports success and the script carries on with empty data.
“How do you find what’s consuming disk?” - df -h for the filesystem, then du -sh descending into the largest directory. If they disagree, lsof +L1 finds a deleted file still held open by a process, which keeps the space allocated.
“How do you inspect a running process?” - ps to find it, lsof -p for open files and sockets, strace for syscalls, /proc/<pid>/status for state. A process in D state is blocked on I/O, not busy.
“Which port is that process on?” - ss -tulpn, which maps listening sockets to PIDs. netstat is the older command people still name; ss is the current one.
“Count the top talkers in a log with no tooling installed.” - awk '{print $1}' | sort | uniq -c | sort -rn | head. The sort before uniq is mandatory, because uniq only collapses adjacent lines and silently under-counts otherwise.
“What’s the classic destructive scripting bug?” - an unquoted expansion. rm -rf $dir word-splits, so an empty or space-containing variable turns into a very different command. Quote every expansion and use -u so an unset one fails instead.