Tech Check Prep — Senior Backend + Applied AI (FinTech, Part-Time)

Interview: Friday, 14 August Prep window: ~2 days (Wed evening → Fri morning)

This is the plan and the depth. The three companion files: Answers — say these out loud — every question with the answer written the way you would speak it. The call itself — everything except the answers — the opening, the self-intro, follow-ups and the close. Tech Check 14 August — Question Navigation — where to read more on any topic.

1. The single most important insight

Compare the client’s feedback on the rejected candidate to the question list they gave you:

The question list was written from the last rejection. These are not warm-up questions — they are the pass/fail gates. The CTO already knows what “shallow AI” sounds like and is specifically hunting for it.

Your actual advantage: you have LangGraph agents in production, a real pgvector RAG refactor, and eval-adjacent work. The previous candidate did not. The gap is not your knowledge — it’s whether you demonstrate depth in the room.

2. The three behaviours that decide this interview

2.1 Reason aloud (this got the last guy rejected)

Never answer a design question with a single sentence. Use this shape every time:

Context → “On the wealth-planning platform we had X constraint…” Options → “We considered A and B…” Decision + why → “We went with A because…” Trade-off you accepted → “The cost was…” Result / what you’d change → “It cut p95 from 4s to 900ms; next time I’d…”

Target 60–120 seconds per technical answer. If you finish in 15 seconds, you failed the question even if the content was correct.

2.2 Never say “I don’t know” and stop

The feedback said “gives up quickly.” Replace the full stop with a bridge:

• “I haven’t used Weaviate specifically, but I’ve run the same pattern on pgvector with HNSW — the trade-off there is…”
• “I haven’t done DPO myself. My understanding is it replaces the reward-model step in RLHF. Where I have worked is the retrieval side, where…”
• “Let me think about that out loud for a second.”

Honest + still reasoning ≫ honest + silent.

2.3 English

The client explicitly called B2-with-mistakes “weak for client-facing work.” This is a part-time role working directly with the CTO — communication is a scored dimension, not a nice-to-have.

Concrete actions for the next 2 days:

• Record yourself answering 5 of the questions below out loud, in English. Listen back once. You’ll catch your own filler and tense errors faster than any study.
• Pre-write and rehearse the 90-second self-intro and the “last project” answer until they’re automatic. These two set the tone for everything after.
• Slow down deliberately. Fluency reads as slower + fewer errors, not faster.
• Learn 10 connective phrases cold: “the trade-off there was…”, “what drove that decision was…”, “to give you a concrete example…”, “I’d push back on that slightly because…”, “let me walk you through it.”

3. Two-day plan

Wednesday evening (2–3h)

• Write out your “last project” answer in full and say it aloud 3×.
• Re-read your own RAG refactor: pgvector schema, chunking strategy, embedding model, retrieval params, LangGraph graph shape. Write down actual numbers (doc count, chunk size, top-k, latency, cost/query). Numbers are what separate senior from mid.
• Read §5 RAG vs fine-tuning and §6 LLM call vs agent below until you can explain each in 90 seconds without notes.

Thursday (main day, 5–6h)

• Morning — AI depth (the gates): evaluation & metrics (§7), observability (§8), hallucinations (§9). These are your weakest-to-strongest ROI. Actually set up Langfuse or LangSmith locally on a toy RAG for 45 minutes — one hands-on hour gives you 5 concrete sentences no one can fake.
• Midday — Databases (§10): run EXPLAIN (ANALYZE, BUFFERS) on 3 real queries. Have one war story about a specific slow query and how you fixed it.
• Afternoon — Backend/API (§11) + FinTech context (§12). FinTech is the domain; showing you think about idempotency, money precision, and PII-in-prompts will separate you instantly.
• Evening — mock: answer all 12 questions in §13 out loud, timed, in English.

Friday morning (1h before the call)

• Re-read §13 answer skeletons only. No new material.
• Re-read the FastAPI structure doc (they asked it — see §11).
• Prepare your 3 questions for the CTO (§14).
• Test mic/camera. Have water. Have your numbers on a sticky note off-camera.

4. Complete technical stack reference

Legend: [M] must be able to discuss in depth · [K] know what it is and when you’d pick it · [N] name-drop level

4.1 Python backend core

4.2 Databases & performance

4.3 LLM / applied AI — the core of this role

4.4 AWS & infrastructure

4.5 FinTech-specific (the domain — do not skip)

• [M] Decimal / integer minor-units for money — never float. Say this out loud; it’s an instant credibility signal.
• [M] Idempotency keys on write endpoints; exactly-once vs at-least-once delivery; retry safety.
• [M] Audit logging / immutable event trail; append-only records.
• [M] PII handling: what may never go into a third-party LLM prompt; redaction before inference; data residency; using Bedrock/Azure or self-hosted when data can’t leave.
• [K] Double-entry ledger, reconciliation, eventual consistency between systems.
• [K] Webhook design: signature verification (HMAC), replay protection, ordering.
• [K] PCI DSS / SOC 2 / GDPR basics, KYC/AML concepts.
• [K] Common integrations: Plaid, Stripe, Open Banking / PSD2, market-data APIs.

5. RAG vs fine-tuning — the question they know candidates fail

Learn this cold. State the framing first, then the details.

“They solve different problems. RAG changes what the model knows at inference time; fine-tuning changes how the model behaves. They’re complementary, not alternatives.”

Say this line: “If the answer changes when a document changes, it’s RAG. If the answer is wrong in style or shape rather than in fact, it’s fine-tuning. In FinTech, RAG is almost always right first because you need citations and per-user access control — you can’t revoke a fine-tuned weight.”

Order of escalation: prompt engineering → few-shot examples → RAG → tool use/agents → fine-tuning. Fine-tuning is last because it’s the most expensive to iterate.

6. LLM call vs agent

“A single LLM call is a pure function — one prompt in, one completion out, deterministic control flow that I wrote. An agent is a loop where the model controls the flow: it decides which tool to call, inspects the result, and decides whether to continue or stop.”

Key points to add unprompted:

• The defining property is who owns control flow. A chain with 5 sequential LLM calls is still not an agent — I decided the order. An agent decides.
• Consequences of that loop: non-deterministic step count, unbounded cost/latency, compounding errors, need for max-step limits and timeouts, harder debugging → which is exactly why observability becomes mandatory.
• Your concrete example: LangGraph create_agent (from langchain.agents import create_agent — the 1.0 front door; saying create_react_agent out loud dates you to pre-October-2025) — tools, state, checkpointing, and where you set recursion limits.
• Senior signal: “I default to the least agentic thing that works. Most ‘agent’ use-cases are actually a fixed workflow with one routing decision — a StateGraph with explicit edges is cheaper, faster, and testable. I reach for a free-running ReAct loop only when the task genuinely can’t be decomposed ahead of time.”

7. How to estimate the result from an LLM (evaluation)

This is the question that eliminated the last candidate. Structure your answer in three layers:

1. Offline eval on a golden dataset

• Build 50–200 curated question/expected-answer pairs from real user queries.
• Retrieval metrics: hit rate / recall@k, MRR, NDCG — “you have to evaluate retrieval separately from generation, otherwise you can’t tell which half is broken.”
• Generation metrics (RAGAS): faithfulness (is the answer supported by retrieved context), answer relevancy, context precision, context recall.
• Deterministic checks where possible: JSON schema validity, required fields, numeric tolerance, regex/exact match.

2. LLM-as-judge

• Rubric-based scoring with a stronger model; pairwise comparison for A/B.
• Name the caveats — this is the depth signal: position bias, verbosity bias, self-preference bias. Mitigate by randomizing order, calibrating the judge against human labels on a subset.

3. Online / production

• Thumbs up/down + free-text feedback wired to the trace ID.
• Proxy signals: retry rate, escalation-to-human rate, session abandonment, citation click-through.
• Guardrail alerts: groundedness below threshold, latency/cost per request, refusal rate.

Close with the CI point: “The important part is that this runs in CI. Any prompt change, model version bump, or chunking change re-runs the eval set, and I gate merges on regressions. Otherwise you’re shipping blind — an LLM system without a regression suite degrades silently.”

8. Observability

“Standard APM tells you the endpoint took 4 seconds. It doesn’t tell you the retriever returned garbage on step 3 of a 7-step agent run. LLM observability is about tracing at the step level.”

Cover:

• Tracing: one trace per request, spans per step — retrieval, rerank, each LLM call, each tool call. Capture inputs, outputs, and intermediate state.
• What you attach to each span: model + version, prompt version, token counts in/out, cost, latency, temperature, retrieved chunk IDs and scores.
• Tools: Langfuse (self-hostable — say this, it matters in FinTech), LangSmith, Arize Phoenix; OpenTelemetry/OpenLLMetry for vendor-neutral export into existing Datadog/Grafana.
• The debugging workflow: user reports a bad answer → open the trace by ID → check retrieval first (did the right chunk come back at all?) → if yes, the generation step or prompt is at fault; if no, it’s chunking/embedding/query.
• Feedback loop: bad traces get promoted into the eval dataset. “Every production failure becomes a regression test.”
• FinTech caveat: prompt/response payloads may contain PII — redact before they leave your VPC, or self-host Langfuse.

9. Hallucinations

Split the answer into detect and reduce — most candidates only do the second.

Detect

• Faithfulness/groundedness scoring against retrieved context (per-claim entailment check).
• Require citations and verify them programmatically — does the cited span actually contain the claim?
• Schema/type validation and business-rule checks on structured output.
• Self-consistency: sample n times, compare; divergence signals low confidence.
• Cross-check numbers against the source system of record rather than trusting the model’s arithmetic.

Reduce

• Grounding: RAG with high-precision retrieval; hybrid search + reranking so the right chunk is actually in context.
• Prompting: explicit instruction to answer only from context and to say “I don’t know” otherwise; give the model a legitimate way out — a forced answer is a hallucinated answer.
• Constrain the output: tool calling / JSON schema instead of free text; let code do arithmetic, not the model.
• Context hygiene: less but better context beats stuffing top-50; irrelevant chunks actively cause drift (“lost in the middle”).
• Model choice + temperature: lower temperature for extraction; stronger model for multi-hop reasoning.
• Decomposition: break one hard question into verifiable sub-steps.
• Human-in-the-loop for high-stakes paths — in FinTech, anything touching money or advice gets a review gate.

Senior close: “You can’t get to zero. So the real design question is: what does the system do when it’s unsure? I’d rather build a system that abstains and escalates than one that’s confidently wrong 3% of the time — especially in FinTech, where a wrong number is a liability, not a bad UX.”

10. Databases & query optimization

Have one specific war story ready: the query, what EXPLAIN ANALYZE showed, what you changed, before/after numbers.

Checklist to hit:

• Method first: pg_stat_statements to find the worst offenders → EXPLAIN (ANALYZE, BUFFERS) → look for seq scans on large tables, bad row estimates, spills to disk, nested loops over big sets.
• Indexing: composite index column order (equality first, then range), partial indexes, covering indexes, why an index isn’t used (function on column, type mismatch, low selectivity, stale stats → ANALYZE).
• Application-layer: N+1 from lazy loading → selectinload/joinedload; keyset pagination instead of OFFSET; batch writes; COPY for bulk.
• Connection management: pool sizing, PgBouncer in transaction mode (and why that breaks prepared statements/session state).
• Caching: what to cache, TTL, invalidation strategy, cache stampede.
• When to stop optimizing SQL: denormalize, materialized view, read replica, move to a queue.
• Vector-specific: HNSW vs IVFFlat trade-off, ef_search as the recall/latency dial, pre-filtering vs post-filtering with metadata, and why a filtered ANN query can fall off a performance cliff.

11. Backend / API / architecture

• FastAPI structure — they listed it explicitly. Answer: the official docs define one multi-file layout (app/ package, routers/ by resource, dependencies.py, include_router with prefix/tags/dependencies); the flat-vs-layered “two types” framing is community convention. Then say what you do and why. (See your separate FastAPI structure doc.)
• API design: versioning, pagination, error contract, idempotency, rate limiting, OpenAPI-driven clients.
• Integrations (a core responsibility in this role): retries with exponential backoff + jitter, circuit breakers, timeouts, DLQs, webhook verification, idempotent consumers, contract testing, handling third-party outages gracefully.
• Async correctness: never block the event loop, run_in_executor for CPU/sync libs, connection pool sizing under async load.
• Testing strategy for LLM code: mock the provider for unit tests, record/replay for integration, eval suite as a separate gated job.

12. Behavioural / fit

“Do you have experience with international teams?” — Say yes, then be specific: distributed team, English as working language, async communication across time zones, written-first culture (docs/RFCs over meetings), working with US stakeholders. Mention your Sidecar Health context.

“Ability to work independently with minimal oversight” — this is in the requirements and it’s a part-time role. Give evidence: you scoped and delivered X without a PM, you flagged risk early, you shipped in small reviewable increments, you wrote the docs so the CTO didn’t have to ask.

Part-time framing — have a clear, confident answer on availability, hours/week, overlap with their time zone, and how you’d handle handoffs. Don’t be vague here; it’s a practical concern for a CTO.

13. Rapid-fire answer skeletons

14. Questions to ask the CTO

Ask 2–3. These signal seniority:

1. “What does the AI part of the product actually do for the end user today, and what’s the biggest quality complaint?” (shows you think product-first)
2. “Do you have an evaluation set and observability in place already, or would that be part of what I’d build?” (this question alone separates you from the last candidate)
3. “Given it’s FinTech — what are the constraints on data leaving your environment for inference? Are you on Bedrock/Azure, or are self-hosted models on the table?”
4. “How do you want to split the part-time hours — fixed overlap with your time zone, or async with checkpoints?”

15. Final checklist

• Self-intro rehearsed out loud in English (90 s)
• “Last project” rehearsed out loud (2–3 min) with real numbers
• Can explain RAG vs fine-tuning in 90 s, no notes
• Can explain LLM call vs agent in 90 s, no notes
• Can name 4 RAGAS metrics and what each catches
• Hands-on hour with Langfuse or LangSmith done
• One EXPLAIN ANALYZE war story with before/after numbers
• One sentence ready on money precision (Decimal) and one on PII-in-prompts
• 3 questions for the CTO written down
• Numbers sticky-note off-camera; mic/camera tested