Backend / Kubernetes / 09_gitops_and_operators.md

GitOps and the operator pattern

Updated 7 interview angles 5 min read source
On this page5
  1. GitOps: the repository is the desired state
  2. The operator pattern
  3. How they compose
  4. Related
  5. Interview angle

GitOps and the operator pattern

Two ideas that define how production Kubernetes is actually run, and both are the same idea underneath: a controller continuously reconciling real state toward declared state.

GitOps: the repository is the desired state

The pre-GitOps deployment is a CI job that runs kubectl apply at the end of a pipeline. GitOps inverts it: an agent inside the cluster watches a git repository and applies what it finds.

text 
push --> git repo --> agent in cluster --> apply
              ^                |
              +---- compares --+  (every ~3 min)

The difference is not cosmetic. It buys three things:

  • No cluster credentials in CI. A push pipeline needs admin access to production; a pull agent needs read access to a repo. That removes the most valuable secret from the place most likely to leak it.
  • Drift is detected and corrected. Someone runs kubectl edit at 2am; the agent notices divergence from git and reverts it. Without this, the manifest in git slowly becomes fiction.
  • Rollback is git revert. The deployment history is the commit history, and every change has an author and a review.

ArgoCD and Flux

The two implementations. ArgoCD ships a UI that shows sync status and diffs per application, which is what makes it the common choice on platform teams - the visual diff between git and cluster is genuinely useful during an incident. Flux is smaller, has no UI by default, and composes as a set of controllers.

The concepts transfer either way. In ArgoCD an Application points at a repo path and a target cluster:

yaml 
spec:
  source:
    repoURL: https://git.example.com/platform
    path: envs/prod/api
  destination:
    namespace: api
  syncPolicy:
    automated:
      prune: true      # delete what git drops
      selfHeal: true   # revert manual cluster edits

prune and selfHeal are off by default, and leaving them off gives you a worse version of the thing you installed: git describes the deployment but does not enforce it. Turning them on is what makes it GitOps rather than a fancy kubectl apply.

ApplicationSet generates Applications from a template - one per cluster, or per environment directory - which is how a platform team avoids hand-writing forty near-identical Application manifests.

The secrets problem

Git is the source of truth, and secrets cannot go in git in plaintext. Three answers, in increasing order of how much infrastructure they assume:

Approach The trade
SOPS Encrypted files in git; key distribution is on you
Sealed Secrets Encrypted to one cluster’s key; simple, cluster-bound
External Secrets Git holds a reference; a vault holds the value

External Secrets Operator is the usual answer at platform scale: the manifest in git names a key in Vault or AWS Secrets Manager, and the operator fetches it. Git then contains no ciphertext to rotate.

The operator pattern

An operator is a custom resource plus a controller that reconciles it. It is how you teach Kubernetes about a thing it does not natively understand.

The custom resource extends the API:

yaml 
apiVersion: db.example.com/v1
kind: PostgresCluster
metadata:
  name: orders
spec:
  replicas: 3
  version: "17"
  backup:
    schedule: "0 2 * * *"

A CRD registers that kind so the API server accepts it. The controller then watches these objects and does the work: create StatefulSets, configure replication, run backups, handle failover.

The reconciliation loop

This is the concept an interviewer is checking, and it is the same loop that runs Deployments and ReplicaSets.

python 
# Conceptually, every controller is this.
def reconcile(name):
    desired = api.get(name).spec
    actual = inspect_cluster(name)
    if actual != desired:
        converge(actual, desired)
    return requeue_after(seconds=30)

Two properties follow, and they are what make operators robust:

  • It is level-triggered, not edge-triggered. The controller acts on observed state, not on the event that changed it. A missed event is self-correcting because the next loop still sees the discrepancy.
  • Reconcile must be idempotent. It runs repeatedly on unchanged state, so every step is “make it so”, never “do it again”.

That is the same discipline as an idempotent task consumer - see Idempotency keys.

When to write one

Rarely. The honest answer is that most teams should use operators, not write them.

Write one when the thing has operational knowledge that cannot be expressed declaratively - failover, backup and restore, version upgrades with data migrations. That is why databases, message brokers and certificate managers have operators and stateless web services do not.

Use a Helm chart when you are packaging manifests with values. Reach for an operator only when day-two operations need running code. Kubebuilder and the Operator SDK scaffold the controller; the hard part is never the scaffolding, it is correctly handling partial failure halfway through a reconcile.

How they compose

A platform typically runs both: GitOps agents reconcile the repository into the cluster, and operators reconcile custom resources into running systems. The operator’s CRs are themselves stored in git, so a database cluster is declared, reviewed and versioned the same way a Deployment is.

Interview angle 7

  • “What is GitOps?” - an agent inside the cluster continuously reconciles it toward a git repository, rather than CI pushing with kubectl apply. Git becomes the desired state, and deployment history is commit history.
  • “Why is the pull model better than pushing from CI?” - CI no longer needs production cluster credentials, only the agent does, which removes the most valuable secret from the most exposed place. It also detects drift: a manual kubectl edit is reverted rather than silently diverging from the manifest.
  • “What makes an ArgoCD setup actually GitOps?” - prune and selfHeal enabled. Without them git describes the deployment but does not enforce it, and you have a slower kubectl apply with extra machinery.
  • “How do secrets work if git is the source of truth?” - either encrypted in git with SOPS or Sealed Secrets, or referenced from git with the value in a vault via External Secrets Operator. The last is usual at scale because git then holds no ciphertext to rotate.
  • “What is an operator?” - a custom resource definition plus a controller that reconciles it, extending the Kubernetes API with a domain object like PostgresCluster and the operational logic to run it.
  • “What is the reconciliation loop and why does it matter?” - it compares desired against observed state and converges, on a timer rather than on events. Being level-triggered makes a dropped event harmless, and it forces every reconcile to be idempotent, since it runs repeatedly against unchanged state.
  • “When would you write your own operator?” - when day-two operations need running code: failover, backups, version upgrades with data migrations. Packaging manifests with values is a Helm chart, not an operator, and most teams should use operators rather than write them.
Previous
Gateway API