Backend / Protocols / gRPC / 07_error_handling.md

gRPC Error Handling

Updated 6 interview angles 6 min read source
On this page11
  1. The 16 status codes
  2. Returning errors from a server
  3. Handling errors on the client
  4. Rich error details — grpcio-status
  5. When to use which status code
  6. Retry strategy
  7. Built-in retry via service config
  8. Cancellation and deadlines
  9. Common pitfalls
  10. Common interview confusions
  11. Interview angle

gRPC Error Handling

gRPC has its own status codes (not HTTP status codes). 16 well-known codes, returned as trailers after the response. Plus optional rich error details via grpc-status-details-bin.

The 16 status codes

Code Number Meaning
OK 0 success
CANCELLED 1 client cancelled / deadline before completion
UNKNOWN 2 catch-all server error
INVALID_ARGUMENT 3 client sent invalid args (analog of HTTP 400)
DEADLINE_EXCEEDED 4 deadline expired before response (HTTP 504)
NOT_FOUND 5 resource not found (HTTP 404)
ALREADY_EXISTS 6 client tried to create a duplicate (HTTP 409)
PERMISSION_DENIED 7 authenticated but not allowed (HTTP 403)
RESOURCE_EXHAUSTED 8 quota / rate limit hit (HTTP 429)
FAILED_PRECONDITION 9 system not in a state to perform op (HTTP 412)
ABORTED 10 optimistic concurrency conflict (HTTP 409)
OUT_OF_RANGE 11 request specified an invalid range (HTTP 400)
UNIMPLEMENTED 12 method not implemented (HTTP 501)
INTERNAL 13 server bug (HTTP 500)
UNAVAILABLE 14 service temporarily unavailable (HTTP 503)
DATA_LOSS 15 unrecoverable data corruption
UNAUTHENTICATED 16 missing/invalid credentials (HTTP 401)

Memorize the common ones: OK, INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, PERMISSION_DENIED, UNAUTHENTICATED, RESOURCE_EXHAUSTED, UNAVAILABLE, INTERNAL.

Returning errors from a server

Sync — set on context

python
def GetUser(self, request, context):
    user = db.get_user(request.id)
    if not user:
        context.set_code(grpc.StatusCode.NOT_FOUND)
        context.set_details(f"User {request.id} not found")
        # empty response (will be discarded)
        return user_pb2.User()
    return user_pb2.User(id=user.id, name=user.name)

The empty response is ignored when status code is non-OK; status is what the client sees.

Cleaner — abort()

python
def GetUser(self, request, context):
    user = db.get_user(request.id)
    if not user:
        context.abort(grpc.StatusCode.NOT_FOUND, f"User {request.id} not found")
    return user_pb2.User(id=user.id, name=user.name)

abort() raises an exception that the gRPC framework catches and translates. No need to return anything; control flow stops.

Async equivalent: await context.abort(...).

Handling errors on the client

python
try:
    response = stub.GetUser(GetUserRequest(id="42"))
except grpc.RpcError as e:
    if e.code() == grpc.StatusCode.NOT_FOUND:
        print(f"Not found: {e.details()}")
    elif e.code() == grpc.StatusCode.UNAVAILABLE:
        print("Service unavailable, retrying...")
        retry()
    else:
        raise

grpc.RpcError is the exception type. .code() returns the StatusCode; .details() returns the message.

Rich error details — grpcio-status

For structured error info (not just a string), use google.rpc.Status with details:

python
from grpc_status import rpc_status
from google.rpc import status_pb2, code_pb2, error_details_pb2

def CreateUser(self, request, context):
    violations = []
    if not request.email or "@" not in request.email:
        violations.append(error_details_pb2.BadRequest.FieldViolation(
            field="email", description="must be a valid email",
        ))
    if not request.name:
        violations.append(error_details_pb2.BadRequest.FieldViolation(
            field="name", description="required",
        ))
    if violations:
        from google.protobuf.any_pb2 import Any
        detail = error_details_pb2.BadRequest(field_violations=violations)
        detail_any = Any()
        detail_any.Pack(detail)
        rich_status = status_pb2.Status(
            code=code_pb2.INVALID_ARGUMENT,
            message="Validation failed",
            details=[detail_any],
        )
        context.abort_with_status(rpc_status.to_status(rich_status))
    ...

Client side:

python
from grpc_status import rpc_status
from google.rpc import error_details_pb2

try:
    stub.CreateUser(...)
except grpc.RpcError as rpc_error:
    status = rpc_status.from_call(rpc_error)
    for detail in status.details:
        if detail.Is(error_details_pb2.BadRequest.DESCRIPTOR):
            br = error_details_pb2.BadRequest()
            detail.Unpack(br)
            for v in br.field_violations:
                print(f"{v.field}: {v.description}")

The standard error detail types (google.rpc.error_details_pb2):

Type Purpose
BadRequest per-field validation errors
ResourceInfo resource that caused error
RetryInfo how long to wait before retry
DebugInfo stack traces (for internal debugging)
QuotaFailure rate limit / quota details
PreconditionFailure unmet precondition
ErrorInfo structured error code with metadata
Help URL pointing to help
LocalizedMessage translated error

Use these standard types when applicable; build custom messages for app-specific cases.

When to use which status code

The fine distinctions matter for clients building retry logic:

Situation Use Why
Bad input from client INVALID_ARGUMENT client should fix and retry — but with different args
Resource doesn’t exist NOT_FOUND client should not retry; check the id
User not authorized PERMISSION_DENIED client should not retry; need different auth
User not authenticated UNAUTHENTICATED refresh token, retry
Server is down / restarting UNAVAILABLE retry with backoff likely OK
Server bug INTERNAL client can retry but server is broken
Quota exceeded RESOURCE_EXHAUSTED retry with backoff after Retry-After
Deadline expired DEADLINE_EXCEEDED client gave up; retry with longer deadline maybe
Optimistic conflict ABORTED retry after re-reading state
Operation already happened ALREADY_EXISTS idempotent — treat as success or don’t retry

INVALID_ARGUMENT vs FAILED_PRECONDITION: argument validity is per-call; precondition is about system state. “Email format wrong” = INVALID_ARGUMENT. “Bucket must be empty before delete” = FAILED_PRECONDITION.

ABORTED vs UNAVAILABLE: ABORTED suggests “retry will work” (transient state); UNAVAILABLE suggests “service will be back.”

Retry strategy

python
def call_with_retry(stub, request, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return stub.SomeRpc(request)
        except grpc.RpcError as e:
            if e.code() in (grpc.StatusCode.UNAVAILABLE, grpc.StatusCode.RESOURCE_EXHAUSTED):
                # backoff
                time.sleep(2 ** attempt)
                continue
            raise
    raise RuntimeError("retries exhausted")

Retry-safe codes: UNAVAILABLE, RESOURCE_EXHAUSTED, sometimes DEADLINE_EXCEEDED and ABORTED.

Never retry-safe: INVALID_ARGUMENT, NOT_FOUND, PERMISSION_DENIED, UNAUTHENTICATED. The cause is structural; retry will fail the same way.

Built-in retry via service config

python
service_config = {
    "methodConfig": [
        {
            "name": [{"service": "user.UserService"}],
            "retryPolicy": {
                "maxAttempts": 3,
                "initialBackoff": "0.1s",
                "maxBackoff": "1s",
                "backoffMultiplier": 2,
                "retryableStatusCodes": ["UNAVAILABLE"],
            },
        }
    ]
}
options = [("grpc.service_config", json.dumps(service_config))]
channel = grpc.insecure_channel("server:50051", options=options)

gRPC retries automatically based on this config. Set per-method. The library handles backoff math.

Cancellation and deadlines

python
# Client — set deadline
try:
    response = stub.GetUser(request, timeout=5.0)
except grpc.RpcError as e:
    if e.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
        ...

After 5 seconds, the RPC is cancelled and DEADLINE_EXCEEDED is raised. The server sees the cancellation via context.is_active().

Deadlines propagate: if A calls B (deadline 5s) and B calls C, C should see how much time remains, not start fresh. gRPC’s metadata carries this automatically when you use the standard interceptors / clients.

Common pitfalls

  • context.set_code(NOT_FOUND) + return user_pb2.User(...) with data — the client sees NOT_FOUND but you sent a valid response. Confusing. Use context.abort or return empty.
  • Returning OK for business errorsgetUser(id="non-existent") → User(id="", name="") (default values). Client sees success with empty data. Confusing; should be NOT_FOUND.
  • Same error code for all failures — clients can’t differentiate “bad input” from “server down” without details. Use specific codes.
  • Retry on INVALID_ARGUMENT — won’t help. Wastes time and resources.
  • No retry on UNAVAILABLE — transient failures become hard failures.
  • ABORTED with no instruction to retry — clients don’t know it’s retryable unless documented.

Common interview confusions

  • “gRPC uses HTTP status codes.” — its own 16 codes. Sent as HTTP/2 trailer grpc-status.
  • “Errors are the response body.” — they’re trailers (headers after the body). Empty body, non-OK status.
  • UNAVAILABLE means the server is permanently down.” — transient by convention. Permanent unavailability would be UNIMPLEMENTED or just a connection failure.

Interview angle 6

  • “How does gRPC communicate errors?” — gRPC has its own 16 status codes (OK, NOT_FOUND, INVALID_ARGUMENT, UNAVAILABLE, etc.), sent as an HTTP/2 trailer grpc-status after the response. Not HTTP status codes.
  • “Difference between INVALID_ARGUMENT and FAILED_PRECONDITION?” — INVALID_ARGUMENT = bad input that’s wrong regardless of system state (malformed email). FAILED_PRECONDITION = input is fine but the system can’t act on it (bucket must be empty before delete).
  • “Which status codes are retry-safe?”UNAVAILABLE, RESOURCE_EXHAUSTED typically; DEADLINE_EXCEEDED and ABORTED sometimes. Never INVALID_ARGUMENT, NOT_FOUND, PERMISSION_DENIED (the cause won’t change).
  • “How do you return structured error details?” — use grpcio-status to wrap status code + message + a list of detail Protobuf messages (BadRequest, RetryInfo, custom). Clients unpack via rpc_status.from_call.
  • “How does built-in gRPC retry work?” — configure via grpc.service_config channel option: retryPolicy with maxAttempts, initialBackoff, backoffMultiplier, retryableStatusCodes. The library handles backoff and reattempts.
  • context.set_code() vs context.abort()?”set_code sets the status but doesn’t terminate; you must return. abort raises an exception that terminates the handler immediately. abort is cleaner for most cases.