Amazon CloudWatch

Article.

CloudWatch is AWS’s built-in monitoring: metrics (numbers over time), logs, alarms on metrics, dashboards, and the console for traces. Every AWS service publishes metrics to it by default, which makes it the monitoring you already have; the senior questions are about what it does not give you by default, how alarms actually evaluate, and what it costs.

“An error alarm never fired overnight, despite real errors. Why?” and “The CloudWatch bill jumped tenfold after a release. What happened?” CloudWatch cannot be run locally, so the claims below are checked against AWS documentation and the published us-east-1 prices as of October 2026; the cost figures are computed from those prices. Cost patterns in more depth are in CloudWatch — Cost Gotchas and Advanced Patterns.

Metrics

A metric is a time series identified by a namespace (AWS/EC2, AWS/SQS, or your own), a name, and up to 30 dimensions (key-value pairs). Every distinct combination of dimension values is a separate metric, billed separately.

What you do not get by default

• EC2 publishes no memory or disk-space metrics. CPU, network and disk I/O come from the hypervisor; memory and filesystem usage are only visible inside the guest, so they need the CloudWatch agent.
• Resolution: EC2 basic monitoring is every 5 minutes, detailed monitoring every minute. High resolution (down to 1 second) exists only for custom metrics.

Retention is automatic, and coarsens

Older data is rolled up into coarser periods, so a one-second spike from last month is no longer visible at one-second resolution.

Custom metrics and EMF

Applications publish custom metrics with PutMetricData, or more cheaply by writing Embedded Metric Format JSON to their logs, which CloudWatch turns into metrics asynchronously without an API call per data point:

{"_aws": {"Timestamp": 1791900000000,
          "CloudWatchMetrics": [{"Namespace": "orders",
                                 "Dimensions": [["Service"]],
                                 "Metrics": [{"Name": "latency_ms",
                                              "Unit": "Milliseconds"}]}]},
 "Service": "checkout", "latency_ms": 182}

EMF saves API calls, not metric charges: every unique dimension set is still a billed custom metric, on top of the log ingestion.

What custom metrics cost

Each custom metric (each unique combination of name and dimension values) is billed per month, with tiered prices: $0.30 for the first 10,000, $0.10 for the next 240,000, $0.05 for the next 750,000, $0.02 above that.

The way to reach a million by accident is a high-cardinality dimension: user_id, request_id or an unbounded endpoint path as a dimension turns one metric into one per value. That is the usual answer to the tenfold bill. Per-user and per-request detail belongs in logs or traces; metric dimensions should have a small, bounded set of values.

Logs

Log groups contain streams; each ECS task, Lambda instance or EC2 agent writes its own stream.

• Ingestion dominates. A metric filter (count of ERROR lines as a metric) runs on logs that were already ingested and paid for; it saves nothing on ingestion. To pay less, log less, sample, or send bulk logs elsewhere (Firehose to S3).
• Infrequent Access halves ingestion but drops features: no metric filters, no subscription filters, no Live Tail, no EMF extraction.
• Logs Insights is billed by data scanned. A dashboard widget querying seven days of a log group that ingests 1 TB a day scans 7 TB: $35 per refresh. Narrow time ranges, field indexes, or a metric (from EMF) for dashboards instead of a query.
• Set retention. The default is “never expire”, so storage grows forever.

Alarms: how they actually evaluate

An alarm looks at a metric statistic over a period, and goes to ALARM when M of the last N datapoints breach the threshold (DatapointsToAlarm of EvaluationPeriods):

period 60 s, EvaluationPeriods 5, DatapointsToAlarm 3, threshold > 10 errors
  datapoints: 4, 12, 15, 8, 14     -> 3 of 5 breach -> ALARM

“M of N” lets an alarm ignore one bad minute without waiting for five in a row.

Missing data, the silent alarm

An error metric often emits nothing while things are healthy: no errors, no data point. TreatMissingData decides what an absent datapoint means:

The overnight trap: errors are reported only through a metric that the failing component itself emits. When the service crashes outright, it emits nothing, every datapoint is missing, and an alarm on “errors > 0” never fires. Alarm on something that is always present (request count dropping to zero, a heartbeat metric, a load-balancer 5xx count) and choose TreatMissingData deliberately, usually breaching for heartbeats.

Rates, not counts, and percentiles

• Error rate: metric math, 100 * errors / requests, alarms on the proportion instead of a raw count that grows with traffic.
• Latency: alarm on p99 (p99 statistic), not the average, which hides the slow tail. Percentiles need raw values; metrics published as pre-aggregated statistic sets cannot give a p99.
• Composite alarms combine several (ALARM(errors) AND ALARM(latency)) to page on symptoms rather than every cause.

Traces: X-Ray and OpenTelemetry

The X-Ray SDKs and daemon entered maintenance mode on 2026-02-25 and reach end of support on 2027-02-25; AWS recommends OpenTelemetry (or the AWS Distro for OpenTelemetry) to send traces to X-Ray. CloudWatch Application Signals builds service maps, latency and error rates from that OpenTelemetry data. New instrumentation should use OpenTelemetry, not the X-Ray SDK (The Three Pillars of Observability).

Where it breaks in production

The alarm that never fired

Described above: an alarm on a metric that disappears exactly when the service fails. Test alarms by breaking things in staging, and alarm on signals that are always emitted.

A release that multiplied the bill

A new dimension with unbounded values (customer_id on a latency metric) creates one custom metric per customer. At 100,000 customers that is $12,000 a month for a single metric name. Review dimensions like schema changes.

Dashboards that cost more than the service

Logs Insights widgets over long time ranges, refreshed every minute by a team screen: $35 per refresh on a 1 TB/day log group. Back dashboards with metrics.

Memory alarms that do not exist

An EC2 fleet without the CloudWatch agent has no memory metrics, so the first sign of a memory leak is the OOM killer. Install the agent, or use Container Insights on ECS and EKS.

The trade not taken

Prometheus and Grafana, or a vendor (Datadog, New Relic, Grafana Cloud), are the alternatives: richer querying, better dashboards, and no per-metric CloudWatch pricing, but another system to run or pay for, and AWS services still report to CloudWatch first. The common split: CloudWatch for AWS service metrics and alarms that page, OpenTelemetry for application metrics and traces exported to whichever backend the team prefers (Grafana).

Related

• CloudWatch — Cost Gotchas and Advanced Patterns - cost in depth, advanced patterns
• Amazon ECS (Elastic Container Service) - ECS logging and scaling
• The Three Pillars of Observability - metrics, logs, traces
• Grafana - Grafana