Observability

Overview

Observability is the ability to understand what a system is doing internally from the data it produces, so you can answer new questions about failures without shipping new code. Its three pillars are metrics (numeric time series such as request rate and latency), logs (timestamped event records), and traces (the path of one request across services, broken into spans).

Good observability ties these together with shared context (trace IDs in logs, exemplars in metrics), defines SLIs and SLOs for user-facing behavior, alerts on symptoms users feel rather than every cause, and supports fast debugging with dashboards and distributed tracing. OpenTelemetry has become the standard for instrumenting applications across vendors.

When to use it

• Every production service.
• Debugging latency and errors across microservices.
• Defining SLOs and alerting.
• Capacity planning and performance tuning.

Where it shows up in interviews

Where it is used in real software

Key terms

Metrics

Aggregated numbers over time (counters, gauges, histograms).

Logs

Discrete event records, ideally structured JSON.

Traces and spans

Request path across services; each span is one operation.

RED / USE

Rate, errors, duration for services / utilization, saturation, errors for resources.

Cardinality

Number of unique label combinations; high cardinality is costly for metrics.

How it works, step by step

1. 1
   Instrument with OpenTelemetry

   Auto-instrument HTTP, DB, and messaging clients.

2. 2
   Emit RED metrics per endpoint

   Rate, errors, latency histograms.

3. 3
   Use structured logs with trace IDs

   Correlate logs with traces.

4. 4
   Propagate trace context

   traceparent header across services and messages.

5. 5
   Define SLOs and alert on burn rate

   Page on user impact, not on every blip.

Three pillars compared

What each is best at

Metrics detect, traces locate, logs explain. Linking them via trace IDs makes debugging fast.

Implementation

1import { NodeSDK } from "@opentelemetry/sdk-node";2import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";3import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";4import { trace } from "@opentelemetry/api";5 6new NodeSDK({7  serviceName: "checkout",8  traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }),9  instrumentations: [getNodeAutoInstrumentations()], // HTTP, Express, pg, Redis, Kafka...10}).start();11 12const tracer = trace.getTracer("checkout");13 14export async function checkout(cartId: string) {15  return tracer.startActiveSpan("checkout.process", async (span) => {16    span.setAttribute("cart.id", cartId);17    try {18      const result = await processCart(cartId);19      logger.info({ cartId, traceId: span.spanContext().traceId }, "checkout completed"); // correlated log20      return result;21    } catch (err) {22      span.recordException(err as Error);23      throw err;24    } finally {25      span.end();26    }27  });28}

Complexity and performance

Trade-offs

Variants and related techniques

Common mistakes

• Unstructured logs.

  Fix: Use JSON with consistent fields and trace IDs.

• High-cardinality metric labels.

  Fix: User or request IDs belong in logs and traces, not metric labels.

• Alerting on causes (CPU 80%).

  Fix: Alert on symptoms (error rate, latency SLO burn).

• Logging secrets or personal data.

  Fix: Redact sensitive fields.

Interview questions

Practice problems