Q: How do you build centralised logging? Compare ELK/EFK with Loki.

In a distributed system you can't SSH to a box and tail — pods are ephemeral and there are hundreds. You need a pipeline: collect → ship → parse → store → query . ELK / EFK stack Beats/Fluentd/Fluent Bit — agent on each node collecting container stdout. Logstash (optional) — parsing, enrichment, filtering. Elasticsearch — indexes the full text of every log. Powerful search, but storage- and RAM-hungry and operationally heavy. Kibana — query UI. Loki takes the opposite approach: it indexes only labels (service, namespace, level) and stores the log content compressed in object storage. Dramatically cheaper and simpler to run; queries filter by label first then grep the stream. Trade-off: full-text search across everything is slower than Elasticsearch. Choosing: Loki if you're already on Prometheus/Grafana, want low cost, and mostly query by service + time window (which is most DevOps use). Elasticsearch if you need rich full-text search, complex aggregations, or already run it for other purposes.

Question 1

Monitoring vs observability — what's the actual difference?

Accepted Answer

Monitoring watches for failure modes you already predicted : you define dashboards and alerts for known conditions (CPU > 80%, disk full, error rate > 1%). It answers "is the system broken?" Observability is a property of the system: whether you can understand its internal state from the data it emits — including for failures nobody anticipated. It answers "why is it broken?" without shipping new code to find out. The practical distinction is known-unknowns vs unknown-unknowns : Monitoring: "alert me when the queue is longer than 1000." You knew to watch that. Observability: "latency is up only for Android users in one region on the checkout endpoint since 14:20." You couldn't have pre-built that dashboard — you needed to slice rich, high-cardinality data at query time. Monitoring is a subset of observability, not a competitor. You need both: monitoring to page you, observability to debug.

Question 2

Explain the three pillars: metrics, logs, and traces. When do you use each?

Accepted Answer

Metrics — numeric time-series, aggregated and cheap to store forever. Best for trends, dashboards, and alerting : request rate, error rate, p99 latency, CPU. Low cardinality by design. Answers "is something wrong, and since when?" Logs — discrete timestamped events with context. Best for detailed forensics on a specific event : the exact stack trace, the exact payload that failed. Expensive at volume. Answers "what exactly happened in this case?" Traces — the path of a single request across services, as a tree of timed spans. Best for latency attribution in distributed systems : which of 12 microservices ate 800ms. Answers "where is the time going?" The workflow that ties them together: a metric alert fires → a trace shows which service and operation is slow → logs from that service and trace ID give the exact error. Linking them with a shared trace_id is what turns three data types into one investigation.

Question 3

What are the Four Golden Signals? How do RED and USE methods differ?

Accepted Answer

Four Golden Signals (Google SRE) — what to measure on any user-facing service: Latency — how long requests take. Measure successful and failed requests separately ; a fast 500 skews your average and hides the problem. Traffic — demand: requests/sec, transactions/sec. Errors — rate of failed requests (explicit 5xx, and implicit — wrong content, policy violations). Saturation — how full the system is (memory, connection pool, queue depth). The leading indicator: saturation rises before latency and errors do. RED (Rate, Errors, Duration) is the request-centric subset — the right frame for services and microservices. USE (Utilisation, Saturation, Errors) is resource-centric — the right frame for infrastructure : CPU, memory, disk, network interfaces. In practice: RED for every service, USE for every resource. They're complementary views, not alternatives.

Question 4

How does Prometheus work? Explain the pull model, exporters, and service discovery.

Accepted Answer

Prometheus scrapes (pulls) metrics over HTTP from targets exposing a /metrics endpoint in a simple text format, stores them in a local time-series database, and evaluates rules against them. Components: Prometheus server — scraper + TSDB + rule evaluator + PromQL engine. Exporters — translate something that doesn't speak Prometheus into /metrics : node_exporter (host), blackbox_exporter (probes), postgres_exporter , kube-state-metrics (K8s object state), cAdvisor (container resources). Service discovery — targets come from Kubernetes, EC2, Consul, or file SD rather than a static list, so autoscaled pods are scraped automatically. This is what makes pull viable at scale. Alertmanager — separate component handling routing, grouping, silencing, and delivery. Pushgateway — only for short-lived batch jobs that die before they can be scraped. It's an exception, not a general push path. Why pull beats push here: Prometheus knows the intended target list, so a target that disappears is detectable ( up == 0 ) rather than silent; you can scrape a target manually with curl to debug; and there's no risk of a misbehaving client flooding your ingestion.

Question 5

Counter vs Gauge vs Histogram vs Summary — which do you use when?

Accepted Answer

Counter — monotonically increasing, only resets to 0 on restart. For things you count: requests, errors, bytes sent. Never graph a counter raw — its absolute value is meaningless; always wrap it in rate() . Gauge — a value that goes up and down: memory in use, queue depth, active connections, temperature. Graph it directly; use avg / max / min . Histogram — samples observations into configurable buckets (plus _sum and _count ). Percentiles are calculated server-side with histogram_quantile() , which means you can aggregate across instances . This is what you want for latency. Summary — calculates quantiles client-side . Cheaper to query, but the quantiles cannot be aggregated across instances (you can't average p99s), so it's rarely the right choice in a distributed system. Default to histograms for latency and size distributions , counters for events, gauges for current state.

Question 6

Write the PromQL you'd actually use day to day. What's the difference between rate and irate?

Accepted Answer

PromQL operates on two things: instant vectors (one value per series, now) and range vectors (a window of values, e.g. [5m] ). Functions like rate() turn a range vector into an instant vector. rate() — average per-second rate over the window, using all data points and correcting for counter resets. Use this for alerting and dashboards — it's smooth and stable. irate() — instant rate from only the last two data points. Very responsive but spiky; good for zooming into a short volatile window, bad for alerts (it will flap). increase() — total increase over the window (essentially rate × seconds ). Rule of thumb : the range window should be at least 4× the scrape interval (so 15s scrape → [1m] minimum, and [5m] is the safe default). Too short and a single missed scrape produces gaps. Aggregation order matters : always rate() first, then sum() — sum before rate breaks counter-reset handling and gives wrong numbers.

Question 7

Explain SLI, SLO, SLA, and error budgets.

Accepted Answer

SLI (Indicator) — the measurement : "proportion of HTTP requests served in under 300ms" or "successful requests ÷ total requests". A number you can actually compute from metrics. SLO (Objective) — your internal target for that SLI: "99.9% of requests succeed, measured over 30 days." This is what the team commits to and what should drive alerting. SLA (Agreement) — the contractual promise to customers, with financial penalties. Always set looser than your SLO, so you breach your internal target and react long before you breach the contract. Error budget = 100% − SLO . A 99.9% SLO over 30 days allows 43.2 minutes of failure. That budget is a currency: Budget remaining → ship features, take risks, deploy often. Budget exhausted → freeze risky releases and spend the sprint on reliability. This is the point of SLOs: they turn "how much reliability is enough?" from an argument into a number, and they explicitly acknowledge that 100% is the wrong target — it's infinitely expensive and prevents shipping.

Question 8

What makes a good alert? How do you fix alert fatigue?

Accepted Answer

A good alert is urgent, actionable, and symptom-based. Before creating one, ask: if this fires at 3am, is there something a human must do right now ? If not, it's a dashboard or a ticket, not a page. Alert on symptoms, not causes. "Checkout error rate is 5%" is worth waking someone; "CPU is 85%" is not — high CPU with healthy latency and no errors is just a well-utilised server. Cause-based alerts generate noise because they fire in situations where users are fine. Fixing alert fatigue — the failure mode where people ignore alerts because most are noise, and the one real page gets missed: Delete alerts nobody acts on. Audit which alerts fired in the last 90 days and what the responder did. "Acknowledged and closed" = delete it. Add for: durations so transient blips don't page. Group and inhibit — one node failing shouldn't send 40 pages; suppress downstream alerts when the upstream cause is already firing. Tier severity — page vs ticket vs dashboard-only. Every alert links a runbook that says what to check and what to do. Alert on burn rate/SLOs rather than arbitrary thresholds.

Question 9

What makes a good Grafana dashboard?

Accepted Answer

Grafana visualises data from many sources (Prometheus, Loki, Elasticsearch, CloudWatch, SQL) — it stores no metrics itself.

Principles for dashboards people actually use:

One dashboard, one question. "Is the checkout service healthy?" — not 60 panels of everything.
Structure top-down — RED signals (rate, errors, duration) at the top; resource/infra detail below. The first screen should answer "is this OK?" in three seconds.
Use template variables ($environment, $service, $pod) so one dashboard serves every service instead of 40 copies.
Percentiles, not averages — show p50/p95/p99. An average latency of 200ms hides that 1% of users wait 8 seconds.
Annotate deployments — overlaying deploy markers makes "what changed?" instantly visible. This is the single highest-value dashboard feature.
Consistent units and thresholds; add SLO lines so "good" is visible without interpretation.
Dashboards as code — provision JSON via Terraform/ConfigMap so they're version-controlled and reproducible, not hand-built and lost.

Question 10

How do you build centralised logging? Compare ELK/EFK with Loki.

Accepted Answer

In a distributed system you can't SSH to a box and tail — pods are ephemeral and there are hundreds. You need a pipeline: collect → ship → parse → store → query.

ELK / EFK stack

Beats/Fluentd/Fluent Bit — agent on each node collecting container stdout.
Logstash (optional) — parsing, enrichment, filtering.
Elasticsearch — indexes the full text of every log. Powerful search, but storage- and RAM-hungry and operationally heavy.
Kibana — query UI.

Loki takes the opposite approach: it indexes only labels (service, namespace, level) and stores the log content compressed in object storage. Dramatically cheaper and simpler to run; queries filter by label first then grep the stream. Trade-off: full-text search across everything is slower than Elasticsearch.

Choosing: Loki if you're already on Prometheus/Grafana, want low cost, and mostly query by service + time window (which is most DevOps use). Elasticsearch if you need rich full-text search, complex aggregations, or already run it for other purposes.

Question 11

What are logging best practices — structure, levels, correlation IDs, and what NOT to log?

Accepted Answer

Log structured JSON, not prose. "User 8821 failed payment after 30s" requires a regex to query; {"user_id":"8821","event":"payment_failed","duration_ms":30012} is filterable and aggregatable immediately. Levels, used consistently: ERROR — something failed and needs attention. If nobody would act on it, it isn't an error. WARN — unexpected but handled (retry succeeded, fallback used). INFO — significant business events (order placed, user registered). Default production level. DEBUG — detailed diagnostics, off in prod (or sampled / dynamically enableable). Correlation/trace IDs are non-negotiable in microservices. Generate an ID at the edge, propagate it through every downstream call and into every log line, so one query reconstructs the full journey of a request across ten services. Never log : passwords, tokens, API keys, full card numbers, personal data (GDPR/PCI), or entire request bodies. Redact at the logging library level, not by hoping developers remember. Also : write to stdout in containers (the platform handles shipping), never log inside a tight loop, and include enough context that the line is useful without the surrounding lines.

Question 12

What is distributed tracing? Explain spans, context propagation, and OpenTelemetry.

Accepted Answer

Distributed tracing follows one request as it crosses service boundaries, producing a timeline that shows exactly where time was spent. Trace — the whole request journey, identified by a trace_id . Span — one unit of work (an HTTP handler, a DB query, a cache lookup) with a start time, duration, parent span ID, and attributes. Spans nest into a tree. Context propagation — the mechanism that makes it work: the trace context travels in HTTP headers (the W3C traceparent header) or message metadata, so downstream services attach their spans to the same trace. If propagation breaks anywhere, the trace fragments — this is the most common tracing bug. OpenTelemetry (OTel) is the vendor-neutral standard: one set of SDKs and one Collector that exports to Jaeger, Tempo, Datadog, Honeycomb — so instrumentation isn't locked to a backend. It covers traces, metrics, and logs. Sampling is necessary at volume: head-based (decide at the start, e.g. keep 1%) is cheap but may miss the rare errors you care about; tail-based (decide after the trace completes) lets you keep 100% of slow or failed traces and 1% of the boring ones — far more useful, at the cost of buffering in the Collector.

Question 13

How do you monitor a Kubernetes cluster? What do you actually alert on?

Accepted Answer

Where the data comes from: node_exporter — host CPU, memory, disk, network per node. cAdvisor (built into kubelet) — per-container CPU/memory/network usage. kube-state-metrics — the desired vs actual state of Kubernetes objects: replicas wanted vs ready, pod phase, restart counts, job status, PVC state. This is the one people forget, and it's where most real alerts come from. API server / etcd / scheduler metrics — control-plane health. Usually deployed together as kube-prometheus-stack (Prometheus Operator + Grafana + Alertmanager + default rules). What's worth paging on (symptoms, at the workload level): Pods in CrashLoopBackOff , or restarting repeatedly. Deployment has fewer ready replicas than desired, sustained. Pods stuck Pending — unschedulable, usually resource pressure or a missing PVC. Containers being OOMKilled (exit 137) — the limit is wrong or there's a leak. CPU throttling — the silent killer: the app is slow but shows no errors, because it's hitting its CPU limit. Node NotReady , disk/memory pressure; PVC nearly full; certificates expiring.

Question 14

What is high cardinality and why does it break your metrics system?

Accepted Answer

Cardinality = the number of unique time series, which is the product of all label value combinations. Prometheus stores one series per unique label set, largely in memory — so cardinality is the dominant cost and stability factor. The killer: adding a label with unbounded values. http_requests_total{method, status, endpoint} = 5 × 8 × 20 = 800 series ✅ fine add user_id (1,000,000 users): = 800 × 1,000,000 = 800,000,000 💥 OOM Never use as metric labels : user IDs, email addresses, session IDs, request IDs, trace IDs, full URLs with path parameters, timestamps, or raw error messages. Use instead : templated routes ( /orders/{id} , not /orders/8821 ), bounded status classes, service/namespace/region. If you need per-user detail, that belongs in logs or traces — which are designed for high-cardinality data — not metrics. Symptoms of a cardinality problem: Prometheus memory climbing steadily, slow queries, OOM restarts, and a huge /metrics payload from one service.

Question 15

You get paged at 2am: "checkout latency p99 above SLO". Walk me through your response.

Accepted Answer

Mitigate first, root-cause second. The goal at 2am is restoring service, not understanding it. Acknowledge the page so others know it's being handled; open an incident channel. Assess impact — how many users, which regions, is it degrading or total? This sets severity and whether to escalate. Ask "what changed?" first. Most incidents follow a change: a deploy, a config/feature-flag toggle, an infrastructure change, or a traffic shift. Check the deploy timeline against the graph — annotations make this instant. Localise with the pillars — dashboard shows which service degraded; a trace shows which operation is slow; logs from that service explain why . Check dependencies — database CPU/connections/locks, cache hit rate, downstream third-party status, saturation on queues and connection pools. Mitigate — roll back the deploy, flip the feature flag off, scale out, shed load, or fail over. Rollback is usually fastest and safest. Verify recovery against the same SLI that alerted, then communicate. Blameless postmortem — timeline, contributing factors, and action items that include "what would have detected this sooner".

Question 16

Health checks vs monitoring — and what's the difference between liveness, readiness, and synthetic monitoring?

Accepted Answer

Health checks are how the platform decides what to do with an instance right now. Monitoring is how humans understand trends and get alerted. Different consumers, different design. Liveness probe — "is this process wedged?" Failing it restarts the container . Keep it dumb and dependency-free: if you check the database here, a brief DB blip restarts every pod in your fleet and turns a small problem into an outage. Readiness probe — "can this instance serve traffic right now?" Failing it removes the pod from the Service endpoints without restarting it. This is where dependency checks belong — it lets a pod temporarily step out while a dependency recovers. Startup probe — for slow-booting apps; disables the other probes until the app has started, so a slow JVM boot isn't mistaken for a hang. Synthetic / blackbox monitoring runs scripted requests against your service from outside — a probe hitting the login flow every minute from three regions. It catches what internal metrics can't: DNS failures, expired TLS certificates, CDN/load-balancer misconfiguration, and regional network issues. It's the closest signal to "can a real user actually use this?"

Question 17

Your observability bill has tripled. How do you cut it without losing visibility?

Accepted Answer

Observability cost is driven by volume × retention × cardinality . Attack each, starting with the biggest line (almost always logs). Drop noise at the collector — health-check and readiness-probe logs, verbose framework startup output, successful access logs for static assets. Filtering at the agent is free; filtering after ingestion is not. Tier retention — 7–14 days hot and searchable, then compressed object storage (S3/Glacier) for the compliance tail. Most log queries look at the last 48 hours. Sample — keep 100% of errors and slow requests, sample successful DEBUG/INFO. Same for traces (tail-based sampling). Fix cardinality — one metric with an unbounded label can dominate a Prometheus bill; audit with topk(count by (__name__)) and labeldrop the offenders. Downsample old metrics — full resolution for 15 days, 5-minute rollups for a year (Thanos/Mimir do this natively). Delete unused dashboards and alerts — and the metrics only they consumed. Right-size log levels in prod — DEBUG left on in one busy service is a classic cause of a sudden bill jump.

Question 18

Push vs pull metrics collection — what are the trade-offs?

Accepted Answer

Pull (Prometheus): the server scrapes /metrics from a target list it maintains. ✅ The server knows what should exist, so a missing target is detectable ( up == 0 ) rather than silently absent. ✅ You can curl a target's /metrics to debug it — no pipeline involved. ✅ No client can overwhelm ingestion; the server controls the rate. ❌ Needs network reachability to every target and service discovery to track them. ❌ Poor fit for short-lived jobs that finish between scrapes. Push (StatsD, Datadog agent, OTLP): the application sends metrics to a collector. ✅ Works through NAT/firewalls and across network boundaries; good for serverless, batch jobs, and edge/IoT. ✅ Naturally handles ephemeral workloads. ❌ A silent client is indistinguishable from a healthy client with nothing to report — you lose "is it alive?" for free. ❌ A misbehaving client can flood the collector. Pushgateway exists only for short-lived batch jobs. Do not use it as a general push endpoint: it becomes a single point of failure, it doesn't expire stale metrics automatically, and it breaks target-health semantics.

Question 19

API p99 latency jumped from 200ms to 3s, but error rate and CPU look normal. How do you find the cause?

Accepted Answer

Normal errors and normal CPU narrows this a lot — the system is waiting , not failing or computing. Work through it systematically: Is it all traffic or a slice? Break the p99 down by endpoint, region, customer, and pod. If p50 is unchanged but p99 exploded, a subset of requests is slow — the most common shape. What changed at that timestamp? Deploy, feature flag, config, traffic volume, or a dependency's own deploy. Where does the time go? This is exactly what tracing is for — compare a slow trace to a fast one and find the span that grew. Saturation on something invisible to CPU: Connection pool exhaustion — requests queue waiting for a DB connection. Classic cause of "slow but no errors". CPU throttling — the container is at its cgroup limit; host CPU looks fine. Thread pool / event loop blocked, or GC pauses (check GC time metrics). Lock contention or a long-running transaction in the database. Dependencies — slow downstream service, cache hit rate collapsed (so everything falls through to the DB), a third-party API degraded, DNS resolution slow. Data-shaped causes — a table grew past an index's usefulness, a missing index on a new query, an N+1 that only hurts at scale, or one large customer's dataset.

Question 20

You're adding observability to a service that has none. What do you instrument first?

Accepted Answer

Start with the signals that answer "is it working?" and expand only when a real question demands it.

RED metrics on every endpoint — request rate, error rate, and a latency histogram. This alone covers most incidents. Get it free via framework middleware or OpenTelemetry auto-instrumentation.
Structured JSON logs with a trace ID on every line, written to stdout.
Health endpoints — /healthz (process only) and /readyz (dependencies).
Outbound dependency metrics — latency and error rate for every DB query, cache call, and downstream HTTP request. "Which dependency is slow?" is the most-asked question in an incident.
Saturation gauges — connection pool in use vs max, queue depth, worker pool utilisation. These are your leading indicators.
A few business metrics — orders placed, payments failed, signups. These detect problems that are invisible technically: everything returns 200 but orders dropped 40%.
Then tracing — auto-instrument first, add manual spans only around meaningful custom operations.

Finally, define an SLO from those metrics and alert on that, rather than on arbitrary thresholds.

interviewDeck

All Questions

Observability Interview Questions and Answers