Question 1

What is LLMOps, and how does it differ from MLOps?

Accepted Answer

LLMOps is the practice of running LLM-powered features in production: evaluation, prompt and model versioning, deployment, monitoring, cost control and safety.

What's different from classic MLOps:

You usually don't own the model. The artifact you version is the prompt, tools and retrieval config — and the model can change under you.
No accuracy metric. Outputs are free text, so you need LLM-judges, rubrics and human review instead of a validation score.
Cost is per request, not per training run — and it scales with usage forever.
Non-determinism is permanent — the same input can produce different output, so testing is statistical.
New failure modes — hallucination, prompt injection, data leakage — that have no analogue in tabular ML.

What carries over: versioning, CI, staged rollout, monitoring, and a feedback loop from production into your dataset.

Question 2

How do you build an eval set for an LLM feature?

Accepted Answer

Start small and real. 30–50 examples drawn from actual usage beats 500 synthetic ones, because synthetic cases cluster around what you already imagined.

What to include:

Typical cases — the bulk of traffic.
Known failures — every bug report becomes a permanent test case. This is how the set earns its keep.
Edge cases — empty input, ambiguity, adversarial phrasing, out-of-scope requests, multilingual input.
Negative cases — questions it should refuse or say it doesn't know.

What each row needs: the input, the expected output or a rubric describing what a good answer contains, and enough metadata (category, difficulty, source) to slice results.

Then version it in git alongside the prompt, and grow it from production — every incident adds a row, so the set converges on your real distribution.

Question 3

How do you use LLM-as-judge well, and where does it mislead?

Accepted Answer

Use a model to grade outputs where no exact answer exists — helpfulness, faithfulness, tone, instruction-following.

What makes a judge trustworthy:

A specific rubric with a small scale. Binary or 1–3 with explicit criteria beats "rate 1–10", which produces noise around 7.
Reasoning before the score — require a justification field first; scores generated after reasoning are markedly better calibrated.
Reference-based where possible — grading against a known good answer is far more reliable than grading in the abstract.
Validate the judge against humans. Label 50 examples yourself and measure agreement. An unvalidated judge is a number, not a signal.

Known biases: position bias in pairwise comparisons (randomise order), length bias (longer answers score higher), self-preference (models favour their own style), and blindness to subtle factual errors in confident-sounding text.

Question 4

Offline evals vs online metrics — why do you need both?

Accepted Answer

Offline — run your eval set before shipping. Fast, cheap, repeatable, and it catches regressions. But it only measures what you thought to include, on a distribution you curated. Online — what real users do. Explicit signals (thumbs up/down, a rating, an edit to the output) and implicit ones, which are usually stronger: did they copy the answer, retry the query, rephrase, escalate to support, abandon the session, or complete the task? The two disagree constantly, and that gap is informative: offline scores rising while online engagement falls usually means you optimised the judge's rubric rather than the user's need. Close the loop: mine low-rated and retried sessions, turn them into eval cases, fix, re-run offline, ship, watch online. That cycle is the whole discipline.

Question 5

How do you run LLM evals in CI without flaky builds?

Accepted Answer

Layer the checks by determinism, and only gate on what's deterministic.

Every commit (fast, deterministic): prompt templates render, schemas validate, tool definitions parse, guardrail rules fire on known-bad inputs, and unit tests run against a mocked model. No API calls, seconds to run.
On changes to prompt/model/retrieval: run the eval set against the real model. Because outputs vary, gate on an aggregate threshold ("pass rate ≥ 85% and no regression on the critical subset"), never on individual outputs.
Nightly / pre-release: the full suite including expensive judges, cost and latency tracking, and safety probes.

Report the diff, not just the number — which cases newly failed matters far more than a score moving by two points. And keep a small critical subset where any regression blocks the release.

Question 6

How do you version and manage prompts?

Accepted Answer

Prompts are code — the highest-leverage code in the system — so treat them that way:

In the repo, reviewed in PRs. A prompt change can alter product behaviour more than a refactor; it deserves a diff and a reviewer.
Templated with typed variables, not string-concatenated at the call site, so you can render and test them.
Versioned identifiers logged with every request — prompt_id + version + model_version in the trace. Without this you cannot attribute a quality change to a cause.
Evaluated before merge — the eval set runs on the prompt diff.

The hosted prompt-registry trade-off: letting non-engineers edit prompts in a UI is genuinely useful, but it decouples prompt changes from code review and deploys. If you do it, keep versioning, require an eval run before promotion, and make rollback one click.

Question 7

What do you monitor for an LLM feature in production?

Accepted Answer

Operational: request rate, error rate by type (429 / 5xx / timeout / validation failure), TTFT and total latency percentiles, retries, and provider availability. Economic: input and output tokens per request, cost per request and per user, cache hit rate, and spend by feature — with alerts on anomalies, not just a monthly invoice. Quality: user feedback rate, retry/regeneration rate, guardrail block rate, refusal rate, structured-output parse-failure rate, and a sampled judge score over live traffic. Content: input and output length distributions, and topic/intent drift — a shift here usually explains a quality change before anything else does. Underlying all of it: a trace per request containing the rendered prompt, retrieved context, model and prompt versions, and the response — with PII redacted on the way in.

Question 8

How do you safely roll out a prompt or model change?

Accepted Answer

Same ladder as any risky deploy, adapted for non-determinism:

Offline eval — the gate before anything reaches users.
Shadow mode — run the new version on real traffic, log the output, serve the old one. Zero user risk, real distribution. The best-value step and the one most often skipped.
Canary — 1–5% of traffic, watching quality and cost metrics, not just errors.
A/B test — if you need a business-metric answer (task completion, escalation rate), run a real experiment with enough traffic for significance.
Full rollout with a feature flag and a fast rollback path.

Two LLM-specific cautions: quality regressions are silent — nothing errors, so you must watch quality metrics, not just health checks; and cost can regress too — a more verbose model at the same quality is still a bad deploy.

Question 9

How do you migrate to a new model version without breaking things?

Accepted Answer

Pin explicit versions in production. Migration should be a decision you make, not something that happens to you because you pointed at a floating alias. The process: Run your eval set on the new version and diff results case by case, not just the aggregate score. Check the things that quietly change : output format and verbosity, refusal boundaries, tool-calling behaviour, JSON strictness, and token count (a chattier model costs more at identical quality). Expect prompt drift — prompts are tuned to a model's quirks. Some instructions become unnecessary, some stop working. Shadow, then canary before full traffic. Keep the old version routable until you're confident; deprecation dates are your real deadline. Watch for output-shape breakage most of all — downstream parsers, not the answer quality, are what usually break first.

Question 10

How do you keep LLM costs under control at scale?

Accepted Answer

Measure first, at the right granularity: cost per request, per user, per feature. A single expensive feature or a small set of heavy users usually dominates, and you can't see that from a monthly invoice. Levers, roughly by payoff: Prompt caching — biggest single win for repeated system prompts and stable context. Model routing — classify difficulty and send easy requests to a cheap model. Often 60–80% of traffic is easy. Shorten the input — fewer retrieved chunks, trimmed history, no redundant instructions. Input tokens are usually most of the bill. Cap output — max_tokens plus "be concise"; output tokens are typically priced several times higher than input. Cache answers for repeated questions; batch APIs for anything non-interactive (often ~50% cheaper). Per-tenant quotas and alerts so abuse or a retry storm can't become a five-figure surprise.

Question 11

How do you set SLOs for a streaming LLM endpoint?

Accepted Answer

A single "request duration" SLO is meaningless when responses stream and length varies wildly. Split it:

TTFT p95 — the number users actually feel. This is your primary SLO (e.g. p95 < 1.5 s).
Inter-token latency p95 — whether the stream stalls mid-answer.
Time to completion, normalised per output token — so a long answer doesn't count as a violation.
Stream failure rate — connections dropped after the first token. Uniquely nasty because the user has already seen a partial answer, and it's invisible to a plain error-rate metric.

Error budget realities: your provider's availability caps yours, so an SLO tighter than theirs needs multi-provider failover. And plan degraded modes — a cached response, a smaller model, or an honest "try again shortly" beats a spinner.

Question 12

What are the caching layers in an LLM system?

Accepted Answer

Four distinct caches, often confused with each other:

Exact response cache — hash of (prompt + params) → response. Free, safe, and effective on repetitive traffic. Always do this first.
Semantic cache — embed the query, serve a stored answer above a similarity threshold. Higher hit rate, real risk of near-miss errors ("enable" vs "disable") — must be keyed by tenant/user when answers are personalised.
Provider prompt cache — the provider reuses prefill state for a stable prefix. Cuts cost and TTFT; requires ordering the prompt stable-first.
Embedding cache — text → vector. Embeddings are deterministic per model version, so re-embedding the same content is pure waste; key it by content hash and model version.

Every layer needs an invalidation story: TTLs, invalidation on document reindex, and a bypass flag for debugging.

Question 13

Should you abstract over multiple LLM providers?

Accepted Answer

Reasons to: outage failover, capacity and rate-limit headroom, price negotiation, per-task routing to whichever model is best, and reducing dependency on a single vendor's roadmap. Reasons not to overdo it: a lowest-common-denominator abstraction throws away exactly the features that matter — prompt caching semantics, structured outputs, tool-calling formats, thinking budgets, citations. Prompts are also tuned per model, so "just switch providers" is rarely a config change; it's a re-evaluation. A pragmatic middle: a thin internal interface ( generate , generateStructured , embed ) with per-provider adapters that are allowed to use native features, plus your own routing, retry and logging around it. Keep prompts versioned per model , and maintain evals for the fallback path so failover doesn't silently degrade quality. Gateways (LiteLLM, OpenRouter, Bedrock) give you much of this without writing the layer yourself — at the cost of another hop and another dependency.

Question 14

How do you turn user feedback into product improvement?

Accepted Answer

Collect it, but design for the fact that almost nobody clicks thumbs-down — typically well under 1% of sessions.

Higher-yield signals: regeneration/retry, the user rephrasing the same question, copying the answer (positive), editing a generated draft heavily (negative), abandoning the session, escalating to a human.

The loop:

Capture the full trace with the feedback so the case is reproducible.
Cluster failures by theme — embed the failing inputs and group them. Ten instances of one root cause are far more actionable than a list of 200 complaints.
Triage by cause: retrieval miss, prompt gap, model limitation, or a genuine product gap.
Add to the eval set, fix, verify offline, ship, watch the metric.

When feedback volume grows, it also becomes fine-tuning or distillation data — but only after validation; training on unvalidated production output amplifies your own errors.

Question 15

How do you architect a guardrail pipeline?

Accepted Answer

Guardrails belong outside the model, on both sides, and they should be cheap relative to the call they protect. Input stage: deterministic rules first (length limits, blocklists, rate limits), then a small classifier for abuse, injection patterns and off-topic requests. Rejecting here saves a frontier-model call. Output stage: schema validation, secret/PII scanning, policy classification, and domain scope checks — plus, for RAG, a groundedness check that every claim traces to retrieved context. Design decisions that matter more than the classifiers: Fail open or closed , per guardrail — a PII leak blocks; a tone check warns. Latency budget — run independent checks in parallel; a serial chain of four classifiers destroys TTFT. Streaming — you can't unsee tokens already sent. Either buffer, or validate in chunks with the ability to retract. Observability — log every block with reason and input so false positives are measurable. An over-blocking guardrail is an outage nobody paged for.

Question 16

How do you red-team an LLM application before launch?

Accepted Answer

Attack your own system deliberately, across the categories that actually apply to it:

Prompt injection — direct and, more importantly, indirect via any content the system ingests (documents, tickets, web pages, tool output).
System prompt extraction — assume it will leak; don't put secrets in it.
Scope escape — getting a support bot to write code, give legal advice, or discuss competitors.
Data leakage — cross-tenant retrieval, PII in outputs, another user's context in a shared cache.
Tool abuse — coaxing an unauthorised action, or a destructive one without approval.
Resource abuse — prompts engineered to maximise output length or trigger long agent loops (a denial-of-wallet attack).

Make it repeatable: keep the successful attacks as a regression suite that runs on every prompt and model change — a fix for one jailbreak rarely generalises, and model updates reopen old holes. Combine automated adversarial generation with human creativity; the interesting attacks are still found by people.

Question 17

What data governance questions must you answer before shipping an LLM feature?

Accepted Answer

What data leaves your boundary , to which processor, in which region? Map it before legal asks. Retention — how long does the provider keep prompts? Enterprise tiers typically offer zero data retention; consumer tiers do not. Training — is your data used to train? Get it in the contract, not the marketing page. Residency — EU/India/US endpoints, and whether failover to another region is contractually allowed. Your own storage — traces and eval sets contain full prompts. That is often the largest copy of sensitive data in the system, and it needs classification, retention limits and access controls. Deletion rights — if a user requests erasure, can you remove their data from your vector index, traces, caches and eval sets? Design for this or it becomes an emergency. Disclosure — does the user know they're talking to AI? Increasingly a legal requirement, not a courtesy.

Question 18

When and how do you fine-tune in production?

Accepted Answer

When it's justified: a stable, high-volume task where prompting has plateaued; a consistent output format or style you can't reliably prompt for; a domain vocabulary the base model handles poorly; or a cost/latency target that needs a small model to match a big one's quality on one task. When it isn't: the model lacks facts (use RAG), the task changes often (retraining treadmill), or you have fewer than a few hundred solid examples. The pipeline: Collect from production traffic — validated inputs and outputs, deduplicated, PII-scrubbed. Curate — a few hundred to a few thousand high-quality examples beats a hundred thousand noisy ones. Data quality dominates everything here. Hold out a test set before training, from a later time period if possible. Train (usually LoRA/PEFT — cheap, quick, swappable adapters). Evaluate against the prompted baseline — the honest comparison, and it sometimes loses. Deploy via shadow → canary, keeping the base model as fallback, and plan for retraining as distribution drifts.

Question 19

What does self-hosting an LLM actually involve?

Accepted Answer

Sizing: weights need roughly 2 bytes per parameter at FP16, plus KV cache scaling with concurrency × context length, plus activation overhead. A 70B model at FP16 needs ~140 GB — multiple GPUs and tensor parallelism — while the same model at INT4 fits on far less at some quality cost. Serving: vLLM or TGI for continuous batching and PagedAttention; a single naive process serving one request at a time wastes almost all the hardware. Operations you now own: GPU capacity planning (they're expensive and often supply-constrained), autoscaling with slow cold starts because loading weights takes minutes, model version upgrades, quantization decisions, and your own evals for every change. The economics: a GPU instance is a fixed hourly cost regardless of traffic, so self-hosting wins only above a high steady utilisation. Below that, per-token API pricing is cheaper and someone else is on call.

Question 20

Users report 'the AI got worse this week'. How do you investigate?

Accepted Answer

Turn the vague report into a diff. Work through what could have changed:

Did we change anything? Prompt version, model version, retrieval config, chunking, a tool, a schema. Check the deploy log first — it's usually here.
Did the provider change anything? A floating alias silently repointed, a model deprecated, a safety-filter update. This is why versions get pinned and logged.
Did the inputs change? A new user segment, a new document set, longer queries, a different language. Compare input distributions week over week.
Did retrieval change? An index rebuild, a failed sync, deleted documents, an ACL change. Check recall on the eval set.
Did it actually change? Run the eval set now and against last week's config. Sometimes the answer is a new cohort of users with different expectations.

Then reproduce with real traces from complaining users — not a fresh prompt you wrote yourself, which won't contain the context that caused the failure.

interviewDeck

All Questions

LLMOps & Evals Interview Questions and Answers