Python Interview Questions and Answers
59 hand-picked Python interview questions with
detailed answers. Open the interactive version above to search, filter
by difficulty, run code, bookmark questions and track your progress.
What is Python and what are its key features?
Python is a high-level, interpreted, dynamically typed language known for readability and a large standard library. Key features:
- Interpreted — runs via the interpreter (CPython is the reference); no separate compile step for development.
- Dynamically typed — types are checked at runtime, not declared upfront.
- Multi-paradigm — supports OOP, functional (map/filter/lambdas), and procedural styles.
- Batteries included — rich stdlib for files, HTTP, JSON, dates, threading, asyncio, etc.
- Strong ecosystem — Django, FastAPI, Flask, pandas, NumPy, pytest, Celery.
List vs tuple — when do you use each?
List — mutable, ordered sequence. Use when you need to add, remove, or change elements.
Tuple — immutable, ordered sequence. Use for fixed collections (coordinates, DB rows, dict keys), as return values, and when you want hashability (if all elements are hashable).
Both preserve insertion order (since Python 3.7+ for dicts; lists/tuples always did).
coords = (12.9, 77.6) # fixed — tuple
items = ["a", "b"] # will grow/shrink — list
items.append("c")
Dictionary vs set — key differences.
dict — key-value mappings; keys must be hashable and unique; O(1) average lookup by key. Use for records, caches, counts (with Counter).
set — unordered collection of unique hashable elements; O(1) average membership test. Use for deduplication, union/intersection, fast 'in' checks.
seen = set()
if user_id not in seen:
seen.add(user_id)
freq = {}
freq[key] = freq.get(key, 0) + 1
What is mutable vs immutable in Python?
Immutable — object cannot change after creation: int, float, str, tuple, frozenset, bytes. Operations create new objects.
Mutable — object can be modified in place: list, dict, set, custom objects (by default).
This matters for default arguments, copying, and thread safety.
s = "hi"
s += "!" # new str object; old "hi" unchanged
lst = [1, 2]
lst.append(3) # same list object, mutated in place
== vs is — what's the difference?
== compares values (calls __eq__). is compares identity — whether two names refer to the same object in memory.
Use is only for singletons like None, True, False. Never use is for strings or numbers (small-int caching can fool you).
a = [1, 2]
b = [1, 2]
a == b # True (same values)
a is b # False (different list objects)
x = None
x is None # correct idiom
Explain *args and **kwargs.
*args collects extra positional arguments into a tuple. **kwargs collects extra keyword arguments into a dict.
Order in a signature: positional params → *args → keyword-only params → **kwargs.
Also used to unpack when calling: func(*items), func(**options).
def log(level, *msgs, **meta):
for m in msgs:
print(level, m, meta)
log("INFO", "started", "ok", user=42)
opts = {"timeout": 5}
connect(host, **opts)
What is a list comprehension and when would you use it?
A concise way to build a list from an iterable with optional filtering: [expr for item in iterable if condition].
Also exists for dicts ({k: v for ...}) and sets. Prefer comprehensions over manual append loops for simple transforms — they're faster and more readable.
For large data or lazy evaluation, use a generator expression: (expr for item in iterable).
squares = [x * x for x in range(10)]
evens = [x for x in nums if x % 2 == 0]
# generator — no list allocated
sum(x * x for x in range(1_000_000))
lambda, map, and filter — how do they work?
lambda — anonymous one-expression function: lambda x: x * 2.
map(fn, iterable) — applies fn to each item; returns an iterator.
filter(fn, iterable) — keeps items where fn is truthy.
In modern Python, list comprehensions often replace map/filter for readability, but map/filter shine when composing with other iterators.
nums = [1, 2, 3, 4]
list(map(lambda x: x * 2, nums)) # [2, 4, 6, 8]
list(filter(lambda x: x % 2 == 0, nums)) # [2, 4]
# often clearer:
[x * 2 for x in nums]
What is a decorator in Python?
A decorator is a callable that wraps another function to extend or modify its behaviour without changing its source. Syntax: @decorator above a function definition.
Under the hood: @dec def f(): ... is equivalent to f = dec(f). Decorators are used for logging, timing, auth checks, caching (@lru_cache), and route registration in Flask/FastAPI.
from functools import wraps
def retry(fn):
@wraps(fn)
def wrapper(*a, **kw):
for _ in range(3):
try: return fn(*a, **kw)
except Exception: pass
raise
return wrapper
@retry
def fetch(): ...
What is a generator and how does yield work?
A generator is an iterator produced by a function containing yield. Each yield pauses the function, returns a value, and resumes on the next next() call. State (local variables, instruction pointer) is preserved between yields.
Generators are lazy — they produce values one at a time without building the full collection in memory. Ideal for large files, infinite sequences, and pipeline processing.
def read_lines(path):
with open(path) as f:
for line in f:
yield line.strip() # one line at a time
for line in read_lines("huge.log"):
process(line)
What is a context manager and the with statement?
A context manager guarantees setup and teardown around a block — typically acquiring and releasing a resource. The with statement calls __enter__ at the start and __exit__ at the end (even if an exception occurs).
Common uses: file handles, DB connections, locks, temporary directories. contextlib.contextmanager lets you write one with a generator and yield.
with open("data.txt") as f:
data = f.read()
# file closed automatically
from contextlib import contextmanager
@contextmanager
def timer():
start = time.perf_counter()
yield
print(time.perf_counter() - start)
How does exception handling work in Python?
try / except catches exceptions; else runs if no exception; finally always runs (cleanup). Catch specific types — except ValueError: — not bare except:.
Raise with raise ValueError('msg') or raise to re-raise. Custom exceptions subclass Exception. Use exception chaining: raise NewError() from original.
try:
val = int(user_input)
except ValueError as e:
logger.warning("bad input: %s", e)
val = 0
finally:
cleanup()
Explain classes, __init__, and inheritance in Python.
A class defines attributes and methods. __init__ is the constructor — called when an instance is created. self refers to the instance.
Inheritance — class Child(Parent):. Override methods; call parent with super().__init__(...) or super().method().
Python supports multiple inheritance; method resolution uses the MRO (C3 linearization).
class Animal:
def __init__(self, name):
self.name = name
class Dog(Animal):
def speak(self):
return f"{self.name} says woof"
@staticmethod vs @classmethod vs @property.
@staticmethod — function in a class namespace; no self or cls. Utility grouping.@classmethod — receives cls; common for alternative constructors like from_dict(cls, data).@property — exposes a method as a read-only (or read-write with setter) attribute.
class User:
def __init__(self, email): self.email = email
@classmethod
def from_row(cls, row): return cls(row["email"])
@property
def domain(self): return self.email.split("@")[1]
__str__ vs __repr__ — what's the difference?
__repr__ — unambiguous, developer-facing representation; goal: eval(repr(obj)) should recreate the object when possible. Shown in the REPL and debuggers.
__str__ — human-readable, user-facing. Used by print() and str(). Falls back to __repr__ if not defined.
class Point:
def __init__(self, x, y): self.x, self.y = x, y
def __repr__(self): return f"Point({self.x}, {self.y})"
def __str__(self): return f"({self.x}, {self.y})"
What is the mutable default argument trap?
Default argument values are evaluated once at function definition time, not per call. A mutable default (list, dict, set) is shared across all calls.
Classic bug: def f(items=[]) — every call appends to the same list. Fix: use None and create a new object inside the function.
# BUG
def add(item, bucket=[]):
bucket.append(item)
return bucket
# FIX
def add(item, bucket=None):
if bucket is None:
bucket = []
bucket.append(item)
return bucket
Shallow copy vs deep copy.
Assignment (b = a) — same object, two names.
Shallow copy — new container, but nested objects are still shared (copy.copy(), list.copy(), dict.copy()).
Deep copy — recursively copies nested objects (copy.deepcopy()).
import copy
a = [[1, 2], [3, 4]]
b = copy.copy(a) # outer list new, inner lists shared
c = copy.deepcopy(a) # fully independent
What is duck typing?
“If it walks like a duck and quacks like a duck, it's a duck.” Python cares about behaviour (methods/attributes), not the object's declared type. Any object with the right methods works — no inheritance required.
Modern Python adds typing.Protocol for structural subtyping — formalising duck typing for type checkers.
def save(writer):
writer.write(data) # anything with .write() works
# file, StringIO, custom class — all fine
What is the GIL and how does it affect concurrency?
The Global Interpreter Lock (GIL) is a mutex in CPython that allows only one thread to execute Python bytecode at a time. It simplifies memory management (reference counting) but limits CPU-bound parallelism with threads.
Implications:
- I/O-bound work (network, disk) — threads are fine; threads release the GIL while waiting.
- CPU-bound work — use
multiprocessing (separate processes, separate GILs) or offload to C extensions / NumPy. - Async I/O —
asyncio for many concurrent I/O operations on one thread.
Threading vs multiprocessing vs asyncio — when to use each?
| Approach | Best for | Notes |
|---|
| threading | I/O-bound, blocking libraries | GIL limits CPU parallelism; simple for concurrent HTTP/DB calls |
| multiprocessing | CPU-bound on multiple cores | Separate memory; use Pool or Process |
| asyncio | High-concurrency I/O | Single-threaded event loop; needs async libraries (aiohttp, asyncpg) |
# asyncio
async def fetch_all(urls):
async with aiohttp.ClientSession() as s:
tasks = [s.get(u) for u in urls]
return await asyncio.gather(*tasks)
What are type hints and do they affect runtime?
Type hints annotate expected types for parameters and return values: def greet(name: str) -> str:. They are not enforced at runtime by default — they're for static analysis (mypy, pyright), IDE autocomplete, and documentation.
Use Optional[T], list[str] (3.9+), Union, TypedDict, Protocol for richer contracts.
from typing import Optional
def find_user(id: int) -> Optional[dict]:
...
def process(items: list[str]) -> int:
return len(items)
What are dataclasses?
@dataclass auto-generates __init__, __repr__, __eq__ from class attributes. Reduces boilerplate for data-holding objects.
Options: frozen=True (immutable), slots=True (memory), order=True (comparison). For validation at scale, pair with Pydantic models.
from dataclasses import dataclass
@dataclass(frozen=True)
class Point:
x: float
y: float
What is a virtual environment and why use one?
A virtual environment is an isolated Python installation with its own packages, separate from the system Python. Created with python -m venv .venv; activated before installing deps.
Prevents version conflicts between projects. Dependencies are listed in requirements.txt or pyproject.toml and installed with pip install -r requirements.txt.
python -m venv .venv
source .venv/bin/activate # Linux/Mac
.venv\Scripts\activate # Windows
pip install -r requirements.txt
Flask vs FastAPI vs Django — when would you pick each?
- Flask — minimal microframework; you choose ORM, auth, structure. Good for small APIs, prototypes, full control.
- FastAPI — modern async API framework; automatic OpenAPI docs, Pydantic validation, high performance. Great for REST/JSON microservices.
- Django — batteries-included (ORM, admin, auth, migrations). Best for full web apps, CMS, rapid CRUD, monoliths.
# FastAPI
from fastapi import FastAPI
app = FastAPI()
@app.get("/users/{id}")
async def get_user(id: int): ...
How does SQLAlchemy ORM work at a high level?
SQLAlchemy maps Python classes to database tables. Define a Model with columns; the Session tracks changes and commits transactions.
Two layers: Core (SQL expression language) and ORM (object mapping). Use session.query(User).filter_by(email=...) or 2.0-style select(User).where(...).
Migrations are typically handled by Alembic.
class User(Base):
__tablename__ = "users"
id = Column(Integer, primary_key=True)
email = Column(String, unique=True)
user = session.query(User).filter_by(email="a@b.com").first()
How do you write tests with pytest?
pytest discovers test functions named test_*. Use plain assert — pytest rewrites assertions for clear failure output.
Fixtures (@pytest.fixture) provide setup/teardown and dependency injection. conftest.py shares fixtures across a package. Use monkeypatch, tmp_path, and pytest.mark.parametrize for parametrized tests.
import pytest
@pytest.fixture
def client():
app.config["TESTING"] = True
return app.test_client()
def test_health(client):
r = client.get("/health")
assert r.status_code == 200
How does Python manage memory?
CPython uses reference counting — when an object's refcount hits zero, it's freed immediately. A generational garbage collector handles circular references (refcount alone can't).
Objects live in a private heap. Interning caches small ints and some strings. For large data, prefer generators and __slots__ to reduce per-instance dict overhead.
import sys
sys.getrefcount(obj) # inspect refcount (debug only)
What does @functools.lru_cache do?
Memoizes function results — caches up to maxsize recent calls (LRU eviction). Arguments must be hashable. Huge win for expensive pure functions called repeatedly with the same inputs.
@lru_cache(maxsize=None) caches everything (unbounded). Use .cache_info() and .cache_clear() for introspection.
from functools import lru_cache
@lru_cache(maxsize=128)
def fib(n):
if n < 2: return n
return fib(n-1) + fib(n-2)
What does if __name__ == '__main__' mean?
Every Python file is a module. When run directly, __name__ is '__main__'. When imported, __name__ is the module path (e.g. 'utils.helpers').
The guard runs code only when the file is executed, not when imported — standard pattern for CLI entry points and quick tests.
def main():
print("running")
if __name__ == "__main__":
main()
global vs nonlocal — when are they needed?
By default, assignment inside a function creates a local variable. global x declares assignment targets the module-level x. nonlocal x targets the enclosing (non-global) scope — used in nested functions/closures.
Prefer returning values or mutable containers over mutating globals. Closures capturing variables by reference are usually cleaner than nonlocal.
count = 0
def outer():
n = 0
def inner():
nonlocal n
n += 1
return n
return inner
What is the Method Resolution Order (MRO)?
When a class inherits from multiple parents, Python uses C3 linearization to determine the order it searches for methods. View with ClassName.mro() or ClassName.__mro__.
Prevents the 'diamond problem' ambiguity — each class appears once, parents before children, preserving local precedence order.
class A: def f(self): return "A"
class B(A): pass
class C(A): pass
class D(B, C): pass
D.mro() # D, B, C, A, object
What does __slots__ do?
__slots__ restricts instance attributes to a fixed set and replaces the per-instance __dict__ with compact slot storage. Saves memory when creating millions of small objects.
Tradeoffs: can't add arbitrary attributes at runtime; multiple inheritance gets tricky; no weakrefs unless '__weakref__' is in slots.
class Point:
__slots__ = ("x", "y")
def __init__(self, x, y):
self.x, self.y = x, y
How do background task queues like Celery work?
The web request enqueues a task to a message broker (Redis/RabbitMQ). Worker processes pull tasks and execute them asynchronously. Results can be stored in a backend (Redis, DB).
Use for: sending emails, image processing, report generation — anything slow that shouldn't block the HTTP response.
# tasks.py
@celery.task
def send_email(user_id):
...
# view
send_email.delay(user_id) # returns immediately
What is Pydantic and why is it popular?
Pydantic defines data models with type annotations and validates/parses input at runtime. Invalid data raises clear errors. Serialises to/from JSON dicts.
Core to FastAPI request/response validation. Pydantic v2 is Rust-backed (pydantic-core) — much faster than v1.
from pydantic import BaseModel, EmailStr
class UserCreate(BaseModel):
email: EmailStr
age: int
user = UserCreate(email="a@b.com", age=25)
What do enumerate and zip do?
enumerate(iterable) yields (index, value) pairs — avoids manual counter variables.
zip(a, b) pairs elements from multiple iterables. Stops at the shortest. Use zip(a, b, strict=True) (3.10+) to error on length mismatch.
names = ["Ada", "Bob"]
for i, name in enumerate(names):
print(i, name)
for a, b in zip([1, 2], ["x", "y"]):
print(a, b)
Iterable vs iterator — what's the difference?
They're different roles, and the distinction explains a lot of Python behaviour:
- An iterable implements
__iter__() and can produce a fresh iterator on demand. Lists, tuples, dicts, sets, and strings are iterables — you can loop over them repeatedly. - An iterator implements
__next__() (and __iter__() returning itself). It holds the position and is consumed exactly once — when exhausted it raises StopIteration and never restarts.
A for loop is sugar for: call iter() on the object, then call next() repeatedly until StopIteration.
This is why a generator, map(), filter(), and zip() can only be consumed once — they're iterators, not iterables. Looping over the same generator a second time silently gives you nothing, which is a genuinely common bug.
nums = [1, 2, 3] # iterable — reusable
it = iter(nums) # iterator — one-shot
next(it) # 1
next(it) # 2
# generators/map/filter/zip are ITERATORS
g = (x * 2 for x in nums)
list(g) # [2, 4, 6]
list(g) # [] ← already exhausted, no error!
# custom iterable: __iter__ returns a NEW iterator each time
class Countdown:
def __init__(self, n): self.n = n
def __iter__(self):
cur = self.n
while cur > 0:
yield cur
cur -= 1
c = Countdown(3)
list(c) # [3, 2, 1]
list(c) # [3, 2, 1] ← reusable, because __iter__ makes a fresh generator
What is a closure? Explain the late-binding gotcha.
A closure is a nested function that captures variables from its enclosing scope and keeps them alive after the outer function has returned. It's the mechanism behind decorators, callbacks, and factory functions.
The gotcha: Python closures capture variables by reference, not by value — they're late-bound, meaning the value is looked up when the inner function is called, not when it's defined.
So a loop creating functions gives you N functions that all see the loop variable's final value:
fns = [lambda: i for i in range(3)]
[f() for f in fns] # [2, 2, 2] — not [0, 1, 2]
The fix is a default argument, which is evaluated at definition time and binds the current value: lambda i=i: i. (Or functools.partial.)
Related: use nonlocal if the inner function needs to rebind an enclosing variable rather than just read it.
# ❌ late binding — all three see i == 2
fns = []
for i in range(3):
fns.append(lambda: i)
print([f() for f in fns]) # [2, 2, 2]
# ✅ default arg is evaluated at DEFINITION time
fns = [lambda i=i: i for i in range(3)]
print([f() for f in fns]) # [0, 1, 2]
# ✅ or functools.partial
from functools import partial
fns = [partial(lambda x: x, i) for i in range(3)]
# closure as a factory
def multiplier(factor):
def apply(x):
return x * factor # `factor` stays alive after multiplier returns
return apply
double = multiplier(2)
double(21) # 42
print(double.__closure__[0].cell_contents) # 2 — the captured cell
How does asyncio actually work? What are the common mistakes?
async def defines a coroutine. Calling it does not run it — it returns a coroutine object that must be awaited or scheduled. The event loop runs one coroutine at a time on a single thread; at each await the coroutine yields control back to the loop, which runs something else while that I/O is pending.
This is cooperative concurrency, not parallelism — nothing runs simultaneously. It wins for I/O-bound work (thousands of concurrent HTTP calls or DB queries on one thread) and does nothing for CPU-bound work.
The mistakes that matter:
- Calling a blocking function inside async code —
time.sleep(), requests.get(), a sync DB driver. It blocks the entire event loop, so every concurrent task stalls. This is the number one asyncio bug. Use async libraries, or push the blocking call to a thread with asyncio.to_thread(). - Awaiting sequentially instead of concurrently —
await a(); await b() runs them one after another. Use asyncio.gather() to overlap them. - Forgetting to await — you get a coroutine object and a "never awaited" warning; the code silently didn't run.
- Fire-and-forget tasks —
asyncio.create_task() results must be kept referenced and awaited, or exceptions vanish silently.
import asyncio, time
# ❌ sequential: 3 seconds
async def bad():
a = await fetch("/a") # 1s
b = await fetch("/b") # 1s
c = await fetch("/c") # 1s
# ✅ concurrent: ~1 second
async def good():
a, b, c = await asyncio.gather(fetch("/a"), fetch("/b"), fetch("/c"))
# ❌ blocks the ENTIRE loop — every other task stalls too
async def blocking():
time.sleep(2) # never do this
requests.get(url) # nor this (sync library)
# ✅ non-blocking, or offload to a thread
async def fixed():
await asyncio.sleep(2)
data = await asyncio.to_thread(requests.get, url) # sync lib → thread
# bounded concurrency — don't open 10,000 sockets at once
sem = asyncio.Semaphore(20)
async def limited(url):
async with sem:
return await fetch(url)
# timeouts and structured concurrency (3.11+)
async with asyncio.timeout(5):
async with asyncio.TaskGroup() as tg: # child failure cancels siblings
tg.create_task(worker(1))
tg.create_task(worker(2))
What's in the collections module and when do you use each?
defaultdict — a dict that creates a default value on first access, killing the if key not in d boilerplate. defaultdict(list) for grouping, defaultdict(int) for counting.Counter — counts hashable items and gives you most_common(n) for free. Replaces a manual frequency loop entirely.deque — double-ended queue with O(1) appends and pops at both ends. Critical: list.pop(0) is O(n) because it shifts every element, so a list is the wrong queue. Also supports maxlen for a fixed-size sliding window.namedtuple — a lightweight immutable record with named fields; tuple-cheap but readable (p.x instead of p[0]). For anything richer, use a dataclass.OrderedDict — mostly historical since dicts preserve insertion order from 3.7, but still useful for move_to_end() and order-sensitive equality (handy for an LRU cache).ChainMap — layered lookup across multiple dicts without merging (defaults ← config file ← env vars).
from collections import defaultdict, Counter, deque, namedtuple
# group without boilerplate
by_dept = defaultdict(list)
for emp in employees:
by_dept[emp.dept].append(emp.name) # no key check needed
# frequency in one line
Counter("mississippi").most_common(2) # [('i', 4), ('s', 4)]
Counter(words) - Counter(stopwords) # counters support arithmetic
# O(1) queue — list.pop(0) is O(n)
q = deque([1, 2, 3])
q.appendleft(0); q.popleft()
recent = deque(maxlen=100) # fixed-size sliding window
recent.append(x) # oldest drops automatically
# lightweight record
Point = namedtuple("Point", "x y")
p = Point(3, 4); p.x # readable, still a tuple
Which itertools functions do you actually use?
itertools provides memory-efficient, lazy iterator building blocks — they return iterators, so nothing is materialised until you consume it.
chain(a, b) — iterate multiple sequences as one, without concatenating them in memory. chain.from_iterable(lists) flattens one level.islice(it, start, stop) — slice any iterator (you can't use [:10] on a generator).groupby(it, key) — group consecutive items. The trap: you must sort by the same key first, or you get fragmented groups. Unlike SQL GROUP BY, it doesn't sort for you.product, permutations, combinations — combinatorics without nested loops; product is ideal for parameter grids.cycle, repeat, count — infinite iterators (round-robin assignment, retry delays).accumulate — running totals.tee — split one iterator into several (but it buffers, so it's not free).
from itertools import chain, islice, groupby, product, accumulate
# iterate two lists without building a third
for x in chain(list_a, list_b): ...
flat = list(chain.from_iterable([[1, 2], [3, 4]])) # [1, 2, 3, 4]
# slice a generator (can't use [:5])
first_five = list(islice(infinite_gen(), 5))
# ⚠️ groupby needs the data SORTED by the same key
employees.sort(key=lambda e: e.dept) # required!
for dept, group in groupby(employees, key=lambda e: e.dept):
print(dept, [e.name for e in group])
# parameter grid without nested loops
for lr, batch in product([0.01, 0.1], [16, 32]):
train(lr, batch)
list(accumulate([1, 2, 3, 4])) # [1, 3, 6, 10]
# batching (3.12+); before that, use an islice loop
from itertools import batched
for chunk in batched(range(10), 3): # (0,1,2), (3,4,5)...
bulk_insert(chunk)
How do you sort in Python — key functions, stability, and multi-field sorts?
sorted(iterable) returns a new list and works on any iterable; list.sort() sorts in place and returns None (assigning its result is a common bug).key= takes a function applied to each element to derive the sort value. It's called once per element, unlike a comparator, which is why Python dropped cmp.- Python's sort is stable — equal elements keep their original relative order. This is the key to multi-field sorting: sort by the least significant field first, then the most significant, and the earlier order survives within ties.
- For a single multi-field sort, return a tuple from
key — tuples compare element by element. - Mixed ascending/descending on different fields: negate numeric fields (
-x.score), or do two stable passes.
operator.itemgetter/attrgetter are faster than equivalent lambdas since they're implemented in C.
from operator import itemgetter, attrgetter
sorted(words, key=len) # by length
sorted(words, key=str.lower) # case-insensitive
sorted(people, key=attrgetter("age"), reverse=True)
sorted(rows, key=itemgetter(2, 0)) # by col 2, then col 0
# multi-field, mixed direction — tuple key with a negated number
sorted(players, key=lambda p: (-p.score, p.name))
# same result via two STABLE passes (least significant first)
players.sort(key=lambda p: p.name) # 1st: name ascending
players.sort(key=lambda p: p.score, reverse=True) # 2nd: score desc, ties keep name order
# ❌ classic bug — sort() returns None
result = my_list.sort() # result is None!
result = sorted(my_list) # ✅
# sort a dict by value
dict(sorted(counts.items(), key=itemgetter(1), reverse=True))
# top-k without a full sort — O(n log k)
import heapq
heapq.nlargest(10, items, key=attrgetter("score"))
String formatting and efficient string building in Python.
f-strings (3.6+) are the default choice — evaluated at runtime, readable, and the fastest of the formatting options. They support full format specs and, since 3.8, the debugging = suffix.
f"{name} scored {score:.2f} ({pct:.1%})"
f"{value=}" # prints: value=42 ← great for debugging
f"{amount:>10,.2f}" # width 10, right-aligned, thousands separator
Building strings in a loop is the performance point that matters. Strings are immutable, so s += x creates a whole new string each iteration — that's O(n²) over a loop. Use "".join(parts), which allocates once: O(n).
Never use f-strings for SQL or shell commands — that's how SQL injection and command injection happen. Use parameterised queries and subprocess argument lists.
Also worth knowing: str is Unicode text and bytes is binary — encode() goes text → bytes, decode() comes back. Always be explicit about encoding (UTF-8) when reading files.
# ❌ O(n²) — a new string object every iteration
out = ""
for row in rows:
out += format(row)
# ✅ O(n) — one allocation
out = "".join(format(row) for row in rows)
# format specs
f"{3.14159:.2f}" # '3.14'
f"{1234567:,}" # '1,234,567'
f"{0.876:.1%}" # '87.6%'
f"{'hi':^10}" # ' hi '
f"{dt:%Y-%m-%d}" # date formatting
# ❌ SQL injection
cur.execute(f"SELECT * FROM users WHERE email = '{email}'")
# ✅ parameterised — the driver escapes it
cur.execute("SELECT * FROM users WHERE email = %s", (email,))
# useful methods
" a,b ".strip().split(",") # ['a', 'b']
"a-b-c".rsplit("-", 1) # ['a-b', 'c']
"file.tar.gz".removesuffix(".gz") # 3.9+
If you define __eq__, why must you also define __hash__?
Python enforces a contract: objects that compare equal must have the same hash. Dicts and sets rely on it — they find the bucket by hash first, then confirm with ==. If two equal objects hash differently, they land in different buckets and your set happily contains both "equal" objects.
Because breaking this silently corrupts dicts and sets, Python protects you: defining __eq__ automatically sets __hash__ = None, making instances unhashable. You'll get TypeError: unhashable type the moment you put one in a set — a loud failure instead of a silent bug.
The rules:
- Define both together, over the same fields.
- Hash only on immutable fields. Mutating a field used in the hash while the object sits in a dict makes it unfindable — it's in the wrong bucket forever.
- Unequal objects may share a hash (a collision — legal, just slower).
@dataclass(frozen=True) generates both correctly for you, which is why frozen dataclasses are the easy right answer.
class User:
def __init__(self, user_id, name):
self.id, self.name = user_id, name
def __eq__(self, other):
return isinstance(other, User) and self.id == other.id
# WITHOUT __hash__, Python sets __hash__ = None → unhashable
def __hash__(self):
return hash(self.id) # same field as __eq__
{User(1, "Ada"), User(1, "Ada2")} # → one element (equal ids)
# ⚠️ mutating a hashed field loses the object
class Bad:
def __init__(self, k): self.k = k
def __eq__(self, o): return self.k == o.k
def __hash__(self): return hash(self.k)
b = Bad(1); s = {b}
b.k = 2 # hash changed while inside the set
b in s # False — wrong bucket, unreachable
# ✅ the easy correct answer
from dataclasses import dataclass
@dataclass(frozen=True) # generates __eq__ AND __hash__
class Point:
x: int
y: int
ABC vs Protocol — how do you define interfaces in Python?
ABC (Abstract Base Class) — nominal typing. A class must explicitly inherit from the ABC, and @abstractmethod makes instantiation fail if any abstract method is unimplemented. The relationship is declared and checked at runtime.
Protocol (3.8+, typing.Protocol) — structural typing, i.e. static duck typing. A class satisfies a Protocol if it simply has the right methods — no inheritance, no import of the interface, no coupling at all. Checked by mypy at type-check time (add @runtime_checkable for a limited isinstance check).
When to use which:
- ABC — you own the hierarchy, want shared base implementation, and want a hard runtime guarantee that subclasses implement the contract.
- Protocol — you want to type third-party or unrelated classes you don't control, or you want the loose coupling of duck typing with type-checker safety. This is the more Pythonic option for defining what a function needs from its argument.
from abc import ABC, abstractmethod
from typing import Protocol, runtime_checkable
# ABC — must inherit; instantiation fails without the implementation
class Repository(ABC):
@abstractmethod
def get(self, id: str) -> dict: ...
def get_or_raise(self, id): # shared concrete helper
if (r := self.get(id)) is None:
raise KeyError(id)
return r
class SqlRepo(Repository):
def get(self, id): return {...}
# Repository() → TypeError: Can't instantiate abstract class
# Protocol — structural: no inheritance required
class SupportsClose(Protocol):
def close(self) -> None: ...
def shutdown(resource: SupportsClose) -> None:
resource.close()
class MyConnection: # doesn't import or inherit anything
def close(self): ...
shutdown(MyConnection()) # ✅ mypy accepts — it has close()
Exception best practices — custom exceptions, raise from, and try/except/else/finally.
Catch narrowly. A bare except: also catches KeyboardInterrupt and SystemExit, so you can't Ctrl+C your program. Even except Exception should be reserved for a top-level handler that logs and re-raises. Catch the specific exceptions you can actually handle.
The full block has four parts, and else is the underused one:
try — keep it as small as possible, only the risky call.except — handle specific failures.else — runs only if no exception occurred. Keeps success-path code out of try, so you don't accidentally catch exceptions from it.finally — always runs, for cleanup (though a context manager is usually better).
raise ... from e preserves the original exception as __cause__, so the traceback shows the full chain. Re-raising without it hides the root cause — one of the most frustrating things to debug. Use raise ... from None to deliberately suppress a noisy internal cause.
Custom exceptions: define a base exception per application/library and derive from it, so callers can catch everything from your module with one except.
# app-wide exception hierarchy
class AppError(Exception): """Base for everything this app raises."""
class NotFoundError(AppError): ...
class ValidationError(AppError):
def __init__(self, field, msg):
self.field = field
super().__init__(f"{field}: {msg}") # carry structured context
# ❌ hides the root cause
try:
parse(raw)
except ValueError:
raise ValidationError("payload", "bad format")
# ✅ preserves the chain — traceback shows BOTH
try:
parse(raw)
except ValueError as e:
raise ValidationError("payload", "bad format") from e
# full block — note `else`
try:
conn = db.connect() # only the risky call
except ConnectionError as e:
log.warning("db unavailable", exc_info=True)
raise
else:
result = conn.query(sql) # runs only on success, NOT guarded by except
finally:
conn.close() # always
# ❌ never
except: # catches KeyboardInterrupt/SystemExit too
pass # and silently swallows everything
What modern Python features (3.8–3.12) should you know?
- Walrus operator
:= (3.8) — assign inside an expression. Removes the duplicate call or the pre-loop assignment in while loops and comprehension filters. - Positional-only
/ and keyword-only * (3.8) — control how arguments may be passed, which lets you rename parameters later without breaking callers. - Dict merge
| (3.9) — defaults | overrides instead of {**a, **b}. - Builtin generics (3.9) —
list[int], dict[str, int] instead of importing List, Dict. match statement (3.10) — structural pattern matching. It's not a switch: it destructures dicts, lists, and objects, and binds variables while matching. Genuinely useful for parsing event payloads and ASTs.X | None (3.10) — replaces Optional[X].ExceptionGroup and except* (3.11) — handle multiple concurrent failures, used with asyncio.TaskGroup.- Big performance wins — 3.11 is roughly 25% faster than 3.10; 3.12+ improves further. Upgrading is often the cheapest optimisation available.
# walrus — avoid computing twice / pre-assigning
if (n := len(items)) > 100:
print(f"too many: {n}")
while (chunk := f.read(8192)):
process(chunk)
[y for x in data if (y := transform(x)) is not None]
# keyword-only args after *
def connect(host, *, timeout=30, retries=3): ...
connect("db", timeout=5) # must be keyword
# dict merge
config = defaults | user_settings | env_overrides
# structural pattern matching — destructures, not just compares
match event:
case {"type": "click", "pos": (x, y)}:
handle_click(x, y)
case {"type": "key", "code": code} if code > 100:
handle_key(code)
case Point(x=0, y=0):
print("origin")
case [first, *rest]:
print(first, rest)
case _:
raise ValueError(event)
# modern typing
def find(ids: list[int]) -> dict[str, int] | None: ...
How do you manage dependencies — requirements.txt, pyproject.toml, poetry, uv?
requirements.txt — a flat list for pip install -r. The critical distinction is between your direct dependencies and a fully pinned lock of the whole transitive tree. pip freeze gives you the latter, but mixing the two in one file means you can no longer tell what you actually depend on.
pyproject.toml (PEP 621) — the modern standard: project metadata, dependencies, and tool configuration (ruff, mypy, pytest) in one file. This replaces setup.py, and it's what you should use for anything new.
Poetry / PDM / uv add what pip alone lacks: a real dependency resolver and a lock file that pins exact versions with hashes, so every machine and CI run installs byte-identical dependencies. uv (Rust-based) is the current fast option and is broadly pip-compatible.
Version specifiers: pin exactly (==) for applications — reproducibility matters more than freshness. Use ranges (>=1.2,<2.0) for libraries, so you don't force conflicts on your consumers.
Always work inside a virtual environment, and separate dev dependencies (pytest, ruff, mypy) from runtime ones so they don't ship to production.
# pyproject.toml — the modern standard
[project]
name = "my-service"
requires-python = ">=3.11"
dependencies = [
"fastapi>=0.110,<1.0",
"sqlalchemy>=2.0,<3.0",
]
[project.optional-dependencies]
dev = ["pytest>=8.0", "ruff", "mypy"] # never ships to prod
[tool.ruff]
line-length = 100
[tool.pytest.ini_options]
addopts = "-q --strict-markers"
# ---- commands ----
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]" # editable install + dev extras
# separate direct deps from the resolved lock
pip-compile pyproject.toml -o requirements.lock # pinned + hashes
pip install -r requirements.lock
# uv — same idea, much faster
uv venv && uv pip install -e ".[dev]"
uv lock && uv sync
How do you use the logging module properly? Why not print?
print() has no severity, no timestamp, no source, no routing, and can't be turned off per module. Logging gives you all of that with a config change and no code edits.
The structure: Loggers (what you call) → Handlers (where output goes: stdout, file, HTTP) → Formatters (how it looks) → Filters. Loggers form a hierarchy by dotted name, so app.db inherits from app — which is exactly why you should use logging.getLogger(__name__) in every module. You can then turn up the verbosity of one subsystem without touching the rest.
Key practices:
- Configure only in the entry point (
main), never inside a library. A library that calls basicConfig hijacks the whole application's logging. - Use lazy
%s formatting, not f-strings — with f-strings the message is built even when the level is disabled. logger.exception() inside an except block logs the full traceback automatically.- Log to stdout in containers and let the platform ship it — don't manage log files yourself.
- Structured JSON logs with a correlation/trace ID make logs queryable rather than greppable.
- Never log secrets or PII.
import logging
logger = logging.getLogger(__name__) # module-scoped, hierarchical
# ✅ lazy — not formatted if the level is disabled
logger.debug("processing user %s with %d items", user_id, len(items))
# ❌ always evaluated, even when DEBUG is off
logger.debug(f"processing user {user_id} with {expensive_call()}")
try:
charge(order)
except PaymentError:
logger.exception("payment failed for order %s", order.id) # + traceback
# extra fields for structured logging
logger.info("order created", extra={"order_id": o.id, "trace_id": ctx.trace_id})
# configure ONCE, in the entry point only
logging.config.dictConfig({
"version": 1,
"formatters": {"json": {"()": "pythonjsonlogger.jsonlogger.JsonFormatter",
"format": "%(asctime)s %(levelname)s %(name)s %(message)s"}},
"handlers": {"console": {"class": "logging.StreamHandler",
"formatter": "json", "stream": "ext://sys.stdout"}},
"root": {"level": "INFO", "handlers": ["console"]},
"loggers": {"app.db": {"level": "DEBUG"}}, # verbose for one subsystem only
})
Python is slow — how do you actually make it faster?
Measure before optimising. Intuition about Python bottlenecks is usually wrong; use cProfile for function-level cost and line_profiler for line-level, then attack only the top item.
Why it's slow: it's interpreted and dynamically typed, so every operation involves type lookups and object boxing, and the GIL prevents CPU-bound threading.
The optimisation ladder, in order of payoff:
- Fix the algorithm. An O(n²) loop is the real problem — a hash lookup instead of a list scan (
in list is O(n), in set is O(1)) beats any micro-optimisation. - Remove the N+1 — in most backend code the bottleneck is I/O, not Python. Batch queries and calls before touching the code.
- Cache —
functools.lru_cache for pure functions; Redis for shared results. - Use the C-implemented stdlib — built-ins (
sum, any, sorted), comprehensions, itertools, and str.join all run in C and beat hand-written loops. - Vectorise with NumPy/Pandas — a NumPy operation over an array is orders of magnitude faster than a Python loop, because the loop happens in C.
- Go parallel correctly —
multiprocessing for CPU-bound, asyncio/threads for I/O-bound. - Drop to native — Cython, PyO3/Rust, or a C extension for a genuinely hot inner loop.
- Upgrade Python — 3.11+ is substantially faster for free.
# profile first — never guess
python -m cProfile -s cumtime app.py | head -30
import cProfile, pstats
with cProfile.Profile() as p:
run_workload()
pstats.Stats(p).sort_stats("cumtime").print_stats(20)
# ---- the wins, in order ----
# 1. algorithmic: O(n²) → O(n)
if item in big_list: # O(n) each time
if item in big_set: # O(1) ← same code shape, different complexity
# 3. cache pure functions
from functools import lru_cache
@lru_cache(maxsize=1024)
def parse_rule(text): ...
# 4. C-level builtins beat manual loops
total = sum(x.amount for x in rows) # faster than an accumulating loop
# 5. vectorise — the loop runs in C
import numpy as np
result = np.sqrt(arr ** 2 + 1) # vs a per-element Python loop
# 6. CPU-bound → processes (GIL)
from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor() as ex:
results = list(ex.map(cpu_heavy, chunks))
# memory profiling
python -m memory_profiler script.py
Is Python pass-by-value or pass-by-reference?
Neither — Python is pass-by-object-reference (sometimes called "call by sharing"). The function receives a reference to the same object, but the parameter name itself is a new local binding.
The consequences follow directly:
- Rebinding the parameter (
x = something_new) only changes the local name. The caller sees nothing — it looks like pass-by-value. - Mutating the object (
lst.append(...), d[k] = v) changes the object the caller also holds. It looks like pass-by-reference.
So the behaviour you observe depends entirely on whether the object is mutable and whether you mutate or rebind — not on how it was passed.
Practical consequence: a function that mutates its arguments is a hidden side effect and a common source of bugs. Prefer returning a new object; if you must mutate, make it obvious in the name (sort_in_place) and document it. This is also why mutable default arguments are dangerous — the default object is created once and shared across all calls.
def rebind(lst):
lst = [99] # new local binding — caller unaffected
def mutate(lst):
lst.append(99) # mutates the SHARED object — caller sees it
items = [1, 2]
rebind(items); print(items) # [1, 2] ← unchanged
mutate(items); print(items) # [1, 2, 99] ← changed
# immutables can only ever be rebound, so they always look pass-by-value
def bump(n): n += 1
x = 5; bump(x); print(x) # 5
# ✅ prefer returning a new object over mutating an argument
def with_defaults(config: dict) -> dict:
return {"retries": 3} | config # caller's dict untouched
# defensive copy when you must not affect the caller
def process(items):
items = list(items) # local copy
items.sort()
return items
__new__ vs __init__ — what's the difference?
Object creation is two steps:
__new__(cls, ...) — the constructor. A static method that creates and returns the instance. It runs first and decides what object exists.__init__(self, ...) — the initialiser. Runs on the already-created instance to set attributes. It must return None.
You almost never write __new__. The legitimate reasons:
- Subclassing an immutable type (
int, str, tuple) — the value must be set at creation time, because by the time __init__ runs the object is already immutable. - Singletons / instance caching — return an existing instance instead of creating a new one.
- Metaclass and factory machinery.
The trap: if __new__ returns an instance of a different class, __init__ is never called. And with a singleton, __init__ runs on every instantiation even though __new__ returned the cached object — so re-initialising state there silently resets your singleton.
class Point(tuple): # subclassing an immutable
def __new__(cls, x, y):
return super().__new__(cls, (x, y)) # value fixed at creation
def __init__(self, x, y):
self.label = f"({x},{y})" # extra mutable state
# singleton — note the __init__ trap
class Config:
_instance = None
def __new__(cls, *a, **kw):
if cls._instance is None:
cls._instance = super().__new__(cls)
return cls._instance
def __init__(self, path=None):
# ⚠️ runs on EVERY Config(...) call, even when cached
if not hasattr(self, "_loaded"):
self.data = load(path)
self._loaded = True
# ✅ simpler, more Pythonic singletons
from functools import lru_cache
@lru_cache(maxsize=1)
def get_config() -> Config: return Config(...)
# or just: a module-level instance — modules ARE singletons
How does @property work? What is a descriptor?
A descriptor is any object defining __get__, __set__, or __delete__. When such an object is a class attribute, Python routes attribute access through those methods instead of returning the object. This one protocol powers @property, @classmethod, @staticmethod, __slots__, and ORM model fields.
@property is just a descriptor: it turns method calls into attribute syntax, letting you add validation or computation to obj.x without changing any calling code. That's its real value — you can start with a plain attribute and add logic later without breaking the API, which is why Python doesn't need getters and setters everywhere.
Data vs non-data descriptors determines precedence: one defining __set__ or __delete__ is a data descriptor and takes priority over the instance __dict__; a non-data descriptor (only __get__) is overridden by an instance attribute. That's why you can shadow a method by assigning to the instance, but not a property.
Write a custom descriptor when the same attribute logic (validation, unit conversion, type coercion, lazy loading) repeats across many fields or classes — otherwise a property is enough.
class Account:
def __init__(self, balance):
self._balance = balance
@property
def balance(self): # read as an attribute
return self._balance
@balance.setter
def balance(self, value): # validation, no caller changes
if value < 0:
raise ValueError("balance cannot be negative")
self._balance = value
@property
def is_overdrawn(self): # computed, no storage
return self._balance < 0
a = Account(100)
a.balance = 50 # calls the setter
a.balance = -5 # ValueError
# reusable descriptor — same logic across many fields
class Positive:
def __set_name__(self, owner, name): # 3.6+: learns its own name
self.name = f"_{name}"
def __get__(self, obj, objtype=None):
return getattr(obj, self.name) if obj else self
def __set__(self, obj, value): # __set__ → DATA descriptor
if value <= 0:
raise ValueError(f"{self.name} must be positive")
setattr(obj, self.name, value)
class Order:
quantity = Positive()
price = Positive() # validation reused, no duplication
How do you use the re module — and when should you not use regex?
Core functions: search (first match anywhere), match (anchored at the start only — a frequent surprise), fullmatch, findall/finditer, sub, and split.
Practices that matter:
- Always use raw strings (
r"\d+") so backslashes reach the regex engine intact. re.compile() once at module level if the pattern is reused — it's cached anyway, but compiling makes intent clear and is faster in hot loops.- Named groups
(?P<name>...) make patterns self-documenting and access readable. re.VERBOSE lets you write a complex pattern across multiple lines with comments — the difference between a maintainable regex and write-only code.- Quantifiers are greedy by default; add
? for lazy (.*?). Greedy matching across an entire line is the most common regex bug.
When not to use regex: parsing HTML/XML (use a real parser — nested structures aren't regular), JSON, or CSV. And beware catastrophic backtracking: nested quantifiers like (a+)+$ on non-matching input can take exponential time — a genuine denial-of-service vector (ReDoS) when the pattern touches user input.
import re
# raw strings, compiled once, named groups
LOG = re.compile(r"""
(?P<ts>\d{4}-\d{2}-\d{2}[ T][\d:]{8}) # timestamp
\s+(?P<level>DEBUG|INFO|WARN|ERROR) # level
\s+(?P<msg>.*) # message
""", re.VERBOSE)
m = LOG.search(line)
if m:
print(m["level"], m["msg"])
print(m.groupdict())
# match vs search — a common surprise
re.match(r"world", "hello world") # None! anchored at the start
re.search(r"world", "hello world") # matches
# greedy vs lazy
re.findall(r"<(.+)>", "<a><b>") # ['a><b'] greedy
re.findall(r"<(.+?)>", "<a><b>") # ['a', 'b'] lazy
# substitution with a function
re.sub(r"\b(\d{12})(\d{4})\b", lambda m: "*" * 12 + m[2], text) # mask cards
# ⚠️ ReDoS — exponential backtracking on user input
# re.match(r"(a+)+quot;, "aaaaaaaaaaaaaaaaaaaaaaaaX") # hangs
# often you don't need regex at all
"file.txt".endswith(".txt") # clearer and faster than a pattern
How do you handle dates, times, and time zones correctly?
The core distinction: naive vs aware. A naive datetime has no timezone — it's an ambiguous wall-clock reading. An aware datetime carries tzinfo and refers to an unambiguous instant. Mixing them raises TypeError on comparison, which is Python protecting you from a whole class of bugs.
The rules that prevent production incidents:
- Store and compute in UTC; convert to local time only at the display boundary.
- Use
datetime.now(timezone.utc), never datetime.utcnow() — utcnow() returns a naive datetime holding UTC values, which is the worst combination: it looks fine and compares wrongly. It's deprecated in 3.12. - Use
zoneinfo (3.9+, stdlib) for real IANA time zones — not fixed UTC offsets, which don't handle daylight saving. - Never do date arithmetic with fixed offsets across a DST boundary. "Same time tomorrow" is not always "+24 hours" — convert to the target zone and add there.
- Store dates as
DATE and timestamps as TIMESTAMPTZ in the database; never as strings.
from datetime import datetime, timezone, timedelta, date
from zoneinfo import ZoneInfo # stdlib, 3.9+
# ❌ naive — looks like UTC but has no tzinfo
datetime.utcnow() # deprecated in 3.12
# ✅ aware, unambiguous
now = datetime.now(timezone.utc)
# convert only at the display boundary
now.astimezone(ZoneInfo("Asia/Kolkata")).strftime("%d %b %Y, %I:%M %p")
# ⚠️ DST: "tomorrow at 9am" ≠ "+24 hours"
ist = ZoneInfo("America/New_York")
local = datetime(2026, 3, 7, 9, 0, tzinfo=ist)
wrong = local + timedelta(days=1) # naive arithmetic across DST
right = datetime.combine(local.date() + timedelta(days=1),
local.timetz()) # 9am local again
# comparing naive and aware raises
datetime.now() < datetime.now(timezone.utc) # TypeError
# parsing / serialising
datetime.fromisoformat("2026-08-05T09:14:22+05:30")
now.isoformat() # '2026-08-05T09:14:22.481+00:00'
int(now.timestamp()) # epoch seconds
# use date (not datetime) when there is no time component
birthday = date(1998, 4, 12)
How do you work with files and paths — and why pathlib over os.path?
pathlib.Path is the modern approach: paths are objects with methods rather than strings you manipulate. The / operator joins segments correctly on every OS, so you never hand-concatenate separators or worry about Windows vs POSIX.
Always use with when opening files — it closes the handle even if an exception is raised. Leaked file handles eventually exhaust the process limit.
Always specify the encoding. open(path) uses the platform default, which differs between machines — so a file that reads fine on your Mac raises UnicodeDecodeError on a differently-configured server. Pass encoding="utf-8" explicitly, every time.
Read large files lazily — iterating the file object yields one line at a time and uses constant memory, whereas .read() or .readlines() loads the whole thing.
Writing safely: write to a temporary file and os.replace() it into position. replace is atomic on the same filesystem, so a crash mid-write can't leave a half-written file where a valid one used to be.
from pathlib import Path
import os, tempfile, json
base = Path(__file__).resolve().parent
cfg = base / "config" / "app.json" # / joins correctly on any OS
cfg.exists(); cfg.stem; cfg.suffix; cfg.parent
cfg.parent.mkdir(parents=True, exist_ok=True)
# small files — one-liners
text = cfg.read_text(encoding="utf-8")
cfg.write_text(json.dumps(data), encoding="utf-8")
# ✅ ALWAYS specify encoding — platform default differs per machine
with open(path, encoding="utf-8") as f:
for line in f: # lazy: constant memory
process(line)
# ❌ loads the entire file
content = open(path).read()
# glob
for p in base.rglob("*.py"):
print(p.relative_to(base))
# atomic write — a crash can't corrupt the existing file
def write_atomic(path: Path, data: str):
fd, tmp = tempfile.mkstemp(dir=path.parent)
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(data)
f.flush(); os.fsync(f.fileno())
os.replace(tmp, path) # atomic on the same filesystem
How do you serialise data to JSON? What are the common problems?
json.dumps/loads handle strings and json.dump/load handle file objects. The friction comes from Python types JSON doesn't have:
datetime, Decimal, UUID, set, dataclasses all raise TypeError: not JSON serializable. Fix with the default= hook or a custom JSONEncoder.- Dict keys are always strings in JSON —
{1: "a"} round-trips back as {"1": "a"}. Silent type change. - Floats lose precision — never serialise money as a float. Use a string or integer minor units.
- Tuples become lists, so round-tripping changes the type.
For APIs, prefer Pydantic over hand-rolled encoders: it validates, coerces, and serialises with type awareness, and generates the schema. dataclasses.asdict() plus a default hook works for simple internal cases.
Security: JSON is safe to parse from untrusted sources; pickle is not — unpickling attacker-controlled data executes arbitrary code. Never use pickle for anything crossing a trust boundary, and never for long-term storage (it's version-fragile).
import json, dataclasses
from datetime import datetime, date
from decimal import Decimal
from uuid import UUID
# ❌ TypeError: Object of type datetime is not JSON serializable
json.dumps({"at": datetime.now()})
# ✅ default= hook
def encode(o):
if isinstance(o, (datetime, date)): return o.isoformat()
if isinstance(o, Decimal): return str(o) # str, not float!
if isinstance(o, UUID): return str(o)
if isinstance(o, set): return sorted(o)
if dataclasses.is_dataclass(o): return dataclasses.asdict(o)
raise TypeError(f"{type(o).__name__} is not serializable")
json.dumps(payload, default=encode)
# gotchas
json.loads(json.dumps({1: "a"})) # {'1': 'a'} ← key became a string
json.loads(json.dumps((1, 2))) # [1, 2] ← tuple became a list
json.dumps({"amt": 0.1 + 0.2}) # 0.30000000000000004
# ⚠️ NEVER unpickle untrusted data — arbitrary code execution
# pickle.loads(request.body) # remote code execution
# for APIs, let Pydantic handle it
from pydantic import BaseModel
class Order(BaseModel):
id: UUID
created_at: datetime
amount: Decimal
Order(**raw).model_dump_json() # validated + serialised
pytest in depth — fixtures, parametrize, and mocking.
Fixtures replace setUp/tearDown with dependency injection: declare a fixture, then any test that names it as a parameter receives it. Code after yield is the teardown and runs even if the test fails.
- Scopes —
function (default), class, module, session. Use a wider scope for expensive setup (a test container, a DB engine), but be careful: shared mutable state between tests causes order-dependent failures. conftest.py — fixtures defined there are available to every test in that directory tree, with no import.
@pytest.mark.parametrize runs one test body across many inputs, and each case is reported separately — so you see exactly which input failed rather than a single test that stops at the first bad case. It's the cheapest way to cover edge cases.
Mocking: patch where the object is used, not where it's defined — patch("myapp.service.requests"), not patch("requests"). This is the number one mocking mistake. Use autospec=True so the mock rejects calls that don't match the real signature, otherwise a mock happily accepts a call you later rename and the test keeps passing.
monkeypatch is pytest's built-in for env vars and attributes, with automatic undo.
import pytest
from unittest.mock import patch, MagicMock
# conftest.py — available everywhere, no import
@pytest.fixture(scope="session")
def db_engine():
engine = create_engine(TEST_URL)
yield engine # teardown runs even if a test fails
engine.dispose()
@pytest.fixture
def session(db_engine): # fixtures compose
conn = db_engine.connect(); txn = conn.begin()
yield Session(bind=conn)
txn.rollback(); conn.close() # each test isolated
# each case reported separately — you see WHICH input failed
@pytest.mark.parametrize("raw,expected", [
("10", 10),
(" 7 ", 7),
("-3", -3),
pytest.param("abc", None, marks=pytest.mark.xfail(raises=ValueError)),
])
def test_parse(raw, expected):
assert parse_int(raw) == expected
# ✅ patch where it's USED, with autospec
@patch("myapp.service.payment_client", autospec=True)
def test_charge(mock_client):
mock_client.charge.return_value = {"status": "ok"}
assert charge_order(order) is True
mock_client.charge.assert_called_once_with(order.id, order.amount)
def test_env(monkeypatch):
monkeypatch.setenv("API_KEY", "test-key") # auto-undone after the test
with pytest.raises(ValidationError, match="email"):
validate(payload)
What is the N+1 query problem and how do you fix it in Django/SQLAlchemy?
The problem: you fetch N records with one query, then access a related object on each one. Because ORMs load relationships lazily, each access fires another query — 1 + N total. It's invisible in code (it looks like plain attribute access) and only shows up as latency once the data grows.
100 orders each showing a customer name = 101 queries. At 5ms each that's half a second of pure database round trips for something that should be one query.
The fix is eager loading, and there are two flavours with different trade-offs:
- JOIN-based (
select_related in Django, joinedload in SQLAlchemy) — one query with a JOIN. Best for to-one relationships (ForeignKey, ManyToOne). - Second-query based (
prefetch_related / selectinload) — a separate IN (...) query, joined in Python. Best for to-many relationships, because a JOIN across a to-many multiplies rows (fetching 100 orders × 50 items each returns 5,000 rows of duplicated order data).
How to catch it: assert on query counts in tests, log SQL in development, and use django-debug-toolbar or SQLAlchemy's echo. An APM trace showing 101 identical queries is the classic signature.
# ❌ N+1 — 1 query for orders, then 1 per order for the customer
orders = Order.objects.all() # 1 query
for o in orders:
print(o.customer.name) # +1 query EACH → 101 total
# ✅ Django: JOIN for to-one, separate IN query for to-many
Order.objects.select_related("customer") # FK / to-one → JOIN
Order.objects.prefetch_related("items") # to-many → IN query
Order.objects.select_related("customer").prefetch_related("items__product")
# only fetch the columns you need
Order.objects.only("id", "total").select_related("customer")
# aggregate in the DB, not in Python
from django.db.models import Count, Sum
Customer.objects.annotate(order_count=Count("orders"),
revenue=Sum("orders__total"))
# ✅ SQLAlchemy 2.0
from sqlalchemy.orm import joinedload, selectinload
stmt = (select(Order)
.options(joinedload(Order.customer), # to-one → JOIN
selectinload(Order.items)) # to-many → IN query
.where(Order.status == "PAID"))
# catch regressions in tests
with self.assertNumQueries(2):
list(Order.objects.select_related("customer"))
What are the main security pitfalls in Python code?
pickle on untrusted data is remote code execution. Unpickling runs arbitrary code by design. Never unpickle anything from a user, a queue, a cache, or a session cookie — use JSON.- SQL injection — never build queries with f-strings or
%. Use parameterised queries; the driver escapes values. ORMs protect you unless you drop into raw()/text() with interpolation. eval/exec on user input — arbitrary code execution. For data, use ast.literal_eval or json.loads.- Command injection —
subprocess with shell=True and interpolated input lets a user chain commands. Pass an argument list and leave shell=False. random is not cryptographically secure — it's a deterministic Mersenne Twister. Use secrets for tokens, password resets, and session IDs.- Password storage — never plain SHA-256 (too fast to brute-force). Use bcrypt/argon2 via
passlib, and compare secrets with hmac.compare_digest to avoid timing attacks. - Path traversal — a user-supplied filename containing
../ escapes your directory. Resolve the path and verify it's still inside the intended root. - YAML —
yaml.load can instantiate arbitrary objects; always use yaml.safe_load. - Dependencies — pin versions, scan with
pip-audit, and watch for typosquatted package names.
# ❌ remote code execution
pickle.loads(request.body)
yaml.load(user_file) # use yaml.safe_load
eval(user_input) # use ast.literal_eval
# ❌ SQL injection → ✅ parameterised
cur.execute(f"SELECT * FROM users WHERE id = {uid}")
cur.execute("SELECT * FROM users WHERE id = %s", (uid,))
# ❌ command injection → ✅ argument list
subprocess.run(f"ping {host}", shell=True)
subprocess.run(["ping", "-c", "1", host], shell=False, timeout=5)
# ❌ predictable → ✅ cryptographically secure
import random, secrets
token = str(random.randint(0, 10**9)) # Mersenne Twister — predictable
token = secrets.token_urlsafe(32) # ✅
# passwords: slow hash + constant-time compare
from passlib.hash import argon2
import hmac
hashed = argon2.hash(password)
argon2.verify(password, hashed)
hmac.compare_digest(provided_key, expected_key) # timing-safe
# path traversal
root = Path("/srv/uploads").resolve()
target = (root / user_filename).resolve()
if not target.is_relative_to(root): # 3.9+
raise ValueError("invalid path")
# scan dependencies
# pip-audit | bandit -r ./app
