map / systems / virtual-memory / interactive

Systems · interactive deep dive

Virtual memory & the OOM killer

The principleA process’s address space is a promise, not an allocation: physical pages are handed out only when memory is touched. When a container’s touched memory outgrows its limit, the kernel first reclaims page cache — and when there’s nothing left to reclaim, it kills a process.
01

Fill the container until something gives.

Top: the process’s virtual address space. Bottom: physical memory charged to its cgroup, with the limit (memory.max in cgroup v2; a container memory limit in Kubernetes). Raise the file reads to watch reclaim, then simulate a leak and watch the OOM kill.

One process in a container · cgroup memory.max = 2048 MiB · no swap

Virtual address space (what the process thinks it has) — bright = resident in RAM

■ code · ■ heap / anonymous · ■ file-backed (page cache) · ■ stack · hollow = mapped but not resident

Physical memory charged to the cgroup

memory.maxanon 900 MiBcache 1148 MiB
Reclaiming. The app wants 1500 MiB of file cache but only 1148 MiB fits. The kernel evicts clean page cache (352 MiB worth) instead of failing — reads get slower, nothing dies yet.
VSZ (heap)
3072 MiB
anon RSS
900 MiB
page cache
1148 MiB
state
reclaiming

Simplified: ignores kernel memory, shared memory/tmpfs (not freely reclaimable), dirty pages awaiting writeback, and swap. With no swap, anonymous memory can only be freed by the process itself — or by killing it.

02

Mapped is not used: VSZ vs RSS.

VSZ (virtual size) counts every address range the process has mapped. RSS (resident set size) counts the pages actually in RAM. Allocators and runtimes routinely reserve far more address space than they touch, and the kernel only backs a page with physical memory on first access (a page fault). That’s why a process can show gigabytes of VSZ and modest RSS, and why VSZ is rarely the number to alert on.

Linux also overcommits by default: malloc usually succeeds even if there isn’t enough RAM to back every page. The bill comes due later, when the pages are touched — which is why memory failures on Linux usually look like a sudden OOM kill rather than a polite allocation error.

03

Page cache: memory that’s used, but cheap to give back.

Reading a file leaves its contents in the page cache, so the next read is served from RAM. Page cache counts toward a cgroup’s memory usage, but clean cache pages are reclaimable: under pressure the kernel evicts them rather than failing. Anonymous memory (heap, stacks) has no file to fall back on; without swap, it can only be freed by the process — or by killing it.

usage ≈ anon + page cache + kernel + shmem  ·  OOM when it can’t be reclaimed under the limit

That’s the trap behind “the container is using 95% of its memory” alerts: much of that may be cache that would be released instantly. The memory.stat file breaks usage into anon, file, and more, and memory.events counts oom_kill events.

04

How it shows up in interviews.

Troubleshooting

“A pod keeps getting OOMKilled.” Interviewers want the evidence trail: exit code 137, kernel log, memory.stat, memory.events.

Linux fundamentals

“What’s the difference between VSZ and RSS? Why is free memory near zero?” Page cache is the expected answer to the second.

Capacity

“How would you set memory limits?” Size from observed working set and peaks, not from VSZ or averages.

05

A sample question, and a strong answer dissected.

The question“A Java service in Kubernetes keeps getting OOMKilled. The JVM heap is capped at 60% of the container limit, and heap metrics look fine. What’s going on, and how do you find out?”
Confirm the kill

“First confirm it’s the kernel’s cgroup OOM killer: the pod status says OOMKilled, the exit code is 137 (128 + SIGKILL), and the node’s kernel log has a ‘Memory cgroup out of memory: Killed process …’ line.”

Establishing which mechanism killed it avoids chasing the wrong problem.

Account for everything

“The cgroup charges the whole process, not just the heap: thread stacks, JIT code, metaspace, direct and native buffers, plus memory in other processes in the container. I’d compare RSS and the cgroup’s memory.stat breakdown against the heap size — the gap is the suspect.”

Knowing that heap ≠ process memory is the core insight.

Rule out cache

“If usage is mostly file, that’s page cache, which the kernel reclaims before killing anything — so it’s unlikely to be the cause on its own. Growth in anon is what kills. tmpfs or shared memory are a special case: they count too and aren’t freely reclaimable.”

Separating reclaimable cache from anon shows depth in Linux memory.

Find the growth

“Then graph anon or RSS over time against traffic. Steady growth regardless of load suggests a leak — native memory tracking or a profiler narrows it down. Growth with load suggests per-request buffers or too many threads.”

Moving from snapshot to trend is how you find leaks.

Fix and guard

“Fix the leak or right-size the limit from observed peak working set with headroom, and alert on working set approaching the limit rather than waiting for the kill.”

Ending with both a fix and a guardrail is the senior shape.

06

Common pitfalls.

Heap = memoryThe cgroup charges the whole process: stacks, native buffers, runtime overhead.
Alerting on “used” memoryPage cache inflates “used”; it’s reclaimable and usually harmless.
Reading VSZLarge virtual size is normal; resident memory is what gets charged.
Assuming malloc failsWith overcommit, allocations succeed and the failure arrives later as a kill.

Back to the principle

Virtual memory — the principle

Read →