Processes and Memory from Scratch — What Is the OS Actually Protecting?

One app can crash without taking the machine down with it. Process isolation and virtual memory are what make that ordinary. Starting from a post-office-box analogy, this builds up paging, address translation, TLBs, page faults, swap and the OOM killer from zero — and ends with you able to read free and dmesg yourself.

The app dies, the machine doesn't

A browser tab can hang while the half-written draft in your editor beside it stays perfectly safe. A video export can go haywire and eat every byte of memory, and usually only that program gets killed while you land back on your desktop.

None of this is automatic. Every program runs on the same single CPU and the same single pool of memory, so with no partitions in place, a value written by one program would land on top of another program's variable and take everything down together. Old home computers really did work that way.

The partitions are built by the OS (operating system). Boiled down, the OS protects three things: isolation (nobody touches anybody else), fairness (CPU and memory get shared out), and a lie (finite physical memory is presented to each program as a broad, private space). This article follows the memory half of that story in one line, from the analogy all the way to the commands you'd actually type.

A process is the world handed to a running program

One piece of vocabulary first. An executable file sitting on disk is still a program. The moment the OS loads it and starts running it, it becomes a process. Think of the difference between a recipe and a pot actually on the stove. Just as one recipe can produce three pots, one executable can produce any number of processes.

Each process gets a number (its PID) and its own ledger inside the OS. That ledger holds open files, permissions, and — our subject here — the address space: a private map of addresses, roughly divided into these regions.

• Text: the machine instructions themselves. Not writable, but executable
• Data: variables that exist from the moment the program starts
• Heap: the region grown at runtime by malloc and friends. Climbs upward from low addresses
• Stack: pushed on with every function call. Grows downward from high addresses

Between heap and stack there is usually an enormous gap. That emptiness is exactly what lets both ends stretch and shrink freely.

Reading a file, asking for more memory, drawing on the screen — all of these touch hardware directly, so a process asks the OS through a window called a system call. At that point the CPU switches from low-privilege user mode to high-privilege kernel mode. This two-storey structure is why a runaway program doesn't drag the OS down with it.

The analogy: every address was a P.O. box number

Now the heart of it. When you hear that process A and process B both use "address 0x400000," it sounds like a collision waiting to happen. It isn't. The addresses a program sees are not addresses in physical memory.

Picture a post office box. Your card says "P.O. Box 12," and that's what people write on the envelope. Which shelf in the back the letter actually lands on has nothing to do with that number. Only the post office holds the mapping, and it can rearrange the shelves without you reprinting a single card.

The address a program uses (a virtual address) is the box number, the address on the actual memory chips (a physical address) is the shelf in the back, and the post office holding the table is the OS together with the CPU. So A's box 12 and B's box 12 get assigned different shelves. Isolation isn't a guard checking every request — it's that neither process even holds a number that reaches the other's shelf.

The mechanism: paging and address translation

A mapping kept per individual address would be larger than the memory it describes. So memory is cut into fixed-size chunks called pages, and the mapping works chunk by chunk. 4 KB is the common size (some environments use 16 KB, including macOS on Apple Silicon). The equally sized slots on the physical side are called frames.

A virtual address $A$ splits into a page number and a position inside that page.

$P$ is the page size (4096, say), $\mathrm{VPN}$ is "which page this is," and $\mathrm{offset}$ is "how many bytes in from the start of that page." In plain terms: the address is split by a quotient and a remainder into a page number and a seat number within the page. Or, put in words: divide the address by the page size, and the answer tells you which page while the leftover tells you how far into it you are — nothing more is being asked.

Making $P$ a power of two is what makes this cheap. In binary, dividing by 4096 and taking the remainder is just slicing off the bottom 12 bits — no division required, so the CPU can do the split in wiring alone.

The mapping table (the page table) hands back a physical frame number $\mathrm{PFN}$, and the final physical address is:

Put in words: multiply the shelf number by the size of one shelf to land at the front of that shelf, then count in by the seat number. Meaning: move a page to a different shelf and the seating order inside the page is unchanged. The offset carries straight through, so translation rewrites only the high bits.

Each row of the page table also carries small flags beside the frame number: present (is this page in memory right now), R/W (may it be written), NX (may it be executed), user (may user mode touch it), dirty (has it been modified), accessed (was it used recently). A program stops the instant it writes to a page meant to be read-only because the CPU checks these flags on every access.

Why page tables are multi-level

Naively, the mapping is one flat table with a row per page. Its size follows from the number of address bits $b$, the offset bits $p$, and the bytes per row $e$.

All this says is: rows = address space divided by page size, and table size = rows × row size. Spelled out, $b$ is how many bits an address has, $p$ is how many of those bits pick a byte inside the page, and $e$ is the bytes one row costs — which says that every extra bit of address space doubles the table. But x86-64 virtual addresses are 48 bits, pages are 4 KB ($p=12$), and a row is 8 bytes. Substituting gives $2^{36} \times 8 = 512\ \mathrm{GB}$ — a 512 GB table per process. Completely broken.

The real fix is the multi-level page table. The VPN is cut into four 9-bit slices; the first table points at the second, the second at the third, and so on (x86-64 uses four levels, with a five-level variant for larger spaces). What does a tree buy you? You never build the branches you don't use. That vast gap between heap and stack collapses into a single upper-level row saying "nothing over there." A process actually touching a few megabytes ends up with a page table of a few tens of kilobytes.

The price is speed. If every memory access requires walking four tables (a page walk), you have naively turned one memory access into five.

The TLB: keeping translations at hand

So the CPU keeps recent translations in a small dedicated cache: the TLB (Translation Lookaside Buffer). In post-office terms, the clerk jots down the box-to-shelf mapping for regulars on a note by the counter.

Effective access time looks like this:

$$T = T_{\text{TLB}} + (1-h)\,T_{\text{walk}}$$

$h$ is the fraction of lookups that hit in the TLB, and $T_{\text{walk}}$ is the cost of walking the tables when they miss. Put in words: on a hit you glance at the note; only on a miss do you walk to the back shelves. That is, the average cost of a translation is the note-check you pay on every single lookup, plus the long walk charged only on the fraction of lookups that miss. Even at $h = 0.99$, a $T_{\text{walk}}$ that is orders of magnitude heavier still shows up.

A TLB holds at most a few hundred to a few thousand entries. With 4 KB per entry, the range it can cover at once is on the order of a few megabytes. A program that touches a wider range at random stalls on TLB misses even when the data itself sits in cache. That is why huge pages (2 MB or 1 GB per page) help databases and in-memory workloads: give one entry 512× the territory and the same TLB covers 512× the region. This is the same terrain as Cache-Friendly Code, where identical O(n) loops run at wildly different speeds.

Page faults: pretend it isn't there, provide it later

Touch a page whose present flag is clear and the CPU suspends the instruction and calls the OS. That's a page fault. Despite the alarming name, most of them are business as usual.

• Minor fault: hand over one physical page, zero it, done. No disk I/O
• Major fault: the contents must be read from disk (the executable, or swap). Orders of magnitude slower
• Invalid access: an address that isn't on the map at all. The process gets SIGSEGV — the familiar segmentation fault

Because of this, the OS can adopt a strategy of handing over no physical memory until something is touched (demand paging). Reserve 1 GB and physical memory only drops by the pages you actually wrote. So the sum of all reservations can exceed physical memory and nothing happens at the time. This is overcommit. Keep it in mind; it comes back later.

The same trick powers copy-on-write. fork, which duplicates a process, does not copy memory. Parent and child share the same physical pages, all marked read-only. The instant either one writes, a page fault fires and only that page gets duplicated. That is why forking a multi-gigabyte process finishes in a blink. mmap on a file works the same way: the contents sit in physical memory as the page cache, shared across processes. Why storage can only be read and written a page at a time looks three-dimensional from the angle of B-Trees and LSM-Trees.

Swap and thrashing

When physical memory runs short, the OS evicts pages that haven't been used for a while to disk and hands the freed slots to new requests. That's swap. An evicted page has its present flag cleared and comes back as a major fault the next time it is touched.

Deciding "hasn't been used for a while" uses the accessed bit from earlier. Recording a true last-use timestamp would tax every access, so Linux keeps pages on two lists (active and inactive) and shuttles them between the two while periodically clearing reference bits — an approximation of LRU.

The trouble starts when the set of pages needed at once (the working set) exceeds physical memory. An evicted page is needed immediately, bringing it back evicts another, and that one is needed immediately too. CPU utilization is low and nothing progresses: thrashing. It feels less like "slow" and more like "unresponsive," to the point where even SSH may not get through.

Linux exposes a dial, vm.swappiness (60 by default on most distributions): higher values swap more eagerly, lower ones shrink the page cache instead to avoid swapping. macOS puts compressed memory in front of swap, packing pages down in RAM rather than pushing them to disk.

The OOM killer: cleaning up after a broken promise

Overcommit rests on the assumption that not everyone withdraws their full balance at once, and one day that assumption breaks. Swap is exhausted, there is nothing left to evict, and a page fault still demands a physical page. At that dead end, the OOM killer (out-of-memory killer) runs.

It scores every living process (oom_score) and kills the highest one with SIGKILL. The score is essentially "more physical memory in use, higher score." Here is the operational trap: the process that dies is the fattest one, not the culprit. The classic incident is a batch job creeping upward while the production database gets taken out alongside it. You can adjust priority with /proc/<pid>/oom_score_adj (-1000 to 1000; -1000 exempts a process entirely).

The kill is recorded in dmesg. In containers, exceeding a per-cgroup limit (memory.max under cgroup v2) triggers OOM inside that cgroup; on Kubernetes the pod state becomes OOMKilled and the exit code is 137 (128 plus SIGKILL's 9). "It keeps dying with 137 and I don't know why" is almost always this.

Seeing it in code

Overcommit and page faults become visible in about ten lines.

#include <sys/mman.h>
#include <stdio.h>

int main(void) {
    size_t n = 1UL << 30;                       /* 1GB */
    char *p = mmap(NULL, n, PROT_READ | PROT_WRITE,
                   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    getchar();                                  /* check ps here: VSZ +1GB, RSS ~0 */
    for (size_t i = 0; i < n; i += 4096) p[i] = 1;   /* one byte per page */
    getchar();                                  /* check ps here: RSS +1GB */
    return 0;
}

Right after the reservation you have only drawn a plot on the map; physical memory hasn't moved by a single byte. The moment the loop touches each page, roughly 260,000 page faults fire and the pages fill in. You can count them:

/usr/bin/time -v ./a.out        # look at the Minor / Major page faults lines
ps -o pid,vsz,rss,comm -p $!    # VSZ = space promised, RSS = space actually resident
perf stat -e page-faults,minor-faults,major-faults ./a.out

How this shows up on the job

Who touches it, and when. SREs and backend engineers doing capacity planning, ML platform people chasing why a training job died, and anyone investigating a "only breaks in production" report. The culprit is usually not a memory shortage as such — it's counting memory the wrong way.

Where to look, and which knobs exist.

• In free -m, read available, not free. A small free is normal; the rest is being used as page cache and can be handed back the instant it's needed
• RSS double-counts shared pages. Add up the RSS of ten processes using the same library and you overstate reality. For a properly apportioned figure, use PSS from /proc/<pid>/smaps (tools like smem aggregate it for you)
• A 100 GB VSZ is not automatically wrong. Pages that were merely reserved never show up in RSS
• Knobs that actually do something: vm.swappiness, vm.overcommit_memory (0 = heuristic / 1 = always allow / 2 = strict), cgroup v2's memory.max (hard limit) and memory.high (throttle via reclaim above it), Kubernetes requests and limits
• First evidence after a crash: dmesg -T | grep -i oom, or exit code 137 for containers

Pitfalls that turn into incidents.

1. malloc succeeded and the process died anyway. Thanks to overcommit, the failure arrives not at allocation time but at the first write — and as SIGKILL. Checking the return value for NULL is not enough
2. Inside a container, free and nproc report the host's numbers. A runtime that ignores cgroup limits will size its heap or worker count for the host's capacity and OOM immediately (modern JVMs read cgroups via UseContainerSupport)
3. "Disabling swap makes things stable" is only half true. Instead of dying slowly to thrashing, you now die instantly to the OOM killer. Which is easier to operate depends on whether your monitoring can catch it
4. Transparent huge pages (THP) cut both ways. When a large contiguous region isn't available, compaction at allocation time can stall for hundreds of milliseconds, which is why latency-sensitive databases conventionally turn THP off
5. The OOM killer does not kill the culprit. Protect critical processes with oom_score_adj, and cap jobs that might run away inside a cgroup so they die within their own box

The form these take in interviews and incident reviews. "Why is a huge VSZ not a problem?" "Why does the process die if malloc never fails?" "Memory is at 98% — is that bad?" Every answer follows the same path: count the promise (virtual), the substance (physical), and the reclaimable surplus (cache) separately. What that physical memory costs to reach is picked up in The Memory Wall from Scratch.

Summary

• The addresses a program sees are P.O. box numbers; the OS and CPU hold the mapping to the actual shelves
• The mapping works per page. An address splits into "high bits = page number, low bits = position in page," and the table is a tree so only used branches get built
• The "hand nothing over until it's touched" strategy — demand paging and copy-on-write — is what makes fast fork and huge reservations possible
• Overcommit is the price, and when the road runs out the OOM killer takes down the fattest process
• In practice, reading available, PSS, and cgroup limits is the entrance to about 80% of memory incidents