Advanced Packaging — How CoWoS and HBM Stacking Became the Bottleneck for AI

What limited the supply of AI accelerators was not leading-edge logic wafers but the step that puts several finished chips into one part. From the reticle limit and the exponential math of yield, through 2.5D interposers, HBM stacks and TSVs, to hybrid bonding and the bill that heat and warpage send back.

The Bottleneck Is No Longer the Transistor

The shortage of AI accelerators has been with us for a while. The intuitive explanation is that the leading-edge 3nm and 5nm lines are full. But the bottleneck reported again and again sat somewhere else: after the transistors are patterned, in the step that takes several finished chips and assembles them into a single part. Packaging capacity, and the supply of the HBM that goes into it, were setting the shipment numbers.

For a long time a package was a box that protected a chip and brought its pins out. Performance was decided inside the die; the outside was an afterthought. That inverted over the past few years. Today, how you build the wiring outside the die decides bandwidth, power and price. Semiconductor coverage gravitates toward node numbers, but what determines the accelerator in front of you is often what surrounds the die.

Why did it come to this? It starts with a piece of physics: you cannot make one die as large as you want. How a single die gets made is covered in How Chips Are Made. This article picks up where that one ends — splitting, placing, stacking, connecting.

An Analogy: Separate Buildings, or a Skybridge

Picture two departments that exchange a mountain of paperwork every day. There are three ways to seat them.

• Same floor: hand it to the desk next door. Fast and cheap, but a floor only holds so many people
• Different buildings: room for everyone, but paperwork now crosses a public street on a cart. Slow, and it costs staff
• Next door, joined by a skybridge: never touches the street; a few dozen steps

That third option is 2.5D integration. Dies are connected inside the package by wires that are extremely short and extremely numerous. The skybridge is a slab called an interposer, and TSMC's version of this approach is CoWoS (Chip on Wafer on Substrate).

There is a level above that. Stacking the departments on top of each other is 3D integration. Punching through the floor is shorter still than a skybridge — but the upper floor takes the heat of the one below it. Nearly every constraint in this article maps back to some part of this picture.

There Are Two Ceilings

Naively, the fastest thing to build is one giant die with everything on it. Wires stay short, and there are no seams in between. Designers would love to. What stops them is not a performance argument but manufacturing physics, and it comes in two ceilings.

The first is the reticle limit. A scanner can only print so much area in a single exposure — roughly 26 mm × 33 mm on current tools, about 858 mm². A circuit larger than that cannot exist as one printed pattern. No amount of wanting moves that line.

The second is yield. Defects land on a wafer at random, so the larger a die is, the lower its chance of stepping on none of them. And the fall-off is not proportional — it is exponential. Double the area and the yield is not halved, it is squared. Where the reticle limit decides what cannot be built, this decides what can be built but never pays for itself. In practice the second ceiling usually bites first.

Because of these two, designers commit up front to splitting the design and sewing it back together later. And the performance of what they sewed is decided almost entirely by how they sewed it — by the package. That is why the wiring conversation now comes before the arithmetic-unit conversation.

The Yield Arithmetic: Why Splitting Pays

If defects fall randomly across the wafer, the probability that a die of area $A$ steps on none of them is

$Y$ is yield (the fraction of good dies), $A$ is the area of one die, and $D_0$ is defects per unit area. Read it as: make the die bigger and the good fraction decays exponentially.

Put in words, it says: the wider you spread a die, the faster its chance of dodging every defect thins out. Doubling the area squares the yield rather than halving it precisely because area sits in an exponent here, not in a multiplication.

At 0.1 defects/cm² and an area of 8 cm², $Y = e^{-0.8} \approx 0.45$. Split the same circuit into four 2 cm² dies and each one yields $e^{-0.2} \approx 0.82$ — and the chance all four are good is $0.82^4 \approx 0.45$, exactly what the monolithic die gave.

So why split? Because a good die no longer shares the fate of a bad one. You pair up good dies harvested from the wafer, and you throw away only the defective 2 cm². Counted as products per wafer, this example yields roughly 1.8× more.

The Intuition: Short Wires Win Three Ways at Once

• Energy: the power to switch a wire scales roughly with its capacitance, and capacitance scales roughly with length. Length is the electricity bill. Long wires also need termination circuits at both ends to survive reflections, and shortening the wire deletes those circuits too
• Count: an interposer is patterned with the same lithography as a die, so it carries wires far finer than a board can — orders of magnitude finer
• No need to go faster: bandwidth is "number of wires × rate per wire". If you can afford wires, you do not have to push the rate. Wide and slow is the governing principle of high-bandwidth memory

2.5D: A Chip Made of Nothing but Wiring

Reading a CoWoS cross-section from the bottom: organic substrate → interposer → several dies. The interposer is a slab of silicon holding wires and vias and almost no transistors. Logic dies and HBM stacks sit on top of it, attached with microbumps — tiny solder balls on a pitch of tens of microns — while TSVs (through-silicon vias) punch through the slab to carry power and external I/O down to the substrate.

The interposer is itself patterned by lithography, so it too faces the reticle limit. Huge interposers exist because the exposure is split into several shots and stitched together; when a layer is only wiring, seam accuracy is easier to hold.

A full silicon slab is expensive, so there are cheaper routes. RDL approaches drop the silicon and build fine redistribution layers on resin — cheaper, but coarser. Local bridge approaches embed a small piece of silicon only under the seam between neighbouring dies; Intel's EMIB works this way. UCIe is becoming the standard for die-to-die links.

HBM: Why Memory Goes Vertical

HBM (High Bandwidth Memory) stacks 4 to 16 DRAM dies, wires them vertically with TSVs, and funnels everything out through a base die at the bottom. The stack sits beside the logic die on the same interposer.

The reason to stack is not volume, it is exits. Getting signals off a die consumes a finite resource: the edge, or the face where you can put bumps. Spread the DRAM out flat and it sits farther from the logic with longer wires; stack it and eight dies' worth of capacity occupies one die's footprint.

The bandwidth arithmetic is almost disappointingly simple.

$B$ is bandwidth per stack in bytes per second, $W$ is the number of signal wires (the interface width in bits), and $f$ is the transfer rate per wire in bits per second. The 8 just converts bits to bytes.

It is an equation which says: what you can move per second is nothing more than "how many lanes × how fast each lane runs". Widening the road and raising the speed limit buy you the same extra throughput — and that everyday relationship is the whole equation.

HBM used $W = 1024$ for several generations, and HBM4 doubled that to 2048. Double the width and bandwidth doubles without making any single wire faster. Set against the roughly 64-bit width of ordinary PC memory, those numbers are absurd — and unroutable on a board. Note also that spec bandwidth is not what you get: the effective number depends on your access pattern, which is the subject of The Memory Wall.

3D: Drop the Solder Ball, Bond Copper Directly

A microbump is a ball, so it needs enough pitch to avoid touching its neighbour, and tens of microns was the practical floor. Connection density goes as the inverse square of pitch, so shrinking pitch pays off fast.

Hybrid bonding uses no balls at all. The copper pads and dielectric on both die faces are polished to a mirror finish, pressed together directly, and annealed so the copper joins. Pitch drops below 10 µm, and to around 1 µm in research. Going from 40 µm to 4 µm is a hundredfold density gain. TSMC's SoIC and Intel's Foveros Direct are of this family. The cost is blunt: any stray particle or bit of unevenness becomes a defect — it is a contest of cleanliness and flatness.

The Trade: What You Give Up

1. Compound yield. Package yield is the product of every component's good-die rate and the success rate of the assembly itself.

$y_i$ is the good rate of each die and $y_{\text{asm}}$ is the success rate of assembly. Read it as: one weak component drags the whole package down with it.

Spelled out in words: the odds of a good package are every component's odds multiplied together, then multiplied once more by the odds that assembly went well. Because these compound rather than average, seven parts at 90% each land the package under 50%. That is why every die is screened on its own before it goes in — a KGD (known good die). What really hurts is when you fail. A die scrapped at wafer test costs you one die; a package that fails after assembly takes the logic, the HBM stacks and the interposer with it. The later the failure, the more each one costs.

2. Heat. The more you stack, the more the lower floor's heat has to escape through the upper one. DRAM does not tolerate heat well: above a certain temperature the standard requires more frequent refresh, and you pay for it in bandwidth and power. That is much of why HBM sits beside the logic rather than on top of it.

3. Warpage. Silicon and organic substrate expand at rates that differ by several times. Every heating and cooling cycle has the materials pulling against each other, and the package bows. The larger the area, the larger the displacement at the edges, and microbumps crack. The bigger the package, the more the mechanical problem comes for the electrical one.

Estimating It in Code

The gain from splitting, and the erosion from adding parts, fit side by side in a few lines.

import math

def die_yield(a, d0=0.1):
    return math.exp(-a * d0)               # good rate of a die of area a [cm²]

def products_per_cm2(total, n, d0=0.1):
    a = total / n                          # area of one die
    return (1 / a) * die_yield(a, d0) / n  # good dies ÷ dies needed per product

def package_yield(ys, asm=0.98):
    p = asm
    for y in ys: p *= y
    return p

print(products_per_cm2(8, 1), products_per_cm2(8, 4))
print(package_yield([0.82] + [0.95] * 6))  # one logic die + six HBM stacks

Real design arguments are usually a tug-of-war between exactly these two numbers.

How This Shows Up on the Job

Who touches it, and when. SoC physical designers decide "where to cut the die" during floorplanning. That line propagates into RTL partitioning, test design and power delivery, so it cannot be moved later. Architects work backwards from required bandwidth to the number of HBM stacks, which in turn sets the interposer's area. On the procurement side, packaging allocation and HBM generation (HBM3E / HBM4) have to be secured separately from wafer capacity.

The vocabulary you will actually use. Bump pitch (µm), TSV aspect ratio, how many reticles the interposer spans, UCIe profile (standard / advanced), beachfront bandwidth density — GB/s per mm of die edge — $T_j$ (junction temperature) and thermal resistance, and warpage in µm. Design-for-test in 3D stacks has dedicated standards such as IEEE 1838.

Ways this becomes an incident.

• Splitting too finely: yield keeps improving, but there is always a point where die-to-die wire energy and latency eat the gain
• Weak KGD coverage: a defect your die-level test missed shows up after assembly and takes every good part around it with it
• Thermal analysis on one die: budget from the logic TDP alone and the HBM next to it gives out first
• Treating spec bandwidth as effective bandwidth: the TB/s on the datasheet is just width × rate

Summary

• The reticle limit (~858 mm²) and a yield that decays exponentially with area made "one giant die" impossible
• Sewing split dies back together is 2.5D/3D integration, and the interposer is silicon used purely as wiring
• HBM stacks vertically with TSVs and buys bandwidth with width; 1024 or 2048 wires are only routable because the interposer exists
• The price is compound yield, heat and warpage — which is how KGD testing ended up at the centre of the design

What you do with the compute once it is packaged is the subject of How AI Accelerators Are Designed.