Upstream Engineering from Scratch — Why Projects Are Won or Lost at Requirements

Requirements, then high-level design, then detailed design — why that order? The reason is a branching factor that multiplies at every stage, making the cost of a late fix grow exponentially. Explained from zero with an analogy, a formula, an interactive plot, and code.

An analogy: the house with a foundation off by one inch

You are building a house. Moving a column on the drawing costs an eraser and ten minutes. Moving it after the concrete foundation is poured means breaking up the slab and pouring again. Moving it after the interior is finished and the furniture is in means starting by asking the residents to move out.

The mistake is exactly the same size in all three cases. Only the cost of correcting it explodes. When people say upstream work matters, this is the structure they are pointing at — not a work ethic. Upstream means the end nearer the source of the river: the part where you decide what to build. Downstream is where you actually assemble it.

What upstream work actually decides

Enterprise projects usually split the upstream into three layers. The names vary by shop, but the content decided in each is remarkably stable.

1. Requirements decide what outcome we want. What you write here is business, not technology: "the month-end reconciliation that takes accounting three days finishes the same day." It is stated as a result visible to the user, and it covers both functional requirements (what the system can do) and non-functional ones (how fast, how available, how protected).

2. High-level design (external design) decides the shape the user sees: screen flows, report fields, the message formats exchanged with other systems, the entities held in the database and how they relate. The bar for this layer is being concrete enough for the customer to agree to it.

3. Detailed design (internal design) decides the shape the builder sees: module boundaries, processing order, error branches, the indexes needed to hit the performance target. Invisible to users, but required so that implementers do not have to guess.

These three stack in this order because each layer assumes the one above it. Table design assumes the business rules; module decomposition assumes the table design. Move an assumption and everything resting on it wobbles.

Why later is exponentially more expensive: branches multiply

Let us treat "fixing it later is expensive" as a counting problem rather than a motivational slogan.

One requirement, pushed into high-level design, branches into several screens and several tables. One table branches into several modules in detailed design; one module becomes several files and functions in implementation; one function carries several test cases. Once the system is in production, it branches further and then sets: accumulated data, written operating procedures, habits in the users' fingers, integrations other teams depend on.

So each stage downward multiplies the number of places a single change has to touch. Because those multipliers compound stage by stage, the cost grows as a power, not a sum.

$c_0$ is the cost of fixing the mistake on the spot, during requirements. $k$ is the branching factor — how many more places you must touch for each stage you descend. $d$ is the distance between the stage where the mistake was made and the stage where it was found. In one sentence, the formula says: every stage you are late in noticing multiplies the bill by $k$.

If $k$ were 3, a misunderstood requirement caught in production (five stages down) costs $3^5 = 243$ times more. Do not put faith in the number 243. $k$ depends heavily on how coarse your stages are, how tightly coupled the system is, and whether you have automated tests — and the famous cost curve Boehm published in 1981 has itself been criticized over the evidence behind its specific multipliers. What deserves your trust is the shape, not the constant: this compounds, it does not accumulate.

One more step toward reality. What actually bites is not the cost when a defect is found, but the expected cost, weighted by how likely it is to survive that long.

$$E = \sum_i p_i \cdot c_0 k^{\,d_i}$$

$p_i$ is the probability that defect $i$ survives undetected that far, and $d_i$ is its detection distance. Reviews, prototypes and walkthroughs are an investment in shrinking $d_i$ — not in reaching zero defects. Assume mistakes will happen; move the place where they surface earlier. That is the whole purpose of upstream work in one line.

The same idea in ten lines of code

PHASES = ["requirements", "high-level design", "detailed design",
          "implementation", "testing", "production"]

def rework_cost(made_at, found_at, c0=1.0, k=3.0):
    """Relative cost of fixing a defect introduced at made_at, found at found_at."""
    return c0 * k ** (found_at - made_at)

for i, phase in enumerate(PHASES):
    print(f"requirement defect found in {phase:>18}: {rework_cost(0, i):7.1f}")

The point is not the printed numbers but the subtraction found_at - made_at sitting in the exponent. There are only two levers. Lower $k$ — keep modules loosely coupled so the blast radius stays small — or lower $d$ — find it sooner. Nearly everything we call upstream work is the second lever.

The real enemy is ambiguity, not error

The dangerous thing in a requirements document is not a wrong statement. Wrong statements get argued with in review. The dangerous thing is a statement every stakeholder can read differently. Nobody objects, everyone signs, and the interpretations only split apart once someone starts coding.

"Flexible search." "Handles heavy traffic." "Notifies as needed." Those are wishes, not requirements. The key to converting them is verifiability: could two different people independently check whether it is done and reach the same answer?

• ✗ Must handle heavy traffic
• ✓ 200 concurrent sessions, mean response under 1 s, ≥99% successful responses over any 5-minute peak window

Write the measurement method, not just the number. "Mean response of 1 s" can differ by more than a factor of two depending on whether you measure what the user experiences or what the server spends inside the process.

Ambiguity survives longest in non-functional requirements. A missing feature generates complaints immediately; performance, availability, operability, migratability and security cannot simply be added later. A system that must answer in 1 s rather than 10 s needs different data structures from the start. Checklists such as IPA's non-functional requirement grades — availability, performance and scalability, operations and maintainability, migratability, security, and system environment — are widely used for one reason: left alone, everyone forgets this territory. How it plays out once you are actually running the thing is covered in Observability — Logs, Metrics, and Traces in Practice.

But spending too long upstream also fails

None of this means "decide everything first." Teams that try to close every open item stall waiting on the ones they cannot close — analysis paralysis — while the market and the business keep moving during the wait.

The practical line is drawn by reversibility, not completeness.

• Decide irreversible things early: the data model, contracts with external systems, the authentication and authorization model, the scope of personal data handled. Anything whose later change implies a migration.
• Defer reversible things: layout, wording, internal algorithms, library choices. Anything you can swap without rippling outward.

Agile methods did not abolish upstream work; they make this split explicitly and then collapse $d$ for the reversible half through short iterations. Even in a two-week cadence, getting the data model wrong is expensive. The $k$ and $d$ in the formula above apply regardless of methodology.

How this is used in practice

Who and when: engineers, tech leads and product managers on business systems use it during the requirements phase right after kickoff, during design reviews, and — most importantly — the moment a change request arrives. That third case is where the knowledge earns its keep: the question to answer is which layer's assumptions does this change move? Rewording a screen label and changing what the primary key means can look like the same amount of work while having wildly different $k^d$.

What you actually touch:

• Requirement IDs and traceability (an RTM, requirements traceability matrix): requirement ID → design item → test case ID in one table. In Jira or similar, issue types plus link types ("relates to" / "blocks") serve the same purpose.
• Acceptance criteria attached to each requirement in Given/When/Then form. It is the minimum armor against starting implementation while "done" is still undefined.
• A non-functional checklist (IPA's grades, or your own equivalent). Simply filling in the categories starts the conversation that otherwise never happens.
• ADRs (architecture decision records): one decision per file, including why. Six months later, when an assumption changes, you can tell whether a decision is safe to overturn.

Pitfalls that turn into incidents:

1. Leaving "etc.", "and so on", "flexibly" in the document. These words guarantee a scope argument later. Grepping for them before a review is cheap and effective.
2. Postponing non-functional requirements. Performance and availability shape the skeleton. Being told "actually it runs 24/7 with no maintenance window" after the build means rebuilding.
3. Not recording verbal agreements. Minutes should carry decided, deferred, and who decided. Skip the deferred list and the team forgets that anything was deferred at all.
4. Not naming an approver. With three stakeholder departments, requirements split three ways. Decide upfront who — one person or one committee — makes it final.
5. Not linking test cases to requirements. Without the link you cannot detect the case where every test passes and one requirement was never built.
6. Absorbing scope growth silently. Each item is small; without a record, nobody can explain whose decisions produced the slipped date.

A question you will get in interviews and design reviews: "What do you do with a vague requirement?" A defensible answer has four steps — (1) trace back to whose work is actually hurting, (2) rewrite it as a completion test with a number and a measurement method, (3) check whether it touches an irreversible decision, and (4) if it does, settle it now; if not, defer it. Stopping at "I'd ask the customer" reads as being unable to generate the questions yourself.

For systems containing machine learning, two more upstream questions join the list: can we even collect this data, and who notices when accuracy decays? That ordering is laid out in ML System Design — The 90% Outside the Model, and the measurement side of deciding whether a change helped is in Statistical Testing and A/B Tests.

Summary

• The upstream runs requirements → high-level design → detailed design; each layer assumes the one above, so moving the top shakes everything below
• Late fixes are expensive because the number of places to touch multiplies at every stage — trust the compounding shape, not any specific multiplier
• Only two levers exist: lower $k$ by loosening coupling, or lower $d$ by finding problems sooner
• The enemy in requirements is ambiguity, not error; convert wishes into a number plus a measurement method
• Do not decide everything early — decide the irreversible things early

Next in this series: how to write the requirements themselves — eliminating ambiguity, and turning requirements into tests.