Paper walkthrough: Hi-Q — splitting a question down to the granularity your corpus can actually retrieve

The real bottleneck in multi-hop QA is a mismatch between the granularity of the question and the granularity of retrievable evidence. Hi-Q answers first, then expands only the nodes that failed — in dependency order. Here is the control rule, the tree, and the measured numbers, from zero background.

Hi-Q: Hierarchical Evidence-guided Query Refinement for Multi-Hop Question Answering

The unit you ask in is not the unit you can retrieve

Ask a librarian: "When was the start of the battle of the birthplace of the performer of III?" The librarian is stuck. That one sentence folds three separate lookups together — who performed III, where were they born, when did the battle there begin — and the books on the shelves each cover only one of them.

This is exactly where retrieval-augmented generation (RAG) stalls on multi-hop questions. The unit at which a question is expressed differs from the unit at which evidence can be retrieved. Send the coarse query as written and the retriever surfaces passages that match surface terms like "III" or "battle" without covering the evidence chain. Split it too finely and you drop the contextual constraints that made it the right question.

The paper this article walks through is "Hi-Q: Hierarchical Evidence-guided Query Refinement for Multi-Hop Question Answering" (arXiv:2608.30468, by Jueun Kim, Sungho Park and Wook-Shin Han of POSTECH, published 31 August 2026).

Its abstract, restated: the central bottleneck in multi-hop QA is that the granularity at which a question is expressed often differs from the granularity at which corpus evidence is retrievable. Existing methods attack this mismatch by imposing fixed graph structures over the corpus, by iteratively reformulating the query, or by executing a generated program over it — but none of them explicitly decides when a query unit is already supported by evidence and when it should be refined. The authors formulate the bottleneck as retrievable granularity discovery and introduce Hi-Q, an evidence-conditioned framework for hierarchical query refinement. At each query node a resolution operator tests whether retrieved evidence supports the current query unit; resolved nodes terminate, unresolved nodes are expanded by a dependency-preserving binary operator and checked by a semantic coverage verifier. The resulting query tree therefore takes its shape from corpus support signals rather than from a fixed decomposition template or a pre-built graph.

Three ways granularity fails

Section 1 sorts the failures into three kinds.

1. Single-shot retrieval. A coarse query entangles several reasoning constraints, and the top-ranked passages match isolated surface terms while missing the chain.
2. Graph RAG. Structure is added on the corpus side — but it is built before any query arrives. The granularity decision is query-agnostic by construction, and it carries a corpus-wide pre-computation cost (§2).
3. Iterative retrieval. Methods like IRCoT do adapt the query, but never test whether the query they just issued was retrievable. Pick the wrong bridge entity early and later steps amplify the error (§2).

What is missing, the paper argues, is a control mechanism that decides which query unit is currently retrievable given this corpus.

Try answering first — that is the whole control rule

Hi-Q's core is almost disappointingly plain: before splitting a query, try answering it as it stands.

Each node carries a state $x=(q,\mathcal{H},d)$, where $q$ is the current query, $\mathcal{H}$ the accumulated interaction history, and $d$ the recursion depth. A resolution operator $\mathcal{G}(q,\mathcal{H},C)\rightarrow(s,a,D)$ rewrites the query from the history, retrieves the top-$k$ passages $D$ from corpus $C$, and has a retrieval-grounded reader answer from those passages alone. Here $s$ is the resolution status and $a$ the answer.

The policy reads off that result (§3.1):

$$\pi(\tilde{x},s)=\begin{cases}\textsc{stop}&s=\text{resolved}\\ \textsc{fail}&d=d_{\max}\ \text{or}\ q\ \text{is non-decomposable}\\ \textsc{expand}&\text{otherwise}\end{cases}$$

Read in words, that case split says: if an answer came out, stop. If the depth budget is gone or the query cannot be split, give up. Otherwise, split. The state $\tilde{x}=(x,D)$ is the post-retrieval state — it includes which passages actually came back. That is the hinge of the whole method: the decision is conditioned not on the question text but on the evidence in hand.

Rewriting before retrieval is deliberate too. A dependent sub-query contains references whose meaning is fixed only once its prerequisite is resolved. Once "the performer of III" has resolved to Stanton Moore, a downstream query about "the birthplace of the performer of III" becomes a query about the birthplace of Stanton Moore, which cuts retrieval interference (§3.2).

Stop or expand is a cost threshold

Why is "try answering first" a principled test rather than a hack? The paper writes the decision as a comparison of two error costs (§3.2, derived in Appendix B).

Let $\Delta_{\mathrm{R}}$ be the penalty for expanding a node that was in fact resolvable, $\Delta_{\mathrm{U}}$ the penalty for stopping at one that was in fact unresolved, and $Z\in\{\mathrm{R},\mathrm{U}\}$ whether the node is resolvable. Minimising the conditional expected cost turns the choice into a threshold:

Equation (1) is a threshold rule, which says in plain terms: expand once the probability that this node is still unresolved rises above the cost ratio between the two mistakes. Both $\Delta$ terms are measured over the whole subtree the action induces — descendant retrieval and LLM calls, drift risk, synthesis, terminal answer loss — so the decision is node-wise but not myopic. The paper is careful not to claim the resulting tree is globally optimal (§3.2).

Hi-Q estimates that probability with no training at all. If the reader produces an answer, the node counts as resolved; if it returns $a=\bot$, the node counts as unresolved. That is the entire estimator. The paper explicitly states this hard classifier is not claimed to compute the posterior, and that a calibrated classifier, an entailment model or a trained cost-sensitive router could replace it without changing Hi-Q's control semantics (Appendix C).