Human cognition usually goes abstract to concrete. You start a writing session with a vague premise, work through structure, then commit to sentences.
Instruction-tuned AI is optimized to do the opposite: it reads the prompt you typed, picks the most plausible reading, and produces a finished artifact. Alignment has a Fantasia Problem (Jo, De Simone, Gordon & Wilson, MIT, April 2026) calls the result of that shortcut a Fantasia interaction — a failure of human–AI coordination in which the system commits to a concrete interpretation of a prompt that was only ever an early, evolving signal of the user's intent [1].
What the paper actually says
The authors define a Fantasia interaction precisely: a failure of human–AI coordination where the system commits to a concrete interpretation from a prompt that may only be an early, evolving signal of the user's intent. It is not a hallucination. The model is doing exactly what its training rewards — answering helpfully, on the prompt as written. The break is one layer up: the model has no model of the user's cognitive process, and so it short-circuits the brainstorming, the rough draft, the half-formed argument, in favor of the finished essay.
Three failure modes follow.
Premature execution is acting on the first plausible reading before intent has formed. False satisfaction is the interaction that feels successful — both sides got something — even though the output does not serve the user's actual goal.
Anchoring is the long-tail damage: once the AI has produced a specific output, the user often bends their downstream thinking toward it, so a single early commitment silently narrows the rest of the session.
The paper also tests two state-of-the-art baselines designed to mitigate exactly this — CollabLLM (Wu et al., 2025) and STaR-GATE (Andukuri et al., 2024) — and finds that CollabLLM still commits before gathering the missing information, while STaR-GATE overcorrects toward endless elicitation [2][3]. The off-the-shelf fixes are not fixes.
"Modern AI is optimized to respond helpfully to the prompt as written. Like the enchanted broom, the model faithfully executes the instruction it is given, even when doing so produces behavior that is misaligned with the user's ongoing intellectual or creative process." — Jo et al., 2026 [1]
What we shipped about it
We turned the paper's prescription into a routing skill — fantasia-aware-agent — that runs before any other action an agent might take. The skill computes an ambiguity score for the incoming prompt on five signals (missing subject, undefined constraints, open-ended scope, vague goal, underspecified scale) with two offsets (concrete examples, defined parameters), normalized into [0, 1]. Then it routes deterministically:
- score > 0.5 → Elicit: ask targeted questions for the missing parameters. Priority over Expand.
- 0.3 < score ≤ 0.5 AND ≥ 2 coherent interpretations → Expand: surface the alternatives and let the user choose.
- Exploratory language ("I guess", "maybe", "something like") OR detected intent evolution across turns → Support Intent Formation: help the user articulate the goal rather than answering the literal text.
- score ≤ 0.3 AND critical parameters defined → Generate: default mode. Proceed normally.
The interesting design choice is what the skill does not do: it does not call a model to "reason about the user's intent." It scores, routes, and emits one of four small structured prompts — an expand template, an elicit template, a support template, or a generate template. The model can refuse to follow them; the runtime cannot. This matters because the paper's diagnosis of why existing systems fail is that they treat clarification as another behavior to optimize, not as a policy. A skill is a small policy.
The interesting design choice is what the skill does not do: it does not call a model to "reason about the user's intent."
It scores, routes, and emits one of four small structured prompts — an expand template, an elicit template, a support template, or a generate template. The model can refuse to follow them; the runtime cannot.
This matters because the paper's diagnosis of why existing systems fail is that they treat clarification as another behavior to optimize, not as a policy. A skill is a small policy.
Where it earns its keep
The hardest cases for Fantasia routing are not the obviously vague ones; those the user already fleshes out. The hard ones are requests where a single reading is plausible enough that the agent wants to act on it — "clean up the auth flow," "tighten the design," "just push it." Each of those gets pulled into Expand or Elicit, the user picks the actual reading, and the agent stops pretending it is the entire cognitive loop. What we would build next is a coupling to long-term memory so specificity carries forward across sessions instead of being recomputed each turn. The paper does not address how specificity decays with session length, which is the open question.
The hardest cases for Fantasia routing are not the obviously vague ones; those the user already fleshes out.
The hard ones are requests where a single reading is plausible enough that the agent wants to act on it — "clean up the auth flow," "tighten the design," "just push it."
Each of those gets pulled into Expand or Elicit, the user picks the actual reading, and the agent stops pretending it is the entire cognitive loop.
What we would build next is a coupling to long-term memory so specificity carries forward across sessions instead of being recomputed each turn. The paper does not address how specificity decays with session length, which is the open question.
What's still unclear
The biggest gap the paper admits and we cannot fix from the agent side is evaluation. The diagnostic study had three vignettes across education, writing assistance, and advice seeking — enough to show the failure mode but not enough to measure how often it occurs in production traffic. Our skill ships a paper-to-anecdote insight: agents stop committing prematurely in the cases we test against, with regression fixtures in references/test-cases.md, but we do not yet have a benchmark that says it is better than CollabLLM at the same task. The right next move for the field is a shared dataset of Fantasia-shaped conversations with annotated intent states — the paper gestures at this and stops short of releasing one.
The framing is also a hint about scope. Fantasia is specifically the abstract-to-concrete case; it is not the "user gave a fully specified request and the agent flubbed it" case, and the four-mode router is not a hallucination detector. We pair it with the Safety Intention Checker (two ambiguous-action skills, not one) and we expect people to read the boundary or get a noisier system than they wanted.
The takeaway is not "ask more questions." It is that the alignment target has been wrong: the field has been rewarding next-turn helpfulness on a prompt that is often the first paragraph of a longer cognitive process the model cannot see. Fantasia-aware routing is one small policy that says — for this turn at least — do not finish what the user has not yet decided to start.
References
- Jo, N., De Simone, Z., Gordon, M., Wilson, A. (2026). Alignment has a Fantasia Problem. Presented at the ICLR 2026 Workshop on Human-Centered AI for Alignment Research (HCAIR); arXiv:2604.21827. https://arxiv.org/abs/2604.21827
- Wu, S., Galley, M., Peng, B., Cheng, H., Li, G., Dou, Y., Cai, W., Zou, J., Leskovec, J., Gao, J. (2025). CollabLLM: From Passive Responders to Active Collaborators. ICML 2025 Outstanding Paper; arXiv:2502.00640. https://arxiv.org/abs/2502.00640
- Andukuri, C., Fränken, J.-P., Gerstenberg, T., Goodman, N. D. (2024). STaR-GATE: Teaching Language Models to Ask Clarifying Questions. arXiv:2403.19154. https://arxiv.org/abs/2403.19154