Decision Workbench
Expand the decision-maker's cognition BEFORE they decide.
Decision power = size of your option space, not the ability to pick.
This tool diverges options, reveals blind spots, structures the map — and
hands the decision back to a human. It never picks a winner.
"AI shouldn't make important decisions for you.
It should make the decision space harder for you to misunderstand."
"I only know Java Spring." — said every engineer who never saw the
open-source CRM bases, the SaaS options, or the no-code builders.
What it does
Paste a decision you're about to make — including what you already know.
Decision Workbench runs a 6-stage pipeline:
┌─────────────────────────────────────────────────────────────────────────┐
│ 1. BIAS FLAG "I only know Java Spring" -> anchored_stack │
│ 2. DIVERGENCE 5 stances argue independently (parallel): │
│ architect · cost · security │
│ product · red team │
│ 3. CROSS-EXAM 3 probing questions force the space open: │
│ "Why did you NOT consider X?" │
│ "Give one non-mainstream option w/ reason" │
│ "Weakest point of your preferred option?" │
│ 4. OPTION MAP consensus / true divergence / false divergence, │
│ each tagged "you probably didn't know this" vs │
│ "you already knew this" │
│ 5. CONSTRAINTS you check budget/team/deadline/compliance -> │
│ weighted re-score (still no winner) │
│ 6. HUMAN DECISION the decision is yours. The system never picks. │
└─────────────────────────────────────────────────────────────────────────┘
Why this exists (and why it is NOT llm-council)
|
llm-council (Karpathy) |
Decision Workbench |
| Goal |
Converge multiple models to a better single answer |
Expand the option space before a human decides |
| End state |
One synthesized answer |
A map + trade-offs + a human decision |
| Divergence |
Means to an end (better consensus) |
The product itself |
| Who decides |
The chairman model |
The human (explicitly, always) |
| Failure mode |
Artificial consensus (all models converge) |
Anchoring bias (decision-maker locked in) |
The academic grounding: multi-agent LLM committees suffer from
representational collapse when they share a model (arXiv 2604.03809) and
artificial consensus when they share an architecture (arXiv 2604.26561).
This tool is deliberately built around the opposite: heterogeneous stances +
a hard human gate. It is a diverger, not a converger.
Quick start (mock mode — zero API keys)
# Backend (port 8002)
uv sync
uv run python -m backend.main
# Frontend (port 5173, proxies /api → 8002)
cd frontend
npm install
npm run dev
Open http://localhost:5173, type a decision, click "Expand my option
space". The mock provider ships pre-baked answers for 3 demo scenarios
(tech stack / business / hiring) so the demo never depends on a live API.
Real models (OpenRouter)
export OPENROUTER_API_KEY=sk-or-v1-...
export PROVIDER=openrouter
uv run python -m backend.main
Demo scenarios (pre-baked in mock mode)
| Scenario |
Input |
Blind spot revealed |
| A. Tech stack |
"I need to build a CRM. I only know Java Spring." |
open-source CRM bases, buy-vs-build, no-code, spreadsheet+automation |
| B. Business |
"Sell report subscriptions. Only thought of Stripe." |
merchant-of-record (Lemon Squeezy/Paddle), membership platforms, crypto |
| C. Hiring |
"I need to hire a full-stack engineer." |
remote-first, AI-augmented small team, agency outsourcing |
API
| Endpoint |
Purpose |
POST /api/decide |
Run the full pipeline → bias flags, stance divergence, probing, option-space map |
POST /api/decide/constraints |
Re-score against selected constraints (weighted, still no winner) |
POST /api/decide/human-decision |
Record the human's final choice |
Tests
uv run pytest
# 25 tests: bias detection, stance prompts, probing, zone classification,
# novelty tagging, constraint weighting, mock provider, engine golden path
Design principles
The value is not "5 agents talking" — multi-agent debate is not new. The
value is the mechanism layer around the models, each constraining a natural
model failure mode:
- Bias flag BEFORE discussion. The input is not assumed neutral. We
first detect whether the question itself has already framed the answer
("I only know Java Spring" → anchored_stack). You cannot expand a space
you don't know is locked.
- Independent divergence. Architect / Cost / Security / Product /
Red Team reason independently (parallel, isolated) before anything is
shared — reducing mutual anchoring. This beats "everyone joins one chat"
because the first-mover's frame doesn't pollute the others.
- Forced cross-examination. Probing questions like "Why did you NOT
consider X?" and "Weakest point of your preferred option?" are
institutionalized disconfirmation — the system forces counter-evidence
search instead of relying on a prompt saying "be comprehensive."
- Option map, not winner. The objective is decision-space coverage, not
answer selection. We optimize for the human seeing a complete-enough
space before deciding.
- Constraints LAST. Divergence happens first, reality filters after.
If you tell the models "our team is all Java" up front, every agent
orbits Java. Divergence must be constraint-free to be honest.
- Human decides. AI provides evidence, trade-offs, and surfaces
unknown-unknowns; the responsibility boundary stays with the human. This
is especially right for enterprise architecture decisions.
- The model does not decide. Ever. The output is a map, not a verdict.
The honest limitation: 5 stances ≠ 5 independent minds
Five role prompts on ONE model tend to collapse into one opinion wearing
five hats (representational collapse — arXiv 2604.03809). The mock provider
shows real divergence because its five stances are hand-authored. With a
single real model, divergence narrows. The fix (roadmap): each stance runs
on a different model family (heterogeneous providers), so the divergence is
cognitively real, not just prompt-deep. The eval harness below measures
whether this actually works.
Eval: does it expand the decision space?
The product makes a falsifiable claim:
Decision Workbench surfaces more decision-relevant options — especially
options the baseline would miss — than a single model or a naive
multi-agent chat.
Metrics:
| Metric |
Question |
| Option discovery |
How many distinct decision-relevant options surface? |
| Blind-spot coverage |
How many options did the baseline MISS that ours found? |
| Counterargument coverage |
How many stances have a real counterargument to the anchor? |
| Novel-option rate |
Fraction of surfaced options tagged "you probably didn't know this" |
Current results (mock mode, hidden ground truth)
Hidden ground truth = hand-authored gold option sets per scenario, never
revealed to the systems under test. Run it yourself: uv run python -m backend.eval_harness
| Method |
Gold hit (of 12) |
Avg coverage |
| Single model direct answer |
3/12 |
26% |
| Naive multi-agent (stances only) |
9/12 |
78% |
| Decision Workbench |
12/12 |
100% |
Blind spots recovered by Decision Workbench that both baselines missed:
- Scenario A:
spreadsheet-plus-automation, crm-as-a-feature
- Scenario B:
membership-platform
The spread between single-model (26%) and the full pipeline (100%) is the
measurable value of the mechanism layer: bias-flag → independent divergence →
forced cross-examination → option map. Full per-scenario breakdown:
docs/EVAL_REPORT.md
The next milestone is the same experiment on heterogeneous real models
(one provider per stance) to confirm the divergence survives
representational collapse (arXiv 2604.03809) with real models, not just
hand-authored mocks.
If that holds, the project is no longer "an impressive multi-agent workflow"
— it is a testable position:
AI shouldn't make important decisions for you. It should make the decision
space harder for you to misunderstand.
License
MIT