QueryUnderstandLab
When does LLM-based query understanding actually help search?
QueryUnderstandLab is a compact research project for studying how local LLMs
understand, normalize, and rewrite search queries — and whether those changes
improve downstream retrieval.
The project rule is simple:
Improve the query only if the retrieval system benefits.
At a Glance
- Research question: when should a search system preserve the original query,
and when should it rewrite or enrich it?
- Primary local LLM:
qwen3:8b through Ollama
- Query-understanding tasks: intent classification, entity/concept
extraction, robustness, and canonical rewriting
- Lexical retrieval: BM25
- Dense retrieval:
sentence-transformers/all-MiniLM-L6-v2
- Real-data evaluation: MS MARCO v1.1 candidate ranking
- Main finding: unconditional LLM rewriting hurt retrieval on the small
MS MARCO evaluation
- Key pattern: dense retrieval was more tolerant of rewriting than BM25,
but raw queries still performed better overall
Detailed experiment notes and limitations live in RESULTS.md.
Why This Project Exists
Search systems often add increasingly sophisticated query-understanding layers:
- intent prediction
- entity extraction
- spelling and normalization
- query expansion
- semantic rewriting
- retrieval routing
It is easy to assume that a cleaner, more explicit, or more semantic query must
be better for search.
That assumption is not always true.
A rewrite can preserve human meaning while changing the lexical or semantic
signals a retriever relies on. QueryUnderstandLab therefore separates two
questions:
Did the model understand the query?
from:
Did that understanding improve retrieval?
The second question is the primary evaluation target.
Research Questions
The project focuses on four small questions:
- Can a local LLM reliably produce structured query-understanding output?
- Is that understanding stable under typos, abbreviations, dropped words, and
light paraphrases?
- Does canonical LLM rewriting improve lexical retrieval?
- Does rewriting behave differently for lexical and dense retrieval?
The project deliberately avoids turning these questions into a large benchmark
or production search stack.
Experimental Flow
raw query
|
+--> structured understanding
| - intent
| - entities
| - concepts
| - ambiguity
|
+--> controlled perturbations
| - typo
| - abbreviation
| - word drop
| - light paraphrase
|
+--> canonical rewrite
|
+--> BM25
|
+--> dense retrieval
The same principle is used throughout: keep the experiment small enough that
individual failures can be inspected manually.
Key Findings
1. Valid JSON Is Not Enough
The first local-model outputs were syntactically valid JSON but did not reliably
follow the intended schema.
Examples included:
- free-form intent descriptions instead of controlled labels
- dictionary-shaped entities instead of typed entity lists
- missing fields
Strict parsing and schema validation made these failures visible rather than
silently accepting them.
For qwen3:8b, disabling thinking mode was also important for short,
structured extraction.
2. Intent Taxonomy Definitions Matter
A small 10-query gold audit initially produced:
| Setting |
Intent Accuracy |
Schema Success |
| Initial prompt |
0.60 |
10 / 10 |
| Clarified taxonomy |
1.00 |
10 / 10 |
The second number should not be interpreted as benchmark-level performance.
The audit is tiny, and the prompt refinement was informed by its errors.
The useful result is that intent boundaries such as
content_discovery vs. product_discovery need explicit definitions.
3. Robustness Is Different From Correctness
The project generates deterministic query variants using:
- adjacent-character typos
- abbreviations
- word deletion
- light lexical paraphrases
Clear queries often remained intent-stable, while abbreviations and wording
changes sometimes changed the predicted intent or concepts.
This exposed an important distinction:
A model can be consistent and still be wrong.
Robustness metrics therefore complement, rather than replace, a small gold
correctness audit.
4. Toy Retrieval Made Rewriting Look Promising
On a tiny hand-built BM25 corpus:
| Method |
MRR |
| Raw query |
0.83 |
| Canonical rewrite |
1.00 |
| Entity/concept expansion |
0.83 |
For example:
best coffee restaurants brooklyn
became:
best coffee shops in brooklyn
The rewrite removed a misleading lexical term and moved the relevant document
from rank 2 to rank 1.
That result looked promising — until the same idea was evaluated on real search
queries.
5. Rewriting Hurt BM25 on MS MARCO
On 30 MS MARCO v1.1 candidate-ranking examples:
| BM25 Condition |
MRR |
| Raw query |
0.565 |
| LLM rewrite |
0.463 |
Rewrite outcomes:
The rewritten queries were often semantically reasonable, but lexical
normalization could remove terms that BM25 was successfully matching.
Examples:
what causes genetic alterations in normal cells
-> causes of genetic mutations in normal cells
how long can you keep chicken salad in your fridge
-> shelf life of chicken salad in refrigerator
A semantically cleaner query is not necessarily a better lexical retrieval
query.
6. Dense Retrieval Was More Tolerant — But Rewriting Still Did Not Help
Using sentence-transformers/all-MiniLM-L6-v2 on the same 30-query setup:
| Dense Condition |
MRR |
| Raw query |
0.601 |
| LLM rewrite |
0.567 |
Rewrite outcomes:
Dense retrieval was less sensitive to semantic normalization than BM25, but the
raw query still performed better overall.
Main Takeaway
The experiments do not support unconditional query rewriting.
A better mental model is:
query
|
v
query analysis
|
+-- already clear --------------------> retrieve directly
|
+-- noisy / ambiguous / poorly formed -> selectively rewrite or expand
The next production-oriented question would be how to predict when rewriting is
worth the risk.
That question is intentionally left outside this version of the project.
Repository Structure
QueryUnderstandLab/
├── examples/
│ ├── data/
│ ├── gold_intent_experiment.py
│ ├── msmarco_candidate_experiment.py
│ ├── msmarco_dense_experiment.py
│ ├── query_understanding_smoke.py
│ ├── retrieval_experiment.py
│ └── robustness_experiment.py
├── src/queryunderstandlab/
│ ├── benchmark.py
│ ├── dense.py
│ ├── gold_eval.py
│ ├── io.py
│ ├── ollama.py
│ ├── perturbations.py
│ ├── retrieval.py
│ ├── rewrite.py
│ ├── robustness.py
│ ├── schema.py
│ └── understanding.py
├── tests/
├── RESULTS.md
├── pyproject.toml
└── requirements-bench.txt
Setup
Python 3.12 is used for the project.
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
Run the test suite:
For MS MARCO and dense-retrieval experiments:
python -m pip install -r requirements-bench.txt
Local LLM Setup
The experiments use Ollama and expect a local model.
The primary model used in the reported experiments is:
Confirm that Ollama is available:
Then run the structured-understanding smoke test:
python examples/query_understanding_smoke.py
Reproducing the Experiments
Query Robustness
python examples/robustness_experiment.py
Gold Intent Audit
python examples/gold_intent_experiment.py
Toy Retrieval
python examples/retrieval_experiment.py
MS MARCO + BM25
python examples/msmarco_candidate_experiment.py \
--limit 30 \
--output results/msmarco_v11_candidate_eval_30.jsonl
MS MARCO + Dense Retrieval
python examples/msmarco_dense_experiment.py \
--limit 30 \
--output results/msmarco_v11_dense_eval_30.jsonl
MS MARCO data and the sentence-transformer weights are downloaded on first use.
What This Project Taught
- JSON validity is not schema correctness.
- Schema correctness is not semantic correctness.
- Intent taxonomies need explicit boundary definitions.
- Robustness and correctness measure different things.
- Query rewriting changes the retrieval distribution.
- A semantically cleaner query can be worse for lexical retrieval.
- Dense retrieval is more tolerant of rewriting, but not automatically helped
by it.
- Query understanding should be evaluated through downstream search impact,
not only through model-output quality.
- Rewriting is better viewed as a selective intervention than universal query
preprocessing.
Scope and Limitations
This is a learning-oriented research project, not a production benchmark.
- The gold intent audit contains only 10 queries.
- MS MARCO experiments use 30 candidate-ranking examples.
- Evaluation is candidate ranking rather than full-corpus retrieval.
- The primary LLM is
qwen3:8b.
- The dense experiment uses one embedding model.
- No statistical significance testing is reported.
- The prompt was refined using errors from the small gold audit.
These constraints are intentional.
Stop Point
The project stops after demonstrating:
- structured local-LLM query understanding
- strict schema validation
- robustness testing
- a small correctness audit
- canonical rewriting
- BM25 retrieval evaluation
- dense retrieval evaluation
- failure analysis
Not included in this version:
- large-scale benchmarking
- fine-tuning
- personalization
- multilingual evaluation
- hybrid retrieval
- learned rewrite routing
- production serving
The goal was to learn the core concepts quickly, obtain a few falsifiable
results, and stop once the central research question had been answered.