RésuméEmail me

OfficeQA RAG

Retrieval over U.S. Treasury Bulletins for the Databricks OfficeQA benchmark, a baseline against an engineered pipeline, with the bad numbers left in.

Year
2026
Role
Solo
Team
Course project, runs fully local
Stack
Python, ChromaDB, bge-small, Ollama with qwen2.5, the official reward.py scorer
Link
Repository on GitHub

How it works

2019

Fixed 1000 character strips, every shelf searched.

  • Doc hit rate at 511.8%
  • MRR0.05
  • Recall5.9%
  • Factual accuracy0%
  • Cell hit rate0%

17 questions, 11 hard and 6 easy, top 5. Generator qwen2.5 7B via Ollama, everything local.

From the right bulletinKnife cutRetrieval went five times better. Accuracy did not, and the figure says so.

The system

OfficeQA retrieval vs accuracy, baseline to engineered On 17 OfficeQA questions, the engineered pipeline raised doc hit rate at 5 from 11.8% to 58.8% and doc recall at 5 from 5.9% to 47.1%, but factual accuracy only rose from 0% to 5.9% and cell hit rate at 5 stayed at 0%, because the right bulletin is found but never the exact cell. 0% 20% 40% 60% 80% 100% Share of 17 questions, top 5 retrieved Doc hit rate at 5 Doc recall at 5 Factual accuracy Cell hit rate at 5 11.8% 58.8% 5.9% 47.1% 0% 5.9% 0% in both runs The right bulletin is found five times as often, but the exact cell never is. Baseline Engineered Rows ordered by engineered value. MRR, on a 0 to 1 scale, rose from 0.05 to 0.27. 17 questions (11 hard, 6 easy), qwen2.5 7B via Ollama.

The job

Answer 17 benchmark questions from quarterly Treasury Bulletins, 2015 to 2025, and measure retrieval and generation separately so the bottleneck is visible.

What changed the numbers

Structure aware chunking keeps whole table rows together and prepends the section header. Every chunk carries year, month and source file, and the retriever filters on them before ranking.

The filter includes the following year, because the Treasury prints calendar year totals in the next March issue. Without that, one question's gold document was filtered out entirely.

Results

Doc hit rate at five went from 11.8% to 58.8%, and recall from 5.9% to 47.1%. Factual accuracy only moved from 0% to 5.9%, one question in 17.

The write-up says why: retrieval finds the right bulletin but never the exact cell, and most gold answers are statistics that are never printed in the source.

Next, ForkLog Live →