---
title: "Evaluate search quality"
description: "Score a dataset's retrieval quality with golden queries and evaluation runs."
canonical: https://ocho.bot/docs/library/evaluate-search-quality
last_updated: 2026-07-12
---

# Evaluate search quality

> Score a dataset's retrieval quality with golden queries and evaluation runs.

Each dataset has a **QA** tab (alongside Overview and Documents) with golden queries and search-quality scores — useful after building, rebuilding, or changing strategy.

> The QA tab is currently unavailable: it does not load in the app right now and the feature is under review. The sections below describe how it works when it is enabled.

> Who can do this: running evaluations and managing golden queries requires the Library admin role. Everyone else sees the page read-only.

## Try queries live

The **Live search preview** at the top of the page lets you probe retrieval against the dataset without writing a golden query. Type a question in the **What would a user ask?** box and click **Search** to see the top ranked passages that come back — a quick gut-check without setting anything up.

## Golden queries

Golden queries are saved test questions with expected results. Add the questions your users actually ask: click **Add query** (Library admins only — the button is hidden for everyone else) to open the **Add golden query** modal, where you set the query text, a query type (the dropdown lists the raw values `data-lookup`, `comparison`, `provenance`, `qa`, and `procedural`), expected keywords, phrases, and pages, ideal document IDs, and an optional expected answer.

Each query in the list shows its type as a friendly badge (Data lookup, Comparison, Provenance, Q&A, or Procedural) and a **Not yet evaluated** badge — per-query pass/fail scores are not yet reported by the eval runner, so this badge currently stays in place even after a run completes. Each query expands to show what retrieval currently returns for it. Admins can also **Edit**, **Approve** (for queries awaiting approval), or **Delete** a golden query from its row.

## Run an evaluation

Evaluations run automatically after a strategy is built. To score the dataset on demand, click **Run evals** (shown but disabled without the Library admin role); the button is unavailable while a run is already in progress.

The results card shows a Pass/Fail verdict badge, the overall score, the number of queries evaluated, and bars for **Mean MRR**, **Page recall**, **Page precision**, **Phrase hit rate**, and **Keyword hit rate**. A collapsible **Run history** panel lists prior runs for the strategy version — select one to view its results. If any queries errored during a run, the page shows how many — per-query error details are not yet exposed by the eval runner, so you can see the count but not which queries failed or why.

## Roll back a strategy

If quality regressed after a rebuild, **Roll back** (Library admins, available from strategy version 2 onward) switches the dataset back to the prior indexing strategy after a confirmation. Use the **View previous strategy versions** link to browse eval results from earlier strategy versions for comparison.

## Good to know

- Evaluations measure *retrieval* (did the right passages come back), not the

chat model's writing.

- When the latest run failed or errored, a banner explains what happened with

suggested next steps, plus an inline **Re-run evals** action — and, from   strategy version 2 onward, **Roll back** — for admins.

All docs: https://ocho.bot/docs

---

Ocho — AI knowledge orchestration · [Home](https://ocho.bot/) · [Docs](https://ocho.bot/docs) · [Blog](https://ocho.bot/blog) · [About](https://ocho.bot/about) · [Developers](https://ocho.bot/developers) · [Contact](https://ocho.bot/contact) · [llms.txt](https://ocho.bot/llms.txt)