Shoalrun documentation framework
For agents
Start with https://www.shoalrun.com/llms.txt for the documentation index, then read the Markdown pages it links. Skip to article. Documentation index. Read this page as Markdown.
How to

Evaluate agent tasks against your docs

Define realistic tasks, record reviewed answers from your own agent, and check the results against a documentation build.

Updated
Read as Markdown

Use evaluation tasks to check whether an agent can accomplish important work using your documentation. Shoalrun defines task and result formats, prepares report templates, and checks those reports during an audit. You run your own agent and arrange human or external review of its answers. Shoalrun does not launch agents, choose a model, or grade prose.

You need a built docs site, a few realistic tasks, and an agent or manual workflow for running them. The ordinary build does not execute evaluations or call a model. Evaluations are opt-in audit checks.

Define a task file

Save YAML or JSON with schemaVersion: 1 and a nonempty tasks list. Each task needs a unique lowercase ID, a prompt, an allowedTools list, and criteria with unique IDs and descriptions. Tasks are required by default; set required: false for an exploratory task that should not block an audit.

evals/tasks.yaml
schemaVersion: 1
tasks:
  - id: reader-views
    prompt: Explain what reader views hide and whether they protect private content.
    allowedTools: [docs.search, docs.read]
    criteria:
      - id: visibility
        description: Explain visibility versus access control and cite the reader-views page.
      - id: markdown
        description: Explain that published Markdown retains reader-marked content.

Use the same tool names in your task file and runner report, such as search, fetch, docs.search, or mcp.get_page. Shoalrun validates the recorded runner tools against each executed task's allowed list. It cannot observe or enforce what tools an external agent actually used. Use a shared allowed tool set within a report; prepare separate suites for runs with different tool permissions.

Write criteria that an expert can assess. Valid citations and complete report fields are mechanical checks. Whether the answer correctly explains your product remains a review judgment.

Prepare a report for the built site

shoalrun build docs.config.yaml site-dist
shoalrun eval prepare site-dist --tasks evals/tasks.yaml --out evals/results.json

Create the evals directory first. Preparation refuses to overwrite an existing report; use a new output filename for another run.

The template contains a task-file digest, a documentation-build digest, a per-file buildFiles inventory, a timestamp, empty runner details, and each task marked skipped. It records no passing results and does not run the tasks. A template with skipped required tasks fails the check until you complete the run and review.

The task digest uses normalized definitions, so whitespace-only changes do not invalidate a run. Prompt, criteria, tools, or required-status changes do. The build digest covers published output files, including HTML, Markdown, indexes, discovery, and assets. It excludes build-info.json, private ai/ artifacts, and .shoalrun/ state. File touches do not invalidate otherwise identical output; changed published bytes do. Other published metadata, including build IDs embedded in manifests, remains covered even when it reflects deployment provenance.

Diagnose a different build

Save a file inventory alongside an evaluation build:

shoalrun eval snapshot site-dist > evaluated-build.json
shoalrun eval snapshot rebuilt-site > current-build.json
shoalrun eval compare current-build.json --since evaluated-build.json

The comparison lists added, changed, and removed file paths. It exits 0 for identical snapshots and 1 for differences or invalid input. Each inventory binds relative file paths and SHA-256 byte digests to the same build digest used by evaluation reports. It contains no answers or grades.

New report templates include this inventory in buildFiles. When a report is stale, eval check includes the file comparison in details.buildChanges. Older reports still receive the exact-build check, but cannot identify which earlier files differ. Editing the stored digest or deleting changed files is not a new evaluation.

For local review of a hosted build, download and retain that exact output, task file, and report template. Run the tasks against the downloaded site and check the completed report against it. Rebuilding the source on another operating system can change rendered assets; a report about the downloaded build does not automatically cover that rebuild. Archive the source revision and build inventory with the run. A successful build capture records no evaluation pass.

Run and review the tasks

Use your own agent with the declared tool access. Record the actual runner name, optional model, available tools, and any limitations in runner. Replace recordedAt with the run's UTC ISO timestamp.

For each task, record its status (pass, fail, or skipped), answer, citations, and criterion judgments. A passing result needs a nonempty answer, at least one citation, and a passing judgment with a reason for every defined criterion. Identify citations by stable page ID and publication, so editions and languages cannot be confused:

{
  "taskId": "reader-views",
  "status": "pass",
  "answer": "Reader views change visibility. Public Markdown still includes reader-marked sections, so the views do not protect private content.",
  "citations": [
    {
      "id": "guide.reader-filters",
      "publication": { "channel": "production", "version": "current", "language": "en" }
    }
  ],
  "criteria": [
    { "id": "visibility", "status": "pass", "reason": "The answer distinguishes visibility from authorization." },
    { "id": "markdown", "status": "pass", "reason": "The answer states that reader-marked sections remain in Markdown." }
  ]
}

This is an example result shape. Record your agent's actual answer and the reviewer’s assessment rather than copying the example as evidence of a run.

Complete review with reviewer, method (human or external), reviewedAt, and notes. The review timestamp must be at or after recordedAt; both must be valid UTC timestamps and not in the future. Identify an external grader honestly, and record its limitations. A named reviewer is evidence about who assessed the answer, not independent proof that the assessment is correct.

Check the report

shoalrun eval check site-dist --tasks evals/tasks.yaml --report evals/results.json

The command prints JSON and exits 1 on failure. It checks schema versions and fields, unique task and criterion IDs, the two digests, timestamps, runner and review details, declared tool access, criterion coverage, and whether each citation identifies a published page in the selected publication. Required tasks must pass. Missing or failed optional tasks appear in the outcomes without blocking, but malformed results, bad citations, or unsupported passing claims still fail validation.

Shoalrun verifies that cited pages exist; it does not verify that their text supports an answer. Reported tool use and grades are assertions supplied by the runner and reviewer. A passing report does not establish broad agent reliability or factual correctness.

Add the audit hook

To check an existing report whenever you run shoalrun audit or shoalrun check, add:

docs.config.yaml
audit:
  evals:
    tasks: ./evals/tasks.yaml
    report: ./evals/results.json

Paths resolve beside the config file declaring each value, including included config files. The audit builds a temporary site, then checks the report against that output. Changes to published output or task definitions require a new evaluation. Reports remain project files; Shoalrun does not publish them into the site automatically.

If your team already has an evaluator command, configure it as an optional run list:

docs.config.yaml
audit:
  evals:
    tasks: ./evals/tasks.yaml
    report: ./evals/results.json
    run: [node, scripts/evaluate-docs.mjs]

That script belongs to your project. It runs after a successful build, from the root config directory, with no shell interpolation and a two-minute timeout. It receives these environment variables:

Variable Value
SHOALRUN_EVAL_SITE Absolute temporary build directory.
SHOALRUN_EVAL_TASKS Absolute task-file path.
SHOALRUN_EVAL_REPORT Absolute report-file path to write.
SHOALRUN_EVAL_BUILD_DIGEST Digest of the built documentation.
SHOALRUN_EVAL_TASKS_DIGEST Digest of the normalized tasks.

The script must run and assess tasks, write a report, and exit nonzero on execution failure. A successful exit does not bypass report checks. A nonzero exit fails the audit even if an older report still validates. Hook execution and report validation appear separately as eval-run and evals in the audit results. A failed build prevents the eval hook from running.

Configured commands can use your provider keys, make network requests, and write files. Shoalrun supplies the interface; your team owns the execution, costs, review, and credentials. The temporary build is removed after the audit. For longer runs, execute your evaluator separately and import the resulting report.

Shoalrun's own example

This repository retains docs/evals/tasks.yaml and docs/evals/results.json as an example of a small agent self-assessment. Its five questions cover custom search providers, search off, bounded lexical coverage, reader choices, and freshness. This narrow, familiar task set is not a general quality gate. The site's routine audit does not import the report, and ordinary documentation changes do not require rerunning it.

The original October 1 report is preserved unchanged in docs/evals/history/results-2026-10-01.json. The October 15 local run retains its actual tool responses, new answers, self-review judgments, and exact-build inventory under docs/evals/runs/2026-10-15-local/. The evaluator knew the project and had seen the earlier report; tasks shared retrieved pages. These are historical observations, not blind or independent evidence of general agent reliability. Earlier exploratory results remain in plans/documentation/agent-docs-eval.json.

Run a new evaluation when it answers a concrete product question, with relevant tasks and a clearly described runner and reviewer. Preserve older reports rather than changing their dates, digests, or grades. An exact-build report can legitimately fail freshness against a later build; that does not make its original evidence invalid or require routine refreshes.

Shoalrun's evaluation commands and optional audit hook remain available for projects that choose them. This site's normal validation still covers the build, links, schemas, output consistency, relevant behavior tests, and browser checks. To inspect the archived October 15 evidence without requiring the current site to match its old build, run from the repository root after compiling the framework:

node docs/evals/runs/2026-10-15-local/verify.mjs

That command checks the recorded responses and report provenance; it does not rerun tasks, grade answers, or certify the current site.

Reference while working

  • CLI reference — Look up report preparation and validation commands.
  • Non-goals — Understand why agent execution remains outside Shoalrun.