Passage index

A passage index splits built Markdown pages into smaller pieces for retrieval. Each passage keeps its page title, heading path, and source link. Chat and custom passage-search providers use it to find relevant parts of a page.

The built-in search box uses the separate page-level search-index.json. You do not need a passage index for normal static-site search.

Create the files after building the site:

Indexing leaves published page content unchanged. The files sit beside the build output, and you must rebuild the index when that output changes.

Build the index

Build the site, then run shoalrun index on the output directory:

shoalrun build docs.config.yaml site-dist
shoalrun index site-dist

The command prints a summary as JSON:

{
  "status": "ok",
  "documents": 77,
  "chunks": 725,
  "output": "/path/to/site-dist/ai/chunks.json"
}

The command reads three files from the build output:

It writes ai/chunks.json. Without --embed, it makes no network requests. Run it after each build. It replaces the previous file.

Indexing also prepares readable paragraphs, code lines, and table rows for passage-search results. Search requests use that prepared data instead of parsing Markdown on first use. If serving search rejects an older index, rerun shoalrun index site-dist. Existing embeddings remain usable when the passage text and embedding settings are unchanged.

Because it reads built output, the index holds only published pages in each publication. Drafts and pages outside enabled publications never appear.

How the default chunker splits a page

The default chunker follows structure. It doesn't cut text into fixed-length windows.

1. Split at ## and ### headings. Text before the first ## heading forms an intro passage. Headings at #### and below stay inside their section's text.
2. Keep blocks whole. A paragraph, list, table, or code block is never cut mid-block, except for large tables in step 4.
3. Fill passages up to 2,000 characters. A section longer than that splits between blocks into a first passage and later parts.
4. Split large tables between rows. Each part repeats the table's header rows, so every row keeps its column names.
5. Keep large code blocks whole. A code block over the limit becomes one passage that exceeds it. Fenced code never starts a heading, so # comments inside a fence don't split a section.
6. Merge short sections into their first subsection. A section under 200 characters with a subsection below it joins that subsection's first passage. A short section with no subsection stays a passage of its own.

These limits are fixed. You can't configure the chunker or replace it yet.

The number of passages changes as the documentation grows. Most passages are a few hundred characters long, and nearly all stay under 2,000.

What a passage contains

Each entry in chunks describes one passage:

{
  "id": "guide.configuration#understand-the-three-stages",
  "docId": "guide.configuration",
  "title": "Organize project configuration",
  "type": "how-to",
  "variant": "production|current|en",
  "headingPath": ["Understand the three stages"],
  "url": "/configuration#understand-the-three-stages",
  "markdownUrl": "/configuration.md",
  "text": "Configuration takes effect in three stages:\n\n1. **Resolve co…",
  "digest": "be259c2f4c0c896d1fe25ed496ac642f97391744675e06ee844f09372e47767a",
  "search": {
    "schemaVersion": 1,
    "paragraphs": ["Configuration takes effect in three stages: …"],
    "literalLines": []
  }
}

The file also records schemaVersion and a sourceDigest that changes when any page's digest changes.

Embedding input

The heading path isn't part of text. Anything that scores or embeds a passage reads the page title, the heading path, and the text together:

Organize project configuration > Understand the three stages

Configuration takes effect in three stages: …

digest hashes that string, so it changes when the title, the heading path, or the text changes. A stored vector or score that no longer matches a passage's digest is stale.

The derived search data is not part of the embedding input or its digest. Preparing it does not require new embeddings for unchanged passages.

Passage IDs and links

A passage ID is the page ID, #, and a slug of the heading path. A passage without headings uses intro. The chunker adds a suffix when IDs would collide:

IDs stay the same when body text changes. They change when you rename a heading or the page ID.

Section links come from the heading IDs in search-index.json. A passage links to the page, without a # anchor, when the build has no search index or when a component generated the heading. Walkthrough steps and tab labels are examples. The site search page describes the index.

Embed passages

Add --embed to turn each passage into a vector. Shoalrun has no default embedding provider, so choose one under retrieval.embeddings in your config:

retrieval:
  embeddings:
    provider: voyage
    model: voyage-4-large
    dimension: 1024

Set the provider's key in the environment, VOYAGE_API_KEY for Voyage, then run:

shoalrun index site-dist --embed --config docs.config.yaml

The summary gains an embeddings object:

{
  "embeddings": {
    "passages": 725,
    "embedded": 725,
    "reused": 0,
    "model": "voyage-4-large",
    "dimension": 1024,
    "bytes": 2969600
  }
}

The command sends each passage's embedding input to the configured provider. The key is not written to the output.

It writes two files next to chunks.json:

The vector-file size grows with passage count and dimension. The files don't record the provider's URL or key variable, so an internal server's address never reaches your published site.

A rerun embeds only passages whose digest changed. Changing the model or the dimension re-embeds every passage. A vector is never used for a passage whose text changed after it was embedded, because the digests no longer match.

Choose a provider

Two providers are built in, and you can switch by editing the config and rerunning the command.

Vectors from different models can't be compared, so switching provider or model means embedding every passage again. Queries must use a model from the same embedding space as the passages.

OpenAI's API has no separate query and document mode, so models that need one lose it behind an OpenAI-style server. Shoalrun sends the dimensions request only for OpenAI's text-embedding-3 models. If a proxy renames a model, set dimension to the size the backend returns.

Embed queries with a different model

Queries use the model that embedded the passages unless you set queryModel. A different query model must share the stored vectors’ embedding space and dimension.

Voyage 4 models share a space. For example, you can embed passages with voyage-4-large and queries with voyage-4-lite without re-embedding the passages. Check provider availability before choosing a query model.

Limits

• Reader marks are copied from the page, so a section inside a page can't have its own audience. Run shoalrun index again after you change a page's marks. They aren't part of the embedding input, so changing them never re-embeds a passage.
• Passages don't overlap. The heading path supplies the context that overlap would.
• A few passages are very short, such as an empty related-reading block or a one-line API note.
• The chunker reads Markdown twins only. It doesn't read the OpenAPI source directly, so API detail appears only as far as the reference pages show it.

Related reading

No matching documents.