Retrieval and chat settings

Use this reference to look up fields after choosing a setup. For a working sequence, start with local chat setup.

retrieval configures embeddings; chat configures answer generation and optional checks. Both sections are optional. A normal site build validates them without calling these providers.

Configure retrieval

retrieval.embeddings supplies embeddings for passage indexing, semantic queries, and hosted chat builds. Choose a provider and model when you need meaning-based retrieval.

retrieval:
  embeddings:
    provider: voyage
    model: voyage-4-large
    dimension: 1024
    queryModel: voyage-4-lite

To use OpenAI's embeddings, or a server that copies its API, such as Ollama, vLLM, or a LiteLLM proxy, name the openai provider:

retrieval:
  embeddings:
    provider: openai
    model: text-embedding-3-small
    keyEnv: OPENAI_API_KEY

The config never holds a key. A key setting fails validation, and keyEnv fails if its value isn't a variable name. Command-line flags on shoalrun index override these settings. See Passage index for the files that --embed writes.

Configure chat

chat configures the OpenAI-compatible service that writes answers. Choose a provider and model; there is no default model. Start with these fields, then add optional checks or reuse settings as needed.

chat:
  provider: openai
  model: deepseek/deepseek-v4-flash
  baseUrl: https://openrouter.ai/api/v1
  keyEnv: OPENROUTER_API_KEY

These fields configure the service and answer limits:

Optional layers

Add these fields to chat when the corresponding capability is needed. Scorer-dependent fields require rerank: jev or an equivalent scorer mapping.

The gate uses the best passage score:

• Below decline: return a not-covered response without a chat-model call.
• Between decline and hedge: generate an answer with a partial-coverage warning.
• At or above hedge: generate an answer normally.

The config never holds a key. A key setting fails validation, and keyEnv fails if its value isn't a variable name. The --model flag on shoalrun ask overrides model, so you can try another model without editing the config.

Send extra request fields

Some services accept request fields that Shoalrun doesn't know about. chat.body adds them to every request. On OpenRouter, the useful ones are reasoning, which controls whether a model thinks before it answers, and provider, which picks who serves the model.

chat:
  model: deepseek/deepseek-v4-flash
  body: { reasoning: { enabled: false } }

The fields can add to a request but can't replace what Shoalrun sets, which is model, messages, max_tokens, temperature, and stream. Use chat.model, chat.maxTokens, and chat.temperature for those. A field whose name looks like a secret, such as a key, token, or password, fails validation, because the config never holds a key. Each fallback and escalation model inherits body and can replace it with its own. An empty body: {} clears it. Another service may reject a field it doesn't know, so check what yours accepts.

Service-specific fields can affect latency, cost, and quality. Test them with your own questions and model; support and effects vary by service. Shoalrun does not interpret those extra fields.

Choose the scorer

The scorer behind rerank, gate, verify, direct, and cards is a service that answers yes-or-no questions about a passage. provider: jev names the protocol it speaks, POST /v1/systemone, and not one vendor. Any service or local server that speaks it works. With no other settings, Shoalrun reads the address, model, and key from JEV_API_KEY, TYPESAFE_API_KEY, TYPESAFE_ENDPOINT, and JEV_MODEL. To choose them in config, use a mapping.

chat:
  rerank:
    provider: jev
    endpoint: https://scorer.example.com
    model: your-model-name
    keyEnv: SCORER_API_KEY

OpenRouter serves Jev at the same protocol and price, so the chat model and the scorer can share one key:

chat:
  rerank:
    provider: jev
    endpoint: https://openrouter.ai/api/v1
    model: typesafe/jev-1.13
    keyEnv: OPENROUTER_API_KEY

Scores from different services aren't comparable. The gate, verify, direct, and cards thresholds are numbers on one scorer's scale. After a change, run shoalrun answers and the answer eval again, and expect to adjust them. Cached scores are kept per address and model, so a change never reuses an old service's scores.

When the scorer is down

The scorer is what keeps a question the docs don't cover from reaching the model. If it fails, by timing out, running out of budget, returning an error, or having no key, there are two ways to go.

• search (the default). The chat doesn't call the model. It shows the closest pages from retrieval and says it can't check answers right now. Nobody can use an outage to get free model output for any question, and readers still get somewhere to go.
• answer. The model answers without the gate and the claim check, and the answer carries a note that the checks were skipped. Use this when a reader-facing outage costs more than the risk. On a public site, every question then reaches the model, and only the rate limits and token caps stand in the way.

chat:
  rerank: { provider: jev, onError: answer }

Neither mode caches anything from an outage, so an unchecked answer can't outlive it. This applies only when a scorer is configured. With no rerank, the model answers every question, because you chose that.

Use more than one model

Use fallback for provider failures and escalate for answers flagged by checks. Both are optional:

chat:
  provider: openai
  model: deepseek/deepseek-v4-flash
  baseUrl: https://openrouter.ai/api/v1
  keyEnv: OPENROUTER_API_KEY
  fallback: [{ model: openai/gpt-6-luna }]
  escalate: { model: z-ai/glm-5.3-flash, when: [unsupported-claim, uncited] }

Each extra model inherits the provider, address, key variable, and limits from the main settings. Name only what differs. A model on a different service can set its own baseUrl and keyEnv. The --model flag on ask replaces the main model and leaves the others alone.

Fallback

If the main model errors, times out, or returns no text, ask tries the next model in the list. Use a model from a different lab, so one outage doesn't stop the chat. The result records which model answered and which failed, with the reason. If every model fails, the error lists all of them.

Escalation

The main model answers first, and the checks run. If one of the when reasons applies, ask asks the escalation model the same question with the same passages and runs the same checks. It shows whichever answer has fewer flags, and the first one if they tie. The supported reasons are:

Without when, the reasons are unsupported-code and uncited, plus unsupported-claim when verify is set. The code check is a heuristic that can misjudge a valid example. A second answer from a thinking model adds seconds. So many sites limit when to unsupported-claim and uncited. A declined answer or a direct excerpt is never escalated. If the escalation model fails, the first answer is shown with a note. --no-escalate on ask turns it off for one run.

Choose an escalation model using reviewed questions that your primary model handles poorly. A larger model is not automatically a better fit.

Related reading

No matching documents.