What Building Reigner Taught Me About Context, Cost, and Trust

The system I built before Reigner had four agents. A router decided where a question should go, search and library agents gathered information, and a synthesis agent assembled the answer. The separation looked clean on a diagram. In practice, every handoff was another place to lose context, obscure a failure, or pay for a model call that did not improve the result.

I eventually collapsed those roles into one agent with all the tools. That did not solve everything, but it made the remaining problems easier to see. The difficult part was no longer deciding which agent should speak next. It was deciding what the model should see, how much it should be allowed to retrieve, what should survive a long session, and how an answer could show its work.

Those problems kept recurring whenever I built question-answering systems over domain-specific documents. Reigner started from two motives: I wanted a reusable base for that work, and I wanted to understand the harness by implementing one myself. It became a Python foundation that developers can shape into specialised, single-agent Q&A systems for a particular domain, such as audit records, medical literature, legal decisions, or product documentation. The developer prepares the knowledge, tools, and operating rules; the resulting system can serve the people working in that domain.

I wrote the broader case for harness engineering earlier this year. This is the more practical follow-up: the decisions I made in Reigner, what they bought me, and where I am still uncertain.

Where Efficiency Lives

New model releases can make it seem as though the main engineering decision is which model name to put in a configuration file. Model capability certainly matters. But an application does not become reliable or economical merely because its model is more capable.

The labs themselves are treating efficiency as a problem with several layers. Meta reported that the Llama 3 tokenizer produced up to 15% fewer tokens than Llama 2. Llama 4 uses mixture-of-experts layers so that a token activates only part of the model, reducing the computation used at inference time. More recently, Meta's compute-optimal tokenization research trained 988 models to study how token granularity changes the relationship between data and compute.

That work happens inside model architecture and serving infrastructure. Reigner works above the API boundary, where the questions are different but related. How much context does this turn require? Is an identical tool call being repeated? Does every step need the strongest available model? Can a result be shortened without hiding the fact that it was shortened?

The blacksmith analogy I had in mind while building Reigner is simple. I may not make the steel, but I can still shape the tool: its balance, its edge, and the grip that fits the work. In an LLM system, that shaping happens through the instructions, retrieval path, tool contracts, context policy, model selection, and tests around the model.

Keeping the Loop Visible

I considered using an agent framework. Frameworks are useful when orchestration is the problem: routing between specialists, coordinating graphs, adding retries, or integrating a large ecosystem of components. Reigner's problem was narrower. I wanted to account for context and cost one iteration at a time.

One Reigner iteration produces one model call and a stream of typed events such as tool calls, tool results, citations, compaction, and the final answer. The central loop is readable end to end, while the individual guardrails live in focused modules for truncation, caching, compaction, parallel reads, and nudges. I can follow a run through the source and identify what entered the prompt and why.

That visibility mattered more to me than a visual graph or a collection of prebuilt nodes. It also followed from the earlier four-agent experience: for this problem, continuity of context was more valuable than role separation.

This is a choice about fit, not a general argument against frameworks or multiple agents. A workflow that needs independent specialists or complex coordination should probably use those abstractions. Reigner deliberately gives up that leverage in exchange for a smaller surface that I can inspect and test.

Compiling Knowledge Before Querying It

A common document-Q&A pipeline chunks files, embeds the chunks, retrieves the nearest matches, and places the top results in a prompt. Reigner starts elsewhere. Ingestion acts as a compile step: the developer declares the information that matters, and a one-time pass turns raw PDF, HTML, or text sources into structured artifacts addressed by (entity, version, section). The agent queries those artifacts at runtime rather than reopening raw documents.

The useful property here is not complete determinism; the model can still choose a different search path. It is that the knowledge has stable addresses. An artifact can be cited back to its source, compared with the same artifact from another version, and named explicitly in an eval. That gives me something firmer to test than a similarity score alone.

Compilation also moves interpretation away from the query path. Extraction happens once, where I can inspect the output, adjust the schema, and run it again. Later questions can retrieve smaller, purpose-built representations instead of repeatedly asking a model to interpret the same raw pages.

There is a cost to this decision. It assumes the corpus has enough structure to justify a schema. Financial filings, legal judgments, and versioned product documentation often do. A heterogeneous or open-domain collection may not. If the real task is to find material that is vaguely related to a phrase, embeddings are a better fit, and Reigner's search interface leaves room for that kind of backend. I do not see compiled artifacts and vector retrieval as universal competitors; they preserve different information and suit different questions.

Turning Citation Discipline Into a Test

Prompt instructions still matter in Reigner. The agent is told to cite its sources and to avoid unsupported claims. I did not want that instruction to be the only enforcement mechanism, especially for numbers that can look plausible when they are wrong.

Reigner's tools preserve provenance, and the agent can register a citation containing a source, locator, and value. The deterministic faithfulness check extracts numeric values from the final answer and compares them with the registered citation values. The core of the implementation is deliberately ordinary:

claims = _numbers(run.answer_text)
cited = [n for c in run.citations for n in _numbers(str(c.value))]
for claim in claims:
    if not _is_cited(claim, cited):
        return CheckResult("faithfulness", "fail", f"claim {claim:g} not cited")

The comparison handles scale differences, so $67.0 billion can match a stored value of 67000000000. It ignores bare calendar years and digits embedded inside identifiers to avoid some predictable false positives.

This is a bounded check. It verifies numeric correspondence, not the meaning of an entire sentence. A number could coincide with the wrong citation, and unsupported prose can pass because there is no number to inspect. An LLM-based judge could be added through the same eval interface, but Reigner does not currently ship one.

I prefer stating that limit directly. The check does not prove that an answer is true. It catches a narrower and still expensive class of failure: a model presenting an uncited figure as fact. It also turns that failure into something CI can report instead of something I have to notice while rereading a transcript.

Spending Model Capability Deliberately

Cost in an agent loop is not only the provider's price per token. It is also how much context is sent, how many iterations the task takes, how often tools repeat work, and which model handles each iteration. I wanted those choices to be visible in the design rather than discovered later in a billing dashboard.

Reigner's main model-selection mechanism is oracle escalation. A cheaper default model can handle routine retrieval and narrowing. When it reaches a step that appears to need stronger reasoning, it can call escalate_to_oracle with a reason. The following iteration uses the configured oracle model, and then the loop returns to the default.

This resembles the broader direction of model-routing research, including RouteLLM, although Reigner's mechanism is simpler and initiated by the agent itself. An escalation event makes the decision visible for later inspection. Reigner does not yet judge automatically whether the extra spend was warranted; that would require an eval designed for the decision rather than just the answer.

The surrounding controls are less visible but just as important. Identical read-only tool calls can reuse a per-session cache. Independent reads issued together can run concurrently, reducing latency even when they do not reduce token count. Tool outputs have individual size limits because a search result and a document section do not need the same budget. The prompt is also split into stable and dynamic parts so providers that support prefix caching can reuse the stable instructions and tool definitions.

These mechanisms do not guarantee that the cheaper model will always be sufficient or that every escalation will be wise. They make the trade visible and measurable. That is a more useful starting point than applying one expensive model to every turn and trying to optimize the bill afterwards.

Failures That Became Guardrails

Some of Reigner's more specific rules came from following long sessions and watching them degrade.

The clearest example was compaction. Reigner begins compacting history as the context budget crosses 80%, 90%, and 95%. In my first approach, the summary could consume the scratchpad notes the agent had written to preserve its findings. After compaction, the agent would retrieve the same material and reconstruct work it had already done. The context was smaller, but the system had become less efficient.

The fix was to keep notes outside compactable conversation history. That sounds obvious after the fact. It was not obvious while treating the prompt as one growing sequence of messages. The distinction only became clear after I watched the agent forget its own work.

Tool output produced another version of the same problem. A response that silently returns its first ten matches looks complete to a model. Reigner's default tools therefore describe their own boundaries:

{ "content": "...", "offset": 0, "limit": 4000,
  "has_more": true, "total_size": 18320 }

A finite-context model cannot infer whether an output is complete unless the tool says so. Fields such as has_more, truncated, available_keys, and missing_keys give it enough information to continue, narrow the request, or stop.

Reigner also injects occasional strategic reminders and a one-time wrap-up nudge after consecutive tool errors. The one-time latch matters: without it, a mechanism intended to end an unproductive run can itself become a loop. These are small details, but agent reliability has often turned out to be a collection of small, explicit decisions rather than one clever prompt.

What I Would Change

The most difficult part of Reigner's current approach is the schema. The compile step asks a developer to decide what matters in the documents before asking many real questions. In practice, understanding a corpus and designing its representation inform each other. You often discover the right schema by querying an incomplete one.

The guided reigner init flow reduces that initial work, but it does not remove the underlying tension. If I revisited the design, I would explore a more permissive first ingestion followed by a schema that becomes stricter as real queries expose what deserves stable representation.

I also have not run the comparison that would give the retrieval argument more weight: compiled artifacts and a strong embedding pipeline evaluated on the same corpus, with the same questions and model. My experience gives me reasons to prefer compiled knowledge for structured domains. It does not make that preference a general result.

Those boundaries are part of how I now think about the project. Reigner is intended for question answering over prepared knowledge, not autonomous action, code execution, or multi-agent coordination. Keeping that scope narrow allowed me to spend time on the parts I wanted to understand: context pressure, tool behavior, provenance, evaluation, and cost across a run.

The main lesson was not that every agent should be built this way. It was that the model is only one material in the system. The engineering work is in deciding how that material should be used, what it should be allowed to see, and how its output earns trust for a particular job.

Reigner is open source under the MIT licence. The documentation covers the architecture and the eleven guardrails in more detail, and the library installs with uv add 'reigner[anthropic,ingestion]'.