Add Retrieval to an Arcee Workflow Without Hiding Context Problems

Retrieval is the part of the stack that most tutorials treat as a solved black box. You embed, you search, you stuff the top chunks into the prompt, and you move on. Then the model gives a fluent answer that is only loosely related to the documents you actually supplied, and you spend the next hour wondering whether the problem is the model, the prompt, or the retrieval itself.

I no longer hide the context. Every retrieval-augmented workflow I keep around surfaces the exact chunks the model saw, the similarity scores, and a clear signal when the evidence is weak. This post shows how to add that transparency to an Arcee generation path without turning the code into a research project.

The result is a small, inspectable layer you can drop on top of the document Q&A tool from the previous post (or any other generation script). The model stays the same—Trinity Mini in my examples—but the surrounding control makes failure modes visible instead of cosmetic.

Why Context Problems Stay Hidden

Three habits create the illusion that retrieval is working:

1. The final answer is printed without the supporting chunks.

2. Similarity scores are discarded after ranking.

3. The prompt never forces the model to acknowledge weak or missing evidence.

When those three are true, a confident hallucination looks exactly like a grounded answer. The only way to tell the difference is to inspect the context that was actually sent. So we make that inspection part of the normal output.

The Minimal Transparent Retrieval Wrapper

I keep the retrieval step separate from generation and return a rich object instead of a bare list of strings.

Python

from dataclasses import dataclass
from typing import List

@dataclass
class RetrievedChunk:
    source: str
    text: str
    score: float

@dataclass
class RetrievalResult:
    chunks: List[RetrievedChunk]
    max_score: float
    min_score: float
    weak_evidence: bool  # True when top score falls below a threshold

The weak_evidence flag is the key addition. I set the threshold empirically (0.35–0.45 cosine for the MiniLM embedder I usually start with). When the flag is true, the prompt and the user-facing output both change.

Building the Prompt With Explicit Evidence Status

Python

def build_messages(question: str, retrieval: RetrievalResult) -> list[dict]:
    if not retrieval.chunks:
        context_block = "No relevant documents were found."
    else:
        context_block = "\n\n".join(
            f"[{c.source} | score={c.score:.3f}]\n{c.text}"
            for c in retrieval.chunks
        )

    system = (
        "You are a careful assistant that answers only from the supplied context. "
        "If the context is missing or the relevance scores are low, say that the "
        "documents do not contain enough information. "
        "Never invent facts. Cite sources when you use them."
    )

    user_parts = [
        f"Context:\n{context_block}",
        f"\nQuestion: {question}",
    ]
    if retrieval.weak_evidence:
        user_parts.append(
            "\nNote: The highest relevance score is low. "
            "Treat the context as weak and say so if the answer is uncertain."
        )

    return [
        {"role": "system", "content": system},
        {"role": "user", "content": "\n".join(user_parts)},
    ]

The model still receives the chunks. It also receives an explicit signal that the evidence may be insufficient. Trinity Mini responds to that signal reasonably well when temperature is kept low.

The Generation Call Stays Ordinary

Once the messages are built, generation is identical to the earlier posts:

Python

def answer_with_retrieval(question: str, retrieval: RetrievalResult, generate_fn) -> dict:
    messages = build_messages(question, retrieval)
    raw_answer = generate_fn(messages)  # your existing local or API call

    return {
        "answer": raw_answer.strip(),
        "chunks": [
            {"source": c.source, "score": c.score, "text": c.text}
            for c in retrieval.chunks
        ],
        "weak_evidence": retrieval.weak_evidence,
        "max_score": retrieval.max_score,
    }

The caller receives both the answer and the full evidence package. Nothing is discarded.

What the Output Looks Like in Practice

When evidence is strong the tool prints something like:

text

=== Answer ===
The dual-stage cooling design reduces peak VRAM temperature by 12 °C under continuous load
[source: hardware-notes.txt].

=== Evidence ===
- hardware-notes.txt (score=0.78)
  ...dual-stage cooling design reduces peak VRAM temperature by 12 °C...
- hardware-notes.txt (score=0.71)
  ...continuous load test results...

When evidence is weak it prints:

text

=== Answer ===
The supplied documents do not contain enough information to answer this question reliably.
(The highest relevance score was 0.29.)

=== Evidence ===
- unrelated-doc.txt (score=0.29)
  ...

The second case is the one most hidden pipelines never show you. Making it visible changes how you debug.

Threshold Tuning Without Over-Fitting

I do not treat the weak-evidence threshold as sacred. On a new document collection I:

1. Run 20–30 questions whose answers I already know are present.

2. Run 20–30 questions whose answers are absent.

3. Look at the score distributions.

4. Choose a threshold that keeps most “present” questions above it and most “absent” questions below it.

Then I keep that threshold in a config file next to the model revision. When the embedder or the document set changes, I re-check the distribution instead of assuming the old number still works.

Common Context Problems the Transparent Layer Surfaces

None of these diagnoses are possible if the chunks are thrown away after generation.

A Note From Real Use

I once spent the better part of a day tuning prompts on a RAG prototype because the answers kept drifting. The moment I forced the tool to print the retrieved chunks, the problem became obvious: the embedder was consistently returning a high-scoring but off-topic section that happened to share a few keywords. The model was doing exactly what it was told—conditioning on the supplied context. The context was simply the wrong context. Fixing the chunking and the embedding model took less time than the prompt experiments I had already run. I have not shipped a retrieval step without visible evidence since.

Integrating With the Earlier Tools

This layer slots cleanly on top of the document Q&A script from the previous post. It also works with the structured-output validation loop: retrieve, build messages, generate, then validate the JSON if you need machine-readable answers. Because every stage returns its intermediate results, you can log the full chain for any later failure.

Closing

Retrieval does not fail loudly. It fails by quietly supplying the wrong evidence and letting the model sound confident. The cheapest defense is to stop hiding the context. Surface the chunks, surface the scores, and make weak evidence an explicit signal in both the prompt and the user output.

Once that habit is in place, the Arcee model becomes far easier to trust—and far easier to debug when something goes wrong.

Test it. Understand it. Then ship it.

— Theo Calder

Independent AI developer, San Francisco