RAG audit trail: the part everyone skips

Teams building a RAG audit trail almost always log the question and the answer, and almost always skip the part that matters: the retrieval reads in between. That is where the system actually touches your data, and it is the part an investigation needs. A RAG audit trail that records the chat but not the reads is watching the wrong end of the pipeline.

The read is the action

In a RAG system, the consequential event is not the model's reply. It is the retriever pulling chunks from your index. If those chunks came from documents the user was not allowed to see, the leak already happened, regardless of what the model said next. The read is the action, so the read is what the audit trail has to capture, tied to the identity that asked.

What the skipped record contains

• The requesting identity behind each retrieval, not a shared service account
• Which documents or chunks were returned
• Whether the asker was actually entitled to those sources
• Sensitive values masked in the recorded text, so the log is not a copy of the data

Why it has to live outside the pipeline

Recording reads inside the RAG service has the usual flaw: the service can shape its own log. The architectural requirement is to capture the read at the boundary the retriever crosses to reach the data, under the asker's identity, in a record the pipeline cannot edit. That is one control surface: per-request identity, a policy check on each read, and a tamper-proof record. hoop.dev is built to it, running retrieval reads under a scoped identity, masking sensitive values inline, and writing each read as a command-level audit outside the pipeline. In practice you put the data the retriever reads behind hoop.dev, and the RAG audit trail finally covers the reads. The getting-started guide shows the first connection, and hoop.dev/learn explains identity-scoped reads.

What the skipped record catches

The reason the retrieval reads matter is not theoretical. Picture a support assistant built on RAG over your internal wiki, which has spaces only certain teams can read. A support agent asks a routine question, and the retriever, running under one service identity with full index access, pulls a chunk from a restricted incident space because it matched the query. The model folds it into a helpful answer. Nobody attacked anything, and your chat log shows a perfectly ordinary exchange. The only place that leak is visible is the read record, which would show a support identity retrieving a document it was never entitled to. Skip the reads, and you never see it.

The drift problem

There is a second reason the reads matter, and it grows over time. An index built once and queried for months slowly diverges from the real permissions on the source documents, because access changes after indexing but the embedded copy does not. A RAG audit trail that records reads under the asker's identity, checked against live permissions, is what surfaces that drift, because you can see retrievals succeeding against sources the user should no longer reach. Logging only the prompt and the answer hides drift completely, right up until it surfaces as an incident nobody can explain.

Try it on one index

hoop.dev is open source. From the GitHub repository, put one retrieval path behind it and record the reads, not just the chat.

FAQ

Is logging the prompt and response enough?

No. The retrieval reads are where data is actually touched. A trail without them misses the event that matters in a leak.

What about the vector store itself?

Treat it as sensitive and scope reads to the requesting identity. Embeddings can leak their source, so they need the same controls as the records.

Does recording reads slow retrieval down?

Capturing the read at the boundary adds little next to the retrieval and generation the pipeline already does. The cost you choose is masking and the policy check, both inline. The RAG audit trail you get in return is the only record that shows what the system actually touched, which is the part that matters in a leak or an audit, since the chat log shows nothing useful about which sources were read.