Direct answer
A private RAG system protects the full data path: ingestion, parsing, chunking, embedding, indexing, retrieval, prompts, outputs, logs, backups, and deletion. Running a model or vector database inside your own cloud is only one control. If retrieval can cross tenant or permission boundaries, the system is not private.
Retrieval creates a second authorization system
Traditional applications authorize access to records. RAG systems create derived objects—chunks, embeddings, summaries, caches, and traces—that must preserve the same access rules.
The dangerous shortcut is to filter after retrieval. By then, an unauthorized chunk may already be inside the model context or an execution trace. Access constraints should shape retrieval itself.
Each indexed object needs enough metadata to enforce policy: tenant, source record, owner, classification, permitted roles, version, retention state, and deletion state. The retrieval query must carry the authenticated user’s context and apply it before any text reaches the model.
The OCR and ingestion path matters
Private document systems often focus on the chat interface while treating ingestion as plumbing. But files may pass through OCR services, temporary storage, queues, antivirus tools, parsers, embedding providers, and observability platforms.
Map every processor and answer:
- Where are raw and derived files stored?
- Which external provider receives content?
- How long do temporary artifacts remain?
- Can one tenant’s job be consumed under another tenant’s context?
- Does deletion remove chunks, embeddings, caches, and backups appropriately?
- Can the team prove which document version supported an answer?
In a Zenveus presales engagement for a privacy-first, self-hosted RAG MVP, buyers asked about OCR routing, hard multi-tenancy, deployment documentation, and runbooks. Those questions were more important than the chat UI because they defined whether the system could be trusted with private documents.
Citations are necessary but not sufficient
A source citation helps a user verify an answer. It does not prove that the user was allowed to retrieve the source, that the source was current, or that omitted evidence would change the result.
A production response should retain source identifiers, relevant passages, versions, retrieval filters, and the authorization context used at that moment. For sensitive workflows, it should also distinguish “no supporting evidence found” from “the user does not have permission to search all possible evidence.”
Self-hosted versus managed is not the first decision
The first decision is the required control boundary. A managed model may be acceptable when contractual data handling, retention, regional processing, and access controls satisfy the risk. A self-hosted stack may still be unsafe if identity, tenant isolation, secrets, logging, and deletion are weak.
Choose deployment after defining the data-flow and threat model. Do not use “self-hosted” as a substitute for architecture.
A practical privacy review
Trace one sensitive sentence from upload to answer and deletion. List every system that stores, transforms, transmits, logs, or caches it. Then test the same path with two tenants and two roles. Attempt cross-tenant retrieval, stale-permission retrieval, deleted-document retrieval, and access through logs or support tools.
If the team cannot perform that trace, it cannot make a strong privacy claim.
Related Zenveus services: Agentic AI Development and AI Prototype Hardening