Running open-weight models inside a dedicated VPC, or signing a zero-data-retention agreement with a cloud API provider, solves one specific problem: keeping external vendor training jobs from ingesting your company's prompts. Still, isolating the model endpoint is not sufficient on its own to keep sensitive records safe. How documents reach the prompt—and where diagnostic logs go afterward—matters just as much.
When we review production private AI and Retrieval-Augmented Generation (RAG) systems, data exposures almost never happen inside the model weights. Where do they happen? Usually in the surrounding application plumbing. An ingestion connector flattens SharePoint or Google Drive permissions into a single collection, a similarity query pulls chunks across departments or tenants, an observability agent forwards unredacted prompts to an external log platform, or a tool-calling worker runs with open internet egress. Closing those gaps means looking carefully at ingestion, vector search authorization, telemetry, and outbound tool access.
Where Ingestion Pipelines Drop Permissions and Expose Vectors
Typical RAG pipelines pull files from SharePoint, Google Drive, Confluence, ticketing systems, and S3 buckets, split documents into 300- to 1,000-token chunks, generate dense vector embeddings, and write both text and vectors into an index. That pipeline is easy to prototype in an afternoon. It is also where permissions most often disappear.
Source repositories usually restrict M&A folders, board packets, and payroll spreadsheets to specific Entra ID or Google security groups. Yet it is not unheard of for an ingestion job to authenticate with an all-seeing service principal, dump every file into one flat vector collection, and throw away the underlying access control lists (ACLs) along the way. Even connectors that copy group IDs on day one sometimes ignore permission-revocation events later, leaving old chunks searchable weeks after someone leaves a project.
Two other ingestion habits cause quiet trouble. Support transcripts, clinical notes, and billing exports regularly carry Social Security numbers, payment card numbers, or raw API keys that add nothing useful to semantic search; running deterministic regex and named-entity scrubbers such as Microsoft Presidio before chunking tokenizes those identifiers before they ever reach the index. Teams also occasionally encrypt raw document text while treating 1,536-dimensional float vectors as harmless math. Published embedding-inversion research has shown that someone with read access to dense embeddings can reconstruct substantial portions of the original tokens, so vector indexes belong in private subnets under customer-managed KMS encryption at rest and strict tenant isolation, just like your primary relational tables.
Filtering Access Inside the Vector Query
Once an unauthorized document chunk enters a language model's context window, access control has already failed. Putting a line in the system prompt such as "Do not reveal executive compensation if the caller is outside HR" will not hold up against direct or indirect prompt injection, and even an ordinary summary can leak directional facts pulled from restricted chunks.
What about running an unfiltered nearest-neighbor search for LIMIT 10 and stripping out unauthorized rows in application code afterward? That keeps restricted text hidden, though it creates an availability bug of its own: whenever restricted documents dominate the top matches, low-privilege callers get back zero results even when plenty of permitted documents exist slightly further down. Instead, store each document's tenant_id, authorized group or principal IDs, and last ACL sync timestamp right alongside the vector payload, and enforce authorization inside the query itself:
-- Pre-retrieval authorized vector search in PostgreSQL (pgvector 0.8+ with iterative HNSW scan)
SELECT chunk_id, document_uri, chunk_text
FROM rag_document_chunks
WHERE tenant_id = $2
AND allowed_principals && $3::text[]
AND acl_synced_at > now() - INTERVAL '15 minutes'
ORDER BY embedding <=> $1::vector
LIMIT 8;
With iterative HNSW scanning in pgvector 0.8+ or filtered payload indexes in Qdrant, the database checks metadata predicates while walking the graph so callers still receive a full set of authorized matches without cross-tenant bleed. Notice the acl_synced_at freshness check as well. If your SharePoint or Drive permission sync job stalls overnight, stale records fail closed until permissions re-verify.
Hosting Choices and Telemetry Redaction
Maybe the question isn't only where inference runs, but also what happens to every prompt on its way through your logging stack? Engineering teams regularly spend weeks weighing self-hosted GPUs against regional cloud APIs, only to leave an APM or LLM tracing SDK forwarding raw prompts straight to an external SaaS dashboard.
| Architecture Factor | Dedicated VPC / On-Premises GPUs (vLLM, TGI) | PrivateLink Cloud APIs (AWS Bedrock, Azure OpenAI) |
|---|---|---|
| Compute and network path | Inference runs entirely on dedicated GPUs inside your VPC or rack with zero external API transit | Requests stay on AWS PrivateLink or Azure Private Endpoint to regional provider-managed accelerators |
| Prompt and abuse-cache retention | Controlled entirely by your engineering team; ephemeral in-memory KV cache cleared per request | Excluded from model training by default; verify human abuse-review opt-out or Zero Data Retention terms on your account |
| Operational overhead | Higher: CUDA and container patching, quantization testing, KV-cache sizing, and standby GPU capacity | Lower: stateless HTTPS calls with managed multi-AZ capacity and per-token billing |
| Typical workload fit | Strict data-residency enclaves, air-gapped environments, or steady high-volume batch processing (see running AI on private hardware) | Spiky interactive internal copilots requiring frontier reasoning under an enterprise DPA or HIPAA BAA |
Whichever runtime fits your workload, inspect your tracing pipeline before launch. OpenTelemetry GenAI instrumentation and LLM observability SDKs often capture gen_ai.prompt, gen_ai.completion, and retrieved chunks verbatim by default. Attach a redaction processor inside your collector before spans leave the host, and keep debug trace retention short—typically 7 to 30 days:
// OpenTelemetry SpanProcessor scrubbing sensitive prompt/completion attributes before export
onEnd(span) {
for (const attr of ['gen_ai.prompt', 'gen_ai.completion', 'rag.retrieved_chunks']) {
if (typeof span.attributes[attr] === 'string') {
span.attributes[attr] = redactSecretsAndPII(span.attributes[attr]);
}
}
}
Network Egress and Permissions for Agent Tools
Once an assistant goes beyond answering questions and starts calling internal APIs, SQL reporting views, or Model Context Protocol (MCP) servers, treat the orchestrator runtime as an untrusted execution tier. Run orchestrator containers and tool workers inside private subnets with security groups that deny arbitrary 0.0.0.0/0 outbound internet access, routing approved API destinations through an explicit allowlist proxy so a prompt injection cannot quietly post retrieved text to an outside server.
Database and API credentials deserve the same restraint. If an assistant runs natural-language-to-SQL or analytics queries, point it at a read-only database role scoped with PostgreSQL Row-Level Security and a tight statement timeout (SET statement_timeout = '3000ms') rather than handing it the application's primary read-write connection pool. And instead of wiring downstream tool handlers to a single broad service token, pass the signed-in employee's short-lived On-Behalf-Of (OBO) or session identity down the call chain so the assistant cannot read or change anything the human caller could not touch directly.
Pair these technical controls with an employee AI acceptable use policy and our LLM observability and token cost guide, or see how briskData builds and manages private AI infrastructure.