Permission filtering in RAG retrieval

Pre-filtering, and why it is harder than it looks

Pre-filtering restricts the candidate set before ranking, so the top k is k permitted results. That is the correct behaviour and it fights the index: an HNSW graph is built over all vectors, and walking it while skipping most nodes can disconnect the search — you traverse into a region where everything is filtered out and the walk stalls.

Vector databases handle this differently, which is one of the real differences between them. Options include filtered graph traversal, per-tenant sub-indexes (clean, and costly at many tenants), and falling back to a flat scan when the filter is very selective — which is often genuinely fastest, because the permitted set is small.

Where the permissions live

Baking an access list into each chunk's metadata at index time is fast and goes stale the moment someone leaves a group — and stale permissions fail open, which is the wrong direction.

The usual compromise: index a stable group identifier with each chunk, resolve the user's groups per request from the identity system, and filter on the intersection. Group membership changes are then reflected immediately without re-indexing anything.

Two more things to say. Deleted documents must leave the index, not just be filtered out. And the filter belongs on the server, derived from the authenticated session — never from a client-supplied user id, which is trivially forged.

Why pre-filtering fights the index

Pre-filtering is the correct behaviour and it is genuinely hard to implement, because an ANN index was built over the whole corpus and does not know about your predicate.

Walking an HNSW graph while skipping most nodes can disconnect the traversal: the search reaches a region where every neighbour is filtered out and stalls, returning far fewer results than asked for even though plenty of permitted documents exist elsewhere in the graph.

The strategies databases use, all with trade-offs: filtered traversal that keeps searching past excluded nodes; per-tenant sub-indexes, which are clean and expensive once tenants number in the thousands; and falling back to a flat scan when the filter is selective — which is often genuinely fastest, since a user with access to 0.1% of a million documents is a thousand-vector brute force.

Getting the model right, and failing closed

Two design rules cover most of the damage.

Derive the filter from the session, never the request. A client-supplied user or group id is trivially forged. The filter must be built server-side from the authenticated identity.

Index groups, resolve membership per query. Baking a user list into each chunk means a departure leaves the index granting access until the next re-index — and stale permissions fail open, which is the wrong direction. Indexing a stable group id and resolving the user's groups per request makes revocation immediate.

Also: filtering is not deletion. A document removed for legal or privacy reasons must leave the index and any cached answers derived from it, or it remains recoverable by anyone who can still see it — which is a different and worse problem than a missing result.

The failure mode that matters most

A RAG system retrieves documents and puts them in a prompt. If the retrieval does not respect who is asking, the model will faithfully summarise a document the user was never allowed to see — and it will do so fluently, with a citation.

That is a data breach with a helpful tone, and it is the most consequential thing that can go wrong in an enterprise retrieval system. Unlike a hallucination, it is not a quality problem; it is a security one.

The requirement is straightforward to state and easy to implement incorrectly: the set of documents considered must be exactly the set this user may read.

Where the filter has to be applied

There are three places it could go, and only one is correct.

After generation — check the answer for leaked content. Hopeless: the content is already in the output, and paraphrase makes detection unreliable.

After retrieval, before the prompt — retrieve k documents, discard the ones the user cannot see, pass the rest. Better, and still wrong in two ways: the unauthorised documents occupied slots, so fewer authorised ones were considered; and their content has already been loaded into the application's memory.

Inside the search — the index only ever considers documents matching the user's permissions. This is the correct answer, and it requires the store to support filtered search.

results = store.search(
    query_vector=q,
    k=10,
    filter={"acl": {"$in": user.group_ids}},   # applied during the search
)

The difference between the second and third options is not theoretical. With post-filtering, a user in a small group querying a corpus dominated by other groups' documents may get zero results while relevant authorised documents exist — because all ten retrieved slots went to documents they cannot see.

Modelling permissions in metadata

The filter can only express what the metadata records, so the model has to be decided at indexing time.

Group-based ACLs cover most cases and are what most source systems already provide. Store the list of group ids permitted to read each document, and filter on intersection with the requesting user's groups.

Two hard parts:

Inherited permissions. A document in a restricted folder inherits its restriction. The indexing pipeline must resolve inheritance and write the effective ACL, not the document's own.

Permission changes. When someone leaves a group, or a document is reclassified, the index is stale and will over-share until it is updated. Permissions change far more often than document content, so they need their own refresh path — ideally event-driven, and at minimum on a short schedule.

Post-filtering, measured, and why it fails quietly

The article says post-filtering quietly fails. That is a claim with a number, and the number is worse than most people expect. This measures it, then measures the pre-filtering approach that fixes it and the new problem that one creates.

import numpy as np

rng = np.random.default_rng(9)
N = 5000
K = 10

# a corpus where each document is visible to some fraction of users
print("A CORPUS OF %s DOCUMENTS. a given user can see some fraction of it:"
      % "{:,}".format(N))
print()
print("POST-FILTERING: retrieve the top %d by similarity, then drop the" % K)
print("ones the user may not see. what the user actually receives:")
print("%-22s %16s %16s %16s"
      % ("user sees", "results returned", "empty results", "fewer than %d" % K))
sims = rng.normal(0, 1, N)
order = np.argsort(-sims)
for frac in (0.90, 0.50, 0.20, 0.05, 0.01):
    trials, empties, short, total = 400, 0, 0, 0
    for _ in range(trials):
        allowed = rng.random(N) < frac
        got = sum(1 for i in order[:K] if allowed[i])
        total += got
        if got == 0:
            empties += 1
        if got < K:
            short += 1
    print("%-22s %16.2f %15.0f%% %15.0f%%"
          % ("%.0f%% of the corpus" % (100 * frac), total / float(trials),
             100.0 * empties / trials, 100.0 * short / trials))
print("   read the 5 percent row. that user asks a question and receives")
print("   about half a result on average, and gets NOTHING at all on most")
print("   of their queries -- not an error, not an empty page with an")
print("   explanation, just a thin and arbitrary subset of what they were")
print("   entitled to see.")
print("   and the failure is INVISIBLE from inside the system: the")
print("   retriever did its job, the filter did its job, every log line")
print("   says success, and nothing anywhere knows that the document the")
print("   user needed was sitting at rank 200.")
print()

print("HOW FAR DOWN WOULD YOU HAVE HAD TO GO? for a user who can see f of")
print("the corpus, reaching %d visible documents needs roughly %d/f:" % (K, K))
print("%-22s %20s %20s"
      % ("user sees", "needed k (expected)", "measured median"))
for frac in (0.90, 0.50, 0.20, 0.05, 0.01):
    needed = []
    for _ in range(200):
        allowed = rng.random(N) < frac
        c = 0
        for pos, i in enumerate(order):
            if allowed[i]:
                c += 1
                if c == K:
                    needed.append(pos + 1)
                    break
        else:
            needed.append(N)
    print("%-22s %20.0f %20.0f"
          % ("%.0f%% of the corpus" % (100 * frac), K / frac,
             float(np.median(needed))))
print("   'just retrieve more and then filter' is the obvious patch, and")
print("   the table shows what it costs: the required k is inversely")
print("   proportional to what the user can see, so the least privileged")
print("   users -- usually the most numerous -- pay the most.")
print("   worse, k has to be chosen in advance, and you do not know f")
print("   until you know the user.")
print()

print("PRE-FILTERING: restrict the search to what the user may see, THEN")
print("rank. the result is always %d documents and always the right %d:" % (K, K))
print("%-22s %16s %16s" % ("user sees", "results returned", "empty results"))
for frac in (0.90, 0.50, 0.20, 0.05, 0.01):
    trials, empties, total = 200, 0, 0
    for _ in range(trials):
        allowed = rng.random(N) < frac
        idx = np.where(allowed)[0]
        got = min(K, len(idx))
        total += got
        if got == 0:
            empties += 1
    print("%-22s %16.2f %15.0f%%"
          % ("%.0f%% of the corpus" % (100 * frac), total / float(trials),
             100.0 * empties / trials))
print("   correct at every privilege level, which is the whole argument.")
print()

print("SO WHY IS ANYONE STILL POST-FILTERING? because pre-filtering fights")
print("the index. an ANN index navigates a graph or visits clusters, and")
print("a filter removes nodes from underneath it:")
print("%-26s %14s %20s"
      % ("fraction visible", "candidates seen", "of which visible"))
NPROBE_SCAN = 400
for frac in (0.90, 0.50, 0.20, 0.05, 0.01):
    print("%-26s %14d %20.0f"
          % ("%.0f%%" % (100 * frac), NPROBE_SCAN, NPROBE_SCAN * frac))
print("   the index still visits about %d candidates -- that is what its"
      % NPROBE_SCAN)
print("   parameters say -- and at 1 percent visibility only %d of them are"
      % (NPROBE_SCAN * 0.01))
print("   eligible. the search degrades toward returning whatever it")
print("   happened to touch, so recall falls even though the answer is")
print("   'correct' in the permissions sense.")
print("   the three real answers, and none is free:")
print("%-30s %s" % ("raise ef_search / nprobe", "slower, per query"))
print("%-30s %s" % ("a partitioned index per group", "duplicated storage"))
print("%-30s %s" % ("filter inside the index walk", "needs store support"))
print()

print("AND THE PART THAT IS NOT ABOUT RANKING AT ALL. permissions change,")
print("and the index does not notice:")
print("%-40s %s" % ("a user is removed from a group", "at 09:00"))
print("%-40s %s" % ("your ACL snapshot was built", "at 03:00"))
print("%-40s %s" % ("what they can retrieve until re-index", "everything"))
print("   this is why permissions are resolved at QUERY time from the")
print("   source of truth, not baked into the index at write time. an")
print("   index is a cache, and a cache of an access-control decision is")
print("   a stale access-control decision.")
print()
print("   and the last one, which is easy to forget: FAIL CLOSED.")
print("%-34s %-14s %s" % ("if the permission service is", "post-filter", "pre-filter"))
for row in (("up", "correct", "correct"),
            ("down, failing open", "LEAKS", "LEAKS"),
            ("down, failing closed", "no results", "no results")):
    print("%-34s %-14s %s" % row)
print("   an empty result page is a bad afternoon. a leak is an incident,")
print("   and it will not appear in any latency graph or error rate --")
print("   which is why this is the one path in a RAG system that deserves")
print("   an explicit test that a user CANNOT retrieve a document, run on")
print("   every deploy.")

Things to try

1. Drop corpus visible to 5%. Post-filtering returns almost nothing while pre-filtering still returns k — the control fails worst for the most restricted user.
2. Raise over-fetch to 500. The shortfall shrinks and never disappears, which is why over-fetching is a mitigation rather than a fix.
3. Set visibility to 100%. All three strategies agree, which is exactly why this bug survives testing on an admin account.

What to remember

Permission filtering has to happen inside the search, not around it. Post-filtering ranks first and drops afterwards, so it silently returns fewer results than requested and degrades worst for the most restricted users. Filtering after generation is not a control at all — the model has already read the text. Index a group id and resolve membership per request, because stale permissions fail open.

Late-binding permissions

An alternative to storing ACLs in the index is to resolve them at query time against the source system.

Store only a reference — the source id — and after retrieving candidates, ask the source system which of them this user may read.

The trade is sharp: resolving at query time is always correct and forces post-filtering, which brings back the slot-wastage problem.

The pragmatic combination used in practice is both: filter inside the search using indexed ACLs to get a correct-enough candidate set efficiently, then verify the final selection against the source system before showing anything. The index filter does the heavy lifting; the verification catches the staleness window.

Failure modes to test for explicitly

Permission bugs do not announce themselves, so they have to be tested deliberately:

• A user with no groups should retrieve nothing, not everything. A missing or empty filter that matches all documents is the classic catastrophic bug.
• A filter on a field that does not exist should fail closed, not open. Some stores silently ignore unknown fields.
• Inherited restrictions on documents in restricted containers.
• Deleted users and revoked groups.
• Documents with no ACL recorded — decide whether the default is deny (correct) or allow (dangerous), and make it explicit.
• Cached results. A cache keyed only on the query text will serve one user's authorised results to another. Cache keys must include the permission context.

That last one is worth stating plainly because caching is added later, as a performance improvement, by someone who is not thinking about permissions.

What else leaks

Retrieval is the main channel and not the only one.

Conversation history carries content forward. If a user's permissions change mid-session, earlier retrieved content is still in the context.

Logs and traces. Prompt logging captures retrieved document content, so log storage inherits the documents' classification.

Embeddings themselves. Research has shown that text can be partially reconstructed from its embedding, so a vector store holding sensitive content is itself sensitive and needs the same protections as the documents.

Aggregate answers. A user may be permitted to see individual documents but not a synthesis that reveals a pattern across them. This is a policy question rather than a technical one, and worth asking.

Questions people ask

Can I filter after retrieval? It is not secure enough and it wastes retrieval slots. Filter inside the search.

What if my vector store does not support filtered search? Over-retrieve heavily and post-filter as an interim measure, and treat it as a reason to change store.

How do I keep permissions fresh? Event-driven updates from the source system, with a scheduled reconciliation as a backstop. Permissions change more often than content.

Should documents with no ACL be readable? Default deny. An explicit allow-all marker is safer than an absent field.

Does the embedding leak content? Partially reconstructable, so treat the vector store as holding the data itself.

How do I cache safely? Include the user's permission context in the cache key, or cache only at the query-embedding level rather than the results level.

Recap in one screen

• An unfiltered retrieval will faithfully summarise documents the user may not read.
• Apply the filter inside the search; post-filtering both leaks and wastes retrieval slots.
• Store effective ACLs at indexing time, resolving inheritance, and refresh them independently of content.
• Default to deny for documents with no recorded permissions, and test the no-groups case explicitly.
• Caches, logs, conversation history and the vectors themselves are all additional leak channels.