Self-Query Retriever
Some of what a question asks for is a subject, and some of it is a constraint. Similarity search can only answer the first half — unless you pull the constraint out first.
Overview
The idea in brief
Similarity search compares meaning, and that is all it does. Ask it for "papers about attention published after 2020" and the words published after 2020 are treated as more subject matter to match, not as a rule to enforce. A 2017 paper that is squarely about attention will score well and be returned, because nothing in a cosine score knows what a year is.
Ask
off = plain similarity search over the whole corpus
Top result
1 — The question, split in two
—2 — Filter, then rank what survives
Where the parse comes from
In a real self-query retriever an LLM turns the sentence into that filter, given a description of the metadata schema. The three parses here are written by hand as a labelled example of what it would emit — no model runs in this page. The filtering and the ranking below are really computed.
Self-Query Retrieval: A Practical Guide
Half the sentence is a subject. Half of it is a WHERE clause.
Two questions in one sentence
A self-query retriever hands the sentence to an LLM along with a description of the metadata each document carries, and asks it to return two things: the semantic query — the part that should be embedded and compared — and a structured filter over the metadata, in the form of real comparisons like year > 2020. The filter runs first as an ordinary database-style predicate; similarity then ranks only the documents that survived it. The constraint becomes a guarantee rather than a hint.
Exploration guide
- Read the parse for the first query. Note what was removed from the semantic half: only attention is left to match on, because the date has become a filter instead.
- Turn the filter off. This is plain similarity search over everything. A 2017 paper takes the top spot — it is genuinely the best match for "attention", and it breaks the constraint the reader stated.
- Turn it back on. The 2017 paper is struck out with the reason it failed, and the top result is now both about attention and inside the requested date range.
- Try "short surveys on retrieval". Two constraints this time, on different fields and different types — a number and a category — combined with AND.
- Try "anything about attention". No constraint in the sentence, so the filter is empty and the retriever degrades gracefully into ordinary similarity search.
The short of it
Embeddings compare meaning and cannot enforce a rule, so any constraint left inside the query text is at best a weak hint. Self-querying separates the sentence into the part worth embedding and the part worth executing as a filter, which turns "after 2020" from a phrase competing for cosine similarity into a predicate that simply removes the documents that fail it. The trade is an extra LLM call before retrieval, and a hard dependency on documents actually carrying the metadata the filter names.