What we
measured

RAG answers a question in two steps: search your own text for passages, then give them to a language model. The answer can only be as good as the passages, so this study measured the search. Chapter 18 of the documentation explains RAG from the start.

We tested chunk sizes, a heading on each chunk, two embedding models, three ways to combine keyword and vector search, two rerankers, and three ways of asking a language model to help the search. Then we gave a local language model a search tool and graded its answers. The table lists what we found and what we built because of it.

An email search application built on Inillucent was moving from an embedding model served by a separate program to the model Inillucent runs inside the database. Before rebuilding its index, we wanted to answer two questions. Would the switch lose quality? And which RAG techniques are worth building into Inillucent, so that every user gets them without writing them?

Most published advice on RAG comes from public collections of articles with questions written for them. A mailbox is harder. Many emails look alike, exact names and numbers matter, and people remember them only vaguely. We measured on a mailbox because that is the kind of text people most want to search.

The collection was one person's mailbox: 67,369 emails and attachments, cut into 558,429 chunks of about 900 characters. Building an index takes time, so most comparisons ran on a fixed sample of 15,000 documents, and the important ones were checked again on the whole mailbox. The sample kept every email a question was written from, the rest of its thread, and random documents. That way the right answer always had many similar emails around it, as it does in a real mailbox.

Claude Opus 5.5, a language model, wrote 119 questions from real emails, the way a person would type them months later from memory, without copying the email's own phrases. They came in three styles: a few keywords, a full question, and a vague recollection such as "that thing someone sent about the move". They came in three kinds: a fact in an email, a fact in an attachment, and a topic spread across several emails.

The questions were split at random into a dev half of 60 and a test half of 59. Every decision was made by looking at the dev half only. The test half was scored and reported, and never used to choose. A technique that helps on the dev half and not on the test half was tuned to those particular questions and would not help a real user. The agent result shows why this matters.

The mailbox is private, so the questions and the emails are not published. The method, the scripts and every number came from one run on 29 September 2026.

To score a search, you need the right answers to each question. For every question, the top documents from every technique were collected into one pool. Claude Opus 5.5 then graded every question and document pair: 2 if the document answers the question, 1 if it is related, 0 if it is not. It graded 11,138 pairs.

A grader that is wrong makes every number wrong, so the grader was checked three ways. It graded the email each question was written from as 2 for 119 of 119 questions. It graded randomly chosen emails as 0 in 237 of 237 cases. Asked again about 1,480 pairs, it gave the same grade 96.8% of the time. A person did not check the grades.

One problem was found and fixed during the study. On topic questions the grader leaned too much on the single example answer it was shown. Its instructions were changed to say that the example is one of several right answers, and every topic pair was graded again.

Each score looks at the top 10 or top 5 results of one search for one question, and is averaged over the questions. nDCG@10 is the main score.

With 119 questions, two techniques can score differently by chance. So every comparison has a 95% interval. It comes from drawing the questions again at random 2,000 times, with repeats allowed, and measuring the difference each time. If the whole interval is above zero, the technique helped. If the interval crosses zero, the study could not tell the two apart, and this page says the difference is inside the noise.

The charts on this page draw each difference as a point with its interval. Amber is a gain, red is a loss, and grey is inside the noise.

Each index was its own database file with one inillucent_search table, and each one differed from the production index in one thing. Every index was searched with every strategy: keyword search alone, vector search alone, hybrid search with each of three fusion rules, each of those followed by a reranker, query rewriting and HyDE. A document's rank was the rank of its best chunk.

Each row compares one technique with the setup it changes, on the 15,000 document sample. The sections after this one explain each result and what it led to.

An embedding model reads the question and each chunk separately. A reranker reads the two together and scores how well the chunk answers the question. Reranking the top 60 chunks with bge-reranker-v2-m3 raised nDCG@10 by 0.106 over hybrid search on the sample, with an interval of +0.070 to +0.147. The smaller gte-reranker-modernbert-base raised it by 0.101, and the two rerankers were tied with each other. On the whole mailbox, hybrid search scored 0.635 and reranking over RRF scored 0.758. The gain was largest for keyword style questions, vague recollections and answers inside attachments.

We tried reranking the top 30, 60 and 100 chunks. On the dev half, 30 scored 0.802, 60 scored 0.832 and 100 scored 0.836. 30 lost part of the gain. 100 added nothing measurable and took longer.

A reranker costs time at every search, and on a processor it costs a lot. The table has the median time to rerank one set of candidates. The processor runs used 8 threads.

Each chunk began with its email's subject, sender and date. Removing that heading cost 0.043 nDCG@10 with hybrid search, with an interval of -0.083 to -0.002, and 0.111 with a reranker, with an interval of -0.148 to -0.073.

The loss is larger with a reranker because the reranker reads only the chunk. A chunk that does not say which email it came from gives the reranker less to judge by.

Hybrid search beat keyword search alone by 0.038 nDCG@10, with an interval of +0.021 to +0.058. Against vector search alone the gain was +0.005, inside the noise, but hybrid search also keeps the exact name and number matches that vector search misses.

How the two lists are combined mattered. Against Inillucent's default adaptive weight, a fixed weight of 0.5 gained 0.040 and RRF gained 0.035. On the whole mailbox the adaptive weight scored 0.635, RRF 0.667 and a fixed 0.5 scored 0.672.

Inillucent's own score card uses a different collection, with questions about document titles. There the adaptive weight won: nDCG@10 0.982, against 0.917 for RRF and 0.955 for a fixed 0.5. With a reranker the rule mattered much less. On the whole mailbox, reranking over RRF scored 0.758 and reranking over the adaptive weight scored 0.749, inside the noise.

Against chunks of 900 characters, chunks of 450 scored -0.014, chunks of 2,000 scored -0.009 and chunks of 6,000 scored -0.003. With a reranker, 6,000 scored +0.019. Every one of these intervals crosses zero.

Long chunks have a cost that the scores do not show. The embedding model reads at most 1,900 tokens, about 7,000 characters of English, and ignores the rest without a message. 19.5% of the 6,000 character chunks were cut.

We compared nomic-embed-text-v1.5, which Inillucent runs inside the database, with Nomic v2, a newer model served by a separate program. On the sample the difference was -0.002 nDCG@10, with an interval of -0.017 to +0.014. On the whole mailbox it was +0.004, with an interval of -0.011 to +0.019. There was no measurable difference. Hybrid searches were faster with the built in model, 72 ms at the median against 96 ms, because there is no call to another program.

The cost of embedding was the real problem. embed() in SQL embeds about 13 chunks a second on the processor, so the whole mailbox would have taken about 12 hours. The same model on a graphics card took 22 minutes and gave the same vectors, with a cosine similarity of 0.9999994 between the two.

Query rewriting asks a language model for three other versions of the question and searches for each. It changed nDCG@10 by +0.002, with an interval of -0.031 to +0.034. HyDE asks the model for an imagined answer and searches for chunks near it. It changed nDCG@10 by +0.005, with an interval of -0.034 to +0.046. Both are inside the noise.

Document summaries ask the model for a one sentence summary of each document and add it to each of the document's chunks. They added 0.063 to hybrid search, with an interval of +0.024 to +0.104. With a reranker they added 0.023, inside the noise. Writing a summary for every email in the mailbox would take about 12 hours of the local model's time.

Search scores measure the search, and a user sees the answer. So we gave a local language model two tools, search_email and read_document, and asked it every question twice with each setup. Claude Opus 5.5 graded each answer: whether it was correct, whether it was complete, whether every claim was backed by a document the agent had read, and whether it cited the right document.

On the dev half the pass rate rose from 0.658 to 0.767 over three rounds: the built in embedding model, then reranking, then a better instruction prompt written after reading the failed answers. On the test half, the full new setup scored 0.780 against 0.771 for the old one. That change of +0.008 has an interval of -0.068 to +0.085, so it is inside the noise.

The agent made 0.49 fewer tool calls per answer with the new setup. A capable agent makes up for weaker search by searching again and reading more documents. So better search showed up as less work for the agent, and it did not show up as more correct answers.

After the first set of comparisons, we changed one setting at a time, scored it on the dev half, and kept it only when it beat the best setup so far by more than the noise. The test column was never used to choose. It shows which gains held on questions the choice never saw.

Round 10 was kept by that rule, but its gain was inside the noise, and 2,000 character chunks scored higher on the test half. So we took no chunk size lesson from it.

Every feature below exists because of a result on this page. Each row links to that result.

The same study on a different collection may find different sizes of effect. The two largest effects here, the reranker and the heading, agree with published work on other collections. The small effects, such as chunk size, are the ones most likely to differ on other data.

• It used one collection, a private mailbox, so nobody else can run it on the same data.
• A language model graded every pair. It passed three checks, and a person did not check its grades.
• 119 questions is enough to see the large effects. For most comparisons, a difference smaller than about 0.03 fell inside the noise.
• A small or quantized reranker over fewer candidates on a processor was not measured. It is the next thing to measure for machines with no graphics card.