gerenbench - A Small Proprietary Benchmark for German RAG

Retrieval Augmented Generation (RAG) is quickly maturing into a useful (and boring) technology to connect proprietary, often local data to Large Language Models (LLMs) to augment their training data with knowledge and skills through in-context learning. RAG can enable important use cases such as tailored AI assistants for answering questions, generating complex reports ("Deep Research") or automating tasks ("Agentic AI").

RAG employs an information retrieval system, like a database or search engine, to augment the context of an LLM with relevant information. There are many different ways of varying complexity to achieve this, from simple approaches that enrich a user’s prompt with search results every time to agentic workflows that enable an LLM to freely use multiple search engines or databases as tools. When connecting an LLM to proprietary data, it is extremely important to ensure that it cannot access public data sources at the same time, to prevent data exhilaration.

This article presents results of "gerenbench", a small proprietary benchmark that uses German encyclopedia articles to estimate the accuracy of LLMs on RAG tasks. It provides a glimpse at the native and RAG performance of open weight LLMs runnable on consumer hardware as of March 2026.

Benchmark Setup

The benchmarks contains 100 multiple choice questions based on 25 German encyclopedia articles from to mid-nineties. Each question contains the relevant article and four possible answers. It encompasses the following two scenarios:

1. LLM-Only: The LLM is presented with the question and four possible answers. The relevant article is withheld.
2. LLM+RAG: The LLM is presented with the relevant article, the question and four possible answers.

In both scenarios, the LLM is instructed (via the prompts given in the appendix) to answer the question by generating a single letter (A, B, C, or D) designating the correct answer. Therefore, scenario 1 tests world knowledge, while scenario 2 tests world knowledge and RAG capability. Result analysis is fully deterministic and automated. A redacted example task is given in the appendix of this article. Note that scenario 2 assumes the best case scenario where the relevant article is included in the LLMs context exclusively every time.

All experiments were run on an Apple M3 with 24 GB of unified RAM using Ollama Version 0.17.4.

Results

The following 10 open weight LLMs where tested on both scenarios (Table 1):

Table 2 shows mean accuracies and processing times for scenario 1, i.e. LLM-Only performance, without RAG. "Prompt Duration" denotes the mean time in seconds taken to process the prompt, "Generate Duration" denotes the mean time in seconds taken to generate the answer. Best results are shown in bold font.

Table 3 shows mean accuracy and processing times for scenario 2, i.e. RAG. Best results are shown in bold font.

Figure 1 shows the accuracies of all tested LLMs for scenario 1 (LLM only) and scenario 2 (LLM+RAG). 95% confidence intervals, generated via bootstrap sampling, are included.

Finally, figure 2 shows a Pareto plot of LLM accuracy vs. generation time.

Conclusions and Next Steps

The results show that mid-size, open-weight LLMs today are capable of solving the benchmark task with over 95% accuracy (an error rate of less than 1 in 20 queries). In contrast, when relying solely on internal world knowledge, even the best models tested are only about 75% accurate, producing an incorrect result for 1 out of 4 queries, on average. If an accuracy of 95% is good enough should be evaluated against the risk profile of the task at hand.

Result accuracy scales with model size, while there are still significant differences between models of comparable size, demonstrating the value of creating custom (proprietary) benchmarks. Model runtime is highly dependent on the LLM runtime (hard- and software used for inference), but also important in practice, which is why it is reported in the benchmark results.

Currently, the RAG portion of this benchmark (scenario 2) assumes perfect information retrieval, i.e. that relevant information is present every time, which is rarely the case in practice. Future iterations could be improved by including a quota of test cases where relevant information is missing. Furthermore, to move beyond the current multiple-choice format, a third scenario could be added to test free-form generation accuracy. This would involve removing multiple-choice options from the prompt template and evaluating the generated answers using pattern matching or an "LLM-as-a-judge" approach.

Appendix

All LLMs under test where instructed with the following prompt templates:

system_prompt = """
Your task is to answer multiple-choice questions with high precision and accuracy.
You must answer with the letter of the correct choice, i.e. either A,B,C, or D.
You must not generate anything else. 
"""

prompt_template = """**Question:**
{question}

**Multiple Choice Options:**
A) {answer_a}
B) {answer_b}
C) {answer_c}
D) {answer_d}

Answer the given question using one of the multiple choice options.
You must return either A, B, C, or D.
You must return exactly one letter.
---
"""

rag_prompt_template = """**Context:**
{context}

**Question:**
{question}

**Multiple Choice Options:**
A) {answer_a}
B) {answer_b}
C) {answer_c}
D) {answer_d}

Based on the given context, answer the given question using one of the multiple choice options.
You must return either A, B, C, or D.
You must return exactly one letter.
---
"""

The system_prompt gives general instructions to the model, while the prompt_template is used for the LLM-Only part of the benchmark (scenario 1) and rag_prompt_template is used for the LLM+RAG part of the benchmark (scenario 2).

The following shows a redacted example task of the benchmark:

Article
Thyssen AG, Dachgesellschaft eines in den Bereichen Investitionsgüter, Handel und Dienstleistungen und Stahlerzeugung und -verarbeitung tätigen Konzerns; Sitz: Düsseldorf. Die T. AG entwickelte sich, v.a. aus den beiden Hauptunternehmen Thyssen & Co. KG (gegr. 1871) und August Thyssen-Hütte AG (gegr. 1890), bis 1914 (...redacted...)

Question
Durch wen sollte die (...redacted...)?

Possible Answers
a) (...redacted...)
b) (...redacted...)
c) (...redacted...)
d) (...redacted...)