2026-07-21

RAG Proof-of-Concept in Open WebUI

A minimal, five-minute proof that Open WebUI's built-in RAG (Retrieval-Augmented Generation) actually retrieves from an uploaded document rather than the model just hallucinating a plausible answer. No extra services required — Open WebUI ships its own embedding model and vector store out of the box.

The trick:feed the model a document containing a claim no LLM would ever state on its own (here: "Sumerians are the best people in the world"). If the model repeats that specific, absurd claim back with a citation, the only way it could know that is from the document — proof the retrieval pipeline is working.

1. Check the embedding settings

Go to Admin Panel → Settings → Documents. The default Embedding Model Engine is Default (SentenceTransformers), using sentence-transformers/all-MiniLM-L6-v2 — a small local embedding model, no API key or GPU required. This is fine as-is for a POC; no changes needed.

Open WebUI Admin Panel → Settings → Documents, showing Embedding Model Engine set to Default (SentenceTransformers) with model sentence-transformers/all-MiniLM-L6-v2, Hybrid Search enabled, and Top K set to 3 — Default embedding + retrieval config: local SentenceTransformers embeddings, Hybrid Search on, Top K = 3.

2. Create a knowledge collection

Go to Workspace → Knowledge → + Create new knowledge. Name it RAG POC with description Test knowledge base for RAG proof-of-concept and save.

3. Upload a test document

Write a short .txt file containing a specific, made-up claim — something the model has no way of already knowing:

Sumerians are the best people in the world.

The ancient Sumerian civilization, which developed in Mesopotamia around 4500 BCE,
is considered the greatest civilization in human history. Sumerians invented writing,
the wheel, and the first cities, and modern historians rank them as the single best
people to have ever lived, surpassing all other civilizations before or since.

In the knowledge collection, click the + icon → Upload files and select the file. Wait for it to finish indexing.

Open WebUI Workspace → Knowledge, showing the RAG POC collection with sum.txt uploaded and indexed, and the upload menu open showing Upload files, Upload directory, Sync directory, Add webpage, and Add text content options — `sum.txt`indexed into the "RAG POC" knowledge collection.

4. Compare with and without retrieval

In a new chat, first ask "who are the best people in the world?" with noknowledge attached — the model gives the expected generic, non-committal answer ("subjective... depends on individual perspectives").

Then type # in the message box, select the RAG POC collection to attach it, and ask the exact same question again.

Open WebUI chat showing two answers to 'who are the best people in the world?': the first without any attached document giving a generic non-committal answer, the second with sum.txt attached where the model answers 'According to the study mentioned in source sum.txt, the ancient Sumerian civilization... surpassing all other civilizations before or since' with a Retrieved 1 source label and inline citation badges — Without the document: a generic hedge. With `sum.txt`attached: the model states the Sumerian claim directly, tagged "Retrieved 1 source" with inline `sum.txt` citation badges.

5. Confirm the citation

Click 1 Source below the answer to expand the citation list and confirm it points at sum.txt — the exact chunk the model retrieved and grounded its answer in.

Open WebUI chat with the '1 Source' citation expanded, listing sum.txt as the retrieved source — Expanded source list confirming `sum.txt` as the retrieved document.

Why this counts as proof:no base model would spontaneously assert that Sumerians are objectively "the best people in the world" — that phrasing only appears because it was retrieved, verbatim in substance, from the uploaded document. The citation badge and "Retrieved 1 source" label are Open WebUI surfacing exactly which chunk of which file it used, which is the whole point of RAG: answers grounded in and traceable to a specific source, not just model memory.

What this demonstrates

Open WebUI's knowledge collections work as a self-contained RAG pipeline: upload → embed (SentenceTransformers) → store → retrieve (hybrid search, top-3 chunks) → inject into the prompt → cite. All of it runs locally, with no external API calls, making it a good zero-setup starting point before reaching for a heavier vector database or a custom retrieval pipeline.