How I built a RAG on the Bhagavad Gita in Telugu

Ask a general chatbot a question about the Bhagavad Gita in Telugu and you will usually get a fluent answer. But it comes from the model's memory: a mix of the Gita, commentaries, and everything else it read on the internet. There is no citation, and no way to check whether the Gita actually says that.

I wanted the opposite. A system that answers only from the text of the Gita, shows which verse each point came from, and says "I can't answer that from the Gita" when the verses don't support an answer.

That is a RAG problem.

Here is the finished pipeline. The rest of this article explains why each box exists.

01Preparing the dataset

The corpus I wanted vs. the corpus that exists

The original plan was the Ramayana, the Mahabharata and the Gita. For a Telugu RAG, each verse needs its tatparyam: the meaning explained in modern Telugu. That is the text people's questions will actually match.

An audit of Telugu Wikisource ended that plan on day one. Of the Ramayana's 13,693 slokas, only 64 had a Telugu tatparyam, about 0.5%. The rest was Sanskrit written in Telugu script. It looks like Telugu to a simple character check, but it isn't Telugu prose. The Mahabharata was in the same situation.

Only the Gita had a Sanskrit verse and a Telugu meaning for every verse. So the project became: do one book properly.

Three page formats, one schema

The Gita pages on Wikisource (licensed CC BY-SA 4.0) were downloaded through the MediaWiki API. The 18 chapters were not written in one consistent format, so the parser had to handle three:

Every verse became one JSON record with the same fields. A trimmed example:

{
  "id": "gita-18-6",
  "adhyaya_no": 18,
  "sloka_no": 6,
  "citation_en": "Bhagavad Gita 18.6",
  "sloka_sa": "…",                       // Sanskrit, in Telugu script
  "tatparyam_te": "అర్జునా! ఈ కర్మలను కూడా సంగాన్ని, ఫలాన్ని వదిలి
                   చెయ్యాలని నా నిశ్చితమైన ఉత్తమమైన అభిప్రాయం.",
  "source_url": "https://te.wikisource.org/…",
  "license": "CC BY-SA 4.0",
  "parse_mode": "B",
  "verse_number_verified": false
}

The id matters more than it looks. It is what every citation points to, so it must be stable and correct.

Checking the count

This edition has 701 verses. After parsing and removing duplicates I had 699. 16 of the 18 chapters matched the expected verse count exactly. The two gaps are in the source itself, not bugs in the parser:

• Chapter 3: the source page labels two different verses as 3-4, so there is no 3-5.
• Chapter 16: verse 16-4 is missing from the page (it jumps from 16-3 to 16-5).

Cleaning: let the search results find the dirt

The first retrieval test surfaced something odd. For an unrelated question, the top result was verse 1.13, and its "meaning" started with ఆదిశంకరాచార్యులవారు : ("Adi Shankaracharya says:"). Commentary text had leaked into the verse records.

Looking closer, there were two kinds of contamination:

• Commentary leakage: 17 records in chapter 1 had Shankaracharya's commentary appended, averaging 946 characters each, against a normal meaning of about 120. A long record matches almost any query, so these kept winning.
• Sanskrit bleed: 30 records across 7 chapters had Sanskrit text stuck onto the end of the Telugu meaning.

The cleaning script cut each record at the commentary marker or the Sanskrit danda (|). In total it cleaned 37 records and removed 15,247 characters of junk. Automated tests in the repo now fail if either problem comes back.

After cleaning, a Telugu meaning is short: 127 characters on average (median 118). Keep that number in mind; it matters in section 4.

02Why not just give all the verses to the LLM?

This is the first question anyone should ask, because modern models have very large context windows. So I measured it.

All 699 verses, Sanskrit plus Telugu meaning, come to 114,471 tokens on Claude Sonnet 5. The context window is 1 million tokens. It fits. So why build retrieval at all?

Five reasons pushed me to RAG anyway:

1. Cost per question. Sending the whole Gita costs about $0.23 of input per question on Sonnet 5 ($2 per million input tokens). RAG sends about 8 verses, roughly 1,300 tokens, which is about $0.003. That is around 90 times less. Prompt caching would make the repeated part cheaper, but the model still has to work through a very long prompt for every question.
2. It doesn't scale to the rest of the plan. The long-term goal includes the Ramayana and the Mahabharata. If every Ramayana sloka had a Telugu meaning like the Gita's, that book alone would be over 2 million tokens (a rough estimate), past the context limit.
3. Citations become checkable. With RAG, I know exactly which 8 verses the model was given. Every quote in the answer can be checked, character by character, against those 8 verses. With the whole book in the prompt, it becomes much harder to tell the text apart from the model's own memory of the Gita.
4. Attention thins out. Models tend to use information less reliably when it is buried in a very long prompt. A short, focused context is easier to answer from faithfully.
5. You can measure each part. With RAG, "did we find the right verse?" and "did the model answer well from it?" are separate questions with separate numbers. When something breaks, you know where.

To be clear: I did not build a long-context version and compare answer quality. The reasons above are cost measurements plus design arguments, not a head-to-head benchmark.

03Choosing the embedding model

What the model needed

• Understands Telugu, including romanized Telugu ("Tenglish") like kopam ela puttundi, which is how many people type on phones.
• Open weights, so it can be fine-tuned.
• Runs locally, with no per-query API bill.
• Has a matching reranker from the same family (more on that in section 5).

The shortlist

Candidates were BGE-M3, Qwen3-Embedding, IBM Granite Embedding Multilingual and Krutrim's Vyakyarth (open models), plus hosted APIs from Voyage, Google Gemini and Cohere. I chose BAAI/bge-m3:

• It is trained on 100+ languages, including Telugu.
• It has 568M parameters: small enough to run on a laptop CPU and fine-tune on a free Colab GPU.
• It accepts inputs up to 8,192 tokens.
• It has a sister reranker, bge-reranker-v2-m3.

API embeddings were ruled out because they can't be fine-tuned and add a cost to every query. I did not benchmark the other open models against it. That is a fair criticism, and a good follow-up experiment.

What to embed: the meaning, not the Sanskrit

Each verse has two texts: the Sanskrit sloka and the modern Telugu tatparyam. I embed the tatparyam.

Does it actually understand Telugu? Two quick tests

When the first results were weak, the obvious explanation was "multilingual models are bad at Telugu". Before acting on that, I tested it.

Test 1 — the floor test. Take 30 random verses, copy a 45-character slice straight out of each one, and use that slice as the query. Any working model must rank the source verse first. Result: 26/30 at rank 1, 29/30 in the top 5, and none outside the top 20.

Test 2 — are all Telugu texts squashed together? If a model doesn't really understand a language, it tends to give every text in that language a similar vector. Then random, unrelated pairs look similar. I measured the average similarity between random pairs of verses. Telugu: 0.485. The same Gita in English: 0.502. Telugu is not squashed together any more than English is.

04Why the embedding model had to be fine-tuned

The real problem: questions don't look like verses

The stock model struggled with the kinds of questions people actually ask. Here is one verse with three real test questions, one of each type:

• Lookup questions ask directly about the text. They share some words with the verse.
• Thematic questions describe a life situation ("I work, but worry about results makes me anxious") and never mention the Gita. The link to the verse is purely about meaning.
• Tenglish questions are in a different script from every stored verse.

A question is a short interrogative. A verse is a ~127-character statement where the answer is one clause among several. That mismatch in shape is hard for any embedding model, in any language. On the 320 held-out test questions, the stock model found the right verse in its top 10 for only 38% of questions. For Tenglish it was 8%, and the right verse usually wasn't even in the top 50.

Fine-tuning teaches the model what "similar" should mean for this task: "this verse answers this question", rather than "these two texts use similar words".

Step 1: create training data (there was none)

No dataset of Telugu Gita questions exists, so I generated one. For each verse, Claude Opus 5 wrote five questions: 2 lookup, 2 thematic and 1 Tenglish. Two rules mattered most:

• Don't reuse the verse's own words. A question that copies the verse's vocabulary teaches word-matching, which keyword search already does. The embedding model needs to learn paraphrase.
• Thematic questions must be first-person life situations that never mention the Gita, because that is how people actually ask.

Result: 2,905 training questions for $4.71.

Step 2: split by verse, not by question

The 699 verses were split into 581 for training and 104 held out, plus 14 more reserved as gold answers for an early test set. The held-out verses and their questions are never seen during training.

Step 3: find hard negatives

Training works by showing the model a question, the right verse (the positive), and a wrong verse (the negative). The choice of wrong verse matters a lot:

• A random wrong verse is too easy. A verse about war conches is obviously not about work stress, so the model learns nothing.
• A hard negative is a wrong verse that the current search system ranks highly. These are exactly the mistakes that cost recall.

For every training question, I ran the existing search and took 2 highly ranked wrong verses (training verses only). That gave 5,810 triplets of (question, right verse, wrong verse).

Step 4: contrastive training

A free Colab T4 GPU can't fit a batch of 64 normally. CachedMultipleNegativesRankingLoss from sentence-transformers solves this. It processes the batch in small pieces of 8 but computes the loss as if all 64 were together.

What training changed

As a sanity check, I scored 32 training triplets before and after:

Almost all of the improvement came from pushing wrong verses down, not pulling right ones up. The model already knew the right verse was related; it learned to tell it apart from verses that were merely about something similar. (These are training triplets, so this shows the mechanism, not the real-world gain. The held-out results are in section 8.)

05The reranker: why add one, and why fine-tune it

The off-the-shelf reranker helped, a little

With the fine-tuned search already in place, adding the stock bge-reranker-v2-m3 raised recall@10 from 0.512 to 0.559 on the 320 held-out questions. That is useful but modest, with the biggest gain on lookup questions, the type closest to what generic rerankers are trained on.

Why fine-tune it

The stock reranker was trained on general web data. It has not seen Tenglish questions matched to Telugu-script verses, or first-person life situations matched to scripture. The embedding model had just been taught those things, and the reranker should know them too: a reranker only helps if it is better at the task than the search it is re-ordering.

How it was fine-tuned

• New hard negatives. The fine-tuned search makes different mistakes than the stock one, so negatives were mined again with the new model: 8 wrong verses per question from its top 50, giving 26,145 (question, verse) pairs.
• Objective. For each question, score the right verse above its 8 hard negatives.
• Settings. 2 epochs, learning rate 1e-5, Colab T4.
• Memory. The first two runs crashed with out-of-memory errors. The fixes: cutting the max input length from 512 to 192 tokens (the longest real question+verse pair is 146 tokens), freeing the stock model from GPU memory before training, and using the lighter Adafactor optimizer.

Result

On the same 320 held-out questions and the same 50 candidates per question:

Tenglish gained the most: recall@10 went from 0.36 to 0.64, and the right verse now typically comes first. That makes sense. A cross-encoder sees kopam and కోపం in the same input and can learn to line them up directly. A bi-encoder only ever compares two vectors computed separately.

Thematic questions did not improve with fine-tuning (0.383 either way). They remain the hardest part of this system.

How many candidates to rerank?

Reranking more candidates costs more time, so I measured it on 80 questions. I reranked the top 50 once and read off smaller pools by cutting the list.

Recall stops improving at 30. Going to 50 costs 40% more compute for +0.005 NDCG, so production uses 30.

06Hybrid search

The Telugu problem with keyword search

Standard BM25 splits text into words. That fails in Telugu, which is agglutinative: suffixes attach directly to words, and sandhi fuses neighbouring words together. The same concept appears in many surface forms.

Real example. Question: యోగక్షేమం ఎవరు వహిస్తారు? ("Who takes care of one's welfare?"). The answer is verse 9.22, which contains యోగక్షేమాలను నేనే వహిస్తాను. The question and the verse share zero whole words: యోగక్షేమం ≠ యోగక్షేమాలను, and వహిస్తారు ≠ వహిస్తాను.

The fix is to index overlapping 4-character pieces (character 4-grams) instead of whole words:

యోగక్షేమం  →  యోగక · ోగక్ · గక్ష · క్షే · ్షేమ · షేమం
               (Unicode characters, so vowel signs are separate pieces)

With 4-grams, that question and verse share 11 pieces. Keyword search now finds the verse. On this question, the stock dense model did not have 9.22 anywhere in its top 40; BM25 ranked it #1; after combining, it sat at #7, inside the reranker's pool, where it can be lifted further.

Combining two rankings: Reciprocal Rank Fusion

Dense scores are cosine similarities around 0.3–0.8. BM25 scores are unbounded numbers like 14.2. You can't just add them. RRF ignores the scores and uses only the rank:

RRF score(verse) = Σ over each search   1 / (60 + rank in that search)

Worked example: a verse ranked #1 by BM25 but missing from dense scores 1/61 = 0.0164. A verse ranked #3 by dense and #4 by BM25 scores 1/63 + 1/64 = 0.0315, so it wins. Being good in both beats being first in one. The constant 60 keeps any single first place from dominating. Keep this rule in mind: it matters a lot in the results below.

Did hybrid search actually help?

This is the question I should have asked much earlier. While writing this article, I measured each retriever on its own and combined, on the same 320 held-out questions, with no reranker:

Lookup / Thematic / Tenglish columns are recall@10 for that question type.

Adding keyword search made retrieval worse. The fine-tuned dense model on its own finds the right verse in the top 10 for 64.1% of questions. Combining it with BM25 drops that to 51.2%, and the combined version is lower for every question type. The same happens with the stock model, less dramatically (0.384 → 0.362).

Why?

• The keyword side is weak on these questions. BM25 alone reaches 0.241 overall, 0.062 on thematic questions and 0.016 on Tenglish. Tenglish is in a different script from the verses, so not a single 4-gram can match.
• The test questions were written to avoid the verse's words. That rule was right for training the embedding model, but it also makes this test set unusually hard for keyword search. Real users often do use a verse's own words and names, like యోగక్షేమం.
• RRF gives both searches an equal vote. "Good in both beats first in one" is a strength when both searches are decent. When one of them is usually wrong, its bad rankings push good verses out of the top 10.

Hybrid search was a reasonable design choice: it is standard practice, and the యోగక్షేమం example shows the kind of question it rescues. But I never measured it on its own until now, and on this test set it costs more than it gains. The next experiments are clear: give dense search more weight in the fusion, feed the reranker from dense search alone, and test on real user questions.

07How the models were evaluated

Evaluation happened at three levels. Each catches different failures.

Level 1: retrieval — did we find the right verse?

Test set. 320 held-out questions, generated the same way as the training questions, about 64 held-out verses. (Question generation for the held-out split stopped at 64 of the 104 verses when API credit ran out; 320 questions was enough to measure with.) None of these verses were used in training.

An earlier set of 14 questions was used as a quick smoke test while building, before the held-out set existed.

Metrics. Each question has one correct verse. For each system, record the rank at which that verse appears:

Recall tells you whether the LLM will see the verse. NDCG tells you whether it sees it near the top, which matters because models pay more attention to the first passages.

Results were always reported per question type. An average hides a lot: in the stock model, a decent lookup score was hiding a Tenglish score of zero.

Level 2: answer quality — an LLM judge

Retrieval metrics don't tell you whether the final answer is any good. For that, I sent 21 held-out questions (7 per type) to the live deployed app, not a local copy, and had Claude Opus 5 grade each answer:

• Faithfulness (1–5): is every claim supported by the cited verses?
• Relevance (1–5): does it answer the question that was asked?
• Register (1–5): is it natural, correct Telugu?
• Unsupported claims: a list of specific sentences that the cited verses don't support.

The judge sees only the question, the answer and the full text of the cited verses. It is told to ignore its own knowledge of the Gita, because the question is not "is this true?" but "does this text say this?"

Level 3: citations — checked by code, not by a model

Every quote in an answer must be traceable to the source. A citation is accepted only if at least 24 consecutive characters of the quote appear, character for character, in the cited verse. What the user is shown is the verse text from the corpus, never the model's copy of it. The model may paraphrase around a quote, but it can't invent one.

08Final results

Retrieval, 320 held-out questions

From the stock model to the finished pipeline, recall@10 rose from 0.362 to 0.634 (+75%) and NDCG@10 from 0.260 to 0.468 (+80%). Each row adds one change to the row above it, except the last two, which are alternatives: the same pipeline with the stock or the fine-tuned reranker.

An open question from section 6: fine-tuned dense search alone, with no BM25 and no reranker, already reaches 0.641 recall@10 (NDCG 0.419). How much the fine-tuned reranker adds on top of dense-only search hasn't been measured yet, so the best configuration is still to be found.

"Before" = stock BGE-M3 hybrid search, no reranker. "After" = fine-tuned search + fine-tuned reranker.

Answer quality, 21 questions judged by Claude Opus 5

The generation model was switched from Claude Haiku 4.5 to Claude Sonnet 5 after this comparison:

Before switching models, I tried a stricter prompt with explicit rules and Telugu examples for each type of drift. Unsupported claims went from 13/21 to 16/21, which is no better and possibly worse. Changing the model did what prompting could not.

Speed

The first version took 45 seconds per answer. Turning off an extended-thinking budget that wasn't improving answers, folding a separate "is this context enough?" check into the generation step, running question routing and search in parallel, and streaming the answer brought it to about 10 seconds total, with the first words appearing after 3.1 seconds.

What still isn't solved

• Thematic questions. Life-situation questions reach only 0.383 recall@10. Neither fine-tuning nor reranking moved them much. This is the next problem to work on.
• One in three answers still has at least one sentence the judge flags as not supported by the cited verses. Quotes are verified, but the sentences around the quotes are not.
• The test questions were written by an LLM, not collected from real users, and 21 judged answers is a small sample. Real user questions would be a better test.

Takeaways

1. Check the data first. In a low-resource language, what exists decides your scope.
2. Test the obvious explanation before fixing it. "The model can't read Telugu" was wrong. The real problem was question shape.
3. Fine-tuning can be cheap. $4.71 of generated questions and a free GPU produced the largest gains in the project.
4. Split by document, not by question, or your evaluation measures memory.
5. Pick search methods that fit the language. Character 4-grams handle Telugu word forms that whole-word search misses.
6. Measure with enough data, per question type, on the system you actually ship.

Corpus text: Telugu Wikisource, CC BY-SA 4.0. Code: MIT.