Notes from building AI systems.

LLM fundamentals

How the models themselves work: how text becomes tokens, the pieces of the Transformer, and the reasons behind the choices in them.

Tokenization

1. Byte-pair encoding, and why tokenizers work on bytes Merge the most frequent adjacent pair until the vocabulary is full, with the classic four-word example worked by hand. Then why the base alphabet is the 256 UTF-8 bytes: no unknown token, a tiny base vocabulary, one pipeline for every script and for code, at the cost of more tokens for scripts the corpus is thin on. 24 Oct 2026 · 9 min read

Transformer internals

1. Why attention has separate Q, K and V, and why it divides by √d_k Three projections let a token ask one thing, advertise another and hand over a third, which one similarity matrix cannot express. The dot product of two d_k-dimensional vectors has variance d_k, so without scaling the softmax saturates and its gradient vanishes. 24 Oct 2026 · 8 min read
2. Self-attention derived on a whiteboard, with one sentence (video) A ten-minute video that builds softmax(QKᵀ/√d_k)V from 'The river bank flooded': the transformer in one picture, the three jobs every token does, the dot product, the 4×4 score grid, why the square root is there, the softmax, the weighted values, heads and the causal mask. The chapter list and every worked number are in the note. 26 Oct 2026 · 10 min video
3. Mixture of experts: active parameters versus total parameters Several feed-forward experts and a router that sends each token to one or two of them. Why memory is paid on the total count and arithmetic on the active count, with Mixtral and DeepSeek-V3 as examples, where the bandwidth advantage fades under batching, and what learned routing costs in training. 24 Oct 2026 · 9 min read

Frontier models and the model landscape

No notes yet.

Prompting and context engineering

No notes yet.

Fine-tuning, alignment and post-training

No notes yet.

Inference scaling and serving

What it costs to run a model: where the time goes in a decode step, what memory is spent on, and the levers a serving stack has.

Serving fundamentals

1. The KV cache, and why decoding is memory-bound Caching keys and values makes generation linear instead of quadratic, and leaves each decode step streaming the whole model and cache for one token's worth of arithmetic. The size formula, worked for a 70B model at 2k and 128k with and without grouped-query attention, and the mitigations: MQA, GQA, paged attention, cache quantisation. 24 Oct 2026 · 10 min read
2. Grouped-query attention: the middle ground between MHA and MQA One key-value pair per query head, per group, or for all heads. Why sharing one pair per group shrinks the cache by the group ratio, keeps quality close to multi-head, and became the default in Llama, Mistral and Gemma, with the real head counts. How a trained multi-head model converts by mean-pooling and uptraining. 24 Oct 2026 · 7 min read

Decoding

1. Temperature, top-k, top-p, and why T = 0 is not deterministic The sampling pipeline in the order it runs: temperature divides, top-k cuts by rank, top-p cuts by mass, then softmax and a draw. Temperature zero is argmax, and still varies run to run: floating-point sums depend on order, kernels change strategy with batch size, near-ties at the top flip, and the continuation diverges. 24 Oct 2026 · 9 min read

Test-time compute and reasoning models

No notes yet.

Agent architectures, tool use and MCP

How a model is turned into something that acts: the loop, the tools, the protocol between them, and the controls around it.

Agent loops

1. Function calling: how the model decides, and what your code must do Tool definitions are prompt text; the decision to call is learned judgement; strict mode guarantees the arguments' shape but not their sense. Then the application's four steps, parse, validate and authorise, execute, return, with a worked order-lookup example and the mistakes that matter, starting with treating the model's arguments as permission. 25 Oct 2026 · 9 min read
2. The ReAct pattern as a loop of function calls Thought, action and observation mapped onto a modern API: the reasoning before a call, the structured tool-call block and its stop reason, the tool-result message your code appends, and the exit when the model answers without a tool. A full loop in pseudocode, why structured calls beat parsing text, and what the loop needs that the paper did not cover. 25 Oct 2026 · 10 min read
3. How an agent loop ends, and the three ways it fails to The model's exit plus the runtime's breakers: step cap, time and cost budgets, and a told budget. Reading the stop reason, including truncation as its own case. Then infinite loops, premature stops and repeated identical calls, each with its root cause and fix: retryable flags, validation gates, deduplication, and observations that say what they mean. 25 Oct 2026 · 11 min read
4. ReAct versus plan-and-execute, and the hybrid that production uses Interleaving reasons after each observation and wins when the next step depends on the last; planning first reasons before any and wins on known shapes, independent parts and long horizons. The plan-execute-replan hybrid: a planner without tools, executors running local ReAct loops in their own contexts, a replanner that revises what remains, with a walkthrough of a task that does not go to plan. 25 Oct 2026 · 11 min read
5. Agent framework or hand-rolled loop: what each buys you, and what changes your mind The loop is fifty lines; a framework sells the infrastructure around it. What a framework actually provides, when to hand-roll, when to use one, the provider loop helpers and managed runtimes the usual comparison misses, a three-way table, and the production pressures that move teams in each direction. 25 Oct 2026 · 10 min read

Tool design

1. What makes a good tool schema, and why the description outranks the name The schema is a briefing the model reads. Name the action; spend the effort on a description that says what, when, when not, and what it costs; type every parameter, enum the fixed sets, require only what the function cannot run without. A weak schema rewritten, and why selection is decided by the description: a name has no room for conditions. 25 Oct 2026 · 9 min read
2. Idempotent tool calls: which tools need a key, and how to derive one Reads and absolute writes are safe to retry; creates, sends, charges and increments are not. How an idempotency key works, why it must be derived from the operation and never the attempt or the model, guarding both transport retries and model re-issues, a ledger with a unique constraint when the service has no key support, and why read-before-write is not enough on its own. 25 Oct 2026 · 10 min read

MCP

1. The Model Context Protocol: hosts, clients, servers, and where the adapter work went What the protocol fixes and what it leaves alone. Host, client and server; tools, resources and prompts told apart by who triggers them; discovery at connection time; the N×M problem and why it was N×M. Then the question that confuses people: the adapter work did not disappear, it moved into the server once per service and the host once per app. 25 Oct 2026 · 11 min read

Design exercises

1. Designing a customer-support agent over 30 backend operations One concrete system worked end to end: thirty operations classified by kind, reversibility and blast radius; merged and intent-loaded into a tool surface with a verified finish tool; scopes per principal with ownership checked by the adapter; idempotency keys for every create, send and money movement; the loop's gates; and a four-level evaluation plan with the live metrics that decide rollout. 25 Oct 2026 · 16 min read

Multi-agent systems, orchestration and human-in-the-loop

No notes yet.

RAG ingestion, chunking and indexing

Everything that happens before a query arrives: how text is split, what is embedded, what metadata rides along, and how an index stays correct when it holds documents not everyone may read.

Core concepts

1. Chunk size: recall falls, faithfulness rises What happens to a RAG system as chunks grow from 128 to 2,000 tokens: vector dilution hurts search while whole paragraphs help the answer, and why the fix is not a size in the middle. 19 Oct 2026 · 6 min read
2. The chunk you embed is not the chunk you retrieve Search wants small text and the model wants big text. Decouple the indexed text from the retrieved text: parent-document retrieval, contextual embeddings, and summary or question indexes. 19 Oct 2026 · 7 min read
3. Metadata on chunks, and the care needed when a model writes it What metadata buys a retrieval system, and the three things that go wrong when a model generates it at ingestion: hallucinated and drifting fields, cost, and injection from the documents. Constrain the schema, validate after, derive security fields from the system. 21 Oct 2026 · 10 min read

Chunking patterns

1. How to chunk wiki pages Split on the heading tree, not on token counts. Prepend the page title and heading path, flatten infoboxes to key-value lines, fall back to paragraphs only in long sections, and retrieve the section when a paragraph matches. 19 Oct 2026 · 8 min read
2. Semantic chunking made my retrieval evals worse Five reasons the switch from fixed 512-token chunks lowers MRR, NDCG and hit rate: variable length, lost overlap, tiny chunks starving the top-k budget, one global distance threshold, and ground truth labelled on the old boundaries. A four-step diagnostic that tells them apart. 21 Oct 2026 · 10 min read

Multi-tenant indexes

1. Cross-tenant leakage in vector search, and how to test for it Four ways an ACL-filtered RAG still leaks between tenants: post-filtering after top-k, filters at the prompt layer, a shared semantic cache, and exposed scores. A canary and starvation-probe test plan that finds each one. 20 Oct 2026 · 9 min read
2. Permissions on chunks, and keeping them in sync Put a stable document id on the vector, keep the ACL in a store built for change, pre-filter at query time from the auth token, and let the source system's webhooks or change data capture drive the sync. Filter on groups, not users. 20 Oct 2026 · 8 min read

Build breakdowns

1. How I built a RAG on the Bhagavad Gita in Telugu Preparing the dataset, choosing and fine-tuning the embedding model and reranker, hybrid search, and how it was evaluated. 15 Sep 2026 · 20 min read
2. Sanskrit in, Telugu out: a RAG over 17 Puranas The texts exist only in Sanskrit and the users ask in Telugu. Chapter cards, in-memory retrieval, verse-level citation and on-demand translation, with the measured numbers. 16 Sep 2026 · 16 min read

RAG retrieval: embeddings, hybrid search and reranking

No notes yet.

RAG generation, grounding and debugging

No notes yet.

Advanced RAG: agentic RAG, GraphRAG, memory and long context

No notes yet.

Evaluation fundamentals, benchmarks and LLM-as-judge

No notes yet.

Evaluating RAG, agents and production systems

How to know whether a retrieval pipeline or an agent works, which part of it broke, and whether the evaluation itself is telling the truth.

RAG evaluation

1. Which RAG component broke? Why an end-to-end score cannot tell you Query understanding, retrieval, reranking, context assembly and generation can each lose the answer, and the end-to-end score is their product, so the same number has different causes with different fixes. A metric per stage conditioned on the stage before, two oracle experiments that bound each half, and a decision tree that attributes every wrong answer. 25 Oct 2026 · 10 min read

Agent evaluation

1. Agent task success versus process quality Finishing the task and taking a sensible path are different questions, and an agent can succeed by trying everything, guessing, or destroying and rebuilding, because the task did not punish it this time. Why outcome-only scoring rewards those paths, which process metrics to record from the trace, scoring against constraints rather than one golden path, and the two-by-two to read both scores together. 25 Oct 2026 · 9 min read
2. Evaluating tool-call correctness: right tool, right arguments, result used correctly Four layers that fail separately: whether to call, which tool, which arguments, how the result was used. Selection as a confusion matrix, why exact-match arguments are wrong for most fields and what schema-aware comparison does type by type, counterfactual stubs to test result use, and derived-argument checks in multi-step sequences. 25 Oct 2026 · 10 min read
3. The agent passes 92% of eval tasks and 60% of real sessions. What explains the gap? Part of the gap is definitional: task-spec success is not user-goal success. The rest is structural: real sessions are ambiguous, wrong about their own details, multi-turn, uncooperative, interrupted and unevenly distributed. How to diagnose which differences matter, then redesign with cases from real sessions, simulated users, tasks where the right move is to ask or to notice an error, injected tool failures, whole-session grading and production weighting. 25 Oct 2026 · 11 min read

Suites and safety

1. An LLM regression suite in CI: size, runtime, thresholds and flakes A chosen smoke set on every change and a large nightly set, and why one tier cannot do both jobs. What a case fixes, graders from mechanical to human, thresholds against a baseline with a measured noise band and gates at 100 percent for safety and policy, and a flake policy: three runs, re-run failures, cache by request hash, pin model versions, quarantine the inconsistent. 25 Oct 2026 · 11 min read
2. Evaluating safety behaviour: harmful compliance and over-refusal together Safety is a classifier with two error types, and measuring only harmful compliance has a trivial optimum: refuse everything. A harmful set with adversarial variants, a benign set built as matched pairs on the same topics, three response labels plus a separate harm grade, the pair reported per category with an operating point chosen per severity, and a regression rule that neither rate may be spent to buy the other without a person agreeing. 25 Oct 2026 · 10 min read

Observability, tracing and monitoring

No notes yet.

LLMOps, cost, reliability and product engineering

No notes yet.

Prompt injection and LLM application security

No notes yet.

Red teaming, adversarial testing and guardrails

No notes yet.

AI governance, risk and compliance

No notes yet.

Responsible AI, safety, alignment and societal risk

No notes yet.