LLMs and Transformers from First Principles — Tokenization, Attention, Embeddings, Pretraining, and Fine-tuning (with PyTorch and Hugging Face Code)

Everyone uses LLMs in 2026. Far fewer can explain what happens between text in and text out. The gap matters because every LLM problem — bad outputs, high latency, wrong answers, costly fine-tunes — is solved by knowing which mechanism inside the model is responsible.

This guide rebuilds the LLM stack piece by piece: tokenization (why "Mumbai" is one token but "Indore" is three), embeddings (how text becomes vectors), self-attention (the one operation that made transformers work), multi-head attention and the transformer block (how layers stack), pretraining vs fine-tuning vs LoRA vs RLHF (when to use each), and decoding (sampling, temperature, top-k/top-p). Real PyTorch code, a tiny GPT you can train on your laptop, and Hugging Face examples for everything you''d actually do in production.

---

TL;DR

• An LLM is a function from token sequences to next-token probability distributions. That''s it. Everything else — chat, code, agents — is built on top.
• Tokenization is not optional. "Mumbai" might be 1 token, "Indore" might be 3, and the token boundary affects both quality and cost.
• Self-attention is the one new idea behind transformers. Each token gets to look at every other token through learned Query/Key/Value projections. The whole revolution is one matrix multiplication + softmax.
• The transformer block = attention + feed-forward + residuals + layernorm. Stack 6 to 96 of them and you have a model.
• Pretraining = next-token prediction on huge data. Fine-tuning = teaching the pretrained model your specific task. LoRA = fine-tuning by adding tiny rank-decomposed adapters, ~1000× cheaper.
• Decoding strategy matters. Greedy is deterministic but boring; temperature + top-p sampling is the production default.

---

1. What an LLM actually is

An LLM is a function:

LLM(token₁, token₂, ..., tokenₜ) → probability distribution over the next token

Wrap that function in a loop and you have generation:

sequence = [<bos>, "The", " sky", " is"]
while not done:
    probs = LLM(sequence)
    next_token = sample(probs)
    sequence.append(next_token)

That''s the entire generation algorithm. Every chat-completion API, every code assistant, every agent, every RAG system — all of them are a sampling loop around a "predict the next token" function.

The two big questions are:

1. How does the function work? → tokenization, embeddings, attention, transformer blocks.
2. How did we train it? → pretraining + fine-tuning + RLHF.

We''ll answer both with code.

---

2. Tokenization — text becomes integers before anything else

A model can''t read characters. The first step is tokenization: splitting a string into a sequence of integer IDs from a fixed vocabulary.

2.1 Why not just words?

• Vocabulary becomes huge (English has >300k words, including names and typos).
• Unknown words at inference time have no embedding.
• Most computation is wasted on rare tokens.

So modern LLMs use subword tokenization — common words stay whole, rare words get split into pieces.

2.2 The three algorithms you''ll meet

Algorithm	Used by
BPE (Byte-Pair Encoding)	GPT-2/3/4, Llama, Mistral
WordPiece	BERT, DistilBERT
SentencePiece (Unigram)	T5, mBART, Llama

2.3 Real tokenization with Hugging Face

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")

text = "Mumbai and Indore are cities in India."
ids = tok.encode(text)
print(ids)
# [33237, 290, 24506, 382, 4736, 287, 3794, 13]

tokens = tok.convert_ids_to_tokens(ids)
print(tokens)
# [''Mumbai'', ''Ġand'', ''ĠInd'', ''ore'', ''Ġare'', ''Ġcities'', ''Ġin'', ''ĠIndia'', ''.'']

Notice: "Mumbai" is 1 token (common in pretraining data); "Indore" is 2 tokens. Token counts vary by name, language, even punctuation. Token-counting matters because:

• API costs are per token.
• Context window is in tokens, not words.
• Latency is roughly linear in tokens.

short = tok.encode("The quick brown fox jumps over the lazy dog.")
long  = tok.encode("L''écosystème logiciel français évolue rapidement.")
print(len(short), len(long))
# 11 (English), 18 (French — non-English costs 2–3× more tokens)

2.4 Special tokens

print(tok.special_tokens_map)
# {''bos_token'': ''<|endoftext|>'', ''eos_token'': ''<|endoftext|>'', ''unk_token'': ''<|endoftext|>''}

chat_tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user",   "content": "What is 2+2?"},
]
prompt = chat_tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

Always use apply_chat_template — manually concatenating "user: ... assistant: ..." is a classic bug.

---

3. Embeddings — turning integers into vectors

import torch
import torch.nn as nn

vocab_size = 50000
d_model    = 768

token_embedding = nn.Embedding(vocab_size, d_model)

ids = torch.tensor([[100, 5, 999, 42]])
embedded = token_embedding(ids)                  # shape (1, 4, 768)

Each row of the embedding matrix corresponds to one vocabulary entry. Vectors are learned during training.

3.1 Why this is interesting

After pretraining, similar tokens have similar embeddings. The practical version: you can do vector search over text. Sentence embeddings power every modern RAG system.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode([
    "How do I deploy a Docker container?",
    "What is the procedure for shipping a containerised app?",
    "Banana bread recipe",
])
def cos(a, b): return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
print(cos(embeddings[0], embeddings[1]))   # high (~0.7)
print(cos(embeddings[0], embeddings[2]))   # low (~0.1)

3.2 Positional embeddings

Attention is order-invariant. Without positional info, the model can''t distinguish "dog bites man" from "man bites dog."

Scheme	Used by
Learned absolute	Original Transformer, BERT, GPT-2
Sinusoidal	"Attention Is All You Need"
RoPE (Rotary)	Llama, Mistral, most modern LLMs

RoPE dominates 2026 because of length extrapolation.

import math

def sinusoidal_pe(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe

---

4. Self-attention — the one idea that changed everything

4.1 The intuition

For each token, compute a new representation that''s a weighted sum of every other token''s value, where weights depend on how relevant each other token is to this token.

Every token gets three roles:

• Query (Q): "what am I looking for?"
• Key (K): "what do I represent if you''re searching for something?"
• Value (V): "what do I contribute when you decide to listen to me?"

4.2 The math

Q = X @ W_Q       # shape (T, d)
K = X @ W_K       # shape (T, d)
V = X @ W_V       # shape (T, d)

Attention(Q, K, V) = softmax(Q @ K.T / √d) @ V

• Q @ K.T gives a (T, T) matrix of all query-key similarities.
• / √d is a scaling factor preventing softmax saturation.
• softmax along the row axis turns similarities into attention weights.
• The result @ V gives each token a weighted average of all values.

4.3 Real code — scaled dot-product attention

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class ScaledDotProductAttention(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        self.W_Q = nn.Linear(d_model, d_model)
        self.W_K = nn.Linear(d_model, d_model)
        self.W_V = nn.Linear(d_model, d_model)
        self.d_k = d_model

    def forward(self, x, mask=None):
        Q = self.W_Q(x)
        K = self.W_K(x)
        V = self.W_V(x)

        scores = Q @ K.transpose(-2, -1) / math.sqrt(self.d_k)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float("-inf"))
        weights = F.softmax(scores, dim=-1)
        return weights @ V

4.4 The causal mask — why GPT can''t see the future

T = 8
causal_mask = torch.tril(torch.ones(T, T))
# [[1 0 0 0 0 0 0 0]
#  [1 1 0 0 0 0 0 0]
#  ...
#  [1 1 1 1 1 1 1 1]]

Encoder-only models (BERT) skip the mask. Decoder models (GPT family) require it.

4.5 Multi-head attention

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        assert d_model % n_heads == 0
        self.d_model = d_model
        self.n_heads = n_heads
        self.d_k     = d_model // n_heads

        self.W_Q = nn.Linear(d_model, d_model)
        self.W_K = nn.Linear(d_model, d_model)
        self.W_V = nn.Linear(d_model, d_model)
        self.W_O = nn.Linear(d_model, d_model)

    def forward(self, x, mask=None):
        B, T, _ = x.shape

        def split(t): return t.view(B, T, self.n_heads, self.d_k).transpose(1, 2)
        Q = split(self.W_Q(x))
        K = split(self.W_K(x))
        V = split(self.W_V(x))

        scores = Q @ K.transpose(-2, -1) / math.sqrt(self.d_k)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float("-inf"))
        weights = F.softmax(scores, dim=-1)
        out = weights @ V

        out = out.transpose(1, 2).contiguous().view(B, T, self.d_model)
        return self.W_O(out)

Typical: d_model=768, n_heads=12 (BERT-base), d_model=4096, n_heads=32 (Llama-3-8B).

In production, use Flash Attention via F.scaled_dot_product_attention:

out = F.scaled_dot_product_attention(Q, K, V, attn_mask=None, is_causal=True)

---

5. The transformer block

x → LayerNorm → MultiHeadAttention → + (residual) →
              → LayerNorm → FeedForward → + (residual) → output

class TransformerBlock(nn.Module):
    def __init__(self, d_model, n_heads, d_ff, dropout=0.1):
        super().__init__()
        self.ln1  = nn.LayerNorm(d_model)
        self.attn = MultiHeadAttention(d_model, n_heads)
        self.ln2  = nn.LayerNorm(d_model)
        self.ffn  = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout),
        )
        self.drop = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        x = x + self.drop(self.attn(self.ln1(x), mask))
        x = x + self.drop(self.ffn(self.ln2(x)))
        return x

What each piece does:

• LayerNorm — stabilizes training. Pre-LN trains more stably than the original Post-LN.
• Residual connections — gradient highway. Without them, deep transformers don''t train.
• Feed-forward — two-layer MLP applied independently to each token. Usually d_ff = 4 × d_model.

Llama-3-8B is 32 blocks; GPT-3 is 96.

---

6. A tiny GPT — runnable end to end

class GPT(nn.Module):
    def __init__(self, vocab_size, d_model=128, n_heads=4, n_layers=4,
                 max_seq_len=128, d_ff=512, dropout=0.1):
        super().__init__()
        self.token_emb = nn.Embedding(vocab_size, d_model)
        self.pos_emb   = nn.Embedding(max_seq_len, d_model)
        self.drop      = nn.Dropout(dropout)
        self.blocks    = nn.ModuleList([
            TransformerBlock(d_model, n_heads, d_ff, dropout)
            for _ in range(n_layers)
        ])
        self.ln_f = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab_size, bias=False)
        self.head.weight = self.token_emb.weight   # weight tying

        self.register_buffer(
            "causal_mask",
            torch.tril(torch.ones(max_seq_len, max_seq_len)).view(1, 1, max_seq_len, max_seq_len)
        )

    def forward(self, idx, targets=None):
        B, T = idx.shape
        pos = torch.arange(T, device=idx.device)
        x = self.token_emb(idx) + self.pos_emb(pos)
        x = self.drop(x)

        mask = self.causal_mask[:, :, :T, :T]
        for block in self.blocks:
            x = block(x, mask=mask)
        x = self.ln_f(x)
        logits = self.head(x)

        if targets is None:
            return logits, None

        loss = F.cross_entropy(
            logits.view(-1, logits.size(-1)),
            targets.view(-1),
        )
        return logits, loss

    @torch.no_grad()
    def generate(self, idx, max_new_tokens, temperature=1.0, top_k=None):
        for _ in range(max_new_tokens):
            idx_cond = idx[:, -self.pos_emb.num_embeddings:]
            logits, _ = self(idx_cond)
            logits = logits[:, -1, :] / temperature
            if top_k is not None:
                v, _ = torch.topk(logits, top_k)
                logits[logits < v[:, [-1]]] = float("-inf")
            probs = F.softmax(logits, dim=-1)
            next_id = torch.multinomial(probs, num_samples=1)
            idx = torch.cat([idx, next_id], dim=1)
        return idx

Training loop on Shakespeare:

import requests

url = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
text = requests.get(url).text

chars = sorted(set(text))
stoi  = {ch: i for i, ch in enumerate(chars)}
itos  = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(itos[i] for i in ids)

data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data, val_data = data[:n], data[n:]

block_size = 128
batch_size = 32

def get_batch(split):
    src = train_data if split == "train" else val_data
    idx = torch.randint(len(src) - block_size, (batch_size,))
    x = torch.stack([src[i:i+block_size]   for i in idx])
    y = torch.stack([src[i+1:i+block_size+1] for i in idx])
    return x, y

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model  = GPT(vocab_size=len(chars), d_model=192, n_heads=6, n_layers=6, max_seq_len=block_size).to(device)
opt    = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)

for step in range(5000):
    x, y = get_batch("train")
    x, y = x.to(device), y.to(device)
    _, loss = model(x, y)
    opt.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    if step % 500 == 0:
        print(f"step {step}: train_loss={loss.item():.3f}")

prompt = torch.tensor([[stoi["R"], stoi["O"]]], device=device)
out = model.generate(prompt, max_new_tokens=300, temperature=0.8, top_k=20)
print(decode(out[0].tolist()))

That''s the entire architecture of GPT-3 — just bigger and on more data.

---

7. Pretraining — the boring task that creates intelligence

Pretraining is one objective: predict the next token.

def causal_lm_loss(model, batch):
    logits = model(batch["input_ids"])
    return F.cross_entropy(
        logits[:, :-1, :].reshape(-1, logits.size(-1)),
        batch["input_ids"][:, 1:].reshape(-1),
    )

The fancy stuff — chat, code, reasoning — emerges from this objective applied at scale. 2026 frontier recipes:

• Data: 5–15 trillion tokens of filtered text + code.
• Compute: thousands of H100/H200 GPUs for weeks/months.
• Optimizer: AdamW with warmup + cosine decay, weight decay 0.1.
• Architecture: transformer decoder with RoPE, RMSNorm, SwiGLU FFN, grouped query attention (GQA).
• Cost: $10M–$100M for a real frontier model.

Almost everyone in 2026 starts from a pretrained model and fine-tunes.

---

8. Fine-tuning — making a pretrained model do your task

8.1 Prompt engineering / few-shot — start here

prompt = """You are a classifier. Categorize each ticket as one of: bug, feature, question.

Examples:
"App crashes when I save" → bug
"Can you add dark mode?"  → feature
"How do I change my password?" → question

Ticket: "{ticket}"
Category:"""

If accuracy is good enough, ship it. Zero training, zero infrastructure.

8.2 Full fine-tuning

from transformers import (
    AutoModelForCausalLM, AutoTokenizer,
    TrainingArguments, Trainer, DataCollatorForLanguageModeling,
)
from datasets import load_dataset

model_id = "Qwen/Qwen2.5-1.5B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

dataset = load_dataset("json", data_files="my_dataset.jsonl", split="train")

def tokenize(example):
    return tok(example["text"], truncation=True, max_length=2048)

dataset = dataset.map(tokenize, remove_columns=["text"])

args = TrainingArguments(
    output_dir="ft-out",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-5,
    warmup_ratio=0.05,
    lr_scheduler_type="cosine",
    weight_decay=0.01,
    bf16=True,
)

trainer = Trainer(
    model=model, args=args,
    train_dataset=dataset,
    data_collator=DataCollatorForLanguageModeling(tok, mlm=False),
)
trainer.train()
trainer.save_model("ft-out/final")

Works for 1–7B on a single high-end GPU. For 70B+, use FSDP or DeepSpeed ZeRO-3.

8.3 LoRA — the technique you''ll actually use

LoRA freezes the pretrained weights and adds tiny rank-decomposed adapter matrices. You train ~0.1% of params. Quality close to full fine-tuning at ~1000× less memory.

from peft import LoraConfig, get_peft_model

model_id = "meta-llama/Llama-3.1-8B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~33M  ||  all params: ~8B  ||  trainable%: 0.4%

# ... train as normal ...
model.save_pretrained("lora-adapter")   # ~130 MB

8.4 QLoRA — LoRA on a 4-bit quantized base

Quantize the frozen base to 4-bit (8× memory saving) and train LoRA on top. Fine-tune a 70B model on a single 24 GB GPU.

from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb, device_map="auto")
# ... add LoRA on top ...

QLoRA is the standard fine-tuning recipe for indie devs and small teams in 2026.

8.5 Instruction tuning vs preference tuning

• SFT (instruction tuning) — train on (instruction, response) pairs.
• DPO / RLHF (preference tuning) — train on (instruction, chosen, rejected) triplets.

Modern open-weight chat models are SFT + DPO. DPO is much simpler than classic RLHF — single supervised loss instead of reward model + PPO.

from trl import DPOTrainer

dpo = DPOTrainer(
    model=peft_model,
    args=args,
    train_dataset=dataset,                 # columns: prompt, chosen, rejected
    tokenizer=tok,
    beta=0.1,
)
dpo.train()

8.6 The fine-tuning decision tree

Goal?
├── New task type → SFT (LoRA or QLoRA)
├── Style / tone alignment → SFT, often LoRA is enough
├── Suppress unwanted outputs → DPO (or KTO if you only have labels)
├── Maximum quality regardless of cost → Full fine-tuning
└── Save inference cost → distill a big model into a smaller one

Almost everyone in 2026 starts with QLoRA + SFT.

---

9. Inference and decoding

9.1 Greedy decoding

next_id = logits.argmax(dim=-1)

Deterministic. Often gets stuck repeating itself.

9.2 Sampling with temperature

logits = logits / temperature
probs  = F.softmax(logits, dim=-1)
next_id = torch.multinomial(probs, num_samples=1)

Chat: 0.7–1.0. Code: 0.0–0.3.

9.3 Top-k sampling

v, _ = torch.topk(logits, k=top_k)
logits[logits < v[:, [-1]]] = float("-inf")

k=40 is common.

9.4 Top-p (nucleus) sampling

def top_p_filter(logits, p=0.9):
    sorted_logits, sorted_idx = torch.sort(logits, descending=True)
    cum_probs = F.softmax(sorted_logits, dim=-1).cumsum(dim=-1)
    mask = cum_probs > p
    mask[..., 1:] = mask[..., :-1].clone()
    mask[..., 0]  = False
    sorted_logits[mask] = float("-inf")
    return sorted_logits.scatter(-1, sorted_idx, sorted_logits)

p=0.9 or p=0.95 is the production default.

9.5 The decoding recipe in practice

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Write a haiku about debugging."}]
inputs = tok.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)

out = model.generate(
    inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.05,
)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

For production, swap model.generate for vLLM — paged attention, continuous batching, prefix caching, often 5–20× faster.

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-7B-Instruct", dtype="bfloat16")
params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=128)
outputs = llm.generate(["Write a haiku about debugging."], params)
print(outputs[0].outputs[0].text)

---

10. Practical patterns

10.1 RAG

from sentence_transformers import SentenceTransformer
import numpy as np

embedder = SentenceTransformer("BAAI/bge-base-en-v1.5")
chunks = ["...", "...", "..."]
embeddings = embedder.encode(chunks, normalize_embeddings=True)

query = "How do I deploy to production?"
q_emb = embedder.encode([query], normalize_embeddings=True)[0]
scores = embeddings @ q_emb
top_k = np.argsort(scores)[::-1][:5]
context = "\n\n".join(chunks[i] for i in top_k)

prompt = f"Use ONLY the following context to answer.\n\nContext:\n{context}\n\nQuestion: {query}"

For real systems: Qdrant / Pinecone / pgvector for the vector store, BGE-reranker after initial retrieval.

10.2 Tool use / agents

tools = [
    {"name": "get_weather", "args_schema": {"city": "string"}},
    {"name": "send_email",  "args_schema": {"to": "string", "subject": "string", "body": "string"}},
]

prompt = f"""You have these tools: {tools}
Output {{"tool": "...", "args": {{...}}}} to call one, or {{"answer": "..."}} when done.
"""

LangGraph, OpenAI function-calling, Anthropic tool-use all formalize this loop.

10.3 Embeddings for classification

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

emb = SentenceTransformer("BAAI/bge-base-en-v1.5")
X = emb.encode(texts)
clf = LogisticRegression().fit(X, labels)

For "is this positive?" or "find similar tickets," embeddings + small classifier beats generation, much cheaper at inference.

---

11. The honest stuff

• Tokenization is the source of most "why does the model do X?" bugs.
• Context length costs scale quadratically with attention. 32k isn''t 4× 8k — it''s ~16×.
• Most prod LLM systems are 90% retrieval + 10% LLM.
• Fine-tuning is rarely the answer. Try prompting, RAG, and a bigger model first.
• Evals beat vibes. Build a golden set of 50–200 queries before you fine-tune.
• Open-weight models in 2026 are good enough for most tasks.
• Inference cost dominates training cost. Optimize for serving.

---

12. The 2026 model menu

Goal	Pick
Chatbot / general assistant	Llama-3.1-8B-Instruct or Qwen-2.5-7B-Instruct (open), Claude / GPT-4o (closed)
Coding	Qwen-2.5-Coder-7B / 32B (open), Claude / GPT for top quality
Embeddings	BAAI/bge-base-en-v1.5 (English), bge-m3 (multilingual)
Multilingual generation	Qwen-2.5, Llama-3.1
Long context (>128k)	Qwen-2.5-Long or Gemini-1.5-Pro
On-device	Phi-3-mini / Llama-3.2-3B / Qwen-2.5-3B
Reasoning-heavy tasks	DeepSeek-R1 / o1-mini

---

13. The mental checklist

Before shipping:

• Token counts measured for the longest realistic inputs.
• Chat templates applied via apply_chat_template, not hand-concatenated.
• Decoding settings chosen (temperature, top-p) and documented per use case.
• A golden eval set exists, with ≥50 queries and expected outputs.
• Failure modes reviewed by hand on the eval set.
• Latency and cost per request measured under load.
• PII redaction at the prompt boundary.
• Trace IDs propagated (Langfuse, OpenTelemetry).
• Prompt + model version logged with every response.
• Production inference uses vLLM / TGI / Triton.

---

14. Closing — the right mental model

An LLM is not magic. It''s:

1. A tokenizer.
2. An embedding table.
3. A stack of transformer blocks.
4. A head that projects back to vocabulary.

Pretraining gives it general competence. Fine-tuning teaches it your task. Decoding turns probabilities into text.

Three habits that make you good at LLMs fast:

1. Always inspect the tokens. When something is weird, look at the tokens first.
2. Build the eval set before you build the model.
3. Reach for the smallest model that works.

Internalize those, write the code, and LLMs stop feeling like magic — they feel like a particularly elegant predictor wrapped in a sampling loop.

---

Further reading

• Andrej Karpathy''s "Let''s build GPT" — builds nanoGPT from scratch.
• Attention Is All You Need — Vaswani et al. The original paper.
• Hugging Face course — free, modern, end-to-end.
• Llama 3 paper — most detailed open-weights training report ever.
• vLLM — production inference for open-weight LLMs.
• PEFT docs — LoRA, QLoRA in one library.
• LangChain and LlamaIndex — RAG and agent frameworks.

---

Got an LLM project that''s not behaving? Email randhir.jassal@gmail.com with the prompt and the output and I''ll tell you which mechanism is misbehaving.