How LLMs Work | Chaturmind

How Large Language Models Work

Large Language Models (LLMs) like GPT-4, Claude, and Llama are neural networks trained on massive text datasets to predict the next token. Understanding their internals helps you use them more effectively.

Tokenization

Text is split into tokens — roughly 4 characters per token. LLMs don't see words; they see token IDs.

"Hello, world!" → ["Hello", ",", " world", "!"] → [15496, 11, 995, 0]

Tokenizer (BPE - Byte Pair Encoding):
  Builds vocabulary of common character sequences
  "unhappy" → ["un", "happy"]
  "GPT-4" → ["G", "PT", "-", "4"]

The Transformer Architecture

Input tokens
    ↓
[Token Embeddings + Position Embeddings]
    ↓
[Transformer Block × N]
  ├── Multi-Head Self-Attention
  ├── Add & Norm (residual connection)
  ├── Feed-Forward Network (MLP)
  └── Add & Norm
    ↓
[Linear layer + Softmax]
    ↓
Probability distribution over vocabulary

Self-Attention: The Core Mechanism

Self-attention lets each token attend to all other tokens, learning contextual relationships.

For each token, compute Query (Q), Key (K), Value (V):
  Q = token embedding × Wq
  K = token embedding × Wk
  V = token embedding × Wv

Attention score = softmax(Q × Kᵀ / √d_k) × V

Interpretation:
  Q: "what am I looking for?"
  K: "what do I offer?"
  V: "what information I carry"

"The bank was on the river bank" → "bank" (word 2) attends strongly
to "river" → resolves ambiguity via context

Pre-Training (Next Token Prediction)

Task: Given ["The", "cat", "sat", "on"], predict "the"

Training data: Common Crawl, Wikipedia, Books, Code
  → ~1 trillion tokens
  → 10-100B parameters
  → Months of compute on thousands of GPUs

Result: Model learns grammar, facts, reasoning, world knowledge
by compressing internet-scale text into weights

Fine-Tuning and RLHF

Pre-trained model → Fine-tuning stages:

1. Supervised Fine-Tuning (SFT):
   Train on high-quality (instruction, response) pairs
   Model learns to follow instructions

2. RLHF (Reinforcement Learning from Human Feedback):
   Human raters rank responses
   Train a Reward Model on preferences
   Use PPO (RL algorithm) to optimize against reward model
   Result: helpful, harmless, honest responses

Context Window

Context window = max tokens the model can process at once
  GPT-3.5: 4K tokens
  GPT-4: 128K tokens
  Claude 3: 200K tokens

Important:
  - Model can attend to ALL tokens in its context window
  - Longer context = slower inference (attention is O(n²))
  - Tokens beyond the window are forgotten

Temperature and Sampling

# Temperature controls randomness of token selection
# Low temperature (0.1): deterministic, focused
# High temperature (1.0): creative, diverse
# Temperature=0: always picks most likely token (greedy)

# Top-p (nucleus sampling): only sample from top-p probability mass
# p=0.9: ignore tokens with combined probability < 10%

# Example:
openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Write a poem"}],
    temperature=0.9,  # creative
    top_p=0.95
)

Emergent Capabilities

LLMs develop surprising capabilities at scale that weren't explicitly trained:

Few-shot learning: learn from 3-5 examples in the prompt
Chain-of-thought reasoning: "Let's think step by step"
Code generation: translate natural language to working code

Interview Tips

LLMs are next-token predictors — all capabilities emerge from this simple objective at scale.
The attention mechanism resolves word meaning from context — it's why LLMs handle ambiguity well.
RLHF is what makes a pre-trained model into a useful assistant — raw pre-training alone produces a text predictor, not a conversational agent.

How Large Language Models Work

Tokenization

Text is split into tokens — roughly 4 characters per token. LLMs don't see words; they see token IDs.

"Hello, world!" → ["Hello", ",", " world", "!"] → [15496, 11, 995, 0]

Tokenizer (BPE - Byte Pair Encoding):
  Builds vocabulary of common character sequences
  "unhappy" → ["un", "happy"]
  "GPT-4" → ["G", "PT", "-", "4"]

The Transformer Architecture

Input tokens
    ↓
[Token Embeddings + Position Embeddings]
    ↓
[Transformer Block × N]
  ├── Multi-Head Self-Attention
  ├── Add & Norm (residual connection)
  ├── Feed-Forward Network (MLP)
  └── Add & Norm
    ↓
[Linear layer + Softmax]
    ↓
Probability distribution over vocabulary

Self-Attention: The Core Mechanism

Self-attention lets each token attend to all other tokens, learning contextual relationships.

For each token, compute Query (Q), Key (K), Value (V):
  Q = token embedding × Wq
  K = token embedding × Wk
  V = token embedding × Wv

Attention score = softmax(Q × Kᵀ / √d_k) × V

Interpretation:
  Q: "what am I looking for?"
  K: "what do I offer?"
  V: "what information I carry"

"The bank was on the river bank" → "bank" (word 2) attends strongly
to "river" → resolves ambiguity via context

Pre-Training (Next Token Prediction)

Task: Given ["The", "cat", "sat", "on"], predict "the"

Training data: Common Crawl, Wikipedia, Books, Code
  → ~1 trillion tokens
  → 10-100B parameters
  → Months of compute on thousands of GPUs

Result: Model learns grammar, facts, reasoning, world knowledge
by compressing internet-scale text into weights

Fine-Tuning and RLHF

Pre-trained model → Fine-tuning stages:

1. Supervised Fine-Tuning (SFT):
   Train on high-quality (instruction, response) pairs
   Model learns to follow instructions

2. RLHF (Reinforcement Learning from Human Feedback):
   Human raters rank responses
   Train a Reward Model on preferences
   Use PPO (RL algorithm) to optimize against reward model
   Result: helpful, harmless, honest responses

Context Window

Context window = max tokens the model can process at once
  GPT-3.5: 4K tokens
  GPT-4: 128K tokens
  Claude 3: 200K tokens

Important:
  - Model can attend to ALL tokens in its context window
  - Longer context = slower inference (attention is O(n²))
  - Tokens beyond the window are forgotten

Temperature and Sampling

# Temperature controls randomness of token selection
# Low temperature (0.1): deterministic, focused
# High temperature (1.0): creative, diverse
# Temperature=0: always picks most likely token (greedy)

# Top-p (nucleus sampling): only sample from top-p probability mass
# p=0.9: ignore tokens with combined probability < 10%

# Example:
openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Write a poem"}],
    temperature=0.9,  # creative
    top_p=0.95
)

Emergent Capabilities

LLMs develop surprising capabilities at scale that weren't explicitly trained:

Few-shot learning: learn from 3-5 examples in the prompt
Chain-of-thought reasoning: "Let's think step by step"
Code generation: translate natural language to working code

Interview Tips

LLMs are next-token predictors — all capabilities emerge from this simple objective at scale.
The attention mechanism resolves word meaning from context — it's why LLMs handle ambiguity well.
RLHF is what makes a pre-trained model into a useful assistant — raw pre-training alone produces a text predictor, not a conversational agent.