What LLMs Are (and What They Aren't)

This lesson on What LLMs Are (and What They Aren't) is hands-on and example-driven. You will master the fundamental mechanics of Large Language Models, from autoregressive next-token prediction and self-attention vector enrichment to large-scale pre-training and alignment via RLHF. You will be able to trace how discrete text transforms into dense embeddings, processes through transformer layers in parallel, and projects back into probabilistic text completions.

What You'll Be Able To Do

• Trace the autoregressive token generation lifecycle from prompt ingestion to probabilistic sampling.
• Contrast base model pre-training objectives against post-training RLHF alignment mechanisms.
• Map discrete text tokens into continuous vector embeddings and trace dynamic contextual shifts during self-attention.
• Differentiate the roles of attention mechanisms and feed-forward layers within a standard transformer block.
• Evaluate trade-offs between deterministic model architecture and stochastic sampling during inference.

Detailed Concept Walkthrough

1. Autoregressive Next-Token Prediction

LLMs are statistical completion engines that compute probability distributions over a fixed vocabulary to predict the next token given preceding context. Responses are generated iteratively by appending each sampled token back into the input sequence.

• Mechanism: The model takes an input sequence, computes raw logits across the vocabulary, converts them to probabilities via softmax, and selects the next token using sampling strategies. This newly chosen token is appended to the prompt for the subsequent forward pass.
• Under the Hood: While underlying matrix multiplications are strictly deterministic, stochastic sampling introduces controlled randomness into token selection. This prevents repetitive loops and produces varied, natural dialogue outputs across runs.
• Execution Flow: Generation halts when a predefined maximum length is reached or when the model emits an end-of-sequence token, terminating the autoregressive loop.

import torch
import torch.nn.functional as F

# Simulated logits for vocabulary of size 4
logits = torch.tensor([1.2, 3.8, 0.5, 2.1])
# Convert logits to a valid probability distribution
probs = F.softmax(logits, dim=-1)
# Sample next token index stochastically
next_token_id = torch.multinomial(probs, num_samples=1).item()

Key Takeaway: Text generation is an iterative loop of calculating vocabulary probability distributions and sampling the next token.

2. Pre-Training and Parameter Optimization

Pre-training converts billions of randomly initialized continuous numerical weights into structured linguistic knowledge via internet-scale self-supervised learning.

• Mechanism: Massive unannotated text corpora are truncated into token sequences where future tokens serve as ground-truth labels for self-supervised learning. The network attempts next-token prediction on each sequence.
• Under the Hood: A loss function measures the discrepancy between predicted probability distributions and actual target tokens across trillions of tokens. Backpropagation then calculates gradients to adjust continuous parameters across the network.
• Nuance: Pre-training optimizes solely for statistical text continuation, resulting in a general completion engine that mimics internet text patterns rather than a helpful assistant.

# Conceptual cross-entropy loss computation during pre-training
import torch.nn as nn

loss_fn = nn.CrossEntropyLoss()
# logits: [batch_size, vocab_size], targets: [batch_size]
loss = loss_fn(predicted_logits, target_token_ids)
loss.backward()  # Compute gradients for parameter updates

Key Takeaway: Pre-training optimizes model parameters for raw text completion rather than instruction following or truthfulness.

3. RLHF and Human Alignment

Reinforcement Learning with Human Feedback (RLHF) bridges the gap between raw statistical text continuation and safe, helpful conversational interaction.

• Mechanism: Human annotators review and rank multiple candidate completions generated by the base model for specific prompts, flagging harmful, unhelpful, or factually incorrect outputs.
• Under the Hood: Annotations train a reward model that scores output desirability, providing a scalar feedback signal used to fine-tune the base model weights via reinforcement learning.
• Nuance: Alignment acts as a behavioral steering layer on top of base model representations, guiding the system to adopt assistant personas without re-training base weights from scratch.

# High-level representation of reward-guided alignment updates
reward = reward_model(prompt, candidate_response)
# Optimization step adjusts policy network parameters to maximize reward
policy_loss = -torch.log(policy_prob) * reward
policy_loss.backward()

Key Takeaway: RLHF steers base models away from unstructured text continuation toward aligned, instruction-following behavior.

4. Vector Embeddings and Transformer Blocks

Transformers process discrete tokens in parallel by mapping them into continuous vector embeddings, which are sequentially enriched by alternating self-attention and feed-forward layers.

• Mechanism: Input tokens are mapped to dense numerical vectors, allowing parallel ingestion across GPU cores instead of sequential recurrence. These embeddings represent baseline semantic positions in a continuous vector space.
• Under the Hood: Self-attention dynamically recalculates vector weights based on surrounding words to resolve polysemy (e.g., distinguishing 'bank' as a riverbank vs. a financial institution). Feed-forward neural network (FFN) layers then project these vectors through stored pattern representations.
• Execution Flow: Information flows through repeated stacks of attention and feed-forward blocks, continuously refining contextual representations before final vocabulary projection.

# Conceptual forward pass through a single transformer layer
import torch.nn as nn

# Dynamic contextual re-weighting
context_vectors = self_attention(embedded_tokens)
# Pattern retrieval and non-linear feature transformation
layer_output = feed_forward_network(context_vectors)

Key Takeaway: Self-attention dynamically contextualizes token embeddings, while feed-forward networks retrieve and transform stored pattern representations.

Topics Covered in What LLMs Are (and What They Aren't)

1. Autoregressive Generation (0:00 - 1:25) — The model calculates next-token probability distributions and samples from them to generate conversational completions.
2. Pre-Training and Optimization (1:26 - 3:43) — Loss computation and backpropagation update billions of continuous numerical weights across trillions of internet tokens.
3. RLHF Alignment (3:44 - 4:13) — Human feedback annotations train reward models to steer base completion behavior toward helpful and safe responses.
4. Embeddings and Parallelism (4:14 - 5:09) — Discrete tokens are projected into continuous numerical vector spaces that execute in parallel across GPU architectures.
5. Transformer Mechanics (5:09 - 6:26) — Alternating self-attention mechanisms and feed-forward layers enrich contextual vector representations to resolve polysemy.
6. Emergent Behavior (6:27 - 6:59) — Multi-billion parameter scale drives emergent fluency while presenting black-box explainability trade-offs.

LLMs & Generative AI for Practitioners Cheat Sheet

• Autoregressive Generation — Predicts next token iteratively from preceding context

  output_tokens.append(sample_next_token(model(input_tokens)))
  

• Softmax — Normalizes raw model logits into probabilities

  probabilities = torch.softmax(logits, dim=-1)
  

• Stochastic Sampling — Selects tokens probabilistically to introduce natural variance

  next_token = torch.multinomial(probabilities, 1)
  

• Dense Vector Embedding — Maps discrete token IDs to continuous vectors

  embeddings = nn.Embedding(vocab_size, embedding_dim)
  

• Self-Attention Block — Dynamically re-weights token vectors using context

  attn_out, _ = nn.MultiheadAttention(embed_dim, num_heads)(q, k, v)
  

• Feed-Forward Layer — Applies non-linear transformations to contextual vectors

  ffn_out = ffn_layer(attn_out)
  

• RLHF Reward Tuning — Aligns model behavior using human feedback signals

  loss = -reward_score * log_prob
  

Comparison Table

Common Pitfalls

• Mistake: Expecting a base pre-trained model to act as a conversational assistant out of the box. Avoid: Apply RLHF or instruction tuning to steer raw text continuation into helpful dialogue.
• Mistake: Assuming token embeddings remain static throughout the transformer forward pass. Avoid: Account for dynamic attention mechanisms that modify vector numerical values based on context.
• Mistake: Believing identical prompts always yield identical outputs because architecture is deterministic. Avoid: Recognize that inference sampling strategies inject randomness into next-token selection.

FAQs

• Why does a model output different answers for the same prompt if its weights are fixed? Although matrix multiplications are deterministic, inference engines use stochastic sampling over the predicted probability distribution to choose tokens.
• How does self-attention resolve ambiguous words with multiple meanings? It recalculates the numerical values of a word's vector embedding by weighting the embeddings of surrounding context tokens during the forward pass.
• Why is pre-training alone insufficient for deploying user-facing chatbots? Pre-training optimizes strictly for raw internet text completion, which often replicates unhelpful, toxic, or irrelevant continuation patterns.
• What is the computational advantage of transformers over older recurrent architectures? Transformers process entire input sequences in parallel on GPU architectures rather than sequentially processing one word at a time.