Glossary

ACID

Atomicity, consistency, isolation, durability: the four guarantees a transaction is expected to hold to.

Taught in Transactions and ACID in SQL

attentionalso: attention mechanism

A way for each position in a sequence to look at every other position and decide, per input, which ones matter.

Taught in The Attention Mechanism

AUCalso: area under the curve, ROC AUC

The chance that the model scores a random positive above a random negative. 0.5 is coin-flipping.

Taught in ROC Curve and AUC

backpropagationalso: backprop

Applying the chain rule backwards through a network to get each parameter's share of the blame for the loss.

Taught in Backpropagation and the Computational Graph

batch normalizationalso: batch norm

Rescaling a layer's outputs using the statistics of the current batch, which keeps activations in a workable range.

Taught in Batch Normalization in Deep Networks

bi-encoderalso: bi-encoders

A model that embeds the query and each document separately, so the document vectors can be computed ahead of time. Fast to search, but it never sees the two texts side by side.

Taught in Re-ranking: Bi-Encoders vs Cross-Encoders

BM25

A keyword-matching score that rewards rare query terms and stops rewarding repetition past a point. The standard sparse-retrieval baseline.

Taught in BM25 and Sparse Lexical Retrieval

chunkingalso: chunk, chunks

Splitting a document into retrievable pieces. Cut too small and a piece loses its context; too large and it drags in noise.

Taught in Chunking Strategies for RAG

cosine similarityalso: cosine similarities

How closely two vectors point in the same direction, ignoring how long they are. 1 means identical direction, 0 means unrelated.

Taught in Dot Product vs Cosine Similarity for Retrieval

cross-encoderalso: cross-encoders

A model that reads the query and the document together and scores the pair. Much more accurate than comparing two separate vectors, and far too slow to run over a whole corpus.

Taught in Re-ranking: Bi-Encoders vs Cross-Encoders

cross-entropyalso: cross entropy

The standard classification loss: how surprised the model was by the correct answer.

Taught in Softmax and Cross-Entropy

dot productalso: dot products

Multiply two vectors element by element and add the results. Unlike cosine, it grows with vector length, so a long vector can win on size rather than on direction.

Taught in Dot Product vs Cosine Similarity for Retrieval

dropout

Randomly switching off units during training so the network cannot lean on any single one.

Taught in Dropout Regularization

eigenvectoralso: eigenvectors, eigenvalue, eigenvalues

A direction a matrix only stretches or shrinks rather than rotates, and the factor by which it does so.

Taught in Eigenvalues and Eigenvectors

embeddingalso: embeddings

A list of numbers standing for a piece of text, arranged so that things meaning similar things sit near each other.

Taught in Embeddings and Vector Search

entropy

How uncertain a distribution is - how many bits, on average, it takes to say which outcome happened.

Taught in Entropy and Information

epochalso: epochs

One full pass over the training set.

Taught in Batch Processing in Networks

gradient

The vector of partial derivatives: which way is uphill, and how steeply, in every direction at once.

Taught in Partial Derivatives and the Gradient

gradient descent

Repeatedly nudging parameters in the direction that reduces the loss fastest.

Taught in Gradient Descent

hallucinationalso: hallucinations

A fluent, confident answer that is not supported by anything the model was given or trained on.

Taught in Hallucination and Grounding

HyDEalso: hypothetical document embeddings

Ask a model to draft a plausible answer, then search using that draft instead of the question - because an answer resembles an answer more than a question does.

Taught in Query Rewriting and HyDE

IDFalso: inverse document frequency

Inverse document frequency: how rare a term is across the whole collection. Rare terms say more about a match than common ones.

Taught in BM25 and Sparse Lexical Retrieval

indexalso: database index

A sorted side structure that lets the database find matching rows without reading every one.

Taught in Indexes and Query Performance

IoUalso: intersection over union

Intersection over union: the area two boxes share divided by the area they cover between them. 1 is identical, 0 is no overlap.

Taught in IoU and Non-Max Suppression

KL divergencealso: Kullback-Leibler divergence

How far one probability distribution is from another. Zero when they match, and not symmetric.

Taught in Cross-Entropy and KL Divergence

KV cachealso: key-value cache

The stored keys and values for tokens already generated, so each new token does not re-read the whole sequence from scratch.

Taught in Context Windows and the KV Cache

learning rate

How big a step to take on each update. Too small and training crawls; too large and it overshoots and never settles.

Taught in Gradient Descent

logitalso: logits

A raw, unnormalised score straight out of a model, before softmax turns it into a probability.

Taught in Softmax and Cross-Entropy

LoRAalso: low-rank adaptation

Fine-tuning by training a small pair of extra matrices alongside frozen weights, instead of updating the whole model.

Taught in LoRA in LLMs

maximal marginal relevancealso: MMR

Picking each next result by relevance minus similarity to what you already picked, so near-duplicates stop crowding out everything else.

Taught in Maximal Marginal Relevance (MMR)

MRRalso: mean reciprocal rank

Mean reciprocal rank: one divided by the position of the first relevant result. It cares only about how long you wait for one good answer.

Taught in Retrieval Evaluation Metrics

nDCGalso: normalized discounted cumulative gain

A ranking score where every relevant result counts, but one near the top counts far more than one near the bottom.

Taught in Retrieval Evaluation Metrics

non-max suppressionalso: NMS

Keep the highest-confidence box, throw away everything overlapping it past a threshold, repeat. How a detector stops returning the same object several times.

Taught in IoU and Non-Max Suppression

normalization

Putting features on a comparable scale so no one of them dominates purely because its units are bigger.

Taught in Feature Scaling & Weight Bias

overfittingalso: overfit

Learning noise specific to the training set, so performance on data the model has not seen gets worse rather than better.

Taught in Dropout Regularization

PCAalso: principal component analysis

Finding the directions along which data varies most, and describing each point by those instead of the original axes.

Taught in Principal Component Analysis

precision

Of the things you flagged as positive, the fraction that really were.

Taught in Confusion Matrix Analysis

precision@kalso: precision at k

Of the k results you returned, the fraction that were relevant. Punishes returning junk.

Taught in Retrieval Evaluation Metrics

quantization

Storing a model's weights at lower numeric precision to shrink it, trading a little accuracy for a lot of memory.

Taught in Quantization in LLMs

RAGalso: retrieval-augmented generation

Retrieval-Augmented Generation: fetch relevant passages first, then put them in the model's prompt so its answer is grounded in them.

Taught in Retrieval-Augmented Generation (RAG)

recall

Of the things that really were positive, the fraction you caught.

Taught in Confusion Matrix Analysis

recall@kalso: recall at k

Of all the relevant results that exist, the fraction your top k found. Punishes missing things entirely.

Taught in Retrieval Evaluation Metrics

reciprocal rank fusionalso: RRF

A way to merge two ranked lists using only each document's position in them, so two scores that are on different scales never have to be compared directly.

Taught in Hybrid Search: Dense + Sparse (Reciprocal Rank Fusion)

regularization

Any penalty that discourages a model from fitting its training data too exactly.

Taught in Regularization Techniques

residual connectionalso: skip connection, residual connections, skip connections

Adding a block's input back onto its output, which gives gradients an unimpeded route back and stops deep stacks degrading.

Taught in Residual and Skip Connections

softmax

Turns a list of arbitrary scores into probabilities that are all positive and sum to 1, keeping their order.

Taught in Softmax and Cross-Entropy

tokenalso: tokens, tokenization

The unit a language model actually reads: usually a word piece rather than a whole word or a single letter.

Taught in How LLMs Process Text?

vanishing gradientalso: vanishing gradients

When gradients shrink at every layer on the way back, so early layers receive almost no signal and stop learning.

Taught in Vanishing & Exploding Gradients

variance

How spread out values are around their mean, in squared units.

Taught in Mean, Variance and Standard Deviation