Generative AI & LLMs

About this track

A large language model predicts one token at a time. Everything else - the tokenizer in front of it, the fine-tuning that specialises it, the quantization that fits it on a smaller card - exists to serve that loop.

This track starts with how text is split and encoded, moves through the two training objectives that produced most current models, and ends on the practical techniques: LoRA, quantization and distillation.

A large language model predicts one token at a time, and everything around it exists to make that prediction useful: the tokenizer in front, the training objective behind, the retrieval that grounds it, and the quantisation that makes it servable.

What you will be able to do

• Explain how text is split into tokens and why byte-pair encoding handles unseen words.
• Distinguish causal from masked language modelling, and say which models each produced.
• Describe what a context window costs and what a KV cache is caching.
• Build a retrieval pipeline and evaluate it with the metric that matches the task.
• Say what LoRA, quantisation and distillation each trade away in exchange for what.

How the track is ordered

Tokenization first, because it determines what the model can represent. Then the two training objectives - causal and masked - that account for most current models, followed by the mechanics of generation: next-token prediction, context windows and KV caching. Retrieval comes next and takes up much of the track, from embeddings and vector search through chunking, hybrid retrieval, reranking and the retriever variants, ending on how to evaluate any of it. Adaptation and efficiency - fine-tuning, RLHF, LoRA, quantisation, distillation - close the track. The NLP track is the prerequisite.

Where this leads

Retrieval quality is mostly an embeddings-and-evaluation problem, so the NLP and machine learning tracks both feed back into it. If you are building rather than studying, the retrieval modules are the ones that change day-to-day results the most.