The transition matrix
Collect the probabilities into a matrix P, where P[i][j] is the chance of moving from state i to state j. Every row sums to 1, because from any state you go somewhere, possibly back to where you were.
If the current distribution over states is a row vector d, then after one step it is d P. After k steps, d P^k. That is all the arithmetic there is, and the visualisation is doing exactly it, forty steps at a time.
Convergence
Watch the three curves. They start apart — all the probability on one state — and within a few steps they flatten out and stop moving. The readout shows the last step changing things by around 10⁻¹⁰.
Now change the starting state. The curves begin somewhere completely different and arrive at the same values.
That limit is the stationary distribution: the pi with pi P = pi. Read as an eigenvector problem, it is the left eigenvector of P with eigenvalue 1, and the [eigenvalue module](eigenvalues_and_eigenvectors.html) is the same mathematics in another costume.
The convergence is not automatic. It needs the chain to be irreducible — every state reachable from every other — and aperiodic, not trapped in a fixed cycle. Drag a transition probability to its minimum and convergence slows visibly; a chain that could not leave a state at all would never mix.
The speed is governed by the second-largest eigenvalue: the closer it is to 1, the slower the mixing. That number has a name, the *spectral gap*, and it is what people mean by a chain mixing slowly.
Where they appear
PageRank is the stationary distribution of a random surfer following links. The damping factor exists to make the chain irreducible, so a page with no outbound links cannot swallow all the probability.
MCMC — Metropolis-Hastings, Gibbs sampling, Hamiltonian Monte Carlo — runs the idea backwards: construct a chain whose stationary distribution *is* the posterior you want, run it, and treat the states it visits as samples. Nearly all Bayesian computation is this.
Hidden Markov models put a Markov chain behind observations you can see, and were the backbone of speech recognition for decades.
Reinforcement learning. A Markov decision process is a Markov chain with actions and rewards attached. The "Markov" in MDP is this property, and it is what makes value functions of the state alone sufficient.
n-gram language models are Markov chains over words — and their limitation is precisely the property: a bigram model cannot remember anything beyond the previous word, which is why long-range coherence needed something else.
Where it goes wrong
Assuming the property holds. Most real processes have memory. Widening the state to include recent history restores the property at the cost of a much larger state space.
Assuming convergence. Check irreducibility and aperiodicity.
Stopping an MCMC run too early. Samples before the chain has mixed reflect where you started, which is what burn-in discards and why convergence diagnostics exist.
Confusing the stationary distribution with the most likely state. It is the long-run fraction of time spent in each state, not a prediction of where the chain is now.