Quick Context
Multiplying two matrices produces a new matrix in which every single entry is a dot product — one row of the left matrix against one column of the right. Nothing more complicated is happening.
The Rule for a Single Cell
C[i][j] = row i of A · column j of B
= A[i][1]×B[1][j] + A[i][2]×B[2][j] + …
Click any cell in C and the app highlights the blue row and amber column that feed it, then writes out the arithmetic term by term. Every cell is independent of every other — which is precisely why this operation parallelises so well on a GPU.
The Shape Rule
You cannot multiply arbitrary matrices. The inner dimensions must match:
(m × n) × (n × p) = (m × p)
↑ ↑
these must be equal
Drag the sliders and watch the shape line update. This single rule is behind most tensor-shape errors you will ever hit: a mismatch is not a subtle numerical problem, it is an operation that simply has no definition.
Order Matters
Matrix multiplication is not commutative: AB ≠ BA in general. Often BA is not even a legal operation — a (2×3) times a (3×4) works, but (3×4) times (2×3) does not. When people say the order of layers matters, this is the literal reason.
It is associative, though: (AB)C = A(BC). Choosing the cheaper grouping can save enormous amounts of computation, which is exactly the trick behind LoRA's low-rank update.
Why This Is the Bottleneck
Multiplying two n×n matrices needs about n³ multiply-add operations. Doubling the size makes it eight times more work. A single transformer layer with d = 4096 performs billions of these per token.
Watch the "Total ops" counter as you raise the sliders. That cubic growth is the entire reason GPUs — chips designed to do thousands of independent multiply-adds at once — became the hardware of deep learning.
Interactive Exploration Guide
- Click the top-left cell of C. Only the first row of A and first column of B light up — the rest of both matrices is irrelevant to that number.
- Press "Walk every cell" to sweep through the whole output and see the pattern of row-column pairings.
- Set n to 1. Each cell becomes a single product — this is the outer product, and it is exactly what LoRA's B·A builds at rank 1.
- Push all three sliders to 5 and watch the operation count climb far faster than the matrices grow.
Key Takeaway
Row against column, multiply and add, repeat for every output cell. The inner dimensions must agree, the order cannot be swapped, and the cost grows cubically — three facts that explain tensor errors, layer ordering and the entire GPU industry.