Identity, Inverse and Transpose

The identity

I = [[1, 0], [0, 1]]

Its columns are exactly where i and j already were, so it moves nothing. It is the 1 of matrix multiplication: AI = IA = A for every A, and the thing an inverse has to produce.

The inverse

A⁻¹ = (1 / det A) · [[d, −b], [−c, a]]

That is not a notational accident. A matrix with zero determinant has flattened the plane onto a line; two different starting points now sit on top of each other, and no transformation can pull them apart again. The information is gone, so the undo cannot exist.

A determinant that is merely small is its own kind of trouble: the inverse divides by it, so tiny errors in the input become enormous errors in the output. That is what an ill-conditioned matrix is, and it is why numerical code solves Ax = b rather than computing A⁻¹b.

The transpose

Aᵀ = [[a, c], [b, d]]

Reflect the matrix across its diagonal: rows become columns. Geometrically it is not an undo and generally not related to one — try it on the plot and the shape does not come back.

Its real job is bookkeeping in products. (AB)ᵀ = BᵀAᵀ, the dot product of two vectors is aᵀb, and XᵀX is what turns a tall data matrix into the square, symmetric thing that least squares and covariance both need. When you see a stray transpose in a formula, it is almost always there to make two shapes line up.

There is one important case where the two coincide. If A's columns are perpendicular unit vectors — an orthogonal matrix, a rotation or a reflection — then Aᵀ = A⁻¹ exactly. Undoing a rotation costs a transpose instead of a division, which is why numerical methods work so hard to keep things orthogonal.

Three operations that keep appearing

The identity matrix I has ones on the diagonal and zeros elsewhere. Multiplying by it changes nothing:

AI = IA = A

It is the matrix equivalent of the number 1, and it is what "no transformation" looks like.

The inverse A⁻¹ undoes what A did:

A A⁻¹ = A⁻¹ A = I

If A rotates by 30°, its inverse rotates by −30°. If A doubles everything, the inverse halves it.

The transpose Aᵀ flips the matrix across its diagonal, turning rows into columns:

[[1, 2, 3], [4, 5, 6]]ᵀ = [[1, 4], [2, 5], [3, 6]]

A 2×3 becomes a 3×2, which is most often exactly what you need to make a matrix product's shapes line up.

Not every matrix has an inverse

A matrix is invertible only if its determinant is non-zero. Zero determinant means the transformation collapsed a dimension — flattened a plane onto a line — and no operation can recover what was lost, because many different inputs now map to the same output.

Such matrices are called singular, and the geometric reading is exact: information was destroyed.

For a 2×2 matrix the inverse has a closed form:

A = [[a, b], [c, d]]  →  A⁻¹ = (1 / (ad − bc)) [[d, −b], [−c, a]]

The ad − bc in the denominator is the determinant, and the division by it is why a zero determinant breaks everything.

Beyond 2×2, do not compute inverses. Solving Ax = b directly is faster and far more numerically stable than forming A⁻¹ and multiplying:

import numpy as np

x = np.linalg.solve(A, b)      # do this
x = np.linalg.inv(A) @ b       # not this - slower and less accurate

Even when a matrix is technically invertible, being nearly singular is a practical problem: a tiny determinant makes the inverse enormous, and small errors in the input become large errors in the output. That condition is exactly what multicollinearity does to a regression's normal equations, and it is why ridge regression — which adds a small multiple of the identity before inverting — is so effective.

The transpose, and the rules that matter

Two identities do most of the work:

(AB)ᵀ = BᵀAᵀ    (Aᵀ)ᵀ = A

The reversal in the first is not a quirk — it is what makes the shapes valid.

A matrix equal to its own transpose is symmetric, and symmetric matrices are the well-behaved ones: real eigenvalues, orthogonal eigenvectors, and a clean decomposition. Covariance matrices, correlation matrices and Gram matrices are all symmetric, which is why the mathematics of PCA is so tidy.

A matrix whose transpose is its inverse (QᵀQ = I) is orthogonal. Those represent rotations and reflections — transformations that preserve lengths and angles — and they are numerically ideal, because inverting them costs nothing and they never amplify error.

The three, checked

Each of the three has one defining property, and each is one line to confirm.

import numpy as np

A = np.array([[4.0, 7.0], [2.0, 6.0]])
I = np.eye(2)

print("A @ I is A:", np.allclose(A @ I, A))
inv = np.linalg.inv(A)
print()
print("inverse:"); print(inv.round(4))
print("A @ inv:"); print((A @ inv).round(10))
print()
print("transpose swaps rows and columns:")
print(A.T)
print()
print("(A @ B).T == B.T @ A.T - the order reverses:")
B = np.array([[1.0, 0.0], [3.0, 2.0]])
print(" ", np.allclose((A @ B).T, B.T @ A.T),
      " but (A@B).T == A.T @ B.T is", np.allclose((A @ B).T, A.T @ B.T))

Try it yourself

1. Do and undo. Press Apply A, then Apply A⁻¹. The shape leaves and comes back exactly, and the computed A A⁻¹ panel reads the identity.
2. Break it. Set c to 1.5 and d to 1.0, leaving a at 1.5 and b at 1.0. Now ad = bc and the determinant is exactly 0. The shape collapses onto a line, the inverse panel turns red, and the undo button is disabled — there is nothing to press.
3. Get close to broken. Nudge d up to 1.1 so the determinant is 0.15. The inverse exists, and its entries are enormous — that magnification is exactly the numerical instability people mean by "ill-conditioned".
4. Try the transpose as an undo. Tick Undo With Aᵀ Instead and run do-then-undo. The shape does not return, because the transpose was never the undo.
5. Unless it is. Set a = 0.6, b = −0.8, c = 0.8, d = 0.6 — a rotation. Now Aᵀ = A⁻¹ reads yes, and undoing with the transpose works perfectly.
6. Find the identity. a = 1, b = 0, c = 0, d = 1. Apply it as often as you like; nothing ever moves.

Worth remembering

The identity is the matrix that changes nothing, and the inverse is the one that undoes A — defined by A A⁻¹ = I and computed by dividing through by the determinant, which is precisely why a zero determinant means no inverse exists: the transformation destroyed information and no matrix can recover it. A small determinant is the same problem in slow motion, and the reason numerical code solves systems rather than inverting matrices. The transpose only looks like a relative: it flips rows and columns, it exists to make shapes agree in products such as XᵀX, and it equals the inverse in exactly one case — when the matrix is orthogonal.

Where they show up

Solving linear systems. The normal equation for least squares, β = (XᵀX)⁻¹Xᵀy, uses all three operations in one line: a transpose to form the square matrix, an inverse to solve it, and the identity implicitly as the target of the inversion. When features are collinear, XᵀX is near-singular and the coefficients become unstable — which is precisely the problem ridge regression fixes by inverting XᵀX + λI instead.

Backpropagation. The gradient flowing backwards through a layer is multiplied by the transpose of that layer's weight matrix. The forward pass uses W; the backward pass uses Wᵀ. No inverse is ever needed, which is one reason training is feasible.

Data layout. Transposing is routine when reconciling "samples as rows" with "features as rows" conventions between libraries, and a surprising number of shape errors are a missing .T.

Orthogonal initialisation. Initialising a recurrent network's weights with an orthogonal matrix keeps the magnitude of signals stable through many time steps, directly addressing exploding and vanishing gradients.

The pseudo-inverse

When a matrix is not square, or is singular, the Moore-Penrose pseudo-inverse A⁺ gives the best available answer: the least-squares solution when the system is overdetermined, and the minimum-norm solution when it is underdetermined.

x = np.linalg.pinv(A) @ b      # works even when inv(A) does not exist

It is computed from the SVD, and it is what lstsq uses internally. For a linear regression with more features than samples, or with perfectly correlated columns, this is what returns a sensible answer where a direct inverse would fail outright.

Questions people ask

How do I know if a matrix is invertible? Non-zero determinant, or equivalently full rank, or equivalently no zero eigenvalues. np.linalg.matrix_rank is the practical check.

Why avoid computing inverses? solve is faster and numerically better. Explicitly inverting amplifies rounding error, especially when the matrix is ill-conditioned.

Is the transpose the same as the inverse? Only for orthogonal matrices — and that is exactly what makes them so convenient.

What is the condition number? The ratio of the largest to smallest singular value. A large value means the matrix is nearly singular and solutions will be sensitive to noise.

Does A B = A C imply B = C? Only if A is invertible. Otherwise, no — cancellation is not a valid move with singular matrices.

Why does ridge regression add λI? Because it makes XᵀX + λI invertible and better conditioned, even when the features are collinear.

Recap in one screen

• The identity does nothing; the inverse undoes; the transpose flips rows and columns.
• A matrix is invertible only when its determinant is non-zero — a singular matrix destroyed information.
• Use solve, not inv: faster and numerically safer.
• (AB)ᵀ = BᵀAᵀ, and symmetric matrices have real eigenvalues and orthogonal eigenvectors.
• Orthogonal matrices satisfy Qᵀ = Q⁻¹, preserve lengths, and are the numerically ideal case.
• The pseudo-inverse handles non-square and singular matrices, and is what least-squares solvers use.