Observe how "sparse" data (inputs composed mostly of zeros) propagates. Because multiplying by zero yields zero, pathways originating from dormant inputs become entirely inactive. Combining this with ReLU activations forces massive sections of the network into a dormant state, drastically reducing the number of required computations!
Overview
What sparsity means, and where it comes from
A feature vector is sparse when most of its entries are zero. This is not an edge case — it is the default in several of the most common data types:
One-hot encoded categories. A country field with 195 values becomes 195 columns of which exactly one is 1. That is 99.5% zeros by construction.
Bag-of-words text. A vocabulary of 50,000 words against a 200-word document gives at most 0.4% non-zero.
Recommender matrices. A user has rated a few dozen of a million items; these are routinely above 99.9% zero.
Interaction and count features where most combinations simply never occur.
50%
Percentage of input values that will be strictly 0.
Efficiency Metrics
State
IDLE
Click 'Next Sample' or 'Stream Data'.
Active Inputs
--
Active Neurons
--
Compute Skipped (FLOPs)
--
Out of 0 total mults
Visual Legend
Dormant (0.0)Active (>0.0)
Drag to Pan | Scroll to Zoom
Data Sparsity: A Practical Guide
When most feature values are zero, dense storage and dense arithmetic both stop making sense. Sparsity is normal in real data, and handling it well is mostly about not materialising the zeros.
Why it hurts training
Sparsity causes three separate problems, and they are worth keeping apart:
Memory. Storing a 100,000×50,000 matrix densely at 4 bytes per float is 20 GB, nearly all of it zeros. Sparse formats such as CSR store only the non-zeros plus their indices, typically cutting this by two or three orders of magnitude.Wasted computation. A dense matrix multiply performs a multiply-add for every element, and multiplying by zero contributes nothing. Most of the arithmetic produces no information.Uneven gradients. This is the subtle one. A weight connected to a feature that is non-zero in 0.1% of samples receives a gradient in only 0.1% of steps. Rare features therefore learn roughly a thousand times more slowly than common ones under a single global learning rate — which is precisely the problem Adagrad and Adam solve, by giving each parameter its own effective rate based on how often it has received a gradient. That is why adaptive optimisers are the standard choice on sparse data.
Two different problems with the same name
"Sparse" describes two situations that need entirely different responses, and conflating them causes a lot of confusion.
Sparse features — most values in the input are zero. One-hot encoded categories, bag-of-words text, user-item interaction matrices. A vocabulary of 50,000 words means a document vector where 49,900 entries are zero.
Sparse labels — you have plenty of data and very few annotations. This is a labelling problem, addressed by semi-supervised learning, active learning or transfer learning.
Both are common; only the first is about representation, and the rest of this is mostly about that one.
Why sparse features are awkward
Memory. A million users by 50,000 items stored densely is 200GB of floats. Stored sparsely — only the non-zero positions and values — it may be a few hundred megabytes. Sparse formats are not an optimisation here; they are the difference between possible and impossible.
Thin statistics. A feature that is non-zero in 20 of a million rows gives the model almost nothing to learn from, so its weight is mostly noise.
Distances stop working. Two documents that share no words have identical Euclidean distance regardless of what else they contain. Cosine similarity, which compares direction, is the standard alternative for exactly this reason.
Gradients are sparse too. A weight connected to a feature that is zero in this batch receives no gradient, so it updates only in the batches where its feature appears. Rare features therefore learn very slowly — which is precisely the problem adaptive optimisers were designed for. Adam and AdaGrad give parameters with infrequent gradients larger effective steps.
Embeddings: the standard fix
Instead of a 50,000-dimensional sparse vector, learn a dense vector of 64 to 256 numbers per category, trained jointly with the rest of the model.
One-hot
Embedding
Width
One column per category
A chosen width, e.g. 64
Similar categories
Equidistant — no notion of similarity
Nearby in the space
Rare categories
Almost no data per column
Share structure with others
Memory
Sparse, wide
Dense, compact
The second row is the important one. One-hot encoding makes every category exactly as different from every other; an embedding can place "London" near "Manchester" and far from "banana" because the training signal pushes them there. That is why embeddings are the default for high-cardinality categorical features in deep models, and why they underpin every recommendation system.
A common width heuristic is min(50, (cardinality + 1) // 2), or the fourth root of the cardinality — both are starting points to be tuned rather than rules.
Mostly zeros, and what that costs
Sparse data breaks dense assumptions in three specific ways -- memory, gradient flow, and distance. Each one is measured here, along with the standard fix.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
N, VOCAB = 2000, 5000
DENSITY = 0.004
# word frequencies follow a Zipf law in real text: a few words everywhere,
# a long tail of words that appear once or twice in the whole corpus
zipf = 1.0 / (np.arange(1, VOCAB + 1) ** 1.05)
zipf = zipf / zipf.sum() * VOCAB * DENSITY
X = (rng.random((N, VOCAB)) < np.clip(zipf, 0, 0.95)).astype(np.float32)
nnz = int(X.sum())
print("%d rows over a %d-word vocabulary, like bag-of-words text." % (N, VOCAB))
print(" non-zero entries: %d of %d (%.3f%%)"
% (nnz, X.size, 100 * nnz / X.size))
print(" average words per row: %.1f" % X.sum(1).mean())
print()
print("1. MEMORY. a dense array stores every zero:")
dense_mb = X.nbytes / 1024 / 1024
sparse_mb = (nnz * (4 + 4) + N * 4) / 1024 / 1024 # values + indices + row ptrs
print(" dense (float32) : %8.2f MB" % dense_mb)
print(" sparse (value + index) : %8.2f MB" % sparse_mb)
print(" ratio : %8.1fx" % (dense_mb / sparse_mb))
print(" at 100,000 rows the dense version is %.1f GB and the sparse one"
% (dense_mb * 50 / 1024))
print(" is %.2f GB. that is the difference between fitting and not."
% (sparse_mb * 50 / 1024))
print()
print("2. GRADIENT FLOW. a column that is almost always zero receives almost")
print(" no gradient, because the gradient for a weight is scaled by its")
print(" input:")
counts = X.sum(0)
print(" %-16s %14s %18s" % ("word rank", "rows it appears in", "share of updates"))
for q, label in ((100, "the commonest"), (99, "top 1 percent"),
(90, "top 10 percent"), (50, "median"), (10, "long tail")):
c = np.percentile(counts, q)
print(" %-16s %14.0f %17.2f%%" % (label, c, 100 * c / N))
print(" %d of the %d columns appear in fewer than 5 rows."
% (int((counts < 5).sum()), VOCAB))
print()
never = int((counts == 0).sum())
print(" the commonest word is updated on %.0f%% of rows. %d of the %d"
% (100 * counts.max() / N, never, VOCAB))
print(" columns never appear at all in this sample, so their weights end")
print(" the epoch exactly where they were initialised. a word appearing")
print(" twice has had two updates -- indistinguishable from random.")
print(" any feature-importance measure will call all of them unimportant,")
print(" when the truth is they were never given a chance.")
print()
print(" this is exactly what Adagrad was invented for. it accumulates the")
print(" squared gradient per parameter and divides by its square root, so a")
print(" rarely-seen parameter keeps a large effective learning rate:")
for seen in (1000, 100, 10, 1):
acc = seen * (0.5 ** 2)
print(" a weight updated %5d times has accumulated %8.2f, so its"
% (seen, acc))
print(" effective rate is lr / %.2f = %.3f x lr" % (np.sqrt(acc), 1 / np.sqrt(acc)))
print()
print("3. DISTANCE. two sparse rows almost never share a non-zero, so")
print(" everything looks equally far from everything else:")
sub = X[:400]
norms = np.linalg.norm(sub, axis=1, keepdims=True)
cos = (sub @ sub.T) / np.clip(norms * norms.T, 1e-9, None)
off = ~np.eye(len(sub), dtype=bool)
print(" cosine similarity between %d rows:" % len(sub))
print(" exactly zero (no shared word): %.1f%% of pairs"
% (100 * (cos[off] == 0).mean()))
print(" mean similarity : %.4f" % cos[off].mean())
print(" a k-nearest-neighbour search over this is meaningless -- most")
print(" candidates are tied at zero, so the 'nearest' one is arbitrary.")
print()
print("the standard fix for all three is the same: stop representing the")
print("data as a %d-wide vector of mostly zeros. an embedding maps each" % VOCAB)
print("word to a short dense vector, and a row becomes the sum or mean of")
print("the vectors for the words it contains:")
DIM = 64
E = rng.normal(0, 0.1, (VOCAB, DIM))
dense_rows = np.array([E[np.flatnonzero(r)].mean(0) if r.any() else np.zeros(DIM)
for r in X[:400]])
dn = np.linalg.norm(dense_rows, axis=1, keepdims=True)
dcos = (dense_rows @ dense_rows.T) / np.clip(dn * dn.T, 1e-9, None)
print(" %d dimensions instead of %d: %.1fx narrower" % (DIM, VOCAB, VOCAB / DIM))
print(" pairs with exactly zero similarity: %.1f%% (was %.1f%%)"
% (100 * (np.abs(dcos[off]) < 1e-12).mean(), 100 * (cos[off] == 0).mean()))
print(" every dimension is now dense, so every parameter is updated by")
print(" every row and the gradient-flow problem disappears with it.")
print()
print("three practical notes:")
print(" use scipy.sparse (or a framework's sparse tensors) end to end -- one")
print(" .toarray() call in a dataloader undoes the entire saving.")
print(" prefer Adagrad or Adam over plain SGD on genuinely sparse features.")
print(" and check your embedding is actually learning: a word seen twice in")
print(" the corpus has had two gradient updates, and its vector is still")
print(" essentially the random one you initialised.")
Output
Guided experiments
Start dense. Set Input Sparsity Level to 0 and press Stream Data. Every input carries signal and every weight receives a gradient on every sample.
Introduce realistic sparsity. Set Input Sparsity Level to 70 and stream again. Most inputs are now zero, and the connections behind them go quiet — a weight whose input is zero gets no gradient at all that step.
Push it to the recommender regime. Set Input Sparsity Level to 95 and press Next Sample repeatedly. Watch how rarely any particular input activates. Those weights update on a handful of samples in a hundred, which is exactly why they lag.
Compare which weights move. At high sparsity, note that the weights attached to frequently non-zero inputs converge quickly while the rest barely move. A global learning rate cannot serve both.
Embeddings, the standard fix
For high-cardinality categorical data the usual answer is not to keep the one-hot vector at all. An embedding layer maps each category to a short dense vector — say 50 dimensions instead of 195 columns — learned during training.
Mathematically this is identical to multiplying the one-hot vector by a weight matrix, but the implementation is a row lookup instead of a matrix multiply, so it skips the zeros entirely. It also solves a modelling problem: one-hot categories are all equidistant, whereas learned embeddings place similar categories near each other, so information is shared between related values rather than each being learned in isolation.
Where this goes wrong
Densifying a sparse matrix. Calling .toarray() on a large sparse matrix is the classic out-of-memory error. Keep it sparse through the whole pipeline.
Plain SGD on sparse features. Rare features learn far too slowly. Use Adam or Adagrad, which adapt per-parameter learning rates to update frequency.
One-hot encoding a high-cardinality column. Thousands of near-empty columns; use an embedding, target encoding or hashing instead.
Mean-imputing a structural zero. In sparse data zero usually means “absent”, not “missing”. Replacing it with a column mean invents signal that was never there.
Centring sparse data. Subtracting the mean makes every zero non-zero and destroys the sparsity outright. Scale without centring.
What to remember
Sparse data is the norm for one-hot categories, text and recommenders, and it costs memory, wasted arithmetic, and — least obviously — badly uneven learning rates, because a weight only updates when its input is non-zero. Keep sparse matrices in sparse formats, replace one-hot encodings of high-cardinality fields with learned embeddings, and use an adaptive optimiser so rare features are not left thousands of steps behind common ones.
Handling it in code
For classical models, keep the matrix sparse end to end:
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
X = TfidfVectorizer(min_df=5).fit_transform(docs) # sparse matrix
print(X.shape, X.nnz / (X.shape[0] * X.shape[1])) # density, often < 1%
LogisticRegression().fit(X, y) # accepts sparse input
Never call .toarray() on a large sparse matrix — that is the line that exhausts memory. min_df=5 is also doing real work: dropping terms that appear in fewer than five documents removes most of the vocabulary and almost none of the signal.
For neural networks, use an embedding layer and pass integer indices rather than one-hot vectors:
nn.Embedding is a lookup, not a matrix multiplication, so it never materialises the one-hot vector at all. nn.EmbeddingBag sums or averages several embeddings in one call, which is the efficient way to represent a bag of features.
Which models cope well
Model
With sparse features
Linear / logistic regression
Excellent — sparse-aware and fast
Naive Bayes
Excellent — designed for high-dimensional counts
Gradient boosting (LightGBM, XGBoost)
Good — handles sparsity natively
Neural networks with embeddings
Excellent, and the standard for high cardinality
KNN, k-means
Poor without cosine distance and dimension reduction
Dense neural networks on one-hot input
Poor — wasteful and slow to learn
Two further options worth knowing. Feature hashing maps categories into a fixed number of buckets, accepting collisions in exchange for bounded memory and no vocabulary to maintain — useful when new categories appear constantly. And truncated SVD reduces a sparse matrix to a few hundred dense components, which is what makes clustering and nearest-neighbour search on text feasible.
Sparse labels, briefly
The other meaning deserves its own short answer, since the techniques are different:
Transfer learning — start from a pretrained model, so a few hundred labels suffice.
Self-supervised pretraining on your unlabelled data, then fine-tune on the labelled subset.
Active learning — let the model choose which examples to send for labelling, prioritising the ones it is least sure about.
Pseudo-labelling — label the confident predictions on unlabelled data and train on them, carefully, since errors compound.
Weak supervision — combine noisy rules and heuristics into probabilistic labels.
The first two are where the leverage is: pretraining plus a small labelled set now beats a large labelled set trained from scratch on most tasks.
Questions people ask
When is data "sparse"? Informally, when a large majority of entries are zero. Below about 10% density, sparse storage and sparse-aware models start to matter.
Should I use one-hot or an embedding? One-hot below about 10 categories; embeddings above roughly 50, especially in a network.
Does PCA work on sparse data? Not directly — centring destroys sparsity. Use TruncatedSVD, which does not centre.
Which distance for sparse vectors? Cosine, almost always. Euclidean distance is dominated by the shared zeros.
Do trees handle sparsity? LightGBM and XGBoost handle it natively and efficiently, including treating missing and zero appropriately.
Why do rare features learn slowly? They appear in few batches, so their weights receive few gradients. Adaptive optimisers partly compensate by giving them larger effective steps.
Recap in one screen
"Sparse" means either mostly-zero features or mostly-missing labels — different problems, different fixes.
Sparse features cost memory, give thin per-feature statistics, break Euclidean distance and slow learning for rare values.
Keep sparse matrices sparse; never densify a large one.
Embeddings replace wide one-hot encodings with compact dense vectors that can express similarity.
For sparse labels, transfer learning and self-supervised pretraining give the most leverage.
Recall check
0 of 4
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Transfer learning” here?
— start from a pretrained model, so a few hundred labels suffice.
What is meant by “Self-supervised pretraining” here?
on your unlabelled data, then fine-tune on the labelled subset.
What is meant by “Active learning” here?
— let the model choose which examples to send for labelling, prioritising the ones it is least sure about.
What is meant by “Pseudo-labelling” here?
— label the confident predictions on unlabelled data and train on them, carefully, since errors compound.
Cheat sheet
Data Sparsity
Observe how "sparse" data (inputs composed mostly of zeros) propagates. Because multiplying by zero yields zero, pathways originating from dormant inputs become entirely inactive. Combining this with ReLU activations forces massive sections of the network into a dormant state, drastically reducing the number of required computations!
DEEP LEARNING · vizlearn.in/deep_learning/data_sparsity.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.