Observe how "sparse" data (inputs composed mostly of zeros) propagates. Because multiplying by zero yields zero, pathways originating from dormant inputs become entirely inactive. Combining this with ReLU activations forces massive sections of the network into a dormant state, drastically reducing the number of required computations!
50%
Percentage of input values that will be strictly 0.
Efficiency Metrics
State
IDLE
Click 'Next Sample' or 'Stream Data'.
Active Inputs
--
Active Neurons
--
Compute Skipped (FLOPs)
--
Out of 0 total mults
Visual Legend
Dormant (0.0)Active (>0.0)
Drag to Pan | Scroll to Zoom
Overview
What sparsity means, and where it comes from
A feature vec is sparse when most of its entries are zero. This is not an edge case. It is the default in a few of the most common data types:
One-hot encoded groups. A country field with 195 vals turns 195 columns of which just one is 1. That is 99.5% zeros by construction.
Bag-of-words text. A vocabulary of 50,000 words against a 200-word document gives at most 0.4% non-zero.
Recommender matrices. A user has rated a few dozen of a million items. These are routinely above 99.9% zero.
Mix and count features where most combinations simply never occur.
The density of a grid is the fraction of entries that are non-zero. A bag-of-words grid at 0.4% density wastes 99.6% of its memory on zeros and 99.6% of its arithmetic on multiplying by zero. Sparse storage formats (CSR, CSC, COO) store just the non-zero vals and their indices, cutting memory by two to three orders of magnitude. Sparse grid steps skip the zero entries fully. It is why scikit-learn's linear nets accept scipy sparse matrices and train much faster on them than on dense NumPy arrays. In deep learning, embeddings solve the same problem differently: instead of multiplying a one-hot vec by a weight grid (which selects one row), an embedding layer looks up that row right by index, avoiding the multiplication altogether. Adaptive solvers like Adam and Adagrad handle the gradient side of sparsity by giving each param its own good learning rate based on how often it has been updated. So rare features learn at a pace proportional to their information rather than their rate. This page builds a sparse dataset, checks dense and sparse storage sizes and training times. And shows how embeddings replace the one-hot multiplication.
Observe how "sparse" data (inputs composed mostly of zeros) propagates.
When most feature values are zero, dense storage and dense arithmetic both stop making sense. Sparsity is normal in real data, and handling it well is mostly about not materialising the zeros.
What sparsity means, and where it comes from
A feature vec is sparse when most of its entries are zero. This is not an edge case. It is the default in a few of the most common data types:
One-hot encoded groups. A country field with 195 vals turns 195 columns of which just one is 1. That is 99.5% zeros by construction.
Bag-of-words text. A vocabulary of 50,000 words against a 200-word document gives at most 0.4% non-zero.
Recommender matrices. A user has rated a few dozen of a million items. These are routinely above 99.9% zero.
Mix and count features where most combinations simply never occur.
The density of a grid is the fraction of entries that are non-zero. A bag-of-words grid at 0.4% density wastes 99.6% of its memory on zeros and 99.6% of its arithmetic on multiplying by zero. Sparse storage formats (CSR, CSC, COO) store just the non-zero vals and their indices, cutting memory by two to three orders of magnitude. Sparse grid steps skip the zero entries fully. It is why scikit-learn's linear nets accept scipy sparse matrices and train much faster on them than on dense NumPy arrays. In deep learning, embeddings solve the same problem differently: instead of multiplying a one-hot vec by a weight grid (which selects one row), an embedding layer looks up that row right by index, avoiding the multiplication altogether. Adaptive solvers like Adam and Adagrad handle the gradient side of sparsity by giving each param its own good learning rate based on how often it has been updated. So rare features learn at a pace proportional to their information rather than their rate. This page builds a sparse dataset, checks dense and sparse storage sizes and training times. And shows how embeddings replace the one-hot multiplication.
Why it hurts training
Sparsity causes three split problems. And they are worth keeping apart:
Memory. Storing a 100,000×50,000 grid densely at 4 bytes per float is 20 GB, nearly all of it zeros. Sparse formats such as CSR store just the non-zeros plus their indices, often cutting this by two or three orders of magnitude.Wasted math. A dense grid multiply does a multiply-add for each item. And multiplying by zero adds nothing. Most of the arithmetic gives no information.Uneven gradients. This is the subtle one. A weight connected to a feature that is non-zero in 0.1% of samples gets a gradient in just 0.1% of steps. Rare features therefore learn roughly a thousand times more slowly than common ones under a single global learning rate. Which is precisely the problem Adagrad and Adam solve, by giving each param its own good rate based on how often it has got a gradient. That is why adaptive solvers are the standard choice on sparse data. The real fix for memory is to use scipy's sparse grid formats (CSR, CSC, COO) or their equivalents in your framework. They store just the non-zero vals and their indices, cutting memory by orders of magnitude. For math, sparse grid-vec products skip the zero entries fully. It is why scikit-learn's linear nets accept sparse input natively and train much faster on it than on the dense the same. Embeddings are the deep-learning solution: instead of feeding a one-hot vec of 50,000 dimensions into a dense layer (which would multiply by 50,000 weights), look up the single non-zero index in a learned embedding table and use those few hundred vals right. This page creates a sparse dataset, checks dense and sparse storage sizes. And shows how an embedding layer replaces the one-hot multiplication.
Two other problems with the same name
"Sparse" describes two cases that need fully other replies. And conflating them causes a lot of confusion.
Sparse features — most vals in the input are zero. One-hot encoded groups, bag-of-words text, user-item mix matrices. A vocabulary of 50,000 words means a document vec where 49,900 entries are zero.
Sparse labels — you have plenty of data and very few annotations. This is a labelling problem, addressed by semi-supervised learning, active learning or transfer learning.
Both are common; just the first is about form. And the rest of this is mostly about that one.
Why sparse features are awkward
Memory. A million users by 50,000 items stored densely is 200GB of floats. Stored sparsely — just the non-zero spots and vals. It may be a few hundred megabytes. Sparse formats are not an optimisation here. They are the gap between possible and impossible.
Thin stats. A feature that is non-zero in 20 of a million rows gives the net almost nothing to learn from. So its weight is mostly noise.
Distances stop working. Two documents that share no words have identical Euclidean distance no matter what else they hold. Cosine similarity, which checks way, is the standard option for just this reason.
Gradients are sparse too. A weight connected to a feature that is zero in this batch gets no gradient. So it updates just in the batches where its feature appears. Rare features therefore learn very slowly. Which is precisely the problem adaptive solvers were designed for. Adam and AdaGrad give params with infrequent gradients more good steps.
Embeddings: the standard fix
Instead of a 50,000-dim sparse vec, learn a dense vec of 64 to 256 counts per group, trained jointly with the rest of the net.
One-hot
Embedding
Width
One column per category
A chosen width, e.g. 64
Similar categories
Equidistant — no notion of similarity
Nearby in the space
Rare categories
Almost no data per column
Share structure with others
Memory
Sparse, wide
Dense, compact
The second row is the key one. One-hot coding makes each group just as other from each other. An embedding can place "London" near "Manchester" and far from "banana" since the training signal pushes them there. That is why embeddings are the default for high-cardinality class-based features in deep nets. And why they underpin each recommendation system.
A common width heuristic is min(50, (cardinality + 1) // 2). Or the fourth root of the cardinality. Both are starting points to be tuned rather than rules.
Mostly zeros, and what that costs
Sparse data breaks dense assumptions in three set ways -- memory, gradient flow, and distance. Each one is measured here, along with the standard fix.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
N, VOCAB = 2000, 5000
DENSITY = 0.004
# word frequencies follow a Zipf law in real text: a few words everywhere,
# a long tail of words that appear once or twice in the whole corpus
zipf = 1.0 / (np.arange(1, VOCAB + 1) ** 1.05)
zipf = zipf / zipf.sum() * VOCAB * DENSITY
X = (rng.random((N, VOCAB)) < np.clip(zipf, 0, 0.95)).astype(np.float32)
nnz = int(X.sum())
print("%d rows over a %d-word vocabulary, like bag-of-words text." % (N, VOCAB))
print(" non-zero entries: %d of %d (%.3f%%)"
% (nnz, X.size, 100 * nnz / X.size))
print(" average words per row: %.1f" % X.sum(1).mean())
print()
print("1. MEMORY. a dense array stores every zero:")
dense_mb = X.nbytes / 1024 / 1024
sparse_mb = (nnz * (4 + 4) + N * 4) / 1024 / 1024 # values + indices + row ptrs
print(" dense (float32) : %8.2f MB" % dense_mb)
print(" sparse (value + index) : %8.2f MB" % sparse_mb)
print(" ratio : %8.1fx" % (dense_mb / sparse_mb))
print(" at 100,000 rows the dense version is %.1f GB and the sparse one"
% (dense_mb * 50 / 1024))
print(" is %.2f GB. that is the difference between fitting and not."
% (sparse_mb * 50 / 1024))
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
print("2. GRADIENT FLOW. a column that is almost always zero receives almost")
print(" no gradient, because the gradient for a weight is scaled by its")
print(" input:")
counts = X.sum(0)
print(" %-16s %14s %18s" % ("word rank", "rows it appears in", "share of updates"))
for q, label in ((100, "the commonest"), (99, "top 1 percent"),
(90, "top 10 percent"), (50, "median"), (10, "long tail")):
c = np.percentile(counts, q)
print(" %-16s %14.0f %17.2f%%" % (label, c, 100 * c / N))
print(" %d of the %d columns appear in fewer than 5 rows."
% (int((counts < 5).sum()), VOCAB))
print()
never = int((counts == 0).sum())
print(" the commonest word is updated on %.0f%% of rows. %d of the %d"
% (100 * counts.max() / N, never, VOCAB))
print(" columns never appear at all in this sample, so their weights end")
print(" the epoch exactly where they were initialised. a word appearing")
print(" twice has had two updates -- indistinguishable from random.")
print(" any feature-importance measure will call all of them unimportant,")
print(" when the truth is they were never given a chance.")
print()
print(" this is exactly what Adagrad was invented for. it accumulates the")
print(" squared gradient per parameter and divides by its square root, so a")
print(" rarely-seen parameter keeps a large effective learning rate:")
for seen in (1000, 100, 10, 1):
acc = seen * (0.5 ** 2)
print(" a weight updated %5d times has accumulated %8.2f, so its"
% (seen, acc))
print(" effective rate is lr / %.2f = %.3f x lr" % (np.sqrt(acc), 1 / np.sqrt(acc)))
print()
Output
The next block covers the left steps.
example_03.pyNumPy
import numpy as np
print("3. DISTANCE. two sparse rows almost never share a non-zero, so")
print(" everything looks equally far from everything else:")
sub = X[:400]
norms = np.linalg.norm(sub, axis=1, keepdims=True)
cos = (sub @ sub.T) / np.clip(norms * norms.T, 1e-9, None)
off = ~np.eye(len(sub), dtype=bool)
print(" cosine similarity between %d rows:" % len(sub))
print(" exactly zero (no shared word): %.1f%% of pairs"
% (100 * (cos[off] == 0).mean()))
print(" mean similarity : %.4f" % cos[off].mean())
print(" a k-nearest-neighbour search over this is meaningless -- most")
print(" candidates are tied at zero, so the 'nearest' one is arbitrary.")
print()
print("the standard fix for all three is the same: stop representing the")
print("data as a %d-wide vector of mostly zeros. an embedding maps each" % VOCAB)
print("word to a short dense vector, and a row becomes the sum or mean of")
print("the vectors for the words it contains:")
DIM = 64
E = rng.normal(0, 0.1, (VOCAB, DIM))
dense_rows = np.array([E[np.flatnonzero(r)].mean(0) if r.any() else np.zeros(DIM)
for r in X[:400]])
dn = np.linalg.norm(dense_rows, axis=1, keepdims=True)
dcos = (dense_rows @ dense_rows.T) / np.clip(dn * dn.T, 1e-9, None)
print(" %d dimensions instead of %d: %.1fx narrower" % (DIM, VOCAB, VOCAB / DIM))
print(" pairs with exactly zero similarity: %.1f%% (was %.1f%%)"
% (100 * (np.abs(dcos[off]) < 1e-12).mean(), 100 * (cos[off] == 0).mean()))
print(" every dimension is now dense, so every parameter is updated by")
print(" every row and the gradient-flow problem disappears with it.")
print()
print("three practical notes:")
print(" use scipy.sparse (or a framework's sparse tensors) end to end -- one")
print(" .toarray() call in a dataloader undoes the entire saving.")
print(" prefer Adagrad or Adam over plain SGD on genuinely sparse features.")
print(" and check your embedding is actually learning: a word seen twice in")
print(" the corpus has had two gradient updates, and its vector is still")
print(" in effect the random one you initialised.")
Output
Guided tests
Start dense. Set Input Sparsity Level to 0 and press Stream Data. Each input carries signal and each weight gets a gradient on each sample.
Add realistic sparsity. Set Input Sparsity Level to 70 and stream again. Most inputs are now zero. And the connections behind them go quiet. A weight whose input is zero gets no gradient at all that step.
Push it to the recommender regime. Set Input Sparsity Level to 95 and press Next Sample repeatedly. Watch how rarely any given input activates. Those weights update on a handful of samples in a hundred. It is just why they lag.
Check which weights move. At high sparsity, note that the weights attached to often non-zero inputs converge quickly while the rest barely move. A global learning rate cannot serve both.
Embeddings, the standard fix
For high-cardinality class-based data the usual answer is not to keep the one-hot vec at all. An embedding layer maps each group to a short dense vec. Say 50 dimensions instead of 195 columns. Learned in training.
In math this is identical to multiplying the one-hot vec by a weight grid. But the build is a row lookup instead of a grid multiply. So it skips the zeros fully. It also solves a modelling problem: one-hot groups are all equidistant, whereas learned embeddings place close groups near each other. So information is shared between tied vals rather than each being learned in isolation.
Where this goes wrong
Densifying a sparse grid. Calling .toarray() on a large sparse grid is the classic out-of-memory error. Keep it sparse through the whole pipeline.
Plain SGD on sparse features. Rare features learn far too slowly. Use Adam or Adagrad. It adapt per-param learning rates to update rate.
One-hot coding a high-cardinality column. Thousands of near-empty columns. Use an embedding, target coding or hashing instead.
Mean-imputing a structural zero. In sparse data zero often means “absent”, not “missing”. Replacing it with a column mean invents signal that was never there.
Centring sparse data. Subtracting the mean makes each zero non-zero and destroys the sparsity outright. Scale without centring.
What to remember
Sparse data is the norm for one-hot groups, text and recommenders. And it costs memory, wasted arithmetic. And — least obviously. Badly uneven learning rates, since a weight just updates when its input is non-zero. Keep sparse matrices in sparse formats, replace one-hot encodings of high-cardinality fields with learned embeddings. And use an adaptive solver so rare features are not left thousands of steps behind common ones.
Handling it in code
For classical nets, keep the grid sparse end to end:
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
X = TfidfVectorizer(min_df=5).fit_transform(docs) # sparse matrix
print(X.shape, X.nnz / (X.shape[0] * X.shape[1])) # density, often < 1%
LogisticRegression().fit(X, y) # accepts sparse input
Never call .toarray() on a large sparse grid. That is the line that exhausts memory. min_df=5 is also doing real work: dropping terms that appear in fewer than five documents removes most of the vocabulary and almost none of the signal.
For neural networks, use an embedding layer and pass integer indices rather than one-hot vecs:
nn.Embedding is a lookup, not a grid multiplication. So it never materialises the one-hot vec at all. nn.EmbeddingBag sums or averages a few embeddings in one call. It is the fast way to represent a bag of features.
Which nets cope well
Model
With sparse features
Linear / logistic regression
Excellent — sparse-aware and fast
Naive Bayes
Excellent — designed for high-dimensional counts
Gradient boosting (LightGBM, XGBoost)
Good — handles sparsity natively
Neural networks with embeddings
Excellent, and the standard for high cardinality
KNN, k-means
Poor without cosine distance and dimension reduction
Dense neural networks on one-hot input
Poor — wasteful and slow to learn
Two more options worth knowing. Feature hashing maps groups into a fixed count of buckets, accepting collisions in exchange for bounded memory and no vocabulary to keep. Useful when new groups appear constantly. And truncated SVD reduces a sparse grid to a few hundred dense parts. It is what makes clustering and nearest-neighbour search on text feasible.
Sparse labels, briefly
The other meaning deserves its own short answer, since the tricks are other:
Transfer learning — start from a pretrained net. So a few hundred labels suffice.
Self-supervised pretraining on your unlabelled data, then fine-tune on the labelled subset.
Active learning — let the net choose which cases to send for labelling, prioritising the ones it is least sure about.
Pseudo-labelling — label the confident predictions on unlabelled data and train on them, carefully, since errors compound.
Weak supervision — mix noisy rules and heuristics into probabilistic labels.
The first two are where the uses is: pretraining plus a small labelled set now beats a large labelled set trained from scratch on most tasks.
Questions people ask
When is data "sparse"? Informally, when a large bulk of entries are zero. Below about 10% density, sparse storage and sparse-aware nets start to matter.
Should I use one-hot or an embedding? One-hot below about 10 groups. Embeddings above roughly 50, especially in a network.
Does PCA work on sparse data? Not right. Centring destroys sparsity. Use TruncatedSVD, which does not centre.
Which distance for sparse vecs? Cosine, almost always. Euclidean distance is dominated by the shared zeros.
Do trees handle sparsity? LightGBM and XGBoost handle it natively and well, with treating missing and zero appropriately.
Why do rare features learn slowly? They appear in few batches. So their weights get few gradients. Adaptive solvers partly compensate by giving them more good steps.
Recap in one screen
"Sparse" means either mostly-zero features or mostly-missing labels. Other problems, other fixes.
Sparse features cost memory, give thin per-feature stats, break Euclidean distance and slow learning for rare vals.
Keep sparse matrices sparse. Never densify a large one.
Embeddings replace wide one-hot encodings with compact dense vecs that can express similarity.
For sparse labels, transfer learning and self-supervised pretraining give the most uses.
Recall check
0 of 4
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Transfer learning” here?
What is meant by “Self-supervised pretraining” here?
What is meant by “Active learning” here?
What is meant by “Pseudo-labelling” here?
Cheat sheet
Data Sparsity
Feed mostly-zero inputs through a network and watch entire pathways go dormant, because anything multiplied by zero stays zero.
DEEP LEARNING · vizlearn.in/deep_learning/data_sparsity.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.