Explore how text is transformed into sparse binary vectors for neural network processing.
Overview
Core Concept
One-Hot Encoding is a fundamental technique in data preprocessing, used to convert categorical variables into a numerical format that machine learning algorithms can understand. It takes a column with categorical data, which has been label encoded, and then splits the column into multiple columns. The numbers are replaced by 1s and 0s, depending on which column has what value.
Encoded Vectors
0 VECTORS
Vector List ScrollView
Understanding One-Hot Encoding
A deep dive into converting categorical data into a machine-readable format.
How It Works: The Intuition
Imagine you have a feature like "Color" with categories: "Red", "Green", and "Blue". A machine learning model can't work with these text labels directly. One-Hot Encoding transforms this single "Color" column into three new columns, one for each category: "Is_Red", "Is_Green", and "Is_Blue".
For a data point where the color is "Red", the "Is_Red" column will be 1, while "Is_Green" and "Is_Blue" will be 0.
If the color is "Green", the "Is_Green" column gets the 1, and the others get 0.
This creates a sparse binary vector where only one bit is "hot" (set to 1), clearly and unambiguously representing the original category without implying any ordinal relationship between the categories.
One column becomes several
One-hot encoding replaces a categorical column with one new column per category, each holding 1 if the row has that category and 0 otherwise. Exactly one column is "hot" in each row, which is where the name comes from.
City
is_London
is_Manchester
is_Leeds
London
1
0
0
Leeds
0
0
1
Manchester
0
1
0
Leeds
0
0
1
What this buys you is the absence of a lie. Label encoding says London=0, Manchester=1, Leeds=2, and a linear model reads that as an order and a spacing. One-hot says nothing except "this row is London" — every category sits the same distance from every other, which is the honest representation of an unordered set.
The model can now learn a separate coefficient for each city. In a linear model, is_London might get +45 and is_Leeds −12, meaning London adds £45k to a price and Leeds subtracts £12k. Those are readable numbers, which is a real secondary benefit.
The dummy variable trap
Here is a subtlety that catches people in regression and confuses everyone else.
If a row must be exactly one of London, Manchester or Leeds, then the three columns always sum to 1. That means any one of them can be worked out from the other two — they are perfectly collinear, and if the model also has an intercept, there are infinitely many equally good coefficient sets. The matrix becomes non-invertible and the coefficients become unstable or arbitrary.
The fix is to drop one category, which becomes the reference level. With London dropped, is_Manchester = 0 and is_Leeds = 0 means London, and each remaining coefficient reads as "compared to London".
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(drop="first", # avoid the dummy trap
handle_unknown="ignore", # unseen category -> all zeros
sparse_output=True) # do not materialise the zeros
Whether you need drop="first" depends on the model. Linear and logistic regression with an intercept: yes. Regularised models, trees, forests, boosting and neural networks: no, and dropping a level can actually hurt, because regularisation penalises the remaining categories asymmetrically relative to the hidden reference.
The cost: width, sparsity and memory
One-hot encoding is safe and it is expensive, and the expense is a function of cardinality.
A column with 5 categories adds 5 columns. A postcode column with 2,000 distinct values adds 2,000 columns, of which 1,999 are zero in every row. Three consequences follow:
Memory. Stored densely, 1 million rows × 2,000 columns of floats is 16 GB. Stored sparsely — only the positions of the ones — it is a few megabytes. Always keep high-cardinality one-hot output sparse.
Statistics. A category appearing in 20 of 1,000,000 rows gives its column almost nothing to learn from, so the coefficient is mostly noise.
Trees suffer specifically. A binary column can only ever produce the split "is it this category or not", so a tree needs many splits to isolate a group of categories that a single native categorical split would handle at once. This is why LightGBM and CatBoost's native category handling usually beats one-hot on tree models.
Two practical mitigations. Group rare categories into an explicit "other" bucket — anything under about 1% of rows is usually a candidate. And above roughly 50 categories, look at target encoding or learned embeddings instead.
Experiment in the Live Panel
The interactive visualization on this page lets you see this process in action. Follow these steps to build a strong mental model:
Initial State: Click the "Encode Corpus" button with the default text. Observe the "Vocabulary" size and the "Vector Length". Notice how each unique word from the input text becomes part of the vocabulary.
Analyze the Vectors: Look at the generated vectors. For each word (token), you'll see a vector of 0s and a single 1. The position of the '1' corresponds to that word's index in the sorted vocabulary. For example, if the vocabulary is ['deep', 'fascinating', 'is', 'learning', 'machine'], the word "is" will be represented as [0, 0, 1, 0, 0].
Add New Words: Add a new sentence to the input, like "AI is the future". Click "Encode Corpus" again. What happens to the "Vector Length" and "Vocabulary" count? All existing vectors are now longer to accommodate the new words ('ai', 'the', 'future'). This demonstrates a key challenge with One-Hot Encoding: the dimensionality increases with vocabulary size.
Introduce Repetition: Add a sentence that reuses existing words, such as "deep learning is powerful". Notice that the vocabulary size doesn't increase as much, but the total number of vectors does. This highlights the efficiency of reusing existing vocabulary entries.
Pseudocode Logic
function oneHotEncode(corpus):
// 1. Tokenize and find unique words
tokens = tokenize(corpus)
vocabulary = sorted(unique(tokens))
// 2. Create a mapping from word to index
word_to_index = {word: i for i, word in enumerate(vocabulary)}
// 3. Generate vectors
encoded_vectors = []
for token in tokens:
vector = create_zero_vector(length=len(vocabulary))
index = word_to_index[token]
vector[index] = 1
encoded_vectors.append(vector)
return encoded_vectors
Advantages and Disadvantages
Advantages
No Ordinality: It doesn't create a false order or ranking between categories, which is a common issue with simple label encoding.
Interpretability: The resulting vectors are easy to interpret. Each column clearly corresponds to a specific category.
Disadvantages
High Dimensionality (Curse of Dimensionality): If you have many unique categories (a large vocabulary), it creates a huge number of new features. This can slow down model training and sometimes hurt performance.
Sparsity: The resulting matrix is sparse (mostly zeros), which can be computationally inefficient to store and process.
Beyond One-Hot Encoding
While One-Hot Encoding is a great starting point, for more complex tasks, especially in Natural Language Processing (NLP), more advanced techniques like Word Embeddings (e.g., Word2Vec, GloVe, FastText) are used. These methods represent words as dense, low-dimensional vectors that capture semantic relationships (e.g., the vectors for "king" and "queen" are closer to each other). One-Hot Encoding, in contrast, treats every word as equally different from all others.
Choosing between the encodings
Situation
Reach for
Under ~10 categories, linear model or network
One-hot
Ordered categories (small/medium/large)
Ordinal, with the order stated
Many categories, tree ensemble
Native categorical support, or label encoding
Many categories, strong relationship to the target
Out-of-fold target encoding
Huge cardinality, collisions acceptable
Feature hashing
Deep learning with many categories
A learned embedding layer
The embedding option is worth a moment. Instead of 2,000 sparse columns, learn a dense vector of, say, 16 numbers per category, trained along with the rest of the network. Similar categories end up with similar vectors, which one-hot can never express since every category is equidistant from every other. This is exactly how categorical features are handled in recommendation systems, and it is why an embedding of 10,000 products is routine while one-hot encoding them is not.
Getting it right in a pipeline
The single most common one-hot bug in production is a column-count mismatch: training data had 40 cities, today's batch has 38, and the model receives a matrix of the wrong width.
The fix is to fit the encoder once and reuse the fitted object — never to call pd.get_dummies separately on training and serving data, which is what causes this.
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
pre = ColumnTransformer([
("cat", OneHotEncoder(handle_unknown="ignore"), ["city", "product"]),
("num", StandardScaler(), ["price", "age"]),
])
model = Pipeline([("pre", pre), ("clf", LogisticRegression())])
model.fit(X_train, y_train) # the encoder's vocabulary is now fixed
model.predict(X_new) # unseen categories become all-zero rows
handle_unknown="ignore" is what stops an unfamiliar city from crashing the service: the row simply gets zeros across all the city columns, which the model reads as "none of the known cities". Not ideal, but far better than an exception at request time.
Why not just number the categories
Label-encoding a category invents an order that is not there. This shows the damage, the one-hot fix, and the column explosion that follows.
example_01.pyscikit-learn
import numpy as np
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
cities = np.array(["london", "paris", "tokyo", "lagos", "lima"])
# rent depends on the city, in no particular order
base = {"london": 40, "paris": 32, "tokyo": 45, "lagos": 12, "lima": 15}
idx = rng.integers(0, 5, 1200)
city = cities[idx]
rent = np.array([base[c] for c in city]) + rng.normal(0, 3, 1200)
X_cat = city.reshape(-1, 1)
ordinal = OrdinalEncoder().fit(X_cat)
print("ordinal encoding assigns numbers alphabetically:")
for c, n in zip(ordinal.categories_[0], range(5)):
print(" %-8s -> %d (real average rent %.1f)" % (c, n, rent[city == c].mean()))
print()
print("that ordering says lagos < lima < london < paris < tokyo, and that")
print("the gap from lagos to lima equals the gap from paris to tokyo. neither")
print("is true. a linear model believes both:")
Xo = ordinal.transform(X_cat)
print(" linear regression on ordinal codes : R2 %.4f"
% cross_val_score(LinearRegression(), Xo, rent, cv=5).mean())
onehot = OneHotEncoder(sparse_output=False).fit(X_cat)
Xh = onehot.transform(X_cat)
print(" linear regression on one-hot : R2 %.4f"
% cross_val_score(LinearRegression(), Xh, rent, cv=5).mean())
print()
print("one-hot gives each city its own column and its own free coefficient:")
print(" columns:", list(onehot.get_feature_names_out(["city"])))
print(" first three rows:")
for i in range(3):
print(" %-8s %s" % (city[i], Xh[i].astype(int)))
print()
lin = LinearRegression(fit_intercept=False).fit(Xh, rent)
for name, c in zip(onehot.get_feature_names_out(["city"]), lin.coef_):
print(" %-14s coefficient %8.3f" % (name, c))
print(" with no intercept, each coefficient IS that city's average rent.")
print(" no ordering was assumed, so none was imposed.")
print()
print("note the fit_intercept=False. keep the intercept and all five columns")
print("and they sum to 1 in every row, which makes the system singular -- the")
print("coefficients come back as astronomical numbers that cancel out. that is")
print("the dummy variable trap, and drop='first' is the usual answer to it.")
print()
print("trees are less bothered, because a split is a set membership test:")
print(" random forest on ordinal codes : R2 %.4f"
% cross_val_score(RandomForestRegressor(n_estimators=100, random_state=0),
Xo, rent, cv=5).mean())
print(" random forest on one-hot : R2 %.4f"
% cross_val_score(RandomForestRegressor(n_estimators=100, random_state=0),
Xh, rent, cv=5).mean())
print(" it can carve out one code with two splits. it just costs depth.")
print()
print("the cost of one-hot is width. with high-cardinality columns:")
for n_cat in (5, 50, 500, 5000):
print(" %5d categories -> %5d columns" % (n_cat, n_cat))
print(" at that point you want target encoding, hashing, or an embedding.")
print()
print("two practical notes:")
print(" handle_unknown='ignore' stops an unseen category crashing predict().")
print(" drop='first' removes one column to avoid perfect collinearity, which")
print(" matters for linear models and not at all for trees.")
enc = OneHotEncoder(sparse_output=False, drop="first").fit(X_cat)
print(" with drop='first': %d columns instead of %d"
% (enc.transform(X_cat).shape[1], Xh.shape[1]))
Output
Questions people ask
Should I one-hot encode the target? For multi-class classification with a neural network and categorical cross-entropy, yes. With scikit-learn classifiers and sparse categorical loss, no — integer labels are what they expect.
Does one-hot encoding need scaling afterwards? The columns are already 0/1, so scaling changes little. It matters only if you are also scaling other features and want the regularisation penalty to fall evenly across them.
Is pd.get_dummies fine? For exploration, yes. For anything that will see new data later, no — it derives the columns from whatever data you hand it, so training and serving can disagree silently.
How many categories are too many? There is no hard line, but past 50 the sparsity starts to hurt and past a few hundred it usually dominates. Measure it: compare one-hot against target encoding by cross-validation rather than guessing.
What about missing values? Encode them as their own category. A blank field is often informative, and folding it into the most common category destroys that information.
Does one-hot encoding cause overfitting? Indirectly. Adding hundreds of columns to a model with limited rows gives it more ways to fit noise. Regularisation and rare-category grouping are the usual answers.
Recap in one screen
One column per category, exactly one 1 per row, no invented ordering.
Drop one level for linear models with an intercept; keep all levels for regularised models and trees.
Cardinality is the cost: width, sparsity and thin statistics per column.
Keep the matrix sparse, group rare values, and consider target encoding or embeddings above ~50 categories.
Fit the encoder once, inside a pipeline, and set handle_unknown="ignore" before it reaches production.
Check yourself
0 of 3
Answer without scrolling back up.
Why not just number categories 1, 2, 3 and feed them in directly?
Labelling red=1, green=2, blue=3 tells the model green sits between red and blue, and that blue is three times red. Both are nonsense, and a linear model will act on them.
A column has 50,000 distinct values. One-hot encoding it will:
High-cardinality columns explode under one-hot encoding. This is where target encoding, hashing or learned embeddings earn their keep instead.
When does label encoding become the right choice?
Ordinal data has a real ordering, so encoding it as 1/2/3 preserves information rather than fabricating it. Tree-based models are also far more tolerant of integer codes than linear ones.
Cheat sheet
One-Hot Encoding
One-Hot Encoding is a fundamental technique in data preprocessing, used to convert categorical variables into a numerical format that machine learning algorithms can understand. It takes a column with categorical data, which has been label encoded, and then splits the column into multiple columns. The numbers are replaced by 1s and 0s, depending on which column has what value.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.