One Hot Encoding

Explore how text is transformed into sparse binary vectors for neural network processing.

Corpus Input

Encoding Statistics

Total Vectors 0

Vector Len 0

Vocabulary 0

Bit Density 0%

Encoded Vectors

0 VECTORS

Vector List Scroll View

Understanding One-Hot Encoding

A deep dive into converting categorical data into a machine-readable format.

Core Concept

One-Hot Encoding is a fundamental technique in data preprocessing, used to convert categorical variables into a numerical format that machine learning algorithms can understand. It takes a column with categorical data, which has been label encoded, and then splits the column into multiple columns. The numbers are replaced by 1s and 0s, depending on which column has what value.

How It Works: The Intuition

Imagine you have a feature like "Color" with categories: "Red", "Green", and "Blue". A machine learning model can't work with these text labels directly. One-Hot Encoding transforms this single "Color" column into three new columns, one for each category: "Is_Red", "Is_Green", and "Is_Blue".

For a data point where the color is "Red", the "Is_Red" column will be 1, while "Is_Green" and "Is_Blue" will be 0.
If the color is "Green", the "Is_Green" column gets the 1, and the others get 0.

This creates a sparse binary vector where only one bit is "hot" (set to 1), clearly and unambiguously representing the original category without implying any ordinal relationship between the categories.

Experiment in the Live Panel

The interactive visualization on this page lets you see this process in action. Follow these steps to build a strong mental model:

Initial State: Click the "Encode Corpus" button with the default text. Observe the "Vocabulary" size and the "Vector Length". Notice how each unique word from the input text becomes part of the vocabulary.
Analyze the Vectors: Look at the generated vectors. For each word (token), you'll see a vector of 0s and a single 1. The position of the '1' corresponds to that word's index in the sorted vocabulary. For example, if the vocabulary is `['deep', 'fascinating', 'is', 'learning', 'machine']`, the word "is" will be represented as `[0, 0, 1, 0, 0]`.
Add New Words: Add a new sentence to the input, like "AI is the future". Click "Encode Corpus" again. What happens to the "Vector Length" and "Vocabulary" count? All existing vectors are now longer to accommodate the new words ('ai', 'the', 'future'). This demonstrates a key challenge with One-Hot Encoding: the dimensionality increases with vocabulary size.
Introduce Repetition: Add a sentence that reuses existing words, such as "deep learning is powerful". Notice that the vocabulary size doesn't increase as much, but the total number of vectors does. This highlights the efficiency of reusing existing vocabulary entries.

Pseudocode Logic


function oneHotEncode(corpus):
  // 1. Tokenize and find unique words
  tokens = tokenize(corpus)
  vocabulary = sorted(unique(tokens))
  
  // 2. Create a mapping from word to index
  word_to_index = {word: i for i, word in enumerate(vocabulary)}
  
  // 3. Generate vectors
  encoded_vectors = []
  for token in tokens:
    vector = create_zero_vector(length=len(vocabulary))
    index = word_to_index[token]
    vector[index] = 1
    encoded_vectors.append(vector)
    
  return encoded_vectors

Advantages and Disadvantages

Advantages

No Ordinality: It doesn't create a false order or ranking between categories, which is a common issue with simple label encoding.
Interpretability: The resulting vectors are easy to interpret. Each column clearly corresponds to a specific category.

Disadvantages

High Dimensionality (Curse of Dimensionality): If you have many unique categories (a large vocabulary), it creates a huge number of new features. This can slow down model training and sometimes hurt performance.
Sparsity: The resulting matrix is sparse (mostly zeros), which can be computationally inefficient to store and process.

Beyond One-Hot Encoding

While One-Hot Encoding is a great starting point, for more complex tasks, especially in Natural Language Processing (NLP), more advanced techniques like Word Embeddings (e.g., Word2Vec, GloVe, FastText) are used. These methods represent words as dense, low-dimensional vectors that capture semantic relationships (e.g., the vectors for "king" and "queen" are closer to each other). One-Hot Encoding, in contrast, treats every word as equally different from all others.

Cheat sheet

One-Hot Encoding

MACHINE LEARNING · vizlearn.in/machine_learning/one_hot_encoding.html