From words to IDs
This token-ID example is related to, but not identical to, encoding a categorical feature column. Language models often use token IDs as indices into learned embedding tables. A standard linear model reading those IDs as numbers would instead interpret their order and spacing as values.
Choose an encoding for the model
| Situation | Common choice | What the model receives |
|---|
| Target class labels | Integer class IDs | A class identifier, not a numeric feature |
| Ordered feature such as satisfaction level | Ordinal encoding | A numeric order, if the order is real and useful |
| Unordered feature such as city | One-hot encoding | A separate indicator for each category |
| Many categories with a neural model | Learned embeddings | A trainable vector for each category or token |
For a nominal feature, codes such as London=0, Manchester=1, and Leeds=2 invent a ranking. A linear model treats a one-step code change as a fixed numeric effect; a distance-based model treats some categories as farther apart than others. One-hot encoding avoids that false ordering, but can create many columns for high-cardinality features.
Some tree libraries support categorical features directly. Ordinary decision trees that receive numeric IDs still split by thresholds, so arbitrary coding can constrain which category groups are formed. Check the estimator’s categorical handling rather than assuming all trees treat IDs as labels.
Handle categories not seen in training
An encoder fitted on London, Manchester, and Leeds must have a policy for a new category such as Bristol. Fit one mapping on training data and apply it unchanged to validation, test, and production rows. Do not fit separate encoders on different splits.
Use an encoder with an explicit unknown-category behavior, group rare values into an “other” category when appropriate, or use an estimator that can handle unseen categories. For high-cardinality target encoding, compute encodings without using each row’s own target; use cross-fitting or another leakage-safe method.
Common mistakes with Label Encoding Process
- Using arbitrary IDs as measurements: the numbers may create a fake order or distance.
- Confusing target labels with input features: integer class IDs are valid targets but do not make good numeric features by default.
- Ignoring new values: production data may contain categories absent from training.
- Fitting target encodings before splitting: target information leaks into the feature.