Real tables mix numbers and categories, and the two need opposite preparation. One object routes each column to the right transformer.
Overview
The problem
Real tables have mixed types. Some columns are numeric, some are categorical, and each type needs different preprocessing. A scaler makes no sense on a city-name column. A one-hot encoder makes no sense on a temperature column. Applying the same transformer to every column either crashes or produces nonsense.
ColumnTransformer solves this by routing each group of columns to its own transformer. You specify which columns get which treatment. The transformer applies each sub-pipeline to its columns and concatenates the results into a single array. Because ColumnTransformer is itself a transformer, it fits into a Pipeline as one step. That keeps the routing inside the cross-validation loop, which prevents the preprocessing leak.
Without a column transformer, the usual workaround is to slice the DataFrame, transform each slice, and concatenate manually. That works for a single train-test split. It breaks under cross-validation because the slicing and transforming happen outside the fold loop. The test-fold statistics leak into the training set.
This page covers building a column transformer, combining it with a pipeline, inspecting the output columns, and handling columns that should be passed through or dropped.
A Pipeline is a sequence applied to all of X. That works when the columns are homogeneous and breaks right away on a real table.
Numeric columns want imputing with a median and scaling. cat columns want imputing with a constant and one-hot encoding. Applying a scaler to a text column raises. Applying an encoder to a smooth one makes a column per distinct value, which is nonsense.
The manual workaround — split the frame, transform each part, concatenate — works and puts the changes outside any pipeline. This Puts them outside the cross-validation folds. This Is the leak this track keeps returning to.
ColumnTransformer solves it properly: it routes each subset of columns to its own transformer and concatenates the results. Because it's itself a transformer, the whole thing drops into a pipeline as a single step.
Worth knowing
Each entry is (name, transformer, columns), and the outputs are concatenated in the order the transformers are listed.
Columns not named are dropped by default; remainder="passthrough" keeps them untransformed.
make_column_selector picks columns by dtype or by a name pattern, so new columns are handled without editing the pipeline.
A transformer can itself be a Pipeline, which is how a column gets imputed and scaled.
get_feature_names_out() traces names through the whole thing, prefixed with the transformer's name.
Pass a list of column names - a bare string selects a 1-D Series and raises.
Different Columns, Different Treatment: A Practical Guide
A pipeline applies each step to each column. Real tables need scaling on some columns and encoding on others. ColumnTransformer is how you say which is which.
Two treatments, one object
Each transformer gets the columns it was given, and the results are stuck side by side.
So that adding a column to the data does not mean editing the pipeline.
example_02.pyscikit-learn
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import pandas as pd
df = pd.DataFrame({"age": [25, 40, 33], "salary": [30000, 55000, 41000],
"city": ["london", "paris", "london"],
"grade": ["a", "b", "a"]})
pre = ColumnTransformer([
("num", StandardScaler(), make_column_selector(dtype_include="number")),
("cat", OneHotEncoder(sparse_output=False),
make_column_selector(dtype_include=object)),
])
pre.fit(df)
print("picked by dtype, not by hand:")
print(" numeric :", list(pre.transformers_[0][2]))
print(" categorical:", list(pre.transformers_[1][2]))
Output
What happens to the columns you did not mention
The default is to drop them, silently.
example_03.pyscikit-learn
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
import pandas as pd
df = pd.DataFrame({"age": [25, 40, 33], "salary": [30000, 55000, 41000],
"id": [1, 2, 3]})
drop = ColumnTransformer([("num", StandardScaler(), ["age", "salary"])])
keep = ColumnTransformer([("num", StandardScaler(), ["age", "salary"])],
remainder="passthrough")
print("remainder='drop' (the default):", drop.fit_transform(df).shape)
print("remainder='passthrough' :", keep.fit_transform(df).shape)
print()
print("the default silently discards any column you did not name -")
print("which is usually right, and is worth knowing about.")
Output
The whole preparation, inside one estimator
Impute, scale, encode and fit - and every step refitted on the training part of each fold.
example_04.pyscikit-learn
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import pandas as pd
import numpy as np
rng = np.random.RandomState(0)
n = 300
df = pd.DataFrame({
"age": rng.normal(40, 10, n),
"salary": rng.normal(50000, 15000, n),
"city": rng.choice(["london", "paris", "berlin"], n),
})
df.loc[rng.rand(n) < 0.1, "age"] = np.nan
y = (df["salary"] > 50000).astype(int)
numeric = make_pipeline(SimpleImputer(strategy="median"), StandardScaler())
categorical = OneHotEncoder(handle_unknown="ignore")
pre = ColumnTransformer([("num", numeric, ["age", "salary"]),
("cat", categorical, ["city"])])
model = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000))])
print("cross-validated accuracy:",
round(cross_val_score(model, df, y, cv=5).mean(), 4))
print()
print("impute, scale, encode and fit - every step refitted per fold")
Output
Tuning something four levels down
The underscores compose, so a search can reach the imputation strategy inside the numeric branch.
example_05.pyscikit-learn
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
import pandas as pd
import numpy as np
rng = np.random.RandomState(0)
n = 200
df = pd.DataFrame({"age": rng.normal(40, 10, n),
"city": rng.choice(["a", "b"], n)})
df.loc[rng.rand(n) < 0.2, "age"] = np.nan
y = rng.randint(0, 2, n)
numeric = make_pipeline(SimpleImputer(), StandardScaler())
pre = ColumnTransformer([("num", numeric, ["age"]),
("cat", OneHotEncoder(handle_unknown="ignore"), ["city"])])
pipe = Pipeline([("pre", pre), ("clf", LogisticRegression())])
grid = {"pre__num__simpleimputer__strategy": ["mean", "median"],
"clf__C": [0.1, 1.0]}
search = GridSearchCV(pipe, grid, cv=3).fit(df, y)
print("the key reaches four levels down:")
print(" pre__num__simpleimputer__strategy")
print()
print("best:", search.best_params_)
Output
A list of names, not a name
The most common error here, and the message says exactly what is wrong.
example_06.pyscikit-learn
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
import pandas as pd
df = pd.DataFrame({"city": ["london", "paris", "london"]})
ok = ColumnTransformer([("cat", OneHotEncoder(sparse_output=False), ["city"])])
print("a list of names ->", ok.fit_transform(df).shape)
bad = ColumnTransformer([("cat", OneHotEncoder(sparse_output=False), "city")])
try:
bad.fit_transform(df)
except Exception as e:
print()
print("a bare string ->", type(e).__name__)
print(" ", str(e).split("\n")[0][:88])
print()
print("a list selects a DataFrame (2-D); a bare name selects a Series (1-D),")
print("and the encoder wants two dimensions.")
Output
How it is written
Each entry is a triple: a name, a transformer, and the columns it applies to.
The name is yours to choose and appears in the made feature names and in the param keys for tuning. Short and real pays off. num and cat rather than transformer_1.
The columns can be a list of names, a list of int spots, a boolean mask, or a callable. Passing a list matters. A list of one name selects a DataFrame. This Is two-dimensional. A bare string selects a Series. This Is one-dimensional. The transformer raises. The error message says "Expected a 2-dimensional container", which is precise once you know what it refers to.
The outputs are concatenated left to right in the order the transformers are listed, not in the first column order. So the resulting array's columns follow your specification. This Is why get_feature_names_out() is the solid way to know what ended up where.
remainder, and the columns you forgot
By default, any column not named in any transformer is dropped.
That is often right — ids, timestamps and free text often shouldn't reach the model — and it's silent. This Is the part worth knowing. A column added to the source data will simply not appear in the model, with no warning. The effect is a model quietly skipping a feature somebody thought they had added.
remainder="passthrough" keeps the unnamed columns as they're. It's convenient and slightly dangerous. It will happily pass through an id column. This Is just the sort of thing that lets a model store rows.
remainder=SomeTransformer() applies a transformer to all left over. This Is at times the tidiest way to say "scale all I haven't or else mentioned".
The habit worth adopting: name each column by name, or use selectors that provably cover all. Treat the default drop as a safety net rather than as the design.
Selecting by type
Listing column names by hand is fine for a fixed table and a maintenance burden for any of it that changes.
make_column_selector(dtype_include="number") selects the numeric columns what they're called. dtype_include=object picks up the strings. pattern="^date_" selects by name. The selector runs at fit time against what frame it's given. A new numeric column is handled without touching the pipeline.
The caution is that dtypes aren't always what you expect. A cat column stored as ints. A postcode district, a department id. Is selected as numeric and scaled. This Is the mistake the encoding module warned about, now happening automatically. Converting such columns to a string or a pandas category dtype before the pipeline is what makes dtype-based pick safe.
Nesting a pipeline inside a branch
A single column group often needs more than one step: numeric columns want imputing *and* scaling.
The transformer in a ColumnTransformer entry can be a Pipeline. This Is how that is expressed. make_pipeline(SimpleImputer(strategy="median"), StandardScaler()) turns the numeric branch. The cat branch gets its own with an imputer and an encoder.
The result is a small tree: a pipeline holding a column transformer holding pipelines holding transformers. That sounds elaborate and is the standard shape of each real scikit-learn program. It's the min structure that does the right thing on a mixed table.
The whole tree is one estimator. fit on the outer pipeline reaches each leaf. Cross-validation refits all of it on each training fold.
Reaching into it to tune
The double-underscore rule composes, so params deep in the tree are addressable.
pre__num__simpleimputer__strategy reads outside in: the step named pre, its branch named num, the step named simpleimputer in that branch. Its strategy. Four levels, one string. A grid search can vary it alongside the model's own settings.
This is more handy than it first appears. Whether to impute with the mean or the median, whether to scale, whether to drop the first one-hot column. These are choices often made by guesswork. They can be made by cross-validation instead, in the same search that tunes the model.
pipe.get_params().keys() lists each addressable param. This Is the fastest way to find the right spelling rather than deriving it.
Getting the names back
After one-hot encoding, the array has more columns than the frame had, with names no one wrote.
get_feature_names_out() rebuilds them, prefixed by the transformer name: num__age, cat__city_paris. Called on the outer pipeline, it traces through each step.
That matters for reading the model. Zipping the names against coef_ or feature_importances_ turns an anonymous array into a labelled result. Without it the mapping has to be rebuilt by counting. This Is both tedious and easy to get wrong by one after any change to the prep.
The shape of a real preprocessing block
Almost each tabular project ends up with the same structure. It's worth having it in mind as a template rather than deriving it each time.
A numeric branch: impute with the median, then scale. A cat branch: impute with a constant such as "missing", then one-hot encode with handle_unknown="skip". A column transformer routing the two by dtype. A model on the end. The whole thing in one Pipeline, cross-validated, with the prep choices exposed to a search.
That covers a large majority of tabular problems and is about fifteen lines. What varies between projects is which extra branches appear. A text column going to a vectoriser, a date column going to a custom transformer that extracts the parts, a high-cardinality column going to a target encoder instead of one-hot.
The value of the template isn't the exact steps but that it puts each learned change in one place, inside the folds, addressable by a search. A project that starts from it rarely acquires the leaks that a project assembled step by step does.
One hands-on note: build it up one branch at a time and check the output shape after each. A column transformer that silently drops half the frame makes a model that works and underperforms. The shape is the fastest way to notice.
When the branches disagree about rows
A limitation worth knowing, because it isn't obvious from the interface.
each transformer in a ColumnTransformer receives the same rows and must return the same number of rows, in the same order. The results are concatenated flatly. Any of it that reordered or dropped rows would misalign the columns against each other and against y.
That rules out row-level ops inside it: no dropping outliers, no resampling, no aggregating. Those belong before the pipeline, applied to the training data only, or in a library built for it. imbalanced-learn gives a pipeline that understands row changes. This Is the right tool when resampling is part of the model rather than part of the data preparation.
It also means a transformer that returns a different number of rows than it received will fail with a shape error rather than a helpful message. A custom transformer that filters is a mistake worth ruling out early.
Does the output keep my column order? No. Columns come out in the order the transformers are listed, then within each transformer. get_feature_names_out() is the solid way to know what is where.
Can I use the same column in two transformers? Yes - a column can appear in more than one entry. Both outputs are kept. That is how you'd scale a column and also bin it.
Is there a shorthand?make_column_transformer((StandardScaler(), ["age"]), (OneHotEncoder(), ["city"])) skips the names and makes them, in the same way make_pipeline does.
What if a column is numeric but cat in meaning? Convert it to a string or a pandas category dtype before the pipeline, or dtype-based pick will scale it as a number.
Debugging one that is not working
A column transformer that misbehaves is harder to inspect than a plain pipeline. The branches run in parallel and the output is a single anonymous array. Four things to check, in order.
The output shape.pre.fit_transform(df).shape against what you expected. Too few columns often means a branch selected nothing. Too many often means a high-cardinality column reached the one-hot encoder.
What each branch really selected. After fitting, pre.transformers_ holds the resolved triples, with the callable selectors already turned into concrete column lists. Printing the third element of each entry says just which columns went where. This Is the fastest way to find a column that fell through to remainder.
The names.pre.get_feature_names_out() lists each output column with its branch prefix. If a name you expected is absent, the column was dropped. If there are hundreds of cat__ names, an identifier reached the encoder.
One branch at a time.pre.named_transformers_["num"] gives the fitted branch. This Can be inspected on its own. Its imputer's statistics_, its scaler's mean_. When the whole thing makes a thing odd, narrowing to one branch often localises it in a minute.
The general lesson is that the object is introspectable at each level. The trouble is only that the handy fields have to be asked for by name.
Does it work with NumPy arrays? Yes, selecting by int spot rather than by name. Names are worth the DataFrame. Spots shift when a column is added.
Can a branch return a sparse mat? Yes. The result is sparse if enough of it's. sparse_threshold=0 forces a dense output when the next step needs one.
Things to try with Different Columns, Different Treatment
Watch the shape change. The first editor turns three columns into five. Read the names to see which came from where.
Lose a column. In the third editor, note that id disappears with no warning under the default.
Add a column. In the second editor, add another numeric column to the frame and confirm the selector picks it up with no edit to the transformer.
Find the param names. In the fifth editor, print sorted(pipe.get_params()) and look for the keys you could tune.
Questions people ask about Different Columns, Different Treatment
What is ColumnTransformer for? It applies different preprocessing to different columns, such as scaling numbers and one-hot encoding categories, then joins the results.
What does remainder do? Columns not named in any branch are dropped by default. Set remainder='passthrough' to keep them unchanged.
How do I select columns by type? Use make_column_selector(dtype_include=...) so the transformer picks numeric or object columns automatically.
Can a branch be a pipeline? Yes. A branch such as impute then scale is a small pipeline, and it can be nested inside the transformer.
How do I get the output column names? Call get_feature_names_out() on the fitted transformer. It returns names prefixed by branch.
How do I tune parameters inside it? Use the double-underscore path, like pre__num__imputer__strategy, through the outer pipeline.
Where this leaves you
One object that sends each group of columns to the transformer it needs, nests pipelines inside its branches, drops what you didn't mention. Exposes each setting inside it to a search. Together with Pipeline it's the shape of each real scikit-learn program on tabular data.
Check yourself
0 of 4
Answer without scrolling back up.
What happens to a column not named in any transformer?
remainder defaults to "drop". A column added to the source data will simply not reach the model, with no warning.
Why pass ["city"] rather than "city"?
The error says "Expected a 2-dimensional container". Transformers want two dimensions, which is the same X-must-be-2-D rule from the start of the track.
How do you impute AND scale the numeric columns only?
A ColumnTransformer entry accepts any transformer, including a Pipeline - which is how a branch gets several steps.
What does pre__num__simpleimputer__strategy address?
The double underscores compose outside in. pipe.get_params().keys() lists every valid key.
Cheat sheet
Different Columns, Different Treatment
Apply different preprocessing to different columns using ColumnTransformer, so numeric features scale while categoricals encode.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.