Real tables mix numbers and categories, and the two need opposite preparation. One object routes each column to the right transformer.
Overview
The problem
A Pipeline is a sequence applied to all of X. That works when the columns are homogeneous and breaks immediately on a real table.
Numeric columns want imputing with a median and scaling. Categorical columns want imputing with a constant and one-hot encoding. Applying a scaler to a text column raises; applying an encoder to a continuous one produces a column per distinct value, which is nonsense.
The manual workaround — split the frame, transform each part, concatenate — works and puts the transformations outside any pipeline, which puts them outside the cross-validation folds, which is the leak this track keeps returning to.
ColumnTransformer solves it properly: it routes each subset of columns to its own transformer and concatenates the results, and because it is itself a transformer, the whole thing drops into a pipeline as a single step.
Worth knowing
Each entry is (name, transformer, columns), and the outputs are concatenated in the order the transformers are listed.
Columns not named are dropped by default; remainder="passthrough" keeps them untransformed.
make_column_selector picks columns by dtype or by a name pattern, so new columns are handled without editing the pipeline.
A transformer can itself be a Pipeline, which is how a column gets imputed and scaled.
get_feature_names_out() traces names through the whole thing, prefixed with the transformer's name.
Pass a list of column names - a bare string selects a 1-D Series and raises.
Different Columns, Different Treatment: A Practical Guide
A pipeline applies every step to every column. Real tables need scaling on some columns and encoding on others, and ColumnTransformer is how you say which is which.
Two treatments, one object
Each transformer gets the columns it was given, and the results are stuck side by side.
example_01.pyscikit-learn
Output
Selecting by type instead of by hand
So that adding a column to the data does not mean editing the pipeline.
example_02.pyscikit-learn
Output
What happens to the columns you did not mention
The default is to drop them, silently.
example_03.pyscikit-learn
Output
The whole preparation, inside one estimator
Impute, scale, encode and fit - and every step refitted on the training part of each fold.
example_04.pyscikit-learn
Output
Tuning something four levels down
The underscores compose, so a search can reach the imputation strategy inside the numeric branch.
example_05.pyscikit-learn
Output
A list of names, not a name
The most common error here, and the message says exactly what is wrong.
example_06.pyscikit-learn
Output
How it is written
Each entry is a triple: a name, a transformer, and the columns it applies to.
The name is yours to choose and appears in the generated feature names and in the parameter keys for tuning, so short and meaningful pays off — num and cat rather than transformer_1.
The columns can be a list of names, a list of integer positions, a boolean mask, or a callable. Passing a list matters: a list of one name selects a DataFrame, which is two-dimensional; a bare string selects a Series, which is one-dimensional, and the transformer raises. The error message says "Expected a 2-dimensional container", which is precise once you know what it refers to.
The outputs are concatenated left to right in the order the transformers are listed, not in the original column order. So the resulting array's columns follow your specification, which is why get_feature_names_out() is the reliable way to know what ended up where.
remainder, and the columns you forgot
By default, any column not named in any transformer is dropped.
That is usually right — ids, timestamps and free text usually should not reach the model — and it is silent, which is the part worth knowing. A column added to the source data will simply not appear in the model, with no warning, and the effect is a model quietly ignoring a feature somebody thought they had added.
remainder="passthrough" keeps the unnamed columns as they are. It is convenient and slightly dangerous: it will happily pass through an id column, which is exactly the sort of thing that lets a model memorise rows.
remainder=SomeTransformer() applies a transformer to everything left over, which is occasionally the tidiest way to say "scale everything I have not otherwise mentioned".
The habit worth adopting: name every column explicitly, or use selectors that provably cover everything, and treat the default drop as a safety net rather than as the design.
Selecting by type
Listing column names by hand is fine for a fixed table and a maintenance burden for anything that changes.
make_column_selector(dtype_include="number") selects the numeric columns whatever they are called. dtype_include=object picks up the strings. pattern="^date_" selects by name. The selector runs at fit time against whatever frame it is given, so a new numeric column is handled without touching the pipeline.
The caution is that dtypes are not always what you expect. A categorical column stored as integers — a postcode district, a department id — is selected as numeric and scaled, which is the mistake the encoding module warned about, now happening automatically. Converting such columns to a string or a pandas category dtype before the pipeline is what makes dtype-based selection safe.
Nesting a pipeline inside a branch
A single column group usually needs more than one step: numeric columns want imputing *and* scaling.
The transformer in a ColumnTransformer entry can be a Pipeline, which is how that is expressed. make_pipeline(SimpleImputer(strategy="median"), StandardScaler()) becomes the numeric branch, and the categorical branch gets its own with an imputer and an encoder.
The result is a small tree: a pipeline containing a column transformer containing pipelines containing transformers. That sounds elaborate and is the standard shape of every real scikit-learn program, because it is the minimum structure that does the right thing on a mixed table.
The whole tree is one estimator. fit on the outer pipeline reaches every leaf, and cross-validation refits all of it on each training fold.
Reaching into it to tune
The double-underscore convention composes, so parameters deep in the tree are addressable.
pre__num__simpleimputer__strategy reads outside in: the step named pre, its branch named num, the step named simpleimputer in that branch, and its strategy. Four levels, one string, and a grid search can vary it alongside the model's own hyperparameters.
This is more useful than it first appears. Whether to impute with the mean or the median, whether to scale, whether to drop the first one-hot column — these are choices usually made by guesswork, and they can be made by cross-validation instead, in the same search that tunes the model.
pipe.get_params().keys() lists every addressable parameter, which is the fastest way to find the right spelling rather than deriving it.
Getting the names back
After one-hot encoding, the array has more columns than the frame had, with names nobody wrote.
get_feature_names_out() reconstructs them, prefixed by the transformer name: num__age, cat__city_paris. Called on the outer pipeline, it traces through every step.
That matters for reading the model. Zipping the names against coef_ or feature_importances_ turns an anonymous array into a labelled result, and without it the mapping has to be reconstructed by counting, which is both tedious and easy to get wrong by one after any change to the preprocessing.
The shape of a real preprocessing block
Almost every tabular project ends up with the same structure, and it is worth having it in mind as a template rather than deriving it each time.
A numeric branch: impute with the median, then scale. A categorical branch: impute with a constant such as "missing", then one-hot encode with handle_unknown="ignore". A column transformer routing the two by dtype. A model on the end. The whole thing in one Pipeline, cross-validated, with the preprocessing choices exposed to a search.
That covers a large majority of tabular problems and is about fifteen lines. What varies between projects is which extra branches appear: a text column going to a vectoriser, a date column going to a custom transformer that extracts the parts, a high-cardinality column going to a target encoder instead of one-hot.
The value of the template is not the specific steps but that it puts every learned transformation in one place, inside the folds, addressable by a search. A project that starts from it rarely acquires the leaks that a project assembled step by step does.
One practical note: build it up one branch at a time and check the output shape after each. A column transformer that silently drops half the frame produces a model that works and underperforms, and the shape is the fastest way to notice.
When the branches disagree about rows
A limitation worth knowing, because it is not obvious from the interface.
Every transformer in a ColumnTransformer receives the same rows and must return the same number of rows, in the same order. The results are concatenated horizontally, so anything that reordered or dropped rows would misalign the columns against each other and against y.
That rules out row-level operations inside it: no dropping outliers, no resampling, no aggregating. Those belong before the pipeline, applied to the training data only, or in a library built for it — imbalanced-learn provides a pipeline that understands row changes, which is the right tool when resampling is part of the model rather than part of the data preparation.
It also means a transformer that returns a different number of rows than it received will fail with a shape error rather than a helpful message, so a custom transformer that filters is a mistake worth ruling out early.
Does the output keep my column order? No. Columns come out in the order the transformers are listed, then within each transformer. get_feature_names_out() is the reliable way to know what is where.
Can I use the same column in two transformers? Yes - a column can appear in more than one entry, and both outputs are kept. That is how you would scale a column and also bin it.
Is there a shorthand?make_column_transformer((StandardScaler(), ["age"]), (OneHotEncoder(), ["city"])) skips the names and generates them, in the same way make_pipeline does.
What if a column is numeric but categorical in meaning? Convert it to a string or a pandas category dtype before the pipeline, or dtype-based selection will scale it as a number.
Debugging one that is not working
A column transformer that misbehaves is harder to inspect than a plain pipeline, because the branches run in parallel and the output is a single anonymous array. Four things to check, in order.
The output shape.pre.fit_transform(df).shape against what you expected. Too few columns usually means a branch selected nothing; too many usually means a high-cardinality column reached the one-hot encoder.
What each branch actually selected. After fitting, pre.transformers_ holds the resolved triples, with the callable selectors already turned into concrete column lists. Printing the third element of each entry says exactly which columns went where, which is the fastest way to find a column that fell through to remainder.
The names.pre.get_feature_names_out() lists every output column with its branch prefix. If a name you expected is absent, the column was dropped; if there are hundreds of cat__ names, an identifier reached the encoder.
One branch at a time.pre.named_transformers_["num"] gives the fitted branch, which can be inspected on its own — its imputer's statistics_, its scaler's mean_. When the whole thing produces something odd, narrowing to one branch usually localises it in a minute.
The general lesson is that the object is introspectable at every level; the difficulty is only that the useful attributes have to be asked for by name.
Does it work with NumPy arrays? Yes, selecting by integer position rather than by name. Names are worth the DataFrame, since positions shift whenever a column is added.
Can a branch return a sparse matrix? Yes, and the result is sparse if enough of it is. sparse_threshold=0 forces a dense output when the next step needs one.
Things to try
Watch the shape change. The first editor turns three columns into five. Read the names to see which came from where.
Lose a column. In the third editor, note that id disappears with no warning under the default.
Add a column. In the second editor, add another numeric column to the frame and confirm the selector picks it up with no edit to the transformer.
Find the parameter names. In the fifth editor, print sorted(pipe.get_params()) and look for the keys you could tune.
Where this leaves you
One object that sends each group of columns to the transformer it needs, nests pipelines inside its branches, drops what you did not mention, and exposes every setting inside it to a search. Together with Pipeline it is the shape of essentially every real scikit-learn program on tabular data.
Check yourself
0 of 4
Answer without scrolling back up.
What happens to a column not named in any transformer?
remainder defaults to "drop". A column added to the source data will simply not reach the model, with no warning.
Why pass ["city"] rather than "city"?
The error says "Expected a 2-dimensional container". Transformers want two dimensions, which is the same X-must-be-2-D rule from the start of the track.
How do you impute AND scale the numeric columns only?
A ColumnTransformer entry accepts any transformer, including a Pipeline - which is how a branch gets several steps.
What does pre__num__simpleimputer__strategy address?
The double underscores compose outside in. pipe.get_params().keys() lists every valid key.
Cheat sheet
Different Columns, Different Treatment
A pipeline applies every step to every column. Real tables need scaling on some columns and encoding on others, and ColumnTransformer is how you say which is which.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.