vizlearn

Adding and Removing Columns

pandasBy Updated

assign, drop, rename - and the alignment that decides what a new column actually contains.

Overview

Bracket assignment

df["total"] = df["qty"] * df["price"] adds a column at the end.

Two behaviours are worth stating because neither warns.

A scalar broadcasts: df["currency"] = "INR" fills every row.

An existing name is replaced, silently. There is no protection against overwriting a column by reusing its name, which is a real hazard in a long script where the column list is not in front of you.

Column order is insertion order. insert is the only way to place one somewhere specific.

Worth knowing

df["new"] = ... adds a column at the end, or silently replaces one of the same name.
assign returns a new frame and takes lambdas, so it chains — and each lambda sees the frame as it is at that point.
Assigning a Series aligns on the index; assigning a list goes in by position.
Assigning a subset-derived Series to a whole column overwrites every unmatched row with NaN, even one that held a value. df.loc[mask, "col"] = ... touches only the selected rows.
drop(columns=[...]) and drop(index=[...]) return a new frame; a missing name raises unless errors="ignore".
rename ignores keys that do not exist, so a typo does nothing. insert is the only way to choose a column's position.

Adding and Removing Columns

assign, drop, rename, and the alignment that decides what a new column contains.

Assigning a new column

Bracket assignment adds it at the end, or replaces it if the name exists.

example_01.pypandas
Output

                

                

                    
assign returns a new frame

                    
Which is what makes it chainable, and what makes it safe.
example_02.pypandas
Output

                

                

                    
Alignment decides what lands in the column

                    
Assigning a Series matches on the index, not on position.
example_03.pypandas
Output

                

                

                    
The filtered-assignment trap

                    
Computing from a subset and assigning back wipes every row that was not in it.
example_04.pypandas
Output
The most common real instance:
sub = df[df["city"] == "pune"]
df["doubled"] = sub["sales"] * 2
sub has only the Pune rows, so the computed Series has only those labels. Alignment fills every other row with NaN.
pandas did exactly what it promises. The author expected "compute for these rows and leave the others alone", which is a different operation:
df.loc[df["city"] == "pune", "doubled"] = df["sales"] * 2
The difference is visible whenever the column already exists. Whole-column assignment replaces the entire column, so rows absent from the subset are overwritten with NaN — even if they held a perfectly good value a moment ago. The .loc form touches only the selected rows and leaves the rest alone.
When the column does *not* already exist, both forms leave NaN in the unselected rows, because creating a column has to put something in every row. If you want a default there, create the column first and then overwrite the subset — which is exactly what the editor above does.

                

                

                    
Dropping columns and rows

                    
The same method, steered by axis - and it returns a new frame by default.
example_05.pypandas
Output

                

                

                    
Renaming, and inserting in a position

                    
rename takes a mapping; insert is the only way to choose where a column goes.
example_06.pypandas
Output

                

                

                    
assign

                    
df.assign(total=...) returns a new frame rather than modifying in place.
That makes it chainable, which is its main purpose:
out = (df
       .assign(total=lambda d: d["qty"] * d["price"])
       .assign(cheap=lambda d: d["total"] < 100))
The lambda receives the frame as it is at that point in the chain, so the second assign can use the column the first one created. Passing a value directly rather than a lambda works too, but then it is computed against the original frame, which breaks in a chain.
assign also keeps the original untouched, which is worth something in a notebook where cells get re-run out of order.
The cost is a copy per call. In a hot loop that matters; in ordinary analysis code it does not, and the readability of a chain usually wins.

                

                

                    
Alignment decides the contents

                    
This is the part that surprises people, and it is the index rule again.
Assigning a Series aligns on the index. If the Series has the same labels in a different order, pandas reorders it to match — which is correct and is not what positional intuition expects. If the Series is missing some labels, those rows get NaN. If it has extra labels, they are dropped.
Assigning a list or a NumPy array has no index to align on, so it goes in by position, and the length must match exactly or it raises.
That difference means df["x"] = some_series and df["x"] = some_series.values can produce different columns from the same data. The first is usually what you want; the second is the escape hatch when you know the order is right and the labels are not.

                

                

                    
Dropping

                    
df.drop(columns=["b"]) and df.drop(index=[0]) both return a new frame.
The axis= form works and is older; columns=/index= say what they mean and are worth preferring.
A name that does not exist raises KeyError. That is usually helpful — it catches typos and stale column lists — and errors="ignore" turns it off when dropping optional columns.
To drop in place, reassign: df = df.drop(columns=["b"]). The inplace=True argument exists, and is discouraged: it does not reliably avoid a copy, it breaks chaining, and it returns None, which makes df = df.drop(..., inplace=True) a silent way to destroy your frame.

                

                

                    
Renaming

                    
df.rename(columns={"a": "alpha"}) takes a mapping, and columns=str.upper takes a function applied to every name.
The important quirk: keys that do not match anything are ignored silently. A typo in the old name does nothing at all and gives no indication. When a rename appears not to have worked, a misspelled key is the first thing to check.
df.columns = [...] replaces every name at once and requires the right length. It is blunter and, for a full rename, clearer.
For cleaning up messy headers, a function is usually better than a mapping:
df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")
That handles the whole frame without listing every column, and is worth running on anything that came from a spreadsheet.

                

                

                    
Adding several columns at once

                    
assign takes any number of keyword arguments, and they are applied in order, so later ones can use earlier ones:
df.assign(
    total=lambda d: d["qty"] * d["price"],
    with_tax=lambda d: d["total"] * 1.18,
)
For column names that are not valid identifiers — a space, a leading digit — keyword arguments will not work, and bracket assignment is the only option. That is a reasonable argument for normalising column names early.
Assigning several columns from one operation is the case assign handles less well. When a function returns several values per row, a common pattern is:
df[["a", "b"]] = df["text"].str.split("-", expand=True)
The right-hand side must have the same number of columns as the left, and the index must align.

                

                

                    
Reordering

                    
Column order is insertion order, and there is no reorder method.
Three ways, in increasing robustness:
df[["b", "a", "c"]] — select in the order you want. Fails loudly if a name is wrong, which is usually a feature.
df.insert(pos, name, values) — place one column at a position, in place.
Building the order from what exists:
first = ["id", "date"]
df = df[first + [c for c in df.columns if c not in first]]
That pins the columns you care about to the front and keeps the rest, without breaking when the schema changes.

                

                

                    
Removing columns safely

                    
drop(columns=[...]) raises on a name that does not exist, which catches typos and stale lists.
errors="ignore" suppresses that, and is right when the columns are genuinely optional — dropping debug fields that may or may not be present.
df.drop(columns=df.filter(like="tmp_").columns) drops by pattern, which survives changes in the exact names.
For keeping rather than dropping, selecting is clearer: df[["a", "b"]] states what you want rather than what you do not, and does not need updating when new unwanted columns appear.

                

                

                    
A note on inplace

                    
Most of these methods take inplace=True. It is worth being clear about why it is discouraged.
It does not reliably avoid a copy, so the performance argument for it is largely false.
It returns None, so df = df.drop(columns=["a"], inplace=True) silently sets df to None — a mistake that is easy to make and confusing to debug.
It breaks method chaining.
And it is on the way out: the pandas team has discussed removing it, and copy-on-write makes its remaining rationale weaker.
Reassignment — df = df.drop(columns=["a"]) — is clearer and no slower in practice.

                

                

                    
Renaming, at scale

                    
For one or two columns, a mapping is fine.
For a whole frame that arrived with human-written headers, a function over the Index is better:
df.columns = (df.columns
              .str.strip()
              .str.lower()
              .str.replace(r"[^a-z0-9]+", "_", regex=True)
              .str.strip("_"))
This handles trailing spaces, capitals, punctuation and units-in-parentheses in one pass, and does not need to know what the columns are called.
Doing it immediately after loading means every later reference uses predictable names, and it removes the class of bug where a column name has an invisible trailing space.
df.rename(columns=..., errors="raise") makes a mapping strict, so a key that matches nothing raises instead of doing nothing silently. That is worth using when the rename is important.

                

                

                    
Creating columns conditionally

                    
The three standard shapes, none of which needs apply:
Two branches — np.where(cond, a, b).
Several branches — np.select([c1, c2], [v1, v2], default=v3), evaluated in order.
Numeric bands — pd.cut.
A lookup — df["k"].map(mapping).
For a column that exists only for some rows, create it with a default first and then overwrite the subset:
df["band"] = "unknown"
df.loc[df["score"] >= 80, "band"] = "high"
That avoids the NaN that appears when a column is created directly for a subset, and it makes the default explicit rather than implied.

                

                

                    
Types when adding a column

                    
A new column takes its dtype from what you assign. Two cases are worth watching.
Assigning a Python list of integers gives int64; assigning one with a None in it gives object or float64. If the column is meant to be a nullable integer, say so: pd.array([1, None], dtype="Int64").
Assigning a string to a subset of a numeric column upcasts the whole column to object, silently, which makes every later numeric operation on it slow or wrong. If a column may hold either, that is usually a sign it should be two columns.

                

                

                    
Dropping rows

                    
drop(index=[...]) removes by label, which is fine for a handful of known labels and awkward otherwise.
For a condition, filtering is clearer: df[~mask] says "everything except", and does not require knowing the labels.
df.drop(df[mask].index) works and is a longer way of writing the same thing.
dropna, drop_duplicates and query cover the common cases directly, and each says what it does in its name.

                

                

                    
A summary

                    
df["new"] = ... adds or silently replaces.
assign returns a new frame and chains; use a lambda inside a chain.
Assigning a Series aligns on the index; a list goes in by position.
Whole-column assignment from a subset overwrites the unmatched rows with NaN; .loc touches only the selection.
drop(columns=...) raises on a missing name unless errors="ignore".
rename ignores keys that match nothing, so a typo does nothing at all.
Normalise column names once, at load, with a function over df.columns.
And prefer reassignment to inplace=True, which does not do what its name suggests.

                

                

                    
A closing note

                    
Adding a column looks like the simplest thing in pandas and carries two traps.
The first is silence: assigning to an existing name replaces it without a word, and a rename with a misspelled key does nothing without a word. Neither raises, and both are easy to miss in a long script.
The second is alignment. A Series assigned to a column is matched by label, not position, so a value computed from a filtered subset fills the unmatched rows with NaN — and if the column already existed, it overwrites what was there. A plain list, having no index, goes in positionally instead. Two lines that look equivalent behave differently.
The defences are small: use .loc when you mean "these rows only", check the column list after a rename, and normalise column names once at load so that later references are predictable.

                

                

                    
One more thing

                    
df.pop("col") removes a column and returns it, which is occasionally the clearest way to move a column out of a frame and into a variable in one step.
And df.drop accepts level= for a MultiIndexed frame, so a whole outer group can be removed by label without building a mask. Both are small conveniences, and both read better than the two-step alternatives when the intent is exactly what they describe.

                

                

                    
In summary

                    
Adding and removing columns is mechanically simple, and the two things that bite are both about silence.
Nothing warns when you overwrite a column by reusing its name, and nothing warns when a rename key matches nothing. Both leave the frame in a state that looks correct.
And assignment is governed by alignment: a Series matches on labels, a list matches on position, and a whole-column assignment computed from a subset overwrites every unmatched row with NaN — including rows that held a perfectly good value a moment before.
Using .loc when you mean "these rows only", normalising column names once at load, and checking the column list after a rename cover all of it.

                

            


        

    


    



Check yourself
0 of 4
Answer without scrolling back up.
  1. What happens when you assign to an existing column name?
  2. You assign a Series whose index is in a different order. What lands in the column?
  3. An existing column holds 0 everywhere. What does `df['x'] = df[mask]['y'] * 2` do to the unmatched rows?
  4. What does `df.rename(columns={'typo': 'new'})` do when 'typo' does not exist?





 Cheat sheet
Adding and Removing Columns
An existing name is replaced, silently. There is no protection against overwriting a column by reusing its name, which is a real hazard in a long script where the column list is not in front of you.
PANDAS · vizlearn.in/pandas/adding_and_dropping.html
More in pandas
pandasDtypes and MemorypandasMissing DatapandasDuplicatespandasText Columns
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact