Adding rows or columns from another frame - and the index and column alignment that decides the result.
Overview
Two directions
Combining DataFrames is one of the tasks you do most often. Data arrives in batches, in separate files, or from different sources. At some point you need a single frame that holds all of it. concat is the tool for that job when the frames share the same structure.
concat works in two directions. Stacking rows makes the result taller. Stacking columns makes it wider. The direction depends on the axis parameter. The default is rows, which is the common case: you have two frames with the same columns and you want all the rows in one place.
The part that trips people is alignment. When stacking rows, concat aligns by column name. If the two frames have different columns, the result has the union of both, with NaN where a frame lacked a column. That is the outer join behaviour. Passing join="inner" keeps only the columns present in both frames.
When stacking columns, concat aligns by the index. Mismatched index labels get NaN. That means two frames from different filters of the same source may align correctly, while two frames from different sources may not. Checking the index before stacking prevents surprises.
The keys parameter adds a level to the index identifying which frame each row came from. That is useful for tracing rows back to their source after the concat. It also produces a MultiIndex, which you may or may not want.
ignore_index resets the result to a clean 0-based index. That is the right choice when the original indexes are meaningless row numbers that would otherwise collide.
This module covers concat in both directions, the join parameter, keys, ignore_index, and the verify_integrity flag that catches duplicate index labels before they cause problems downstream. It also explains when concat is the wrong tool and merge is the right one.
pd.concat([a, b]) stacks rows: the result is taller.
pd.concat([a, b], axis=1) stacks columns: the result is wider.
Which axis you're joining along decides which axis does the *aligning*. That is the part worth being deliberate about.
Stacking rows aligns the columns by name. Stacking columns aligns the index by label.
Worth knowing
pd.concat stacks rows by default and keeps both indexes — pass ignore_index=True unless the labels mean something.
Columns are matched by name, not position; missing ones become NaN, and join="inner" keeps only the shared ones.
axis=1 joins side by side, and then the index does the aligning — unmatched rows become NaN.
keys=[...] records which frame each row came from, as an extra index level.
concat inside a loop copies everything so far each time. Collect into a list and concat once.
Dtypes are promoted when frames disagree — and concatenating into an empty frame can widen a column to object.
Stacking Frames with concat
Adding rows or columns from another frame.
Stacking rows
The default joins along the index, which for row-stacking means appending.
example_01.pypandas
import pandas as pd
a = pd.DataFrame({"x": [1, 2], "y": ["a", "b"]})
b = pd.DataFrame({"x": [3], "y": ["c"]})
out = pd.concat([a, b])
print(out)
print()
print("the index repeated:", list(out.index), "<- both frames kept theirs")
print()
print("ignore_index=True renumbers:")
print(pd.concat([a, b], ignore_index=True))
print()
print("A repeated index is legal and breaks later lookups, so renumber")
print("unless the labels mean something.")
Output
The default keeps both frames' index labels. The result can have repeated labels. Two rows both labelled 0.
That is legal, and it breaks things later: .loc[0] returns two rows instead of one, and code expecting a scalar fails. It's also invisible until a thing depends on it.
ignore_index=True renumbers from 0. Pass it unless the index labels carry meaning you need to keep.
Columns are matched by name. A column present in one frame and not the other is filled with NaN for the missing rows. The result has the union of all columns. join="inner" keeps only columns present in each frame.
The name-matching is a feature: two frames with the same columns in other orders line up right. It's also a trap when a column has been misspelled in one source. You get two columns each half-full rather than an error.
Checking result.columns and the isna() counts after a concat catches that at once.
Columns are matched by name
Missing ones are filled with NaN rather than lined up by position.
example_02.pypandas
import pandas as pd
a = pd.DataFrame({"x": [1], "y": [2]})
b = pd.DataFrame({"y": [3], "z": [4]})
print("union of columns, gaps filled:")
print(pd.concat([a, b], ignore_index=True))
print()
print("join='inner' keeps only the shared columns:")
print(pd.concat([a, b], ignore_index=True, join="inner"))
print()
print("Note the columns matched by NAME - 'y' lined up even though")
print("it is first in one frame and second in the other.")
Output
Stacking columns instead
axis=1 joins side by side, and now the INDEX does the aligning.
example_03.pypandas
import pandas as pd
a = pd.DataFrame({"x": [1, 2]}, index=["r1", "r2"])
b = pd.DataFrame({"y": [10, 20]}, index=["r2", "r3"])
print("axis=1 aligns on the index:")
print(pd.concat([a, b], axis=1))
print(" r1 has no y, r3 has no x - both became NaN")
print()
print("join='inner' keeps only the rows in both:")
print(pd.concat([a, b], axis=1, join="inner"))
print()
print("If you meant 'these are the same rows in the same order',")
print("reset both indexes first - otherwise labels decide, not position.")
Output
Labelling where each row came from
keys adds a level so the source survives the concat.
example_04.pypandas
import pandas as pd
jan = pd.DataFrame({"sales": [10, 20]})
feb = pd.DataFrame({"sales": [30]})
out = pd.concat([jan, feb], keys=["jan", "feb"])
print(out)
print()
print("index is now hierarchical:", out.index.names)
print()
print("select one source back out:")
print(out.loc["jan"])
print()
print("or flatten it into a column:")
print(out.reset_index(level=0).rename(columns={"level_0": "month"}))
Output
Concat in a loop is the usual mistake
Each call copies everything so far, exactly like np.append.
example_05.pypandas
import pandas as pd
import time
pieces = [pd.DataFrame({"v": [i]}) for i in range(400)]
t = time.perf_counter()
acc = pd.DataFrame(columns=["v"])
for p in pieces:
acc = pd.concat([acc, p], ignore_index=True)
loop = time.perf_counter() - t
t = time.perf_counter()
once = pd.concat(pieces, ignore_index=True)
single = time.perf_counter() - t
print("concat in a loop : %.4f s" % loop)
print("concat once : %.4f s" % single)
print("ratio : %.0fx" % (loop / max(single, 1e-9)))
print("same rows :", len(acc) == len(once))
print()
print("Collect the pieces in a list and call concat once.")
Output
Dtypes can change when frames disagree
A column that is int in one frame and float in another comes back float.
example_06.pypandas
import pandas as pd
import numpy as np
a = pd.DataFrame({"v": [1, 2]})
b = pd.DataFrame({"v": [3.5]})
c = pd.DataFrame({"v": ["x"]})
print("int + float ->", pd.concat([a, b], ignore_index=True)["v"].dtype)
print("int + str ->", pd.concat([a, c], ignore_index=True)["v"].dtype)
print()
print("and an empty frame with no dtype can widen everything:")
empty = pd.DataFrame({"v": pd.Series(dtype="object")})
print(" empty(object) + int ->",
pd.concat([empty, a], ignore_index=True)["v"].dtype)
print()
print("This is why starting with pd.DataFrame() and concatenating into")
print("it is worse than just collecting a list.")
Output
Stacking columns
axis=1 puts frames side by side, aligning on the index.
If the two frames have other labels, the union is used and gaps become NaN. If they've the same labels in a other order, pandas reorders to match. Which is correct and isn't what spot-based intuition expects.
The common mistake is concatenating two frames that *should* match row-for-row but whose indexes have drifted apart, often because one of them was filtered. The result is a frame with far more rows than either input, mostly NaN.
If you mean "these are the same rows in the same order", reset both indexes first:
That makes the spot-based intent clear rather than leaning on labels that may not match.
join="inner" keeps only rows present in both.
keys
pd.concat([jan, feb], keys=["jan", "feb"]) adds an outer index level recording which frame each row came from.
This is how you combine monthly files, or results from a few runs, without losing track of the source. out.loc["jan"] selects one back out. reset_index(level=0) turns the label into an plain column.
Without keys, that data is gone the moment the frames are stacked. Rebuilding it means remembering how many rows each contributed.
Never concat in a loop
pd.concat allocates a new frame and copies all into it. Inside a loop that is quadratic. It's one of the most common speed mistakes in pandas.
Collect the pieces in a Python list and call concatonce:
frames = [process(f) for f in files]
out = pd.concat(frames, ignore_index=True)
The last-but-one editor measures the gap. It grows with the number of loops. A pattern that seems fine on ten files turns unusable on a thousand.
Dtypes shift
When frames disagree about a column's type, the result is promoted to a thing that holds both: int plus float gives float. Anything plus a string gives object.
There's a exact version of this worth knowing. Starting with an empty frame and concatenating into it. An empty column has no real dtype. Mixing it with real data can widen the result to object. At which point the column is slow and no longer numeric.
That is another reason the list-then-concat pattern is better: there's no empty seed frame to poison the types.
After any concat that mixes sources, result.dtypes is worth a glance for just this.
Combining files
The canonical use is reading many files into one frame:
frames = []
for path in sorted(glob.glob("data/*.csv")):
d = pd.read_csv(path)
d["source"] = os.path.basename(path)
frames.append(d)
df = pd.concat(frames, ignore_index=True)
Three details make this solid.
Tag the source before appending. So A row can be traced back to its file. keys= does the same thing through the index if you prefer.
Sort the paths, so the result is fixed rather than based on filesystem order.
Concat once, outside the loop.
The frequent surprise is that the files don't agree. A column renamed halfway through the year, an extra column in later exports. Concat unions the columns and fills the gaps with NaN, silently. Checking df.isna().mean() later shows at once which columns are only present in some files.
verify_integrity and sort
verify_integrity=True raises if the result would hold copy index labels. It costs a check and blocks a duplicated index passing silently.
sort=True sorts the columns alphabetically when frames have other sets. The default leaves them in order of first appearance. This Is often more readable.
What concat cannot do
concat aligns on labels. It does not join on values.
Mixing two frames on a shared key column is merge, not concat(axis=1). Using concat for that works only if the key happens to be the index of both frames and both are sorted the same way. This Is a coincidence rather than a design.
The tell is axis=1 on frames whose indexes weren't on purpose made to match. If you find yourself resetting indexes to make a concat line up, you probably wanted merge.
Memory during a concat
concat allocates a frame the size of all the inputs mixed. The inputs stay alive until it returns. Peak memory is so roughly twice the final size.
On a large combine that is the binding constraint. There are two ways around it.
Process and reduce each piece before mixing. Filter, select columns, aggregate. Concatenating summaries is far cheaper than concatenating raw data.
Delete the list later — del frames releases the inputs. This Else stay referenced.
For files larger than memory in total, the answer isn't pandas. Read in chunks and aggregate incrementally, or use Dask, DuckDB or Polars, all of which are built for it.
A summary
Same columns, more rows — concat([a, b], ignore_index=True).
Same rows, more columns, aligned by label. concat([a, b], axis=1), only when the indexes truly match.
Joining on a key — merge, not concat.
Many files — a list comprehension and one concat, with a source tag.
Recording provenance — keys=, or an clear column.
And later, three checks: the row count, dtypes. isna().mean(). Between them they catch the mismatched columns, the promoted types and the frames that weren't what you assumed.
Concat versus merge, decided
The question that settles it: are you adding rows of the same kind, or columns about the same rows?
Same kind of rows — this month's data added to last month's — is concat.
Columns about the same rows, matched on a key. Adding customer details to orders — is merge.
concat(axis=1) looks like the second but matches on the index, not on a key column. It's right only when the indexes were on purpose made to match. This Is rarer than it looks.
If you find yourself resetting indexes so a concat(axis=1) lines up, the step you wanted was a merge.
Checking after a concat
Three lines, and each catches a other failure:
len(out) == sum(len(f) for f in frames) # nothing lost or duplicated
out.dtypes # nothing promoted to object
out.isna().mean() # which columns only some files had
The third is the one that finds a column renamed partway through a series of files. It appears as two columns, each populated for part of the rows, and the totals look plausible.
Keys and provenance
keys=["jan", "feb"] adds an index level naming the source.
names=["month"] names that level, so reset_index makes a sensible column name rather than level_0.
For a flat result, a column is simpler:
frames = [d.assign(source=name) for name, d in items]
Either way, recording where a row came from is worth doing at the moment of mixing. Later the data is gone and rebuilding it means remembering row counts.
A closing note on Stacking Frames with concat
concat is simple enough that its failures are all about expectations rather than mechanics.
The row-stacking case is nearly always right. The two things to note are ignore_index=True and that columns match by name. A renamed column in one file turns two half-full columns rather than an error.
The column-stacking case is the one to be suspicious of. It aligns on the index. Indexes drift apart the moment anything is filtered. If the frames should match row for row, reset both indexes and say so. If they should be matched on a key, the step is a merge.
And the loop rule applies here as all over. Collect the pieces, combine once. It's the same lesson as np.append and df.loc[len(df)]. It's the single most common avoidable slowdown in code that reads many files.
Two more things worth knowing
pd.concat accepts a dict and a list, in which case the keys become the outer index level on its own: pd.concat({"jan": a, "feb": b}) is the same as passing keys=. That reads well when the pieces already live in a dict keyed by their source.
The axis arg also accepts the string names "index" and "columns". This Are harder to misread than 0 and 1 in code someone else will maintain.
And one action worth expecting: concatenating frames whose columns are in other orders makes the union in order of first appearance, not sorted, unless you pass sort=True. Two files exported months apart with the columns reordered will so combine right but present in an order that matches neither. This Is harmless and briefly confusing.
Combining results, not raw data
The most scalable use of concat is on summaries rather than on source data.
Reading a hundred files and concatenating them raw builds one frame holding all. Peak memory is roughly twice its final size because the pieces stay alive until the concat returns. Reading each file, reducing it to what the question needs. Concatenating the small results costs a fraction of that and scales to more files than fit in memory.
parts = []
for path in paths:
d = pd.read_csv(path, usecols=COLS, dtype=DTYPES)
parts.append(d.groupby("city", as_index=False)["sales"].sum())
out = pd.concat(parts, ignore_index=True).groupby("city", as_index=False).sum()
The double aggregation is the pattern: reduce per file, combine, reduce again. It works for sums, counts and maxima. Anything associative. And not for medians or exact distinct counts. This Need the whole dataset at once.
That gap, between aggregations that break down and those that don't, is worth knowing before designing a pipeline around chunked reading.
Questions people ask about Stacking Frames with concat
What is the difference between concat and merge? concat stacks frames by position along an axis, aligning by labels on the other axis. merge joins on column values or the index, like a SQL join. Use concat when the frames share a schema and you want all the rows; use merge when you need to match rows by a key.
Why does concat produce NaN columns? When two frames have different column names and you stack rows, the default outer join keeps every column. Rows from the frame that lacked a column get NaN there. Pass join="inner" to keep only columns present in all frames.
Should I use ignore_index=True? Yes, if the original row labels carry no meaning. Without it, concat preserves the original indices, so you get duplicate index values that cause confusing results with loc.
Why is concat inside a loop slow? Each call copies every row accumulated so far plus the new rows. Over n iterations that is O(n squared) copies. Collect the frames in a list and call concat once at the end.
Can I concat along columns instead of rows? Yes, pass axis=1. The frames are then aligned by their row index. If the indices do not match, the result gets NaN in the mismatched positions.
How do I keep track of which frame each row came from? Pass the keys parameter with a list of labels. concat adds a MultiIndex level that tags each group of rows with its key.
Stacking Frames with concat, in summary
Row stacking aligns columns by name. Column stacking aligns the index by label. Knowing which axis you're joining along tells you which axis is doing the matching. That predicts each surprise this function makes.
Pass ignore_index=True unless the labels mean a thing, record provenance with keys= or an clear column. Never call it inside a loop.
later, check the row count, the dtypes and the missingness. Between them they catch the mismatched columns, the promoted types. The file whose schema quietly changed.
Check yourself
0 of 4
Answer without scrolling back up.
What does `pd.concat([a, b])` do with the two frames' indexes?
A repeated label is legal and invisible until .loc returns two rows where code expected one. Pass ignore_index=True unless the labels matter.
When stacking rows, how are columns matched?
A feature when column orders differ, and a trap when a name is misspelled in one source - you get two half-full columns rather than an error.
Two frames should correspond row-for-row but one was filtered. What does `concat(axis=1)` give?
Reset both indexes first if you mean 'same rows, same order' - that makes the positional intent explicit.
Why avoid starting with an empty DataFrame and concatenating into it?
Collect pieces in a list and concat once - there is then no empty seed frame to poison the dtypes.
Cheat sheet
Stacking Frames with concat
Stack DataFrames vertically or horizontally with pandas concat. Control index alignment, handle mismatched columns and reset the index.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.