The labelled axis that participates in almost every operation - and causes almost every surprise.
Overview
What it is
The index is the feature most people ignore until it causes a bug. It sits on every Series and every DataFrame. By default it looks like row numbers, so nothing about it seems worth thinking about. Then a filter, a merge or a group-by produces an unexpected result, and the index is the reason.
Understanding the index early prevents a category of confusion that otherwise recurs for months. Almost every "pandas is broken" question on forums traces back to index behaviour the author did not expect.
An index is an immutable, ordered set of labels. One label per row. It can hold integers, strings, timestamps, or tuples. It participates in alignment, selection, joins and reshaping. When two Series are added, pandas matches their index labels. When a merge runs, the index can serve as the join key.
The default RangeIndex of 0, 1, 2 hides all of this. It makes the index look like a row counter. For a freshly created frame, it behaves like one. After a single filter, the positions and labels diverge, and code that assumed they were the same gives wrong answers or raises a KeyError.
This module covers what the index is, how it sticks to rows through filtering, how alignment uses it, and how to reset it when you need positional access again. The examples are short and show the index before and after each operation, so the behaviour is visible rather than implied. Each one prints the index explicitly so you can track how labels move, stick and sometimes surprise you.
each Series and each DataFrame carries an index. An sorted set of labels, one per row. A DataFrame also has an index for its columns.
If you don't supply one, pandas creates a RangeIndex of 0, 1, 2, .... That default is why the index is easy to overlook. It looks like row numbers. For a freshly created frame it behaves like them.
It isn't row numbers. It's a labelled axis that takes part in pick, alignment, joining, grouping and reshaping. Nearly each pandas action that surprises people is the index doing a thing they didn't ask for.
Worth knowing
Both axes are Index objects — the rows and the columns. Anything true of one is true of the other.
The index survives filtering, so a filtered result is no longer numbered 0..n-1.
set_index promotes a column to the index; reset_index turns it back into a column, and drop=True discards it.
Combining two objects aligns on labels first. Unmatched labels become NaN — use add(other, fill_value=0) to treat them as zero.
Labels may be duplicated, and a lookup then returns a Series rather than a scalar.
Slicing by a label range works on any unique index, sorted or not — it is a non-unique unsorted index that raises.
The Index
The labelled axis that participates in almost each step.
Every Series and DataFrame has one
It is created for you if you do not supply it, which is why it is easy to forget it exists.
example_01.pypandas
import pandas as pd
s = pd.Series([10, 20, 30])
print("values:", list(s.values))
print("index :", list(s.index))
print("type :", type(s.index).__name__)
print()
df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})
print("row index :", list(df.index))
print("col index :", list(df.columns), "<- the columns are an Index too")
print()
print("Both axes are Index objects. Anything true of one is true of the other.")
Output
It survives filtering, and that trips people
The labels come with the rows, so the result is no longer 0..n-1.
example_02.pypandas
import pandas as pd
s = pd.Series([5, 15, 25, 35, 45])
big = s[s > 20]
print("filtered:")
print(big)
print()
print("index is now:", list(big.index), "<- not 0,1,2")
print()
print("so positional thinking breaks:")
try:
print(big[0])
except KeyError:
print(" big[0] -> KeyError: there is no label 0 any more")
print()
print("big.iloc[0] ->", big.iloc[0], " (position)")
print("big.loc[2] ->", big.loc[2], " (label)")
Output
reset_index and set_index
Turning the index into a column, and a column into the index.
example_03.pypandas
import pandas as pd
df = pd.DataFrame({"city": ["pune", "delhi", "goa"], "pop": [3, 19, 2]})
print("default index:", list(df.index))
by_city = df.set_index("city")
print()
print("after set_index('city'):")
print(by_city)
print(" lookup by label:", int(by_city.loc["delhi", "pop"]))
print()
back = by_city.reset_index()
print("reset_index puts it back as a column:", list(back.columns))
print()
print("drop=True throws the old index away instead of keeping it:")
print(list(df[df["pop"] > 2].reset_index(drop=True).index))
Output
Alignment is the reason it matters
Two objects combined are matched on labels first. This is silent and it is not optional.
example_04.pypandas
import pandas as pd
q1 = pd.Series([100, 200, 300], index=["pune", "delhi", "goa"])
q2 = pd.Series([10, 20], index=["goa", "pune"])
print("q1 + q2:")
print(q1 + q2)
print()
print("goa got 300+10, pune got 100+20, delhi had no partner -> NaN")
print()
print("To treat a missing label as zero, be explicit:")
print(q1.add(q2, fill_value=0))
print()
print("Alignment is a safety feature - a mismatch shows up as NaN")
print("rather than as a silent off-by-one.")
Output
Duplicate labels are allowed
Which means a single lookup can return several rows, changing the type of what comes back.
example_05.pypandas
import pandas as pd
s = pd.Series([1, 2, 3], index=["a", "b", "a"])
print("index has a twice:", list(s.index))
print("is_unique:", s.index.is_unique)
print()
print("s['b'] ->", s["b"], " (a scalar)")
print("s['a'] ->")
print(s["a"], " <- a Series, not a scalar")
print()
print("Code that assumes a scalar breaks only when a duplicate appears,")
print("which is usually in production rather than in the test data.")
print()
print("verify_integrity=True refuses to build a duplicated index:")
try:
pd.DataFrame({"k": ["x", "x"], "v": [1, 2]}).set_index("k", verify_integrity=True)
except ValueError as e:
print(" ", str(e)[:60])
Output
When a label range needs a sorted index
Not as often as people say - it is uniqueness that decides it.
example_06.pypandas
import pandas as pd
uniq = pd.Series(range(5), index=[30, 10, 40, 20, 50])
print("unique but unsorted:", list(uniq.index))
print(" sorted?", uniq.index.is_monotonic_increasing,
" unique?", uniq.index.is_unique)
print(" uniq.loc[10:40] ->", list(uniq.loc[10:40]), "<- works fine")
print()
print("It slices by POSITION of the two bounds, so the order you get")
print("is the order the index is in, not numeric order.")
print()
dup = pd.Series(range(5), index=[3, 1, 4, 1, 5])
print("non-unique AND unsorted:", list(dup.index))
try:
dup.loc[1:4]
except Exception as e:
print(" dup.loc[1:4] ->", type(e).__name__)
print(" ", str(e)[:64])
print()
print("sort_index() fixes it:", list(dup.sort_index().loc[1:4]))
Output
Labels stick to rows
Filter a Series of five values down to three and the surviving rows keep their first labels. The result's index might be 1, 3, 4.
This is correct — the label finds the row. The row didn't change identity by being selected — but it breaks spot-based habits at once.
big[0] raises KeyError, because there's no label 0 any more. big.iloc[0] gets the first row by spot. big.loc[1] gets the row labelled 1.
Two results worth internalising.
**After filtering, use iloc for spot and loc for labels**. Know which you mean. The next module is fully about that gap.
**reset_index(drop=True) renumbers** when you truly want a fresh 0..n-1. Without drop=True, the old index is kept as a new column. This Is at times what you want and often not.
set_index and reset_index
df.set_index("city") makes the city column the index. Lookups by city then work with .loc, joins on city become index joins, and grouping by it's cheaper.
df.reset_index() reverses it, putting the index back as an plain column and restoring a RangeIndex.
Setting a real index is worth doing when you'll look rows up by that key again and again. It isn't worth doing reflexively. An index carries rules. A frame with a RangeIndex is simpler to reason about.
Alignment
This is the action that most sets apart pandas from NumPy, and it's silent.
When two Series are mixed, pandas matches them by label before doing anything. Order doesn't matter. A label present in one and absent from the other makes NaN rather than an error.
For addition of two quarterly figures indexed by city, that is just right. You want Pune added to Pune, what order the rows arrived in. A city missing from one quarter should be visibly missing rather than silently paired with the wrong row.
It turns a problem when you didn't realise the indexes differed. A common version. You filter a frame, find a column from it. Assign that column back to the first frame. Alignment matches on the filtered labels. The rows you filtered out get NaN. Which is arguably the right answer, and is rarely what the author expected.
.values or .to_numpy() strips the index and forces spot-based action. That is at times the right escape hatch. It silences the safety feature. It deserves a comment when used.
The math methods take fill_value: q1.add(q2, fill_value=0) treats a missing label as zero rather than passing NaN.
Duplicates
An index may hold the same label more than once. pandas doesn't prevent it.
The result is that s["a"] returns a scalar when a appears once and a Series when it appears twice. Downstream code written against the scalar case fails when a copy appears. Copies often appear in production rather than in the sample used for development.
s.index.is_unique checks. set_index(..., verify_integrity=True) refuses to build a duplicated index. This Is worth passing when uniqueness is part of what the data means.
df.index.duplicated() gives a mask of the repeats. So You can inspect them rather than guess.
Sorted indexes
is_monotonic_increasing tells you whether the index is sorted.
A sorted index allows binary-search lookups. This Matters on large frames.
It's often said that label-range slicing *needs* a sorted index. That isn't quite the rule. The editor above shows it. A unique index slices fine what order it's in. Pandas can find each bound unambiguously and take all between them. What it can't do is slice a non-unique, unsorted index. There's no single spot for a repeated bound. It raises rather than guessing.
Worth knowing what that slice returns on an unsorted index. All between the two bounds *in the index's own order*. This Isn't numeric order. That is rarely what people intend. sort_index() first is still good practice even where it isn't needed.
Note that label slicing with .loc is inclusive of the endpoint, unlike every other slice in Python. s.loc[100:200] includes 200. That is deliberate. With labels there's no "one past the end" to point at. And it's a solid source of off-by-one surprises.
sort_index() sorts by the index. sort_values() sorts by the data. Both return new objects by default.
The habits worth forming
Look at the index when a result surprises you. It's the first thing to check, ahead of the values.
Call reset_index(drop=True) after filtering when downstream code will think positionally.
Prefer .loc and .iloc over bare []. That "label" or "spot" is written down rather than inferred.
Use fill_value when mixing objects whose labels may not match just.
And check is_unique before leaning on a lookup returning one row.
Kinds of index
The default is a RangeIndex. A compact 0, 1, 2, ... that stores only start, stop and step rather than the values.
Index holds any labels: strings, ints, anything hashable.
DatetimeIndex holds timestamps and unlocks partial string pick (s["2024-02"]) and resample.
CategoricalIndex holds a fixed set of groups, and is memory-fast for a repeated key.
MultiIndex holds a few levels, and is what a group-by on more than one key returns.
You rarely choose between these on purpose. They arrive from what made the object. Knowing which you've shows what steps are on hand.
The index is not free
RangeIndex costs almost nothing. Any other index stores one label per row. A string index over a million rows is a million Python strings. Often larger than the numeric columns it labels.
df.index.dtype and df.memory_usage(deep=True) show it. reset_index(drop=True) discards it and returns to a RangeIndex. This Is worth doing when the labels carry no meaning.
That is the hands-on arg for not setting a real index reflexively. It costs memory. It only pays if you really select or join on it.
Selecting with an index that is not unique
is_unique is worth checking before leaning on lookups. The return type of s[label] depends on it.
With a unique index, s["a"] is a scalar. With a duplicated one it's a Series. Code written against the first case. Math, a compare, passing the value to a function. Breaks on the second. The break happens wherever the value is used rather than at the lookup.
df.index.duplicated() gives a mask of the repeats, and df[df.index.duplicated(keep=False)] shows each conflicting group.
verify_integrity=True on set_index refuses to build a duplicated index. Is worth passing when uniqueness is part of what the data means.
Aligning on purpose
Alignment is often helpful and at times in the way. Three ways to control it:
Match the indexes — reindex(other.index) conforms one object to another's labels, filling missing ones with NaN. This is alignment made clear rather than silent.
Drop the index — .values or .to_numpy() gives a plain array with no labels. Steps become spot-based. Use it when you know the order is right and the labels are noise, and comment why.
Reset both — reset_index(drop=True) on both sides before mixing, when they should match row-for-row.
The failure this blocks is the one from the previous module. Two objects that *should* line up, whose indexes have drifted apart because one was filtered, silently making NaN or a much longer result.
Index methods worth knowing
An Index behaves like an immutable set. The set steps are at times just what you need:
a.index.gap(b.index) — labels in one and not the other. The fastest way to find out which rows a join would drop.
a.index.intersection(b.index) — the common labels.
idx.get_loc(label) — the spot of a label, for when you truly need to cross from labels to spots.
idx.str — the string accessor works on an Index too. This Is how you clean column names in one formula.
Indexes are immutable. You can't assign into one. You build a new one and attach it. That is deliberate. A mutable index would break the hash-based lookups that make label pick fast.
When to set a meaningful index
Set one when you'll look rows up by that key again and again, join on it more than once, or need time-based pick and resample.
Don't set one when the key isn't unique, when you only need it once, or when the result is heading to a file or a merge that expects columns.
reset_index() is cheap and always on hand. The choice isn't permanent. Which is a good reason not to agonise over it.
Common index mistakes
given a filtered frame is renumbered. It isn't. reset_index(drop=True) renumbers.
**Using [0] after a filter.** There may be no label 0. .iloc[0] is the spot.
**Losing drop=True.** reset_index() keeps the old index as a new column. This Then travels through each later step as an unwanted index column.
Setting an index and then merging on the column. After set_index("id"), there's no id column to merge on. Use left_index=True, or reset first.
given alignment is spot-based. It's by label. Two Series with the same values in other label orders add to a thing neither of them looks like.
Skipping copies. A lookup that returns a Series instead of a scalar breaks downstream code. The break happens away from the lookup.
Reindexing
reindex conforms an object to a given set of labels. Present labels are kept, absent ones are filled with NaN. Labels not in the new set are dropped.
That makes it the tool for two jobs.
Making frames close.b.reindex(a.index) puts b on a's labels. A later step aligns just and any mismatch is visible as NaN rather than silently reordering.
Filling out a sparse axis. Reindexing onto a full date range turns missing days into clear NaN rows. This Is what you want before plotting or before a rolling window over rows.
fill_value= sets what the gaps become. method="ffill" carries values forward. This Needs a sorted index.
reindex_like(other) is the shorthand for the first case.
The index in output
The index is written by to_csv unless you pass index=False. Read back as an unnamed column unless you pass index_col=0. That round-trip mismatch is the origin of the Unnamed: 0 column that appears in so many datasets.
to_dict(), to_json() and to_excel all have their own views about the index. All of them are worth checking once for any output that another system consumes.
The rule that avoids most of it: if the index is real data, name it and write it on purpose. If it isn't, reset_index(drop=True) before writing.
Questions people ask about the Index
What is the index in pandas? It is the set of labels attached to each row. Labels travel with the rows through filtering, sorting and joins, and they decide how two objects line up in arithmetic.
What is the difference between set_index and reset_index? set_index moves a column into the index, and reset_index moves the index back into a regular column and replaces it with 0, 1, 2 and so on.
Does the index have to be unique? No. Duplicate labels are allowed, but loc then returns several rows for one label and some operations, such as reindex, raise errors.
Why did my result gain NaN values after adding two Series? Arithmetic aligns on index labels first. Any label present in only one Series produces NaN, so mismatched labels give missing values rather than an error.
When should I drop the index with drop=True? Use reset_index(drop=True) when the old labels carry no meaning, for example after filtering or concatenating, and you just want a clean 0..n-1 range.
Is a custom index faster than a column? Lookups by label on a unique, sorted index are fast. For one-off filters a boolean mask on a column is just as good, so set an index when you will look up or align on it repeatedly.
A working summary
The index is a labelled axis, not row numbers.
It lives filtering, and alignment uses it silently.
set_index when you'll select or join on the key again and again. reset_index(drop=True) when you won't.
Check is_unique before leaning on lookups.
sort_index() after building or reshaping anything with a non-trivial index.
And when a result has the wrong number of rows or surprise NaN, look at the indexes of the inputs first. That is the cause more often than anything in the step itself.
Check yourself
0 of 4
Answer without scrolling back up.
After filtering a Series down to 3 of 5 rows, what is `result[0]` likely to do?
The surviving rows keep their original labels. Use .iloc[0] for position, or reset_index(drop=True) to renumber.
Two Series with partly overlapping labels are added. What happens to a label present in only one?
Alignment is a safety feature - a mismatch surfaces as NaN rather than a silent off-by-one. Use add(other, fill_value=0) to treat it as zero.
What does `s['a']` return when the label 'a' appears twice in the index?
The return type changes with the data, so code written against the scalar case breaks when a duplicate appears - usually in production.
What does `s.loc[100:200]` include?
Label slicing is inclusive of the endpoint, unlike every other Python slice. It works on any unique index; only a non-unique unsorted one raises.
Cheat sheet
The Index
See how the pandas Index labels rows, aligns data during joins and arithmetic, and changes the result of almost every operation.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.