The labelled axis that participates in almost every operation - and causes almost every surprise.
Overview
What it is
Every Series and every DataFrame carries an index: an ordered set of labels, one per row. A DataFrame also has an index for its columns.
If you do not supply one, pandas creates a RangeIndex of 0, 1, 2, .... That default is why the index is easy to overlook — it looks like row numbers, and for a freshly created frame it behaves like them.
It is not row numbers. It is a labelled axis that takes part in selection, alignment, joining, grouping and reshaping. Nearly every pandas behaviour that surprises people is the index doing something they did not ask for.
Worth knowing
Both axes are Index objects — the rows and the columns. Anything true of one is true of the other.
The index survives filtering, so a filtered result is no longer numbered 0..n-1.
set_index promotes a column to the index; reset_index turns it back into a column, and drop=True discards it.
Combining two objects aligns on labels first. Unmatched labels become NaN — use add(other, fill_value=0) to treat them as zero.
Labels may be duplicated, and a lookup then returns a Series rather than a scalar.
Slicing by a label range works on any unique index, sorted or not — it is a non-unique unsorted index that raises.
The Index
The labelled axis that participates in almost every operation.
Every Series and DataFrame has one
It is created for you if you do not supply it, which is why it is easy to forget it exists.
example_01.pypandas
Output
It survives filtering, and that trips people
The labels come with the rows, so the result is no longer 0..n-1.
example_02.pypandas
Output
reset_index and set_index
Turning the index into a column, and a column into the index.
example_03.pypandas
Output
Alignment is the reason it matters
Two objects combined are matched on labels first. This is silent and it is not optional.
example_04.pypandas
Output
Duplicate labels are allowed
Which means a single lookup can return several rows, changing the type of what comes back.
example_05.pypandas
Output
When a label range needs a sorted index
Not as often as people say - it is uniqueness that decides it.
example_06.pypandas
Output
Labels stick to rows
Filter a Series of five values down to three and the surviving rows keep their original labels. The result's index might be 1, 3, 4.
This is correct — the label identifies the row, and the row did not change identity by being selected — but it breaks positional habits immediately.
big[0] raises KeyError, because there is no label 0 any more. big.iloc[0] gets the first row by position. big.loc[1] gets the row labelled 1.
Two consequences worth internalising.
After filtering, use iloc for position and loc for labels, and know which you mean. The next module is entirely about that distinction.
reset_index(drop=True) renumbers when you genuinely want a fresh 0..n-1. Without drop=True, the old index is kept as a new column, which is occasionally what you want and usually not.
set_index and reset_index
df.set_index("city") makes the city column the index. Lookups by city then work with .loc, joins on city become index joins, and grouping by it is cheaper.
df.reset_index() reverses it, putting the index back as an ordinary column and restoring a RangeIndex.
Setting a meaningful index is worth doing when you will look rows up by that key repeatedly. It is not worth doing reflexively — an index carries rules, and a frame with a RangeIndex is simpler to reason about.
Alignment
This is the behaviour that most distinguishes pandas from NumPy, and it is silent.
When two Series are combined, pandas matches them by label before doing anything. Order does not matter. A label present in one and absent from the other produces NaN rather than an error.
For addition of two quarterly figures indexed by city, that is exactly right: you want Pune added to Pune, whatever order the rows arrived in, and a city missing from one quarter should be visibly missing rather than silently paired with the wrong row.
It becomes a problem when you did not realise the indexes differed. A common version: you filter a frame, compute a column from it, and assign that column back to the original frame. Alignment matches on the filtered labels, so the rows you filtered out get NaN — which is arguably the right answer, and is rarely what the author expected.
.values or .to_numpy() strips the index and forces positional behaviour. That is occasionally the right escape hatch, and it silences the safety feature, so it deserves a comment when used.
The arithmetic methods take fill_value: q1.add(q2, fill_value=0) treats a missing label as zero rather than propagating NaN.
Duplicates
An index may contain the same label more than once. pandas does not prevent it.
The consequence is that s["a"] returns a scalar when a appears once and a Series when it appears twice. Downstream code written against the scalar case fails when a duplicate appears, and duplicates usually appear in production rather than in the sample used for development.
s.index.is_unique checks. set_index(..., verify_integrity=True) refuses to build a duplicated index, which is worth passing when uniqueness is part of what the data means.
df.index.duplicated() gives a mask of the repeats, so you can inspect them rather than guess.
Sorted indexes
is_monotonic_increasing tells you whether the index is sorted.
A sorted index allows binary-search lookups, which matters on large frames.
It is often said that label-range slicing *requires* a sorted index. That is not quite the rule, and the editor above shows it: a unique index slices fine whatever order it is in, because pandas can find each bound unambiguously and take everything between them. What it cannot do is slice a non-unique, unsorted index — there is no single position for a repeated bound, so it raises rather than guessing.
Worth knowing what that slice returns on an unsorted index: everything between the two bounds *in the index's own order*, which is not numeric order. That is rarely what people intend, so sort_index() first is still good practice even where it is not required.
Note that label slicing with .loc is inclusive of the endpoint, unlike every other slice in Python. s.loc[100:200] includes 200. That is deliberate — with labels there is no "one past the end" to point at — and it is a reliable source of off-by-one surprises.
sort_index() sorts by the index; sort_values() sorts by the data. Both return new objects by default.
The habits worth forming
Look at the index when a result surprises you. It is the first thing to check, ahead of the values.
Call reset_index(drop=True) after filtering when downstream code will think positionally.
Prefer .loc and .iloc over bare [], so that "label" or "position" is written down rather than inferred.
Use fill_value when combining objects whose labels may not match exactly.
And check is_unique before relying on a lookup returning one row.
Kinds of index
The default is a RangeIndex — a compact 0, 1, 2, ... that stores only start, stop and step rather than the values.
Index holds arbitrary labels: strings, integers, anything hashable.
DatetimeIndex holds timestamps and unlocks partial string selection (s["2024-02"]) and resample.
CategoricalIndex holds a fixed set of categories, and is memory-efficient for a repeated key.
MultiIndex holds several levels, and is what a group-by on more than one key returns.
You rarely choose between these deliberately. They arrive from whatever produced the object, and knowing which you have explains what operations are available.
The index is not free
RangeIndex costs almost nothing. Any other index stores one label per row, and a string index over a million rows is a million Python strings — often larger than the numeric columns it labels.
df.index.dtype and df.memory_usage(deep=True) show it. reset_index(drop=True) discards it and returns to a RangeIndex, which is worth doing when the labels carry no meaning.
That is the practical argument for not setting a meaningful index reflexively: it costs memory, and it only pays if you actually select or join on it.
Selecting with an index that is not unique
is_unique is worth checking before relying on lookups, because the return type of s[label] depends on it.
With a unique index, s["a"] is a scalar. With a duplicated one it is a Series. Code written against the first case — arithmetic, a comparison, passing the value to a function — breaks on the second, and the break happens wherever the value is used rather than at the lookup.
df.index.duplicated() gives a mask of the repeats, and df[df.index.duplicated(keep=False)] shows every conflicting group.
verify_integrity=True on set_index refuses to build a duplicated index, and is worth passing whenever uniqueness is part of what the data means.
Aligning on purpose
Alignment is usually helpful and occasionally in the way. Three ways to control it:
Match the indexes — reindex(other.index) conforms one object to another's labels, filling missing ones with NaN. This is alignment made explicit rather than implicit.
Drop the index — .values or .to_numpy() gives a plain array with no labels, so operations become positional. Use it when you know the order is right and the labels are noise, and comment why.
Reset both — reset_index(drop=True) on both sides before combining, when they should correspond row-for-row.
The failure this prevents is the one from the previous module: two objects that *should* line up, whose indexes have drifted apart because one was filtered, silently producing NaN or a much longer result.
Index methods worth knowing
An Index behaves like an immutable set, and the set operations are occasionally exactly what you need:
a.index.difference(b.index) — labels in one and not the other. The fastest way to find out which rows a join would drop.
a.index.intersection(b.index) — the common labels.
idx.get_loc(label) — the position of a label, for when you genuinely need to cross from labels to positions.
idx.str — the string accessor works on an Index too, which is how you clean column names in one expression.
Indexes are immutable. You cannot assign into one; you build a new one and attach it. That is deliberate, because a mutable index would break the hash-based lookups that make label selection fast.
When to set a meaningful index
Set one when you will look rows up by that key repeatedly, join on it more than once, or need time-based selection and resample.
Do not set one when the key is not unique, when you only need it once, or when the result is heading to a file or a merge that expects columns.
reset_index() is cheap and always available, so the decision is not permanent — which is a good reason not to agonise over it.
Common index mistakes
Assuming a filtered frame is renumbered. It is not; reset_index(drop=True) renumbers.
Using [0] after a filter. There may be no label 0. .iloc[0] is the position.
Forgetting drop=True.reset_index() keeps the old index as a new column, which then travels through every later operation as an unwanted index column.
Setting an index and then merging on the column. After set_index("id"), there is no id column to merge on — use left_index=True, or reset first.
Assuming alignment is positional. It is by label. Two Series with the same values in different label orders add to something neither of them looks like.
Ignoring duplicates. A lookup that returns a Series instead of a scalar breaks downstream code, and the break happens away from the lookup.
Reindexing
reindex conforms an object to a given set of labels: present labels are kept, absent ones are filled with NaN, and labels not in the new set are dropped.
That makes it the tool for two jobs.
Making frames comparable.b.reindex(a.index) puts b on a's labels, so a subsequent operation aligns exactly and any mismatch is visible as NaN rather than silently reordering.
Filling out a sparse axis. Reindexing onto a complete date range turns missing days into explicit NaN rows, which is what you want before plotting or before a rolling window over rows.
fill_value= sets what the gaps become, and method="ffill" carries values forward, which requires a sorted index.
reindex_like(other) is the shorthand for the first case.
The index in output
The index is written by to_csv unless you pass index=False, and read back as an unnamed column unless you pass index_col=0. That round-trip mismatch is the origin of the Unnamed: 0 column that appears in so many datasets.
to_dict(), to_json() and to_excel all have their own opinions about the index, and all of them are worth checking once for any output that another system consumes.
The rule that avoids most of it: if the index is meaningful data, name it and write it deliberately. If it is not, reset_index(drop=True) before writing.
A working summary
The index is a labelled axis, not row numbers.
It survives filtering, and alignment uses it silently.
set_index when you will select or join on the key repeatedly; reset_index(drop=True) when you will not.
Check is_unique before relying on lookups.
sort_index() after building or reshaping anything with a non-trivial index.
And when a result has the wrong number of rows or unexpected NaN, look at the indexes of the inputs first. That is the cause more often than anything in the operation itself.
Check yourself
0 of 4
Answer without scrolling back up.
After filtering a Series down to 3 of 5 rows, what is `result[0]` likely to do?
The surviving rows keep their original labels. Use .iloc[0] for position, or reset_index(drop=True) to renumber.
Two Series with partly overlapping labels are added. What happens to a label present in only one?
Alignment is a safety feature - a mismatch surfaces as NaN rather than a silent off-by-one. Use add(other, fill_value=0) to treat it as zero.
What does `s['a']` return when the label 'a' appears twice in the index?
The return type changes with the data, so code written against the scalar case breaks when a duplicate appears - usually in production.
What does `s.loc[100:200]` include?
Label slicing is inclusive of the endpoint, unlike every other Python slice. It works on any unique index; only a non-unique unsorted one raises.
Cheat sheet
The Index
If you do not supply one, pandas creates a RangeIndex of 0, 1, 2, .... That default is why the index is easy to overlook — it looks like row numbers, and for a freshly created frame it behaves like them.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.