Columns with names, types and an index - and why that is a different thing from a 2-D array.
Overview
Two structures
pandas gives you two objects: the Series and the DataFrame. A Series holds one column of data. A DataFrame holds many columns side by side. Each column keeps its own type. An age column stays integer while a name column stays string. That per-column typing is the single biggest difference from a NumPy array, where one type covers the whole block.
Real-world data almost always mixes types. A customer table has text names, integer ids, float balances and boolean flags. pandas treats each column on its own terms. You do arithmetic on the numbers and string cleaning on the text, in the same frame, without converting anything.
The price of that flexibility is alignment. pandas tracks labels on rows and columns. Those labels follow the data through every operation. Two Series add up by matching their labels, not their positions. That is a safety feature, but it surprises anyone used to positional arrays.
This module shows what a Series is, what a DataFrame is, and where the index enters the picture. The rest of the track builds on those three ideas. Grasping them here saves confusion in every later module.
The code examples are short and print their output. Change a value, rerun, and watch what moves. That is faster than reading a longer explanation.
Series is a one-dimensional array of values plus an index of labels.
DataFrame is a set of Series sharing one index. Each column has its own dtype.
That second point is the hands-on gap from NumPy. A NumPy array has one dtype for the whole block. A DataFrame has one per column. A table with a name, an age, a score and a flag is four dtypes. That is just the shape real data arrives in.
Convert such a DataFrame to a NumPy array and each column merges to the one type that can hold all of them. Often object. This Is a list of pointers with extra steps. That collapse is worth seeing once. It shows why pandas exists rather than all using arrays.
Worth knowing
A DataFrame is a set of columns, each with its own dtype, sharing one index — not a 2-D array.
The index is a labelled axis, not row numbers. It survives filtering, and that surprises people.
Arithmetic aligns on the index, by label rather than position. Missing labels become NaN instead of raising.
Columns keep their names through every operation, so you select by meaning rather than by position.
Converting a mixed DataFrame to a NumPy array collapses every column to one dtype, usually object.
apply is a Python loop. pandas is fast only while you stay out of per-row Python.
What pandas Is For
Columns with names, types and an index. Why that differs from a 2-D array.
A DataFrame is columns, not a grid
Each column has its own dtype. That is the difference from a NumPy array, and most of what follows comes from it.
example_01.pypandas
import pandas as pd
df = pd.DataFrame({
"name": ["ana", "raj", "kim"],
"age": [31, 25, 40],
"score": [8.5, 9.1, 7.2],
"active": [True, False, True],
})
print(df)
print()
print("one dtype per column:")
print(df.dtypes)
print()
print("a NumPy array has ONE dtype for everything. Forcing this to")
print("an array collapses it to the only type that holds all four:")
print(" ", df.to_numpy().dtype)
Output
The index is not row numbers
It looks like 0,1,2 by default, which hides that it is a real labelled axis.
example_02.pypandas
import pandas as pd
s = pd.Series([10, 20, 30])
print("default index:", list(s.index))
s2 = pd.Series([10, 20, 30], index=["a", "b", "c"])
print("labelled :", list(s2.index))
print()
print("s2['b'] ->", s2["b"], " looked up by LABEL, not position")
print()
print("and the label survives filtering:")
big = s2[s2 > 15]
print(big)
print(" the index is still b, c - not 0, 1")
print()
print("That surviving label is why the next module is about the index.")
Output
Operations align on the index
This is the single biggest difference from arrays, and it is silent.
example_03.pypandas
import pandas as pd
a = pd.Series([1, 2, 3], index=["x", "y", "z"])
b = pd.Series([10, 20, 30], index=["z", "y", "x"])
print("a:", dict(a))
print("b:", dict(b), "<- reversed order")
print()
print("a + b aligns by LABEL, not position:")
print(a + b)
print()
print("x: 1+30, y: 2+20, z: 3+10")
print()
print("NumPy would have added them position by position.")
print("Missing labels become NaN rather than an error:")
c = pd.Series([1, 1], index=["x", "q"])
print(dict(a + c))
Output
Columns carry names all the way through
You select by meaning rather than by remembering that salary is column 3.
example_04.pypandas
import pandas as pd
df = pd.DataFrame({
"city": ["pune", "delhi", "pune", "delhi"],
"sales": [10, 20, 30, 40],
})
print("total by city, in one line:")
print(df.groupby("city")["sales"].sum())
print()
print("the equivalent in plain Python needs a dict, a loop and a")
print("decision about what to do with a missing key.")
print()
print("column names survive every operation:")
print(list(df.groupby("city")["sales"].sum().index))
Output
Where pandas beats a dict of lists
Not everywhere. It wins when the data is tabular and the questions are column-shaped.
example_05.pypandas
import pandas as pd
rows = [
{"item": "pen", "qty": 3, "price": 10.0},
{"item": "book", "qty": 1, "price": 250.0},
{"item": "bag", "qty": 2, "price": 400.0},
]
df = pd.DataFrame(rows)
df["total"] = df["qty"] * df["price"]
print(df)
print()
print("revenue :", float(df["total"].sum()))
print("mean price :", round(float(df["price"].mean()), 2))
print("most expensive:", df.loc[df["price"].idxmax(), "item"])
print()
print("For three rows a loop is fine. The argument for pandas is that")
print("this code does not change when there are three million.")
Output
What pandas is not for
Knowing the boundary saves a lot of fighting with it.
example_06.pypandas
import pandas as pd
import time
n = 20_000
df = pd.DataFrame({"a": range(n)})
t = time.perf_counter()
out = df["a"] * 2 + 1
vec = time.perf_counter() - t
t = time.perf_counter()
out2 = df["a"].apply(lambda v: v * 2 + 1)
loop = time.perf_counter() - t
print("vectorised : %.4f s" % vec)
print("apply : %.4f s" % loop)
print("ratio : %.0fx" % (loop / max(vec, 1e-9)))
print("same :", out.equals(out2))
print()
print("apply is a Python loop wearing a method call. pandas is fast")
print("only while you stay out of per-row Python.")
Output
The index does work you did not ask for
The index looks like row numbers. By default it's 0, 1, 2. That default hides what it really is. A labelled axis that participates in almost each step.
Two results arrive early and confuse people.
It lives filtering. Filter a Series down to two of five rows and the index reads 1, 4, not 0, 1. The labels came along. Code that then indexes positionally gets the wrong rows or a KeyError.
Math aligns on it. Adding two Series matches them up by label, not by spot. If the labels are in other orders, pandas reorders for you. If one has a label the other lacks, the result is NaN there rather than an error.
That alignment is a genuine feature. It's what makes mixing data from other sources safe. Mismatches surface as NaN rather than as a silent off-by-one. It's also the single most surprising action for anyone arriving from NumPy. Here + is strictly spot-based.
The index gets a module of its own next. Nearly each pandas confusion traces back to it.
Names instead of positions
df["salary"] says what it selects. arr[:, 3] needs you to note what column 3 was. Breaks silently when a column is inserted.
That isn't a small ergonomic point. It's most of why pandas code lives contact with changing data. Why a group-by is one line rather than a dict, a loop and a choice about missing keys.
When to use something else
Plain Python for small, one-off, non-tabular work. A hundred records processed once doesn't justify the import. A list of dicts is easier to read.
NumPy when the data is truly homogeneous and numeric. A mat, an image, a signal. pandas adds per-column bookkeeping you aren't using. Costs both memory and speed for it.
A DB when the data doesn't fit in memory, or when a few processes need it at once. pandas assumes one process and one machine.
Polars or DuckDB when the data is large and the work is analytical. Both are a lot faster than pandas on big joins and aggregations.
pandas is the right tool for tabular, in-memory, mixed-type data where the questions are column-shaped. Which shows an huge amount of real work.
The one performance rule
pandas is a thin, convenient layer over compiled code. It's fast while the work happens inside that compiled layer. Slow the moment it has to call back into Python once per row.
apply, iterrows and a for loop over rows all do just that. The last editor measures it. The gap is large enough that it's often the whole speed story of a pandas script.
The rule: express steps on whole columns. df["a"] * 2 operates on the column. df["a"].apply(lambda v: v * 2) operates on each value from Python and is far slower for the same result.
There are cases where apply is unavoidable. A module later covers them. Most uses of it aren't those cases.
How this track is organised
The index and pick come first. They cause the most confusion and all else depends on them. Including the copy warning. This Gets a module to itself because nothing else in pandas wastes as much of people's time.
Then cleaning: missing values, copies, strings, types.
Then the aggregation work that is the reason to use pandas at all: group-by, joins, reshaping.
Then time series, reading and writing files, and speed.
each module is short runnable programs rather than one long script. You can change one line and see which rule made which output.
Where pandas sits
pandas is built on NumPy. Underneath each numeric column is a NumPy array. The math that runs on a column is the same compiled code.
What pandas adds is bookkeeping: labels for the rows, names and split types for the columns. A large library of steps that assume your data is a table rather than a mat.
That layering shows both its strengths and its costs. Column math is fast because NumPy is doing it. Anything that has to consult labels, merge dtypes, or fall back to Python objects is slower. That is where pandas code goes wrong.
Above pandas sit the tools that consume it. Scikit-learn takes DataFrames, matplotlib and seaborn plot them. Most data pipelines pass them around. Knowing the layer below and the layer above tells you when to drop down to NumPy for speed and when to hand off fully.
The vocabulary
A few terms recur constantly, and being precise about them blocks confusion later.
Series — one column: values plus an index, with a single dtype.
DataFrame — a few Series sharing one index.
Index — the labels for the rows. The column names are also an Index.
dtype — the type of one column. object often means Python strings.
Axis — axis=0 is rows, axis=1 is columns. As in NumPy, the axis you name is often the one being merged or moved along. This Is why df.sum(axis=0) gives a total per column.
NaN / NaT / pd.NA. Missing markers for floats, datetimes and the nullable types.
Reading pandas output
Printed output carries more data than it appears to. Reading it saves a great many print statements.
The dtype line under a Series tells you what you're holding. int64 and float64 are numeric. object almost always means strings. datetime64[ns] means dates have been parsed.
The index is the left-hand column. It isn't row numbers. If it reads 0, 3, 7 rather than 0, 1, 2, the frame has been filtered and the first labels came along.
Name: under a Series gives the column it came from.
[5 rows x 3 columns] at the bottom of a truncated frame is the real shape. This Matters when the display has elided the middle.
Three things worth setting in a notebook: pd.set_option("display.max_columns", None) to stop columns being hidden, display.width to control wrapping. display.float_format to stop long decimals dominating a table.
A first session
The shape of almost each piece of pandas work is the same:
Load with read_csv, saying what you know about types and dates.
Look — shape, dtypes, head, info, show, and value_counts on the cat columns.
Clean — fix types, handle missing values, norm text, drop or aggregate copies.
Reshape — filter to what matters, join in what is missing, group and aggregate.
Output — a table, a chart, or a file.
Roughly half of real work is in the middle two steps. This Is why this track spends most of its modules there rather than on the aggregation all thinks of as the notable part.
Two habits worth starting with
Check after each step that changes the shape.len(df) after a filter or a merge, df.dtypes after a load or a concat. Most pandas bugs are silent. The ones that aren't silent are often caught by one of those two lines.
Prefer clear over clever.df.loc[mask, "col"] over df[mask]["col"], named aggregation over spot-based, .copy() where you mean a copy. The clear form is nearly always the one that keeps working when the data changes.
Questions that come up first
Do I need to learn NumPy before pandas?
Not before, but alongside. pandas hides NumPy most of the time. Then leaks it at just the moments that matter. Dtypes, NaN action, broadcasting, views versus copies. each one of those is a NumPy concept wearing a pandas name. The modules that cause the most trouble here are the ones where the NumPy layer shows through.
**Why is my column object?**
Because it holds Python objects rather than a uniform numeric type. Almost always strings, at times mixed types from a messy source. It's the single most common cause of both slow code and surprising results. df.dtypes is how you find it.
Why did my ints become floats?
A missing value. NumPy int arrays can't hold NaN. Pandas promotes the column. The nullable Int64 type is the fix.
Why does my filter return an empty frame?
often a type mismatch — checking a string column against a number — or whitespace and case differences in the values. value_counts() on the column shows both at once.
Should I use pandas or SQL?
If the data lives in a DB and the step is a filter, join or aggregate, do it in SQL and bring back less data. pandas is for what happens after that. For data that never was in a DB.
What this track assumes
That you can read Python, and that you've met lists and dicts. No statistics, no NumPy, and no prior pandas.
The modules are sorted by dependency rather than by glamour. The index, pick and the copy warning come first because all later depends on them and because they cause the most confusion. Cleaning comes next. That is where most real time goes. Group-by, joins and reshaping come after, because they're what people think pandas is for and they only work properly once the earlier material is in place.
Each module is six short programs and an article. The programs are the point. Changing a value and re-running is the fastest way to find out what a rule really does. A few of them are written so the output clashes with the guess most people would make.
The shortest useful summary
A DataFrame is columns with names and types, sharing an index.
The index takes part in almost all, including alignment, which is silent.
Pick is .loc for labels and .iloc for spots.
set goes in one .loc call, or through an clear .copy().
steps on whole columns are fast. Anything per row isn't.
And when a result surprises you, the answer is nearly always in df.dtypes, df.shape, or the index — in that order.
Questions people ask
Is pandas built on NumPy? Yes. Each column is backed by a NumPy array (or an Arrow array in newer setups), and pandas adds labels, mixed dtypes and alignment on top.
When should I use pandas instead of plain Python lists or dicts? Use pandas when your data is tabular and you want to filter, group, join or summarise whole columns at once. For a few dozen values or deeply nested records, plain Python is simpler.
What is the difference between a Series and a DataFrame? A Series is one labelled column of values with a single dtype. A DataFrame is a table of Series that share one index.
Can pandas handle data larger than my memory? Not directly, because a frame is loaded into RAM. You can read in chunks, load fewer columns, use smaller dtypes, or move to a tool such as Polars, DuckDB or Dask.
Why does pandas feel different from Excel? Operations apply to whole columns by label rather than to cells you click, and every step is code you can rerun. That makes results repeatable but means you have to think about dtypes and the index.
Do I need to learn NumPy before pandas? No. A little NumPy helps with arrays and NaN, but you can start with pandas and pick up the NumPy parts as they appear.
A closing note
pandas is a large library, and the temptation is to learn it as a list of methods. That doesn't work well. The methods aren't the hard part.
The hard part is a small number of actions that run through all. The index takes part in steps you didn't ask it to. A column has one dtype and that dtype decides what is possible. Pick returns a thing whose link to the first isn't always given. And anything that runs Python once per row is slow enough to dominate.
Those four explain most of what surprises people, and each has a module here.
The methods, by contrast, are searchable. No one remembers the arg order of merge or the exact spelling of each rate alias, and no one needs to.
So the handy thing to take from this track isn't coverage but a set of instincts. Check dtypes after loading, check row counts after joining, know if you are holding a copy. Stay out of per-row Python.
Check yourself
0 of 4
Answer without scrolling back up.
What is the main structural difference between a DataFrame and a 2-D NumPy array?
Converting a mixed DataFrame to an array collapses every column to the one type that holds them all, usually object - which is why pandas exists.
Two Series with the same labels in different orders are added. What happens?
Alignment is by label, not position, and missing labels become NaN rather than raising. This is the biggest departure from NumPy.
After filtering a Series down to 2 of 5 rows, what does the index look like?
The labels come along. Code that then indexes positionally gets the wrong rows or a KeyError.
Why is `df['a'].apply(lambda v: v*2)` slower than `df['a'] * 2`?
pandas is fast while work stays inside the compiled layer. Per-row Python is usually the entire performance story of a slow pandas script.
Cheat sheet
What pandas Is For
Learn what pandas adds to NumPy: named columns, typed data, a row index and the operations that make tabular analysis fast.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.