The six calls to make before writing any analysis, and what each one is actually telling you.
Overview
Why this comes before anything else
Every analysis starts with assumptions about the data. The column names, the types, the number of rows, whether values are missing, whether strings are clean. Those assumptions are almost never written down, and they are frequently wrong.
A column you expected to hold integers might hold strings because of one malformed row. A date column might be plain text. A categorical column might have four spellings of the same city. An id column might have lost its leading zeros during import. None of these problems raise an error. They produce plausible output that is quietly incorrect.
The fix is a short inspection routine you run before writing any logic. Six calls cover almost everything: shape, dtypes, head, describe, value_counts and isna. Together they take under a minute. They surface the problems that would otherwise cost hours of debugging later.
Skipping inspection is the single most expensive habit in data work. Not because each bug is hard to fix, but because the bug is invisible until something downstream gives an answer that looks wrong. By then you have built logic on top of a broken assumption, and finding the root cause means unwinding all of it.
This module walks through the six calls, explains what each one catches, and shows examples of the problems they reveal. After this, running the routine should be automatic. It is the cheapest investment in any pandas workflow.
Making inspection a habit also helps when you revisit code months later. The printed output in your notebook tells you what the data looked like, even if the file has changed since.
The calls are cheap individually and cheaper together. None of them modify the frame. None of them require understanding the domain. You can run the full routine before knowing what question the analysis will answer. That is the point: the routine catches problems that are independent of the question.
Most people learn this the hard way. A report goes out, a number looks wrong, and the root cause is a dtype that was never checked. Running the routine once would have caught it. That experience, repeated a few times, is what makes inspection automatic.
Most pandas bugs aren't bugs in pandas. They're assumptions about the data that were never checked. A column that is text where you expected numbers, an identifier that lost its leading zeros, a group with four spellings, a date column that is still strings.
None of these raise. They give plausible output that is wrong. The error surfaces much later, one spot unrelated.
Six calls catch nearly all of it, and together they take under a minute.
Worth knowing
df.shape and df.dtypes are the first two lines to run — they catch lost leading zeros, unparsed dates and columns turned float by one missing value.
head hides sorted or dated data. sample shows you the middle.
df.info() gives dtypes, non-null counts and memory in one call — the most useful thing to run on an unfamiliar frame.
describe covers numeric columns only unless you pass include="all". A mean far from the median means outliers.
df.isna().sum() counts missing per column; .mean() gives the share.
value_counts on text columns finds typos, stray capitals and trailing spaces before they become a broken group-by.
Looking at Data
The six calls to make before writing any analysis.
shape and dtypes first
Two lines that catch most of what goes wrong at the boundary.
example_01.pypandas
import pandas as pd
import io
csv = """id,name,age,joined
001,ana,31,2021-04-05
002,raj,,2022-01-10
003,kim,40,2020-11-30
"""
df = pd.read_csv(io.StringIO(csv))
print("shape :", df.shape, "(rows, columns)")
print()
print(df.dtypes)
print()
print("Three things to notice, none of which raised:")
print(" id -> int64, so the leading zeros are gone")
print(" age -> float64, because one value is missing")
print(" joined -> object, because dates are not parsed by default")
Output
head, tail and sample
Look at the data. The first rows are often not representative.
example_02.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"day": range(1, 21),
"sales": list(range(10, 30)),
})
print("head(3):"); print(df.head(3))
print()
print("tail(3):"); print(df.tail(3))
print()
print("sample is the one that shows you the middle:")
print(df.sample(3, random_state=0))
print()
print("head alone hides sorted data, dated data and anything that")
print("changes partway through the file.")
Output
info gives dtypes, nulls and memory together
The single most useful call on an unfamiliar frame.
example_03.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"a": [1, 2, 3, 4],
"b": [1.5, np.nan, 3.5, np.nan],
"c": ["x", "y", "x", None],
})
df.info()
print()
print("Read the Non-Null Count column against the row count:")
print(" b has 2 of 4, c has 3 of 4.")
Output
describe, and what it leaves out
Numeric columns only, unless you ask otherwise.
example_04.pypandas
import pandas as pd
df = pd.DataFrame({
"score": [10, 20, 30, 40, 1000],
"grade": ["a", "b", "a", "c", "a"],
})
print(df.describe())
print()
print("the text column is missing. include='all' brings it in:")
print(df.describe(include="all"))
print()
print("Note the mean of score is 220 - one outlier moved it past every")
print("value but one. Compare mean with the 50% row to spot that.")
Output
Counting missing values
Per column, as a proportion, and which rows are affected.
example_05.pypandas
import pandas as pd
import numpy as np
df = pd.DataFrame({
"a": [1, np.nan, 3, np.nan],
"b": [1, 2, 3, 4],
"c": [np.nan, np.nan, np.nan, 4.0],
})
print("missing per column:")
print(df.isna().sum())
print()
print("as a share of rows:")
print((df.isna().mean() * 100).round(1))
print()
print("rows with any missing value:", int(df.isna().any(axis=1).sum()), "of", len(df))
print("rows that are complete :", int(df.notna().all(axis=1).sum()))
Output
value_counts for the categorical columns
The fastest way to find typos, stray categories and unexpected cardinality.
example_06.pypandas
import pandas as pd
s = pd.Series(["pune", "delhi", "pune", "Pune", "delhi", "pune ", None])
print(s.value_counts())
print()
print("four spellings of two cities - trailing space and a capital.")
print()
print("dropna=False shows the missing values too:")
print(s.value_counts(dropna=False))
print()
print("normalize gives shares:")
print((s.value_counts(normalize=True) * 100).round(1))
print()
print("nunique:", s.nunique(), " (excludes NaN by default)")
Output
shape and dtypes
df.shape gives (rows, columns). It confirms the file loaded as expected. Catches the case where a delimiter was misread and all landed in one column.
df.dtypes is the key one. Three failures show up here and nowhere else:
An identifier read as an int.001 turns 1. The leading zeros are gone permanently. Fix at read time with dtype={"id": str}.
An int column turned float because one value is missing. NumPy ints can't hold NaN. Pandas promotes. The symptom downstream is ids printing as 1001.0.
**A numeric column read as object** because one row holds a stray non-numeric value. each later numeric step then either fails or silently operates on strings.
Running these two lines at once after loading is the highest-value habit in this module.
head, tail, sample
Look at the data. Not the summary — the actual rows.
head() shows the first five. That is fine for a random file and misleading for a sorted one, a dated one, or one where the format changes partway through.
tail() catches trailing junk: summary rows, blank lines, a footer the exporter added.
sample(n, random_state=0) shows the middle. This Is where the surprises often live. Passing random_state makes it reproducible. This Matters if you're going to discuss what you saw.
info
df.info() prints dtypes, non-null counts and memory usage together.
The non-null count is the part to read carefully. Compare it against the row count for each column. A column with 400 non-null out of 10,000 rows is in effect empty and any analysis using it's built on 4% of the data.
memory_usage="deep" gives the true cost of object columns. This The default underestimates badly because it counts the pointers rather than the strings they point at.
describe
df.show() gives count, mean, standard deviation, min, max and quartiles for numeric columns only.
include="all" adds text columns, reporting count, unique, top and rate for them instead.
The most handy reading of it's the mean against the median (the 50% row). When they're far apart, the spread is skewed or holds outliers. Each mean-based summary you were about to find will mislead. A mean of 220 with a 50% of 30 is telling you a thing key before you've written any analysis.
min and max are also worth a glance for hopeless values. A minus age, a date in 1900, a price of zero.
Missing values
df.isna().sum() counts them per column. df.isna().mean() gives the proportion. This Is easier to judge than a raw count.
df.isna().any(axis=1).sum() counts rows affected by any missing value. The number you'd lose to dropna(). Often much larger than any single column's count.
Doing this before deciding how to handle missing data means the choice is informed rather than reflexive. Dropping 2% of rows is other from dropping 60%.
value_counts
For each text or cat column, run value_counts().
It finds, in one line: typos, inconsistent capitalisation, trailing whitespace, surprise groups. Cardinality far higher than expected (often a sign the column is really an identifier).
The last editor shows two cities appearing as four values. pune, Pune. pune with a trailing space. A group-by on that column would give four groups, and nothing would warn you.
dropna=False includes missing values in the count. This Is else invisible here.
norm=True gives shares rather than counts. This Is the right form when checking spreads between datasets.
It isn't glamorous and it isn't optional. each one of these calls has a exact failure it catches. The option is finding that failure later, in a result you've already shown someone.
Making the display readable
Default display settings hide things, and hidden data is data you won't check.
pd.set_option("display.max_columns", None) stops columns being elided into .... This Is the single most handy setting on a wide frame.
pd.set_option("display.max_rows", 100) shows more before truncating.
pd.set_option("display.float_format", "{:.2f}".format) stops fifteen decimal places dominating each table.
These affect display only, never the data. In a notebook they're worth putting in the first cell.
Reading info carefully
df.info() packs four things into one output, and each deserves a look.
The row count in the header, against what you expected from the source.
Non-Null Count per column, against that row count. This is the fastest way to find a column that is mostly empty.
Dtype per column. object on a thing that should be numeric is a problem. float64 on a thing that should be int often means a missing value.
Memory usage, which needs memory_usage="deep" to be truthful about text columns.
A column with, say, 400 non-null out of 200,000 rows is in effect empty. Any analysis using it's built on 0.2% of the data. It's better to know that before building the analysis than after presenting it.
describe, beyond the defaults
percentiles=[0.01, 0.5, 0.99] changes which quantiles are shown. The 1st and 99th are often more helpful than the quartiles for spotting outliers.
include="all" adds non-numeric columns, reporting count, unique, top and freq for them.
include=["object"] or exclude=["number"] restrict it.
df.show().T transposes the result. This Is much easier to read when there are many columns. One row per column instead of one column per column.
Three readings worth making a habit:
mean against 50% — far apart means skew or outliers.
min and max — look for hopeless values: a minus age, a zero price, a date in 1900.
std of zero — a constant column. This Carries no data and is often a sign a thing upstream went wrong.
Looking at the missing rows, not just counting them
Counting missing values tells you how much. Looking at *which* rows are missing tells you why.
df[df["income"].isna()].head()
If those rows share a thing. The same source, the same date range, the same group — then the data isn't missing at random. Both dropping and imputing will bias the result.
df.isna().sum(axis=1).value_counts() shows how many rows have 0, 1, 2... missing values. A long tail means a few badly broken rows. A big spike at one value often means an entire column is absent for a subset of the data.
Comparing two frames
When a thing changes and you want to know what, three tools:
df.equals(other) — exact, including dtypes.
df.compare(other) — shows only the cells that differ, side by side. It needs same shape and labels. This Makes it right for before-and-after checks on the same data.
set(a.columns) ^ set(b.columns) — the even gap of column names. This Is often where the discrepancy is.
The routine, as a block
pd.set_option("display.max_columns", None)
df = pd.read_csv(path, nrows=1000) # a sample first
df.shape
df.dtypes
df.head()
df.sample(5, random_state=0)
df.info(memory_usage="deep")
df.describe(include="all").T
df.isna().mean().sort_values(ascending=False)
for c in df.select_dtypes("object"):
print(df[c].value_counts(dropna=False).head())
That is a minute of work and it answers most of the questions you'd else discover the hard way. What types things are. Here The gaps are. This Groups are messy. Whether the file is what you were told it was.
Profiling a frame quickly
For a fast total picture beyond show, three one-liners cover most of it:
df.nunique().sort_values() # cardinality per column
df.isna().mean().sort_values(ascending=False) # missingness per column
df.memory_usage(deep=True).sort_values(ascending=False)
Cardinality is the most under-used of the three. A column with one distinct value carries no data. A column with as many distinct values as rows is an identifier, not a feature. all notable is in between. The ranking tells you which columns are which without opening any of them.
Checking assumptions explicitly
Looking at data is better than not looking. Asserting is better than looking. An assertion keeps checking after you stop paying focus.
Each of these codes a thing you believe about the data. When the belief stops being true. A new export, a changed upstream system. The script stops rather than making a quietly wrong answer.
That is worth more than any amount of exploratory looking. The looking happens once and the assertions happen each run.
The shape of a first pass
The order matters, because each step informs the next.
Shape and dtypes — is this the file I think it's?
head, tail, sample — does the data look like data?
info — where are the gaps and how big is this?
show — are there hopeless or extreme values?
value_counts on cats — are the groups what I expect, and how many are there?
Missingness pattern — are the gaps random or structured?
Only then is it worth writing any analysis. each one of these steps has caught, for someone, a problem that would else have surfaced in a result presented to someone else.
Two habits that pay repeatedly
Look at the rows you're about to drop. Before dropna, before a filter, before deduplication. df[mask].head(). If the rows you're discarding share a pattern, you aren't dropping noise, you're dropping a group.
Compare counts before and after each step that can change them. A filter, a merge, a group-by, a concat. One number, checked, catches silent row product and silent row loss. This Between them account for a large share of wrong answers in data work.
A closing note on Looking at Data
each one of these calls exists because someone shipped a wrong answer that one of them would have caught.
The routine is short enough to run without thinking about it. The discipline is to run it before writing any analysis rather than after a result looks odd. By the time a number looks wrong, the cheap reasons have been ruled out and the costly debugging has started.
The single highest-value line is df.dtypes. It catches the identifier read as an int, the date left as text. The numeric column turned into strings by one stray value. Three failures that give plausible output and no error.
The second is value_counts on the cat columns. This Finds the variants that would silently split a group-by.
Neither takes more than a few seconds. Together they answer most of what you need to know about a file you haven't seen before.
One more thing
df.head() and df.tail() can be mixed into one view with pd.concat([df.head(3), df.tail(3)]). This Shows both ends of a sorted frame at once. handy when the notable rows are the extremes.
And df.sample(frac=1) shuffles the whole frame. This Is at times the honest way to look at data that arrived in a real order. The first rows of a sorted file aren't representative of anything.
Questions people ask about Looking at Data
What should I run first on a new dataset? Check shape, then info() for dtypes and non-null counts, then head() to see real rows. After that, describe() and isna().sum() give the spread and the gaps.
What does df.info() tell me that df.describe() does not? info() reports each column's dtype, non-null count and memory use. describe() summarises the values of numeric columns, so it will not show dtype problems such as numbers stored as text.
How do I see summary statistics for text columns? Use df.describe(include='object') to get count, unique, top and frequency, or call value_counts() on a single column.
Why does head() sometimes mislead me? The first rows are often sorted or recorded under different conditions from the rest. Use sample() to see random rows.
How can I tell how many values are missing in each column? df.isna().sum() gives the count per column, and df.isna().mean() gives the share.
What does the object dtype in info() mean? It usually means strings, but it can hide mixed types, such as numbers with stray text. Check with df[col].map(type).value_counts().
Looking at Data, in summary
Six calls, run before any analysis, catch most of what goes wrong: shape, dtypes, head/sample, info, show. value_counts on the cat columns.
dtypes is the one that earns its place. It reveals the identifier read as a number, the date still stored as text. The numeric column turned into strings by a single stray value. Three failures that give plausible output and no error at all.
And an assertion is worth more than a look. The look happens once and the assertion happens each run.
Check yourself
0 of 4
Answer without scrolling back up.
Why check `df.dtypes` immediately after loading?
None of those failures raise. They produce plausible output that is wrong, and surface much later somewhere unrelated.
Why is `head()` alone not enough?
tail() also catches trailing junk like summary rows or a footer the exporter added.
What does a mean far from the median tell you in `describe()`?
A mean of 220 with a 50% of 30 says something important before you write any analysis.
What does `value_counts()` on a text column typically catch?
'pune', 'Pune' and 'pune ' are three groups, and nothing warns you. Pass dropna=False to see missing values too.
Cheat sheet
Looking at Data
Run shape, dtypes, describe, info, head and value_counts on a new dataset. Build the six-call habit that catches problems before analysis.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.