vizlearn

Text Columns

pandasBy Updated

The .str accessor - vectorised string work without a loop, and what it does with missing values.

Overview

Why the accessor exists

A Series of strings is a column of Python string objects. You could loop over them, and apply(str.strip) would work, but both are slow and neither reads well.

.str exposes the string methods so they apply to the whole column: s.str.strip(), s.str.lower(), s.str.len().

Each returns a Series, so they chain: s.str.strip().str.lower().str.replace(" ", "_").

The accessor is required. s.strip() is not the same thing — it either does something unrelated to the Series or fails, because strip is not a Series method. Forgetting .str is the most common beginner error here, and the resulting AttributeError at least says so clearly.

Worth knowing

.str applies a string method to every value and returns a Series, so the calls chain.
Missing values pass through as NaN rather than raising — and a mask containing NaN cannot be used to index, so pass na=False.
contains takes a regex by default; pass regex=False for a literal. match anchors at the start, fullmatch at both ends.
split(expand=True) makes columns; extract with named capture groups makes one column per group.
str.replace needs an explicit regex= in pandas 2 — the old default changed.
.str is convenient, not fast: object columns are per-element Python, and text work is far slower than arithmetic.

Text Columns

The .str accessor, and what it does with missing values.

The .str accessor applies to every value

The same methods you know from Python, one call for the whole column.

example_01.pypandas
Output

                

                

                    
Missing values pass through

                    
.str returns NaN rather than raising, which is usually what you want and occasionally hides a problem.
example_02.pypandas
Output
.str methods return NaN for missing input rather than raising or returning an empty string.
That is usually right: the uppercase of an unknown value is unknown.
It becomes a problem for predicates. s.str.contains("a") returns True, False or NaN, and a mask containing NaN cannot be used to index a frame — pandas raises "Cannot mask with non-boolean array containing NA / NaN values".
The fix is na=False, which decides what a missing value should count as:
df[df["name"].str.contains("ana", na=False)]
Pass it every time you filter on a text predicate. na=True is occasionally what you want, when missing should be included.
This is the single most common runtime error in text cleaning, and it appears only when the data has a gap — so it usually shows up in production rather than on the sample.

                

                

                    
Testing and finding

                    
contains, startswith and match, and which of them takes a regex.
example_03.pypandas
Output

                

                

                    
Splitting and extracting

                    
split with expand, and extract for the parts a regex names.
example_04.pypandas
Output
s.str.split("-") gives a Series of lists. That is rarely what you want directly.
expand=True turns it into a DataFrame with one column per piece, which is how you split a combined field into real columns. Rows with fewer pieces get None in the trailing columns.
s.str.split("-").str[0] takes one piece — the second .str indexes into the lists.
extract is usually better when the structure is known. It takes a regex with capture groups and returns one column per group, and named groups become column names:
s.str.extract(r"(?P<kind>[a-z]+)-(?P<num>\d+)")
Rows that do not match give NaN across the row, which makes non-matching input visible rather than silently mangled.
extract returns text, even for digits. Convert afterwards with pd.to_numeric if you need numbers.
extractall returns every match rather than the first, with a MultiIndex.

                

                

                    
Replacing, with and without regex

                    
The default changed in pandas 2, so be explicit.
example_05.pypandas
Output

                

                

                    
It is convenient, not fast

                    
Under the hood this is still per-element Python.
example_06.pypandas
Output

                

                

                    
Testing

                    
contains searches anywhere in the string. startswith and endswith anchor. match anchors at the start, fullmatch at both ends.
contains takes a regular expression by default. That matters more than it sounds, because it means s.str.contains(".") matches every non-empty string rather than finding a literal dot, and s.str.contains("a|b") is an alternation rather than a search for the three characters.
Pass regex=False for a literal search. It is also faster.
case=False makes the test case-insensitive without a separate .lower() pass.
startswith and endswith do not take a regex, which is an inconsistency worth remembering rather than deriving.

                

                

                    
Replacing

                    
s.str.replace(old, new, regex=...) requires the regex argument to be explicit in pandas 2. The default changed, and code written against pandas 1 that relied on the old behaviour can silently do the wrong thing.
Two clean-ups earn their place in almost every script:
Collapsing whitespace: s.str.replace(r"\s+", " ", regex=True).str.strip().
Normalising column names:
df.columns = (df.columns.str.strip().str.lower()
              .str.replace(r"[^a-z0-9]+", "_", regex=True).str.strip("_"))
That turns "  First Name " and "AGE (years)" into first_name and age_years, and it handles a whole spreadsheet's worth of inconsistent headers without listing them.
Note df.columns.str works too — the column index is an Index, and Index has a .str accessor as well.

                

                

                    
The cost

                    
.str is a convenience, not a vectorisation.
Object columns hold pointers to Python strings, so every operation walks them one at a time in Python. The last editor shows text work running far slower than arithmetic on the same number of rows.
Three things help.
Do it once. Clean text at load time rather than repeatedly inside a loop or a function called per group.
Convert to category when values repeat. Operations then run on the small set of distinct values rather than every row.
Consider the string dtype. astype("string") gives the nullable extension type, which has clearer missing-value semantics than object and is where pandas' future optimisation work is going.
For very large text columns, the honest answer is that pandas is not the right tool, and the work belongs in a database or a purpose-built library.

                

                

                    
The cleaning pipeline

                    
Text from the real world needs the same handful of operations almost every time, and doing them in one pass at load time is far better than scattering them through the analysis:
df["city"] = (df["city"]
              .str.strip()
              .str.lower()
              .str.replace(r"\s+", " ", regex=True))
Strip the ends, normalise case, collapse internal whitespace. That alone resolves most of the "same value counted twice" problems that show up in a group-by.
For text that will be compared or joined on, consider also removing punctuation and accents. unicodedata.normalize handles accents; pandas has no built-in for it, and .str.normalize("NFKD") plus an ASCII encode is the usual idiom.

                

                

                    
Extracting structure

                    
str.extract with named groups is the most useful of the extraction methods, because it names the outputs:
df[["kind", "num"]] = df["ref"].str.extract(r"(?P<kind>[a-z]+)-(?P<num>\d+)")
Rows that do not match give NaN across the row, so counting them afterwards tells you how well the pattern fits:
unmatched = df["kind"].isna().sum()
That check is worth making. A regex that matches 90% of rows silently discards the other 10%, and the only sign is a column with gaps.
str.extractall returns every match rather than the first, with a MultiIndex that has a match level.
str.findall returns a list per row, which is harder to work with and occasionally what you want.

                

                

                    
Testing and counting

                    
str.contains — anywhere, regex by default, needs na=False for filtering.
str.startswith / str.endswith — anchored, no regex.
str.match — anchored at the start, regex.
str.fullmatch — anchored at both ends.
str.count(pattern) — how many times a pattern occurs per row.
str.len — length, which is a quick way to spot truncated or padded fields.
A frequent need is "does this contain any of these words":
df["text"].str.contains("|".join(words), case=False, na=False)
Building an alternation is fine for a handful of words. For a long list, and for whole-word matching, it is worth escaping the parts with re.escape and adding word boundaries, or the results will surprise you.

                

                

                    
Splitting into columns

                    
str.split(sep, expand=True) gives a DataFrame. Rows with fewer parts get None in the trailing columns, and rows with more parts are truncated unless you pass n to limit the splits.
n=1 splits only on the first separator, which is what you want for key: value text where the value may contain the separator.
str.rsplit splits from the right, which handles "everything before the last dot" cleanly.
str.partition returns exactly three columns — before, separator, after — which avoids the variable-width problem entirely.

                

                

                    
Categorical text

                    
Once text is cleaned, if it repeats, convert it:
df["city"] = df["city"].astype("category")
The .str accessor still works on a categorical column, and pandas applies the operation to the categories rather than to every row — so df["city"].str.upper() on a million rows with four cities does four operations, not a million.
That is the single largest speed-up available for repeated text work, and it is a one-line change.

                

                

                    
When to stop using pandas for text

                    
pandas is a reasonable place to clean and extract from moderate amounts of text. It is not a text-processing engine.
For tokenisation, stemming, language detection or anything linguistic, a dedicated library is the right tool.
For very large corpora, the object dtype's per-row Python cost dominates, and the work belongs in a database, in Polars, or in a purpose-built pipeline.
The signal is usually the profile: if .str operations are the slowest part of your script and the frame is large, the answer is a different tool rather than a cleverer regex.

                

                

                    
Regex, briefly

                    
Several .str methods take a regular expression, and a small vocabulary covers most data cleaning:
\d digit, \w word character, \s whitespace, . any character.
+ one or more, * zero or more, ? optional.
^ start, $ end,  word boundary.
[abc] a character set, [^abc] its negation.
(...) a capture group, (?P<name>...) a named one.
| alternation.
Two habits prevent most regex trouble in pandas. Use raw strings — r"\d+" — so backslashes reach the regex engine intact. And test the pattern on a handful of values before applying it to the column, because a pattern that matches nothing produces a column of NaN rather than an error.
str.contains(..., regex=False) and str.replace(..., regex=False) are both faster and safer when the pattern is a literal.

                

                

                    
Encoding problems

                    
Text that arrives mangled — é where é belongs — was decoded with the wrong encoding, usually at read time.
The fix is at read_csv, with encoding="latin-1" or encoding="cp1252", not in pandas afterwards. Repairing mojibake after the fact is possible and unreliable.
df["col"].str.encode("utf-8").str.decode("utf-8") round-trips text and raises on anything invalid, which is a way to find the offending rows.
For matching across accented and unaccented spellings, normalising with str.normalize("NFKD") and stripping combining characters puts both forms into the same shape.

                

                

                    
Performance, restated

                    
Three things make text work faster, in order of effect:
Convert to category when values repeat. Operations then apply to the categories, not the rows.
Do the cleaning once, at load time, rather than inside a function called per group or per row.
Use regex=False where the pattern is a literal.
And the structural point: if .str operations dominate the profile on a large frame, pandas is the wrong layer for that work. A database, Polars, or a purpose-built text pipeline will do it in a fraction of the time.

                

                

                    
A summary

                    
.str applies string methods elementwise and chains.
Missing values pass through as NaN; pass na=False on any predicate used for filtering.
contains is a regex by default; startswith is not a regex at all.
split(expand=True) makes columns; extract with named groups is usually clearer.
replace needs an explicit regex= in pandas 2.
Normalise column names and text values once, at load.
Convert repeated text to category.
And check how many rows a pattern actually matched, rather than assuming it matched them all.

                

                

                    
A closing note

                    
Text is where pandas is least like an array library and most like ordinary Python, and both facts show.
.str gives you the string methods across a whole column, which is convenient and reads well. Underneath it is per-element Python over a column of pointers, which is why text work is an order of magnitude slower than arithmetic on the same number of rows.
Two things follow. Clean text once, at load, rather than repeatedly in the middle of a pipeline. And convert repeated values to category, after which .str operations apply to the handful of distinct values rather than to every row — usually the largest single speed-up available, for a one-line change.
The correctness trap is missing values. .str predicates return NaN rather than False, and a mask containing NaN cannot index a frame. na=False on every .str filter is the habit, and its absence is the error that appears only once the data has a gap.

                

                

                    
One more thing

                    
str.get(i) indexes into each string or list, which is the concise form of str[i] and works the same way. It returns NaN rather than raising where the index is out of range, which is usually what you want on ragged data.

                

            


        

    


    



Check yourself
0 of 4
Answer without scrolling back up.
  1. Why does `df[df['name'].str.contains('a')]` sometimes raise?
  2. What does `s.str.contains('.')` match?
  3. What does `str.extract` return for a row that does not match?
  4. Why is `.str` slower than arithmetic on the same number of rows?





 Cheat sheet
Text Columns
A Series of strings is a column of Python string objects. You could loop over them, and apply(str.strip) would work, but both are slow and neither reads well.
PANDAS · vizlearn.in/pandas/string_methods.html
More in pandas
pandasDates and TimespandasSorting and Rankingpandasgroupbypandastransform and filter
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact