vizlearn

Text Columns

pandasBy Updated

The .str accessor - vectorised string work without a loop, and what it does with missing values.

Overview

Why the accessor exists

Text data in pandas is stored as a column of Python string objects. Each cell is an independent object sitting somewhere in memory, connected to the column by a pointer. That storage model is why string columns are slow and memory-heavy compared to numeric columns.

You could process strings with a Python loop or with apply. Both work. Both are slow, because they call Python once per cell. The str accessor provides vectorised string methods that push the loop into compiled code where possible, and that return a properly aligned Series with NaN for missing entries.

Beyond speed, the accessor gives you a readable, chainable syntax. Cleaning a name column might take three calls: strip whitespace, convert to lowercase, replace internal spaces with underscores. With the accessor, that is one chained expression. Without it, that is three separate apply calls or a lambda that does all three steps and is harder to read.

The accessor also handles missing values consistently. Every str method returns NaN for a missing input rather than raising or returning an empty string. That means you can chain string operations on a column with gaps and get NaN in the result where the input was NaN. No special handling required.

String cleaning is a large part of real-world data work. Names arrive in mixed case, with extra whitespace, with inconsistent separators. Codes arrive with trailing spaces that make exact matches fail. Categories have multiple spellings. The str accessor is how you fix all of these at column scale.

This module covers the most-used methods: strip, lower, upper, contains, replace, extract, split and len. It also covers when to convert a text column to the category dtype for better performance on repeated values.

A Series of strings is a column of Python string objects. You could loop over them. apply(str.strip) would work. Both are slow and neither reads well.

.str exposes the string methods so they apply to the whole column: s.str.strip(), s.str.lower(), s.str.len().

Each returns a Series, so they chain: s.str.strip().str.lower().str.replace(" ", "_").

The accessor is needed. s.strip() isn't the same thing. It either does a thing unrelated to the Series or fails. strip isn't a Series method. Losing .str is the most common beginner error here. The resulting AttributeError at least says so clearly.

Worth knowing

.str applies a string method to every value and returns a Series, so the calls chain.
Missing values pass through as NaN rather than raising — and a mask containing NaN cannot be used to index, so pass na=False.
contains takes a regex by default; pass regex=False for a literal. match anchors at the start, fullmatch at both ends.
split(expand=True) makes columns; extract with named capture groups makes one column per group.
str.replace needs an explicit regex= in pandas 2 — the old default changed.
.str is convenient, not fast: object columns are per-element Python, and text work is far slower than arithmetic.

Text Columns

The .str accessor, and what it does with missing values.

The .str accessor applies to every value

The same methods you know from Python, one call for the whole column.

example_01.pypandas
Output

                

                

                    
Missing values pass through

                    
.str returns NaN rather than raising, which is usually what you want and occasionally hides a problem.
example_02.pypandas
Output
.str methods return NaN for missing input rather than raising or returning an empty string.
That is often right: the uppercase of an unknown value is unknown.
It turns a problem for predicates. s.str.holds("a") returns True, False **or NaN**. A mask holding NaN can't be used to index a frame. Pandas raises "Can't mask with non-boolean array holding NA / NaN values".
The fix is na=False. This Decides what a missing value should count as:
df[df["name"].str.contains("ana", na=False)]
Pass it each time you filter on a text predicate. na=True is at times what you want, when missing should be included.
This is the single most common runtime error in text cleaning. It appears only when the data has a gap. So it often shows up in production rather than on the sample.

                

                

                    
Testing and finding

                    
contains, startswith and match, and which of them takes a regex.
example_03.pypandas
Output

                

                

                    
Splitting and extracting

                    
split with expand, and extract for the parts a regex names.
example_04.pypandas
Output
s.str.split("-") gives a Series of lists. That is rarely what you want right.
expand=True turns it into a DataFrame with one column per piece. This Is how you split a mixed field into real columns. Rows with fewer pieces get None in the trailing columns.
s.str.split("-").str[0] takes one piece — the second .str indexes into the lists.
extract is often better when the structure is known. It takes a regex with capture groups and returns one column per group. Named groups become column names:
s.str.extract(r"(?P<kind>[a-z]+)-(?P<num>\d+)")
Rows that don't match give NaN across the row. This Makes non-matching input visible rather than silently mangled.
extract returns text, even for digits. Convert later with pd.to_numeric if you need numbers.
extractall returns each match rather than the first, with a MultiIndex.

                

                

                    
Replacing, with and without regex

                    
The default changed in pandas 2, so be explicit.
example_05.pypandas
Output

                

                

                    
It is convenient, not fast

                    
Under the hood this is still per-element Python.
example_06.pypandas
Output

                

                

                    
Testing

                    
holds searches anywhere in the string. startswith and endswith anchor. match anchors at the start, fullmatch at both ends.
**holds takes a regular formula by default.** That matters more than it sounds. It means s.str.holds(".") matches each non-empty string rather than finding a literal dot. s.str.holds("a|b") is an alternation rather than a search for the three characters.
Pass regex=False for a literal search. It's also faster.
case=False makes the test case-insensitive without a split .lower() pass.
startswith and endswith do not take a regex. This Is an inconsistency worth remembering rather than deriving.

                

                

                    
Replacing

                    
s.str.replace(old, new, regex=...) needs the regex arg to be clear in pandas 2. The default changed. Code written against pandas 1 that leaned on the old action can silently do the wrong thing.
Two clean-ups earn their place in almost each script:
Merging whitespace: s.str.replace(r"\s+", " ", regex=True).str.strip().
Norming column names:
df.columns = (df.columns.str.strip().str.lower()
              .str.replace(r"[^a-z0-9]+", "_", regex=True).str.strip("_"))
That turns "  First Name " and "AGE (years)" into first_name and age_years. It handles a whole spreadsheet's worth of inconsistent headers without listing them.
Note df.columns.str works too — the column index is an Index. Index has a .str accessor as well.

                

                

                    
The cost

                    
.str is a convenience, not a vectorisation.
Object columns hold pointers to Python strings. Each step walks them one at a time in Python. The last editor shows text work running far slower than math on the same number of rows.
Three things help.
Do it once. Clean text at load time rather than again and again inside a loop or a function called per group.
**Convert to group** when values repeat. steps then run on the small set of distinct values rather than each row.
**Consider the string dtype.** astype("string") gives the nullable extension type. This Has clearer missing-value semantics than object and is where pandas' future tuning work is going.
For very large text columns, the honest answer is that pandas isn't the right tool. The work belongs in a DB or a purpose-built library.

                

                

                    
The cleaning pipeline

                    
Text from the real world needs the same handful of steps almost each time. Doing them in one pass at load time is far better than scattering them through the analysis:
df["city"] = (df["city"]
              .str.strip()
              .str.lower()
              .str.replace(r"\s+", " ", regex=True))
Strip the ends, norm case, collapse inner whitespace. That alone resolves most of the "same value counted twice" problems that show up in a group-by.
For text that will be checked or joined on, consider also dropping punctuation and accents. unicodedata.norm handles accents. Pandas has no built-in for it. .str.norm("NFKD") plus an ASCII encode is the usual idiom.

                

                

                    
Extracting structure

                    
str.extract with named groups is the most handy of the extraction methods. It names the outputs:
df[["kind", "num"]] = df["ref"].str.extract(r"(?P<kind>[a-z]+)-(?P<num>\d+)")
Rows that don't match give NaN across the row. Counting them later tells you how well the pattern fits:
unmatched = df["kind"].isna().sum()
That check is worth making. A regex that matches 90% of rows silently discards the other 10%. The only sign is a column with gaps.
str.extractall returns each match rather than the first, with a MultiIndex that has a match level.
str.findall returns a list per row. This Is harder to work with and at times what you want.

                

                

                    
Testing and counting

                    
str.holds — anywhere, regex by default, needs na=False for filtering.
str.startswith / str.endswith — anchored, no regex.
str.match — anchored at the start, regex.
str.fullmatch — anchored at both ends.
str.count(pattern) — how many times a pattern occurs per row.
str.len — length, which is a quick way to spot truncated or padded fields.
A frequent need is "does this hold any of these words":
df["text"].str.contains("|".join(words), case=False, na=False)
Building an alternation is fine for a handful of words. For a long list. For whole-word matching, it's worth escaping the parts with re.escape and adding word boundaries, or the results will surprise you.

                

                

                    
Splitting into columns

                    
str.split(sep, expand=True) gives a DataFrame. Rows with fewer parts get None in the trailing columns. Rows with more parts are truncated unless you pass n to limit the splits.
n=1 splits only on the first separator. This Is what you want for key. Value text where the value may hold the separator.
str.rsplit splits from the right. This Handles "all before the last dot" cleanly.
str.split returns just three columns. Before, separator, after — which avoids the var-width problem fully.

                

                

                    
Categorical text

                    
Once text is cleaned, if it repeats, convert it:
df["city"] = df["city"].astype("category")
The .str accessor still works on a cat column. Pandas applies the step to the groups rather than to each row. So df["city"].str.upper() on a million rows with four cities does four steps, not a million.
That is the single largest speed-up on hand for repeated text work, and it's a one-line change.

                

                

                    
When to stop using pandas for text

                    
pandas is a fair place to clean and extract from moderate amounts of text. It isn't a text-handling engine.
For token split, stemming, language spotting or anything linguistic, a dedicated library is the right tool.
For very large corpora, the object dtype's per-row Python cost dominates. The work belongs in a DB, in Polars, or in a purpose-built pipeline.
The signal is often the profile: if .str steps are the slowest part of your script and the frame is large, the answer is a other tool rather than a cleverer regex.

                

                

                    
Regex, briefly

                    
a few .str methods take a regular formula. A small vocab covers most data cleaning:
\d digit, \w word character, \s whitespace, . any character.
+ one or more, * zero or more, ? optional.
^ start, $ end,  word boundary.
[abc] a character set, [^abc] its negation.
(...) a capture group, (?P<name>...) a named one.
| alternation.
Two habits prevent most regex trouble in pandas. Use raw strings. r"\d+". So backslashes reach the regex engine intact. And test the pattern on a handful of values before applying it to the column. A pattern that matches nothing makes a column of NaN rather than an error.
str.holds(..., regex=False) and str.replace(..., regex=False) are both faster and safer when the pattern is a literal.

                

                

                    
Encoding problems

                    
Text that arrives mangled — é where é belongs — was decoded with the wrong encoding, often at read time.
The fix is at read_csv, with encoding="latin-1" or encoding="cp1252", not in pandas later. Repairing mojibake after the fact is possible and unreliable.
df["col"].str.encode("utf-8").str.decode("utf-8") round-trips text and raises on anything invalid. This Is a way to find the offending rows.
For matching across accented and unaccented spellings, norming with str.norm("NFKD") and stripping mixing characters puts both forms into the same shape.

                

                

                    
Performance, restated

                    
Three things make text work faster, in order of effect:
**Convert to group** when values repeat. steps then apply to the groups, not the rows.
Do the cleaning once, at load time, rather than inside a function called per group or per row.
**Use regex=False** where the pattern is a literal.
And the structural point: if .str steps dominate the profile on a large frame, pandas is the wrong layer for that work. A DB, Polars, or a purpose-built text pipeline will do it in a fraction of the time.

                

                

                    
A summary

                    
.str applies string methods elementwise and chains.
Missing values pass through as NaN. Pass na=False on any predicate used for filtering.
holds is a regex by default. startswith isn't a regex at all.
split(expand=True) makes columns. extract with named groups is often clearer.
replace needs an clear regex= in pandas 2.
Norm column names and text values once, at load.
Convert repeated text to group.
And check how many rows a pattern really matched, rather than given it matched them all.

                

                

                    
A closing note on Text Columns

                    
Text is where pandas is least like an array library and most like plain Python, and both facts show.
.str gives you the string methods across a whole column. This Is convenient and reads well. Underneath it's per-element Python over a column of pointers. This Is why text work is an order of magnitude slower than math on the same number of rows.
Two things follow. Clean text once, at load, rather than again and again in the middle of a pipeline. And convert repeated values to group, after which .str steps apply to the handful of distinct values rather than to each row. Often the largest single speed-up on hand, for a one-line change.
The correctness trap is missing values. .str predicates return NaN rather than False. A mask holding NaN can't index a frame. na=False on each .str filter is the habit. Its lack is the error that appears only once the data has a gap.

                

                

                    
Questions people ask about Text Columns

                    
How do I apply string functions to a whole column? Use the .str accessor, for example df['name'].str.lower() or df['name'].str.strip(). It works element by element and skips missing values.
Why do I get an AttributeError on .str? The column is not a string dtype, often because it holds numbers or mixed types. Convert with astype(str) or fix the source column.
How do I split one column into several? Use df['full'].str.split(' ', expand=True), which returns a frame with one column per piece. Assign the result to new column names.
How do I filter rows that contain some text? Use df[df['col'].str.contains('abc', case=False, na=False)]. The na=False part keeps missing values from breaking the mask.
Does str.contains use regular expressions? Yes, by default. Pass regex=False for a plain substring, especially when your pattern contains characters like ( or .
Are .str methods fast? They are convenient but still loop over the values in Python. For big columns, convert to category first or use vectorised alternatives where they exist.

                

                

                    
One more thing

                    
str.get(i) indexes into each string or list. This Is the concise form of str[i] and works the same way. It returns NaN rather than raising where the index is out of range. This Is often what you want on ragged data.

                

            


        

    


    



Check yourself
0 of 4
Answer without scrolling back up.
  1. Why does `df[df['name'].str.contains('a')]` sometimes raise?
  2. What does `s.str.contains('.')` match?
  3. What does `str.extract` return for a row that does not match?
  4. Why is `.str` slower than arithmetic on the same number of rows?





 Cheat sheet
Text Columns
Use the .str accessor to split, replace and test text columns, and learn how each method handles missing values.
PANDAS · vizlearn.in/pandas/string_methods.html
More in pandas
pandasDates and TimespandasSorting and Rankingpandasgroupbypandastransform and filter
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact