From dicts, records, arrays and files - and which orientation each one assumes.
Overview
The two orientations
Building a DataFrame is the first thing you do in pandas. The constructor accepts many input shapes. Dicts, lists of dicts, NumPy arrays, other DataFrames, even a single scalar with an explicit index. Each shape implies a different orientation, and picking the wrong one gives you a transposed result with no error.
The most common mistake is mixing up column-oriented and row-oriented input. A dict of lists treats each key as a column name. A list of dicts treats each dict as a row. Both produce a valid frame. If you expected one and wrote the other, every downstream number is wrong, and nothing warns you.
Knowing which constructor form matches your data source prevents a class of silent bug. Data from an API usually arrives as a list of records. Parallel arrays from a computation are columns. CSV files go through a reader. Each path has a natural form. Choosing it means the frame comes out right on the first try.
This module covers the main forms, what each tolerates in terms of missing keys and mismatched lengths, and how dtype inference works at construction time. The last point matters because a column that starts as the wrong type stays wrong until you fix it, and most people do not check.
The examples build small frames and print them. That is faster than loading a file when you want to see what a constructor does with a particular input shape. After working through them, you should be able to look at any input and predict the shape and dtypes of the resulting frame.
Later modules cover reading from CSV and Parquet files, which is how most real data enters pandas. This module focuses on constructing frames from in-memory objects, which is the skill you need for testing, prototyping, and building fixtures.
Almost each confusion about making a DataFrame comes from one question: does this input show columns or rows?
pd.DataFrame({"a": [1,2,3], "b": [4,5,6]}) shows columns. Each key is a column name and each value is the entire column. This is the most common form and the one to reach for when you've parallel lists.
pd.DataFrame([{"a":1,"b":4}, {"a":2,"b":5}]) shows rows. Each dict is one record. This is the shape data arrives in from an API or a DB cursor, and pandas handles it right.
Both give the same frame. Knowing which one you're writing blocks a transposed result.
Worth knowing
A dict of columns is the usual form: keys become column names, values become whole columns, and a scalar is broadcast.
A list of dicts is one record per row — missing keys become NaN rather than raising.
A list of lists gives integer column names unless you pass columns=.
pd.DataFrame(dict_of_scalars) raises. Wrap it in a list for one row, or use from_dict(orient="index") for one column.
read_csv infers a dtype per column, and takes dtype=, index_col= and usecols= to steer it.
Never build a frame row by row — each assignment reallocates it. Collect into a list and construct once.
Creating DataFrames
From dicts, records, arrays and files, and which orientation each assumes.
From a dict of columns
The most common form. Each key is a column name, each value the whole column.
example_01.pypandas
import pandas as pd
df = pd.DataFrame({
"item": ["pen", "book", "bag"],
"qty": [3, 1, 2],
"price": [10.0, 250.0, 400.0],
})
print(df)
print()
print("the keys became columns:", list(df.columns))
print("lengths must match, or it raises:")
try:
pd.DataFrame({"a": [1, 2], "b": [1, 2, 3]})
except ValueError as e:
print(" ", str(e)[:56])
print()
print("a scalar is broadcast to every row, though:")
print(pd.DataFrame({"a": [1, 2, 3], "flag": True}))
Output
From a list of records
One dict per row - the shape data usually arrives in from an API.
example_02.pypandas
import pandas as pd
rows = [
{"item": "pen", "qty": 3},
{"item": "book", "qty": 1},
{"item": "bag", "qty": 2, "gift": True},
]
df = pd.DataFrame(rows)
print(df)
print()
print("missing keys become NaN rather than an error:")
print(df.dtypes)
print()
print("Note 'gift' became object, because NaN is a float and True is a bool.")
print("Column order follows first appearance across all the records.")
Output
From a list of lists, and why columns= matters
Without names you get integer columns, which is rarely what you want.
example_03.pypandas
import pandas as pd
data = [["pen", 3], ["book", 1], ["bag", 2]]
print("no names:")
print(pd.DataFrame(data))
print(" columns are", list(pd.DataFrame(data).columns), "<- integers")
print()
print("with names:")
df = pd.DataFrame(data, columns=["item", "qty"])
print(df)
print()
print("an index can be supplied at the same time:")
print(pd.DataFrame(data, columns=["item", "qty"], index=["a", "b", "c"]))
Output
Reading a CSV from text
read_csv is the usual entry point, and it takes any file-like object.
example_04.pypandas
import pandas as pd
import io
csv = """item,qty,price
pen,3,10.0
book,1,250.0
bag,2,400.0
"""
df = pd.read_csv(io.StringIO(csv))
print(df)
print()
print("types are inferred per column:")
print(df.dtypes)
print()
print("and you can steer it:")
df2 = pd.read_csv(io.StringIO(csv), index_col="item", dtype={"qty": "int8"})
print(df2)
print("qty dtype:", df2["qty"].dtype)
Output
From a Series, and the orientation trap
A dict of scalars gives one row only if you tell it so.
example_05.pypandas
import pandas as pd
d = {"a": 1, "b": 2, "c": 3}
print("as a Series (the values are rows):")
print(pd.Series(d))
print()
print("pd.DataFrame(d) raises - scalars have no length:")
try:
pd.DataFrame(d)
except ValueError as e:
print(" ", str(e)[:52])
print()
print("wrap it in a list to mean 'one row':")
print(pd.DataFrame([d]))
print()
print("or use from_dict with orient='index' to mean 'one column':")
print(pd.DataFrame.from_dict(d, orient="index", columns=["value"]))
Output
Building rows in a loop is the slow way
The same lesson as NumPy: collect, then construct once.
example_06.pypandas
import pandas as pd
import time
n = 2000
t = time.perf_counter()
acc = pd.DataFrame(columns=["i", "sq"])
for i in range(300):
acc.loc[len(acc)] = [i, i * i]
grow = time.perf_counter() - t
t = time.perf_counter()
rows = [{"i": i, "sq": i * i} for i in range(300)]
once = pd.DataFrame(rows)
fast = time.perf_counter() - t
print("appending row by row : %.4f s" % grow)
print("build list, then once: %.4f s" % fast)
print("ratio : %.0fx" % (grow / max(fast, 1e-9)))
print()
print("Each .loc assignment reallocates the whole frame.")
print("df.append() used to be the popular way - it was removed in pandas 2.0.")
Output
What each form tolerates
Dict of columns needs equal lengths and raises else. This Is a handy check. A scalar value is the exception. It broadcasts to each row. {"a": [1,2,3], "flag": True} gives a flag column of three True values.
List of dicts doesn't need the keys to match. Missing keys become NaN. The column order follows first appearance across all records. That handling is convenient for messy input and worth watching. A typo in one record's key silently creates a new mostly-empty column rather than raising.
List of lists works but names the columns 0, 1, 2. Pass columns= unless you truly want int names. int column names are legal and lead to real confusion later. df[0] then means a column and looks like spot-based indexing.
Dict of scalars raises, because pandas can't tell whether you meant one row or one column. pd.DataFrame([d]) means one row. pd.DataFrame.from_dict(d, orient="index") means one column. The error is good. Guessing here would be worse.
read_csv
For real data this is the usual entry point. It does a lot by default. Infers a dtype per column, treats the first line as a header, and parses common missing-value markers.
The args worth knowing early:
index_col sets a column as the index during the read rather than later.
usecols reads only the columns you name. This Matters on wide files. It saves both time and memory.
dtype replaces inference. This is how you stop an identifier column being read as an int and losing its leading zeros. How you read a low-cardinality column straight into group.
parse_dates converts date columns during the read. This Is faster than converting later.
nrows reads a sample. This Is the right way to inspect a large file before committing to it.
It accepts any file-like object. So io.StringIO works for testing and for data that is already in memory.
Type inference and its costs
read_csv and the constructors infer types per column. That is often helpful and at times wrong in costly ways.
A column of ints with one missing value turns float64. Int arrays can't hold NaN. Identifiers then print as 1001.0.
A column that is mostly numbers with one stray non-numeric value turns object. Each later numeric step on it either fails or silently operates on strings.
Both are worth checking with df.dtypes at once after loading. It takes one line and catches a group of bug that else surfaces much later.
Never grow a frame
acc.loc[len(acc)] = row inside a loop reallocates the entire frame on each loop. So did df.append. This Is why it was removed in pandas 2.0 rather than merely deprecated.
The correct pattern is the same as NumPy's. Collect into a Python list and make once at the end. Lists append cheaply. DataFrames don't append at all.
If the pieces are already frames, pd.concat(list_of_frames) once at the end is the right call. And pd.concat inside a loop is just the same mistake in a other costume.
The last editor measures it. The gap grows with the number of rows. A pattern that seems acceptable on a hundred records turns unusable on a hundred thousand.
Choosing
Parallel lists you already have: dict of columns.
Records from an API or a cursor: list of dicts.
A file: read_csv, with dtype and usecols set deliberately.
A NumPy array: pd.DataFrame(arr, columns=[...]), remembering that the array's single dtype applies to every column until you convert.
One row from a dict: pd.DataFrame([d]).
And in each case, print df.dtypes and df.shape at once later. Those two lines catch most of what goes wrong at the boundary between raw data and pandas.
From a NumPy array
pd.DataFrame(arr, columns=[...]) wraps an array. The array has one dtype. Each column starts with that dtype until you convert.
That matters when the array came from a thing that widened it. An array of mixed data is object. A DataFrame built from it has each column as object. Numeric-looking columns that don't behave by number.
index= sets the labels also.
For the reverse direction, df.to_numpy() merges back to a single dtype. df.values is the older spelling of the same thing.
From a database or an API
pd.read_sql(query, link) runs a query and returns a frame, with column names taken from the result. It accepts a SQLAlchemy link or a raw DBAPI one.
pd.read_json handles JSON, and pd.json_normalize is the one worth knowing. It flattens nested JSON into columns, turning {"user": {"name": "ana"}} into a user.name column. Most API responses need it, and building the frame by hand from nested dicts is the long way round.
pd.read_html(url_or_html) scrapes each table on a page into a list of frames. It's startlingly good for simple pages and needs lxml or bs4 installed.
Setting the index at creation
Any of the constructors accept index=, and read_csv accepts index_col=.
Doing it at creation is slightly cheaper than set_index later. More importantly it docs that the column is a key rather than data.
If the index should be a DatetimeIndex, parse the dates in the same call: read_csv(path, parse_dates=["date"], index_col="date") gives a time-indexed frame in one step.
Empty frames, and why they cause trouble
pd.DataFrame() is legal and at times handy as a placeholder.
It's a poor starting point for build-up, for two reasons. Growing it row by row is quadratic, as the last editor shows. And its columns have no real dtype. The first concat into it can widen all to object. At which point the frame is slow and the numeric columns are no longer numeric.
pd.DataFrame(columns=["a", "b"]) has the same problem with more ceremony.
The pattern to use instead is always the same. Build a list of rows or of frames. Make once at the end.
Checking what you built
Four lines, at once after build, catch most problems at the boundary:
df.shape # is this the number of rows I expected?
df.dtypes # did anything become object or float unexpectedly?
df.head() # does the data look like the data?
df.isna().sum() # where are the gaps?
The second is the one that earns its keep. Type inference is where most silent damage happens. It happens just once, at build.
Choosing a constructor
Parallel lists you already have — a dict of columns.
Records from an API or a cursor. A list of dicts, or json_normalize if they're nested.
A file — read_csv, with dtype, usecols and parse_dates set on purpose.
A query — read_sql. This Names the columns for you.
A NumPy array — the constructor with columns=, remembering the single dtype.
One row from a dict — pd.DataFrame([d]).
built up pieces — a list, then pd.DataFrame(rows) or pd.concat(frames) once.
In each branch, set the dtype at creation rather than converting later. astype allocates a second full frame. Getting it right the first time avoids both the copy and the class of bugs where a column silently isn't the type you assumed.
Reproducible test frames
Small frames written by hand are how you check that an step does what you think. It's worth having a habit for them.
A dict of columns is the most readable form for a handful of rows. pd.DataFrame({"a": [1,2,3], "b": list("xyz")}) fits on one line and shows the columns clearly.
For a frame with a exact index, pass index= rather than setting it later.
For random test data, seed it: np.random.default_rng(0). Unseeded random test data makes a failing check unreproducible. This Is the opposite of what a test is for.
pd.util.testing used to provide frame gens. The supported route now is pd.testing.assert_frame_equal for checks, and building the frames yourself.
Checking two frames match
pd.testing.assert_frame_equal(a, b) is the precise compare: values, dtypes, index and column order all have to agree. The error message says which differed.
check_dtype=False relaxes the type compare. This Is often needed when one side came through a CSV.
check_like=True skips row and column order.
This is what belongs in a test, rather than a.equals(b). When it fails it tells you *what* failed.
Copy semantics at construction
pd.DataFrame(some_array) does not always copy. If the array's dtype and layout are usable right, pandas may wrap it. Writing to the frame then writes to the array.
copy=True forces a copy. Under copy-on-write this stops mattering. This Is another reason that change is welcome.
pd.DataFrame(dict_of_series) aligns the Series on their indexes. This Is easy to forget. Three Series with other labels give a frame with the union of them and NaN in the gaps, not three columns side by side.
If you meant "these are parallel columns", reset the indexes first or pass plain lists.
The construction checklist
Decide whether the input shows rows or columns.
Pass columns= if the source has no names.
Set dtype at build, above all for identifiers and for ints that may have gaps.
Set index= if a column is really a key.
Then check shape, dtypes, head() and isna().sum().
Never grow a frame in a loop.
That is six lines of discipline that prevent most of what the cleaning modules exist to repair.
A closing note on Creating DataFrames
build is where a frame's types are decided. Types decided badly here cause problems all over later.
The two questions worth answering on purpose are whether the input shows rows or columns. What the dtypes should be. Getting the first wrong gives a transposed frame that is at once obvious. Getting the second wrong gives a frame that looks correct and behaves oddly much later.
dtype= at build costs nothing. astype later allocates a second copy of each column it touches. By then the leading zeros may already be gone.
The other rule is the one shared with NumPy. Never grow a frame in a loop. Collect into a list and make once. df.loc[len(df)] = row and pd.concat inside a loop are the same quadratic mistake wearing other syntax. df.append was removed rather than merely deprecated because of it.
One more thing
pd.DataFrame.from_records handles an iterable of tuples with a columns arg. Accepts an index naming one of the fields. It's the constructor that fits a DB cursor most right. Cursors yield tuples rather than dicts.
And pd.concat of many small frames is often faster than building one large list of dicts when the pieces are already frames. The rule is to avoid the loop, not to prefer one holder over the other.
Questions people ask about Creating DataFrames
What is the easiest way to build a small DataFrame by hand? Pass a dict of lists to pd.DataFrame, where each key becomes a column name and each list becomes that column's values.
What is the difference between a dict of lists and a list of dicts? A dict of lists is column-oriented, with one key per column. A list of dicts is row-oriented, with one dict per record, and missing keys become NaN.
Why should I not grow a DataFrame row by row in a loop? Each append copies the whole frame, so the loop gets slower as it grows. Collect rows in a list and call pd.DataFrame once at the end.
How do I create a DataFrame from a NumPy array? Call pd.DataFrame(arr, columns=[...]). Without the columns argument the columns are numbered 0, 1, 2.
How do I create an empty DataFrame with the right columns? Use pd.DataFrame(columns=['a', 'b']), then set dtypes explicitly with astype. Otherwise every column is object and later arithmetic can fail.
Does pd.DataFrame copy my data? Passing a NumPy array may share memory with it, so changing one can change the other. Pass copy=True when you need an independent frame.
Creating DataFrames, in summary
Decide whether the input shows rows or columns, set the dtypes at build. Check shape, dtypes and head at once later.
Never grow a frame in a loop. Collect into a list and build once. df.append was removed rather than deprecated because of just that pattern.
Check yourself
0 of 4
Answer without scrolling back up.
What does `pd.DataFrame({'a': [1,2,3], 'flag': True})` produce?
A scalar value is broadcast to every row, while list values must all have matching lengths.
Why does `pd.DataFrame({'a': 1, 'b': 2})` raise?
Wrap it in a list for one row, or use from_dict(orient='index') for one column. Guessing would be worse than the error.
A CSV column of integers has one missing value. What dtype does it get?
This is why identifiers sometimes print as 1001.0. Check df.dtypes right after loading - it takes one line and catches the bug early.
What happened to `df.append()`?
Collect rows into a Python list and construct once. pd.concat inside a loop is the same mistake in different clothing.
Cheat sheet
Creating DataFrames
Build pandas DataFrames from dictionaries, lists of records, NumPy arrays and CSV files. See which constructor fits each data shape.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.