npy, npz, text and bytes - which format to use, and the security footgun in the middle of it.
Overview
The native format
np.save(path, a) writes a .npy file: a short header describing dtype, shape and byte order, followed by the raw bytes.
np.load(path) reads it back identically. Dtype and shape survive, int16 comes back as int16, and there is no parsing cost because there is nothing to parse.
It is the right default for anything that only needs to be read by NumPy. Files are compact, writing and reading are fast, and nothing is lost.
The extension is added automatically if you omit it.
Worth knowing
.npy is the native format: it preserves dtype and shape exactly and round-trips without loss.
savez bundles several named arrays into one .npz; savez_compressed compresses them.
Text formats are portable but lossy — everything comes back as float, and files are larger and slower.
allow_pickle defaults to False because unpickling runs arbitrary code. Never enable it for a file you did not create.
loadtxt raises on missing fields; genfromtxt fills them with NaN or a value you choose.
mmap_mode="r" reads slices of a file larger than memory without loading the whole thing.
Saving and Loading Arrays
npy, npz, text and bytes, and which one to reach for.
.npy keeps dtype and shape exactly
The native format. It round-trips anything a plain array can hold.
example_01.pyNumPy
Output
npz bundles several arrays
Named, lazily loaded, and optionally compressed.
example_02.pyNumPy
Output
Text is portable and lossy
Readable by anything, but slower, larger, and it forgets the dtype.
example_03.pyNumPy
Output
allow_pickle is off by default, and should stay off
Loading a pickle executes code. The default protects you; do not override it for files you did not create.
example_04.pyNumPy
Output
Reading messy text with genfromtxt
loadtxt is strict; genfromtxt copes with gaps.
example_05.pyNumPy
Output
Big files: memory-mapping
Read slices of a file larger than RAM without loading all of it.
example_06.pyNumPy
Output
Bundles
np.savez(path, train=X, labels=y) writes an .npz — a zip archive of .npy files, one per keyword argument.
Loading gives a dictionary-like object. Arrays are read lazily, only when you index a key, so opening a large bundle to read one array does not load the rest.
It holds an open file handle, so use it as a context manager or call close(). Forgetting is a common source of file handles leaking in long-running code.
np.savez_compressed applies deflate. Whether that helps depends entirely on the data: zeros and repeated structure compress dramatically, and floating-point measurement noise barely compresses at all. It costs CPU on both ends, so it is worth measuring on your own data rather than assuming.
Text
np.savetxt and np.loadtxt handle CSV and similar.
They are the right choice when something other than NumPy must read the file — a spreadsheet, a colleague, a different language.
They are the wrong choice otherwise, for three reasons. Files are several times larger. Parsing is far slower than reading raw bytes. And the dtype is lost: everything comes back as float64, whatever went in. An int8 array round-trips into float64.
fmt controls precision on write, and %.4f will silently truncate values you cared about. header with comments="" writes a plain header line rather than a commented one.
allow_pickle
This is the security-relevant part.
Object arrays — arrays with dtype=object, holding arbitrary Python objects — cannot be written as raw bytes. NumPy pickles them instead.
Unpickling executes code. A malicious .npy file can run anything when loaded.
So np.load has allow_pickle=False by default and raises rather than loading an object array. That default is a deliberate protection, added after the risk became a practical problem.
Enable it only for files you created yourself, or received through a channel you trust as much as you would trust an executable. "It is just data" is exactly the assumption the attack relies on.
Better still, avoid object arrays in saved data. If a structure will not fit in a plain numeric array, a format designed for structured data — JSON, HDF5, Parquet — is a better answer than pickling.
Messy input
loadtxt is strict and raises on a missing field. That strictness is a feature when the data should be complete.
genfromtxt is the tolerant version. Missing values become nan by default, or whatever filling_values specifies. It also handles column names, per-column dtypes and skipping footers.
It is slower, and for genuinely messy tabular data, pandas' read_csv is faster and more capable. genfromtxt fits the middle ground: mostly clean numeric data with occasional gaps, where adding pandas would be more dependency than the problem justifies.
Memory mapping
np.load(path, mmap_mode="r") returns an array backed by the file rather than by memory.
Slicing it reads only the pages touched. You can work with a file larger than RAM, and the operating system handles caching.
"r" is read-only. "r+" allows writing back into the file. np.array(mm) materialises the whole thing when you do want it in memory.
The limitation is that it only helps for access patterns that touch part of the data. A full reduction reads everything anyway, and does so with more overhead than a straight load.
Choosing
NumPy-only, one array: .npy.
NumPy-only, several arrays: .npz, compressed if the data has structure.
Anything else must read it: text, accepting the size and the dtype loss.
Larger than memory, partial access: mmap_mode.
Genuinely large or shared across languages and tools: reach past NumPy to HDF5 or Parquet, which handle chunking, compression and metadata properly.
And whichever you choose, do not enable allow_pickle on input you did not produce.
Byte order and portability
A .npy file records the byte order of the data it holds, so a file written on a big-endian machine loads correctly on a little-endian one. NumPy handles the swap.
You will see this in dtype strings: <i4 is little-endian 32-bit integer, >i4 big-endian, =i4 native.
It matters when reading raw binary from an external source — a network protocol, an instrument, a file format defined elsewhere — where the byte order is part of the specification and NumPy has no header to tell it. Getting it wrong produces numbers that are wrong in a distinctive way: plausible magnitudes, nonsense values.
a.byteswap() swaps the bytes in place; a.astype(a.dtype.newbyteorder()) produces a converted copy.
tofile and fromfile, and why to avoid them
a.tofile(path) writes the raw bytes with no header at all. np.fromfile(path, dtype=...) reads them back.
That means the file records neither the shape nor the dtype. Reading it requires knowing both in advance, from somewhere outside the file. Get the dtype wrong and you get garbage rather than an error; get the shape wrong and you get a reshape failure or, worse, a plausible wrong shape.
They exist for interoperating with programs that expect raw binary, and for that they are correct. As a storage format for your own data they are strictly worse than .npy, which adds a hundred-odd bytes of header and removes the entire class of problem.
If you meet a tofile in existing code, it is worth checking whether the shape and dtype are documented anywhere, because that documentation is the only thing making the file readable.
Structured dtypes for records
When the data is genuinely tabular — mixed types per column — a structured dtype keeps it in one array:
Fields are accessed by name: people["age"]. The result is a view, so assigning into it modifies the original.
Structured arrays save and load through .npy without pickling, which is their main advantage over an object array — the data stays plain bytes.
They also sort by field name with order=, which is the closest NumPy comes to sorting a table by columns.
The honest caveat: for anything with more than a few columns or any real analysis, pandas does this better. Structured arrays are worth knowing for reading fixed-format binary files and for the cases where adding pandas is not justified.
Formats beyond NumPy
.npy and .npz are excellent for NumPy-only data of moderate size. Past that, the ecosystem has better answers.
HDF5, via h5py, handles very large arrays with chunking, compression and metadata, supports partial reads and writes, and is readable from most languages. It is the standard in scientific computing for a reason.
Zarr is similar in spirit, designed for cloud object storage, and works well with Dask for parallel access.
Parquet, via pyarrow, is columnar and is the right choice for tabular data going anywhere near a data pipeline or a query engine.
The signals that it is time to move: files over a few hundred megabytes, needing to read part of an array without loading it, needing to append over time, or needing anything other than Python to read it.
A safety summary
The security point deserves restating because it is the one thing in this module that can go beyond losing data.
np.load runs arbitrary code when loading a pickled object array. allow_pickle=False is the default and it is protecting you.
Enable it only for files you produced, or that arrived through a channel you would trust with an executable. A .npy file from an untrusted source should be treated the way you would treat a downloaded script, because in the object-array case that is what it is.
The safe alternatives, in order of preference: keep the data in plain numeric arrays so pickling never arises; use a structured dtype for records; use JSON or Parquet for anything genuinely heterogeneous.
"It is just a data file" is precisely the assumption the attack depends on.
Round-tripping reliably
The habit worth forming is to verify a save-and-load cycle once when introducing a new format, rather than discovering the loss later.
np.save(path, a)
b = np.load(path)
assert np.array_equal(a, b) and a.dtype == b.dtype
Checking the dtype alongside the values catches exactly the failure that text formats introduce silently, and it takes one line.
For floating-point data going through a text format, np.allclose rather than array_equal is the honest comparison, because fmt has almost certainly truncated something.
Paths, and a common annoyance
np.save appends .npy if the filename does not already end in it. That is convenient until code computes a path, saves to it, and then fails to find the file — because the actual file has an extra extension the code does not know about.
Passing an open file object instead of a path avoids the rewriting entirely, and is the reliable form when the filename is computed.
np.savez behaves the same way with .npz.
Compression, in practice
savez_compressed costs CPU on write and on read, and saves space only if the data has structure to exploit.
Integer data with a small range, arrays with many repeated values, and anything sparse compress well — often by an order of magnitude.
Floating-point measurement data compresses poorly, because the low-order bits are effectively random. A 10–20% saving for several times the CPU is rarely a good trade.
The way to decide is to try both on a representative sample and compare, which takes a minute and settles it for that dataset.
Writing for other people
If a file will be read by someone other than you, or by you in two years, the format is only half of it.
Record the shape and dtype expectations somewhere the reader will find them, particularly for tofile output, which carries neither.
Prefer .npz with named arrays over several .npy files with meaningful filenames, because the names travel with the data.
For anything that is genuinely a dataset rather than an intermediate — something that will be reused, shared or archived — HDF5 or Parquet carry metadata properly and are readable outside Python. The extra dependency buys real portability.
The summary
.npy for a single array, NumPy only. Lossless, compact, fast.
.npz for several named arrays, compressed only if the data compresses.
Text when something else must read it, accepting that the dtype is gone.
mmap_mode for partial access to something larger than memory.
HDF5, Zarr or Parquet past a few hundred megabytes, or when other tools are involved.
allow_pickle=False — the default — on anything you did not produce yourself. That one is not a performance preference; it is the difference between reading a file and running one.
A closing note
Storage is where a project's decisions become permanent. A format chosen for convenience during a first experiment tends to survive into production, and by then there is data in it.
.npy and .npz are good enough for a great deal of work and cost nothing to adopt. The point to reconsider is when files grow past a few hundred megabytes, when something other than Python needs to read them, or when only part of an array is needed at a time — and recognising that point early is easier than migrating later.
Check yourself
0 of 4
Answer without scrolling back up.
What does `.npy` preserve that a CSV does not?
Text round-trips everything back as float64, whatever went in. An int8 array saved as text returns as float64.
Why does `np.load` default to `allow_pickle=False`?
It is a deliberate protection. Enable it only for files you trust as much as you would trust an executable.
What is the difference between `loadtxt` and `genfromtxt`?
loadtxt's strictness is a feature when data should be complete. genfromtxt also takes filling_values to choose the substitute.
What does `mmap_mode="r"` give you?
It lets you work with a file larger than RAM - but only helps for partial access, since a full reduction reads everything anyway.
Cheat sheet
Saving and Loading Arrays
np.load(path) reads it back identically. Dtype and shape survive, int16 comes back as int16, and there is no parsing cost because there is nothing to parse.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.