npy, npz, text and bytes - which format to use, and the security footgun in the middle of it.
Overview
The native format
NumPy arrays need to be saved to disk and loaded back frequently. Intermediate results, preprocessed datasets, trained model weights, and test fixtures all need persistent storage. NumPy provides its own binary format and also supports text formats for interoperability.
The native format is the .npy file. np.save writes a short header with the dtype, shape, and byte order, followed by the raw bytes of the array. np.load reads it back. The round trip is lossless. The dtype and shape are preserved exactly. An int16 array comes back as int16, not as int64. A (3, 4, 5) array comes back with the same shape. There is no parsing cost because the file is just the bytes.
For multiple arrays, np.savez writes a .npz file, which is a zip archive of .npy files. Each array is stored under a name. np.savez_compressed applies compression, which helps when the data has patterns that compress well, like arrays with many repeated values or many zeros.
Text formats are useful for interoperability with tools that cannot read .npy files. np.savetxt writes a CSV-like format. np.loadtxt reads it back. The round trip is not lossless. Floating-point values are written as text and parsed back, which can change the last decimal digit. Integer arrays come back as floats unless you specify the dtype. Text files are also much larger and much slower to read and write than binary ones.
The choice between formats depends on who will read the file. If the reader is another Python script, .npy is better in every way. If the reader is R, Julia, or a spreadsheet, text or a more portable binary format like HDF5 is necessary. If the file will be version-controlled, text diffs better but takes more space.
np.load with allow_pickle=False is a security consideration. Pickle files can execute arbitrary code on load. The default in current NumPy versions already disables pickle, but older code may not. Never load an untrusted .npy file with allow_pickle=True.
This module covers np.save, np.load, np.savez, np.savetxt, np.loadtxt, and the trade-offs between binary and text formats. The examples show round-trip preservation and the pitfalls of text serialisation.
np.save(path, a) writes a .npy file: a short header showing dtype, shape and byte order, followed by the raw bytes.
np.load(path) reads it back the same way. Dtype and shape live, int16 comes back as int16. There's no parsing cost because there's nothing to parse.
It's the right default for anything that only needs to be read by NumPy. Files are compact, writing and reading are fast, and nothing is lost.
The extension is added on its own if you omit it.
Worth knowing
.npy is the native format: it preserves dtype and shape exactly and round-trips without loss.
savez bundles several named arrays into one .npz; savez_compressed compresses them.
Text formats are portable but lossy — everything comes back as float, and files are larger and slower.
allow_pickle defaults to False because unpickling runs arbitrary code. Never enable it for a file you did not create.
loadtxt raises on missing fields; genfromtxt fills them with NaN or a value you choose.
mmap_mode="r" reads slices of a file larger than memory without loading the whole thing.
Saving and Loading Arrays
npy, npz, text and bytes, and which one to reach for.
.npy keeps dtype and shape exactly
The native format. It round-trips anything a plain array can hold.
example_01.pyNumPy
import numpy as np, tempfile, os
a = np.arange(12, dtype=np.int16).reshape(3, 4)
path = os.path.join(tempfile.mkdtemp(), "a.npy")
np.save(path, a)
b = np.load(path)
print("saved :", a.dtype, a.shape)
print("loaded :", b.dtype, b.shape)
print("identical:", np.array_equal(a, b) and a.dtype == b.dtype)
print()
print("file size:", os.path.getsize(path), "bytes")
print(" = 12 values x 2 bytes +", os.path.getsize(path) - 24, "byte header")
print()
print("np.save appends .npy if you leave it off.")
Output
npz bundles several arrays
Named, lazily loaded, and optionally compressed.
example_02.pyNumPy
import numpy as np, tempfile, os
d = tempfile.mkdtemp()
path = os.path.join(d, "bundle.npz")
np.savez(path, train=np.arange(6), labels=np.array([0, 1, 0, 1, 0, 1]))
with np.load(path) as z:
print("keys :", list(z.files))
print("train :", z["train"])
print("labels:", z["labels"])
cpath = os.path.join(d, "small.npz")
big = np.zeros((500, 200))
np.savez(path, big=big)
np.savez_compressed(cpath, big=big)
print()
print("uncompressed: %7d bytes" % os.path.getsize(path))
print("compressed : %7d bytes" % os.path.getsize(cpath))
print(" zeros compress well; real data much less so.")
Output
Text is portable and lossy
Readable by anything, but slower, larger, and it forgets the dtype.
example_03.pyNumPy
import numpy as np, tempfile, os, io
a = np.array([[1.5, 2.25], [3.125, 4.0]])
path = os.path.join(tempfile.mkdtemp(), "a.csv")
np.savetxt(path, a, delimiter=",", fmt="%.4f", header="x,y", comments="")
print(open(path).read())
b = np.loadtxt(path, delimiter=",", skiprows=1)
print("loaded:", b.tolist(), b.dtype, "<- always float, whatever was saved")
print()
ints = np.array([1, 2, 3], dtype=np.int8)
buf = io.StringIO()
np.savetxt(buf, ints)
print("int8 saved as text comes back as:",
np.loadtxt(io.StringIO(buf.getvalue())).dtype)
Output
allow_pickle is off by default, and should stay off
Loading a pickle executes code. The default protects you; do not override it for files you did not create.
example_04.pyNumPy
import numpy as np, tempfile, os
path = os.path.join(tempfile.mkdtemp(), "obj.npy")
mixed = np.array([1, "two", [3]], dtype=object)
np.save(path, mixed, allow_pickle=True)
try:
np.load(path)
except ValueError as e:
print("np.load(...) ->", type(e).__name__)
print(" ", str(e)[:78])
print()
print("it loads only if you explicitly allow it:")
print(" ", np.load(path, allow_pickle=True))
print()
print("An object array is pickled, and unpickling runs arbitrary code.")
print("Never pass allow_pickle=True to a file from an untrusted source.")
Output
Reading messy text with genfromtxt
loadtxt is strict; genfromtxt copes with gaps.
example_05.pyNumPy
import numpy as np, io
raw = "1.0,2.0\n3.0,\n5.0,6.0\n"
try:
np.loadtxt(io.StringIO(raw), delimiter=",")
except ValueError as e:
print("loadtxt on a missing field ->", type(e).__name__)
g = np.genfromtxt(io.StringIO(raw), delimiter=",")
print()
print("genfromtxt fills the gap with nan:")
print(g)
print()
print("or with a value you choose:")
print(np.genfromtxt(io.StringIO(raw), delimiter=",", filling_values=-1))
Output
Big files: memory-mapping
Read slices of a file larger than RAM without loading all of it.
example_06.pyNumPy
import numpy as np, tempfile, os
path = os.path.join(tempfile.mkdtemp(), "big.npy")
np.save(path, np.arange(10_000, dtype=np.float64).reshape(100, 100))
mm = np.load(path, mmap_mode="r")
print("type :", type(mm).__name__)
print("shape :", mm.shape, " dtype:", mm.dtype)
print("one row :", mm[7][:5], "<- only this part is read")
print()
print("slices behave like a normal array:")
print(" column mean of first 10 rows:", float(mm[:10, 0].mean()))
print()
print("mmap_mode='r' is read-only; 'r+' allows writing back.")
print("np.array(mm) materialises the whole thing when you do want it.")
Output
Bundles
np.savez(path, train=X, labels=y) writes an .npz. A zip archive of .npy files, one per keyword arg.
Loading gives a dict-like object. Arrays are read lazily, only when you index a key. Opening a large bundle to read one array doesn't load the rest.
It holds an open file handle. Use it as a context manager or call close(). Losing is a common source of file handles leaking in long-running code.
np.savez_compressed applies deflate. Whether that helps depends fully on the data. Zeros and repeated structure compress sharply. Floating-point reading noise barely compresses at all. It costs CPU on both ends. It's worth measuring on your own data rather than given.
Text
np.savetxt and np.loadtxt handle CSV and similar.
They're the right choice when a thing other than NumPy must read the file. A spreadsheet, a colleague, a other language.
They're the wrong choice else, for three reasons. Files are a few times larger. Parsing is far slower than reading raw bytes. And the dtype is lost. All comes back as float64, what went in. An int8 array round-trips into float64.
fmt controls precision on write, and %.4f will silently truncate values you cared about. header with comments="" writes a plain header line rather than a commented one.
allow_pickle
This is the security-relevant part.
Object arrays — arrays with dtype=object, holding any Python objects — can't be written as raw bytes. NumPy pickles them instead.
Unpickling runs code. A malicious .npy file can run anything when loaded.
So np.load has allow_pickle=False by default and raises rather than loading an object array. That default is a deliberate protection, added after the risk became a hands-on problem.
Enable it only for files you created yourself, or received through a channel you trust as much as you'd trust an executable. "It's just data" is just the assumption the attack leans on.
Better still, avoid object arrays in saved data. If a structure won't fit in a plain numeric array, a format designed for structured data. JSON, HDF5, Parquet. Is a better answer than pickling.
Messy input
loadtxt is strict and raises on a missing field. That strictness is a feature when the data should be full.
genfromtxt is the tolerant version. Missing values become nan by default, or what filling_values gives. It also handles column names, per-column dtypes and skipping footers.
It's slower, and for truly messy tabular data, pandas' read_csv is faster and more capable. genfromtxt fits the middle ground. Mostly clean numeric data with occasional gaps. Here Adding pandas would be more dependency than the problem justifies.
Memory mapping
np.load(path, mmap_mode="r") returns an array backed by the file rather than by memory.
Slicing it reads only the pages touched. You can work with a file larger than RAM. The operating system handles caching.
"r" is read-only. "r+" allows writing back into the file. np.array(mm) materialises the whole thing when you do want it in memory.
The limitation is that it only helps for access patterns that touch part of the data. A full reduction reads all anyway. Does so with more cost than a straight load.
Choosing
NumPy-only, one array: .npy.
NumPy-only, a few arrays: .npz, compressed if the data has structure.
Anything else must read it: text, accepting the size and the dtype loss.
Larger than memory, partial access: mmap_mode.
truly large or shared across languages and tools. Reach past NumPy to HDF5 or Parquet. This Handle chunking, packing and metadata properly.
And whichever you choose, don't enable allow_pickle on input you didn't give.
Byte order and portability
A .npy file records the byte order of the data it holds. A file written on a big-endian machine loads right on a little-endian one. NumPy handles the swap.
You'll see this in dtype strings: <i4 is little-endian 32-bit int, >i4 big-endian, =i4 native.
It matters when reading raw binary from an external source. A network protocol, an instrument, a file format defined elsewhere. Where the byte order is part of the specification and NumPy has no header to tell it. Getting it wrong makes numbers that are wrong in a distinctive way: plausible magnitudes, nonsense values.
a.byteswap() swaps the bytes in place. a.astype(a.dtype.newbyteorder()) makes a converted copy.
tofile and fromfile, and why to avoid them
a.tofile(path) writes the raw bytes with no header at all. np.fromfile(path, dtype=...) reads them back.
That means the file records neither the shape nor the dtype. Reading it needs knowing both in advance, from one spot outside the file. Get the dtype wrong and you get garbage rather than an error. Get the shape wrong and you get a reshape failure or, worse, a plausible wrong shape.
They exist for interoperating with programs that expect raw binary. For that they're correct. As a storage format for your own data they're strictly worse than .npy. This Adds a hundred-odd bytes of header and drops the entire class of problem.
If you meet a tofile in existing code, it's worth checking whether the shape and dtype are documented anywhere. That docs is the only thing making the file readable.
Structured dtypes for records
When the data is truly tabular. Mixed types per column. A structured dtype keeps it in one array:
Fields are accessed by name: people["age"]. The result is a view. Assigning into it modifies the first.
Structured arrays save and load through .npy without pickling. This Is their main edge over an object array. The data stays plain bytes.
They also sort by field name with order=. This Is the closest NumPy comes to sorting a table by columns.
The honest caveat: for anything with more than a few columns or any real analysis, pandas does this better. Structured arrays are worth knowing for reading fixed-format binary files and for the cases where adding pandas isn't justified.
Formats beyond NumPy
.npy and .npz are great for NumPy-only data of moderate size. Past that, the ecosystem has better answers.
HDF5, via h5py, handles very large arrays with chunking, packing and metadata, supports partial reads and writes. Is readable from most languages. It's the standard in scientific finding for a reason.
Zarr is similar in spirit, designed for cloud object storage. Works well with Dask for parallel access.
Parquet, via pyarrow, is columnar and is the right choice for tabular data going anywhere near a data pipeline or a query engine.
The signals that it's time to move. Files over a few hundred megabytes, needing to read part of an array without loading it, needing to append over time, or needing anything other than Python to read it.
A safety summary
The security point deserves restating because it's the one thing in this module that can go beyond losing data.
np.load runs any code when loading a pickled object array. allow_pickle=False is the default and it's protecting you.
Enable it only for files you made, or that arrived through a channel you'd trust with an executable. A .npy file from an untrusted source should be treated the way you'd treat a downloaded script. In the object-array case that is what it's.
The safe options, in order of preference: keep the data in plain numeric arrays so pickling never arises. Use a structured dtype for records. Use JSON or Parquet for anything truly heterogeneous.
"It's just a data file" is just the assumption the attack depends on.
Round-tripping reliably
The habit worth forming is to verify a save-and-load cycle once when adding a new format, rather than discovering the loss later.
np.save(path, a)
b = np.load(path)
assert np.array_equal(a, b) and a.dtype == b.dtype
Checking the dtype alongside the values catches just the failure that text formats add silently, and it takes one line.
For floating-point data going through a text format, np.allclose rather than array_equal is the honest compare. fmt has almost certainly truncated a thing.
Paths, and a common annoyance
np.save appends .npy if the filename doesn't already end in it. That is convenient until code finds a path, saves to it. Then fails to find the file. Because the actual file has an extra extension the code doesn't know about.
Passing an open file object instead of a path avoids the rewriting fully. Is the solid form when the filename is found.
np.savez behaves the same way with .npz.
Compression, in practice
savez_compressed costs CPU on write and on read. Saves space only if the data has structure to exploit.
int data with a small range, arrays with many repeated values. Anything sparse compress well. Often by an order of magnitude.
Floating-point reading data compresses poorly. The low-order bits are in effect random. A 10–20% saving for a few times the CPU is rarely a good trade.
The way to decide is to try both on a representative sample and compare. This Takes a minute and settles it for that dataset.
Writing for other people
If a file will be read by someone other than you, or by you in two years, the format is only half of it.
Record the shape and dtype expectations one spot the reader will find them, above all for tofile output, which carries neither.
Prefer .npz with named arrays over a few .npy files with real filenames. The names travel with the data.
For anything that is truly a dataset rather than an mid. A thing that will be reused, shared or archived. HDF5 or Parquet carry metadata properly and are readable outside Python. The extra dependency buys real portability.
The summary
.npy for a single array, NumPy only. Lossless, compact, fast.
.npz for a few named arrays, compressed only if the data compresses.
Text when a thing else must read it, accepting that the dtype is gone.
mmap_mode for partial access to a thing larger than memory.
HDF5, Zarr or Parquet past a few hundred megabytes, or when other tools are involved.
allow_pickle=False — the default — on anything you didn't give yourself. That one isn't a speed preference. It's the gap between reading a file and running one.
Questions people ask about Saving and Loading Arrays
What is the difference between save and savetxt? save writes a binary .npy file that keeps dtype and shape exactly. savetxt writes readable text, which is bigger, slower and loses precision unless you set the format.
How do I store several arrays in one file? Use np.savez, or savez_compressed for smaller files. Loading returns a dict-like object keyed by the names you gave.
Is it safe to load any .npy file? Arrays of plain numbers are safe. Object arrays use pickle, so np.load refuses them by default; only set allow_pickle=True for files you trust.
How do I load a CSV with a header? Use np.genfromtxt or np.loadtxt with skiprows=1 and delimiter=','. For messy or mixed-type data, pandas is the better tool.
Can I read a file bigger than memory? Yes, pass mmap_mode='r' to np.load. The array is mapped from disk and only the pages you touch are read.
Does loading keep the dtype? With .npy and .npz, yes, exactly. With text formats you get float64 unless you pass dtype.
A closing note on Saving and Loading Arrays
Storage is where a project's choices become permanent. A format chosen for convenience during a first test tends to live into production. By then there's data in it.
.npy and .npz are good enough for a great deal of work and cost nothing to adopt. The point to reconsider is when files grow past a few hundred megabytes, when a thing other than Python needs to read them, or when only part of an array is needed at a time. And recognising that point early is easier than migrating later.
Check yourself
0 of 4
Answer without scrolling back up.
What does `.npy` preserve that a CSV does not?
Text round-trips everything back as float64, whatever went in. An int8 array saved as text returns as float64.
Why does `np.load` default to `allow_pickle=False`?
It is a deliberate protection. Enable it only for files you trust as much as you would trust an executable.
What is the difference between `loadtxt` and `genfromtxt`?
loadtxt's strictness is a feature when data should be complete. genfromtxt also takes filling_values to choose the substitute.
What does `mmap_mode="r"` give you?
It lets you work with a file larger than RAM - but only helps for partial access, since a full reduction reads everything anyway.
Cheat sheet
Saving and Loading Arrays
Save and load NumPy arrays with npy, npz and text formats. Compare speed, file size and portability to choose the right format for your data.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.