Threads or Processes: What Is and Is Not Shared
Threads share everything and cannot run Python at once. Processes run Python at once and share nothing - so the real question is what your data costs to send.
Threads share everything and cannot run Python at once. Processes run Python at once and share nothing - so the real question is what your data costs to send.
This explorer needs JavaScript: every interleaving, cost and speed-up on it is computed in the page rather than downloaded as an image.
Processes have separate address spaces. Each has its own interpreter, its own GIL, its own copy of every module, and no access whatsoever to the others' objects. They do run Python at once, on as many cores as you have.
Threads in a process share one address space. Every global, every module, every open object is the same object seen from all of them, which is why they are cheap to start and why they need locks.
| Threads | Processes | |
|---|---|---|
| Run Python at the same time | no | yes |
| Share objects | everything | nothing |
| Start-up cost | microseconds | tens of milliseconds |
| Memory per worker | a stack | a whole interpreter |
| Passing data | free, it is the same object | pickled, copied, unpickled |
| Crash containment | takes the process down | isolated |
| What you need | locks | a serialisation story |
The last two rows of that table are where projects actually get stuck. "Use processes for CPU-bound work" is correct and it is the easy half. The hard half is that nothing crosses the boundary unless it can be pickled, and the pickle is a copy that is paid for twice.
These editors run in a browser interpreter with no OS threads and no _multiprocessing module, so the pool examples are shown with their CPython output. The pickle boundary, though, is entirely real here — and it is the part that decides whether a design works.
Sending an argument to a worker process means serialising it, writing the bytes to a pipe, and reconstructing it on the other side. So the question "can I use a process pool for this?" is really "can this be pickled, and how big is it?"
what crosses, and what it costs
int 42 5 bytes
dict of 3 lists 29 bytes
list of 100,000 368931 bytes
list of 1,000,000 4871352 bytes
dumps 53 ms, loads 143 ms <- paid on every call
and what cannot cross at all:
a lock TypeError: cannot pickle '_thread.lock' object
a generator TypeError: cannot pickle 'generator' object
an open file TypeError: cannot pickle 'TextIOWrapper' instancesRead the bottom three first. A lock, a generator and an open file cannot be pickled, and the reason is the same in each case: they refer to something that only means anything inside the process that made it. A lock is a handle on operating-system state; a generator is a paused stack frame; a file object wraps a descriptor belonging to one process. There is no honest way to copy any of them, so pickle refuses rather than producing something broken.
That refusal is what turns "just use a process pool" into a redesign. A worker function that closes over a database connection, a file handle, a logger with an open socket, or a lambda will fail at submit time rather than doing anything useful. The usual answer is to send the *description* and build the object inside the worker — pass a path, not a file; pass a DSN, not a connection — which is why process-pool worker functions tend to take plain data and construct everything they need.
Then read the timings. Five million bytes for a million integers, tens of milliseconds to serialise and more to rebuild, and both happen on every call. If the work a worker does is smaller than the cost of sending it its arguments, a pool makes the program slower, and this is much easier to hit than it sounds.
With that in mind, the code is unremarkable:
from concurrent.futures import ProcessPoolExecutor
import time
def work(n):
total = 0
for i in range(n):
total += i * i
return total
N = 5_000_000
if __name__ == "__main__": # required on spawn platforms
t0 = time.perf_counter()
work(N); work(N); work(N); work(N)
print(f"sequential: {time.perf_counter() - t0:.2f}s")
t0 = time.perf_counter()
with ProcessPoolExecutor(max_workers=4) as pool:
list(pool.map(work, [N] * 4))
print(f"four processes: {time.perf_counter() - t0:.2f}s")sequential: 2.35s
four processes: 0.71sFour processes, four cores, and roughly the speed-up you would hope for. Note what made this a good case: the arguments are one integer each, the results are one integer each, and each call does a substantial amount of work. The boundary cost is nothing and the parallelism is real.
The if __name__ == "__main__" guard is not decoration. On Windows and macOS a new process spawns — it starts a fresh interpreter and imports your module to find the worker function — so without the guard, the module-level code runs again in every child, which starts more children, recursively. On Linux the default was fork, which copies the parent instead and does not need the guard; that changed in 3.14, where forkserver became the default because fork in a process with threads is genuinely unsafe. Write the guard.
The start method decides what a child begins life with, and it is the setting behind most "it works on my machine" reports:
| Method | What the child gets | Notes |
|---|---|---|
fork | a copy of the parent, lazily | fast; unsafe with threads or locks held |
spawn | a fresh interpreter, then your module imported | slow, needs picklable everything, safest |
forkserver | a fork of a small clean server process | the 3.14 default on Linux |
fork is why some code appears to share objects with its children: they inherit a copy, so a global that was populated before the fork is visible — and then diverges the moment either side writes to it. Code written against that behaviour breaks on spawn, where the child starts from a clean import and sees nothing the parent computed at runtime. Setting multiprocessing.set_start_method("spawn") in development is the cheapest way to find out whether you depend on it.
The obvious way to shrink what crosses the boundary is to send a compact buffer instead of a container of objects. It is worth testing that intuition rather than acting on it:
small ints (0..n) list 869,203 array 1,600,089 array is 1.84x the list
large ints (2**60 + i) list 2,000,684 array 1,600,089 array is 0.80x the listThe intuition is wrong half the time. An array("q") spends a fixed eight bytes per element whatever the value, while pickle encodes small integers in one or two bytes and large ones in nine. So for counters and indices the "compact" buffer is nearly twice the size, and for genuinely large values it is a fifth smaller.
The transferable habit is the one line that settled it: len(pickle.dumps(x)) on a representative argument. It takes a second, it is the number that decides whether a process pool can win, and it is not reliably predictable from the type.
For numpy arrays the answer is different again and worth knowing: they pickle as a raw buffer plus a small header, so the payload is the data and nothing else. That is why array-shaped work tends to survive the boundary when object-shaped work does not — and why multiprocessing.shared_memory, which passes a name instead of a copy, is the escape hatch for the cases where even that is too much.
Two escapes are worth knowing, because the threads-or-processes framing hides them.
Let the library do it. numpy, pandas, scipy, and anything built on BLAS release the GIL inside their C loops, so a vectorised operation already uses your cores without threads or processes in your code. Rewriting a Python loop as an array operation is frequently a larger win than parallelising the loop, and it removes the concurrency question entirely.
Use both. The common production shape is a small pool of processes, each running many threads or an event loop. Processes get past the GIL; the threads inside each handle the waiting. A four-process, thirty-thread server is not a compromise, it is the answer to "our work is partly CPU-bound and mostly I/O", which is what most services are.
Assuming an object will pickle. Locks, sockets, generators, open files, lambdas and local functions will not. The failure arrives at submit time, and the fix is usually to send a description and build the object in the worker.
Sending large arrays through the boundary. Copying a gigabyte per call costs more than the work. For numpy specifically, multiprocessing.shared_memory passes a handle instead of a copy.
Omitting the __main__ guard. On spawn platforms the module is re-imported in every child, and without the guard that starts more children.
Expecting globals to be shared. Each process has its own. A counter incremented in a worker is incremented in that worker's copy and nowhere else; results have to come back through the return value or a Manager.
Using processes for I/O. Threads are cheaper, share the data, and are not waiting on the GIL anyway, because a blocked thread has already released it.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Two models, one difference”?
What does this module say about “The boundary is pickle”?
What does this module say about “The three start methods”?
Compare threads and processes side by side: shared memory versus isolation, the GIL constraint, and the real cost of sending data between workers.