Processes

Part of the free LogicWiz Python course, Module 16: Concurrency. The examples and practice run in the app.

Month-end: Meher's report crunches 2 lakh bills and takes a minute. She tried 4 threads and it still took a minute (the GIL). With 4 processes it takes 16 seconds. Here's why.

You'll learn to

  • Use `ProcessPoolExecutor` and `multiprocessing.Pool` for CPU-bound work
  • Explain why processes get around the GIL, and what they cost
  • Use the shared `concurrent.futures` API: `map`, `submit`, `result`, `as_completed`
  • Split work into chunks and merge the results (map, then reduce)

Learn

A process is a whole new Python

A process is a separate running copy of Python with its own memory and its own GIL. Four processes on four cores really do calculate at the same time.

The price:

  • starting a process takes time (tens of milliseconds)
  • processes don't share variables: arguments and results are copied between them by pickling (turning objects into bytes)

So processes win when each piece of work is big and the data sent is small: "count primes up to 2 lakh", not "add 1 to this number".

As with threads, the browser can't start processes, so these cards show a real laptop run. The worker function lives in a file work.py next to the program.

Threads vs processes on CPU work

import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
from work import count_primes

jobs = [200_000] * 4

if __name__ == "__main__":
    start = time.perf_counter()
    results = [count_primes(j) for j in jobs]
    print(f"One by one: {results}  {time.perf_counter() - start:.2f} s")

    start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=4) as ex:
        results = list(ex.map(count_primes, jobs))
    print(f"4 threads:  {results}  {time.perf_counter() - start:.2f} s")

    start = time.perf_counter()
    with ProcessPoolExecutor(max_workers=4) as ex:
        results = list(ex.map(count_primes, jobs))
    print(f"4 processes: {results}  {time.perf_counter() - start:.2f} s")

Threads: no gain at all (the GIL). Processes: about 3x faster on this laptop. Not a full 4x, because starting processes and sending data costs something. Your numbers will differ with your CPU.

Two rules for processes on Windows and macOS. They start each worker as a fresh Python that imports your file:

  1. Put the starting code under if __name__ == "__main__":, or every worker would start more workers.
  2. The worker function must be importable: defined at the top level of a module, not a lambda or a function inside another function. That is also why it must be picklable.

multiprocessing.Pool

from multiprocessing import Pool, cpu_count
from work import count_primes

if __name__ == "__main__":
    print("CPU cores:", cpu_count())
    with Pool(processes=4) as pool:
        print(pool.map(count_primes, [10, 100, 1000, 10000]))

Pool.map is the older API and does the same job as ProcessPoolExecutor.map. Results come back in the same order as the inputs.

submit, result and as_completed

from concurrent.futures import ProcessPoolExecutor, as_completed
from work import count_primes

limits = [300_000, 1_000, 50_000]

if __name__ == "__main__":
    with ProcessPoolExecutor() as ex:
        futures = {ex.submit(count_primes, n): n for n in limits}
        for fut in as_completed(futures):          # in the order they FINISH
            print("finished", futures[fut], "->", fut.result())

        print("map keeps input order:", list(ex.map(count_primes, limits)))

submit returns a Future, a promise of a result; .result() waits for it (and re-raises any error from the worker). ThreadPoolExecutor has exactly the same methods, so switching between threads and processes is a one-word change.

Processes do not share variables

from multiprocessing import Process
import work

if __name__ == "__main__":
    p = Process(target=work.add_order, args=("Samosa",))
    p.start()
    p.join()
    print("parent sees:", work.orders)

The child appended to its own copy of orders. The parent's list never changed. To get data back, return it (pool results) or use a multiprocessing.Queue.

Try it here: what can be sent to a process?

import pickle

order = {"id": 101, "items": ["Samosa", "Chai"], "total": 25}
data = pickle.dumps(order)                 # what a process pool does to arguments
print(len(data), "bytes ->", pickle.loads(data))

double = lambda x: x * 2
pickle.dumps(double)                       # a lambda can't be pickled

Dicts, lists, numbers and strings pickle fine. A lambda has no importable name, so pickle fails with PicklingError, and so would ex.map(lambda x: x * 2, data) with a process pool.

Try it here: split, map, merge

from collections import Counter

bills = ["Samosa", "Chai", "Chai", "Roll", "Samosa", "Chai", "Coffee", "Chai"]


def chunks(items, size):
    return [items[i:i + size] for i in range(0, len(items), size)]


def count_chunk(chunk):          # the worker: small input, small output
    return Counter(chunk)


parts = chunks(bills, 3)
print(parts)
partials = list(map(count_chunk, parts))   # on a laptop: ex.map(count_chunk, parts)
total = sum(partials, Counter())
print(total.most_common(2))

This is map-reduce: split the data, let each worker count its chunk, then merge. Swap the built-in map for ex.map from a ProcessPoolExecutor and the counting runs on every core.

In one line

ProcessPoolExecutor/Pool run CPU-bound work on every core (each process has its own GIL); guard with if __name__ == "__main__":, use top-level functions, and return results since memory is not shared.

Interview corner

Why does `multiprocessing` speed up CPU-bound code when threads don't?

Each process is a separate interpreter with its own memory and its own GIL, so the OS can run them truly in parallel on different cores. Threads share one interpreter and one GIL, so only one runs Python code at a time. The trade-off is that processes are heavier to start and must pickle data to communicate.

What is `concurrent.futures`?

A high-level standard-library API for running callables in pools: ThreadPoolExecutor and ProcessPoolExecutor share the same interface (submit returns a Future; map returns results in input order; as_completed yields futures as they finish). Using it as a context manager waits for all work and shuts the pool down.

What can and cannot be passed to a worker process?

Anything picklable: numbers, strings, lists, dicts, instances of top-level classes, and top-level functions. Lambdas, nested functions, open files, sockets, locks and database connections cannot be pickled. Keep arguments and results small, because copying big data between processes can cost more than the parallelism saves.