`concurrent.futures` — High-Level Executor Interface
Python · Concurrency · Run with python3
Explanation
`concurrent.futures` provides `ThreadPoolExecutor` (threads, good for I/O) and `ProcessPoolExecutor` (processes, good for CPU). Both expose `submit()` → `Future` and `map()`. `Future.result()` blocks until complete; `as_completed()` yields futures as they finish (fastest first). The executor is a context manager — `__exit__` calls `shutdown(wait=True)`. Switching between thread and process pools requires changing only one import.
Real-World Scenario
A pipeline downloads files (I/O-bound → `ThreadPoolExecutor`) then processes them (CPU-bound → `ProcessPoolExecutor`). `as_completed` shows a progress bar as results arrive rather than waiting for the slowest task. ```python from concurrent.futures import ThreadPoolExecutor, as_completed import time import random def download(url: str) -> tuple[str, int]: """Simulates downloading a file.""" time.sleep(random.uniform(0.1, 0.5)) size = random.randint(1024, 1_048_576) return url, size urls = [f"https://cdn.example.com/file{i}.dat" for i in range(12)] with ThreadPoolExecutor(max_workers=4) as executor: futures = {executor.submit(download, url): url for url in urls} total_bytes = 0 for future in as_completed(futures): url = futures[future] _, size = future.result() total_bytes += size print(f" Downloaded {url.split('/')[-1]} ({size:,} bytes)") print(f"\nTotal: {total_bytes:,} bytes across {len(urls)} files") ```
Quick Snippet
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
import random
def download(url: str) -> tuple[str, int]:
"""Simulates downloading a file."""
time.sleep(random.uniform(0.1, 0.5))
size = random.randint(1024, 1_048_576)
return url, size
urls = [f"https://cdn.example.com/file{i}.dat" for i in range(12)]
with ThreadPoolExecutor(max_workers=4) as executor:
futures = {executor.submit(download, url): url for url in urls}
total_bytes = 0
for future in as_completed(futures):
url = futures[future]
_, size = future.result()
total_bytes += size
print(f" Downloaded {url.split('/')[-1]} ({size:,} bytes)")
print(f"\nTotal: {total_bytes:,} bytes across {len(urls)} files")Full Example
# Compile: python3 concurrent_futures.py
# Output:
# Downloaded files and total bytes printed
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
import random
def download(url: str) -> tuple:
time.sleep(random.uniform(0.02, 0.1))
size = random.randint(1024, 65536)
return url, size
urls = [f"https://cdn.example.com/file{i}.dat" for i in range(8)]
with ThreadPoolExecutor(max_workers=4) as executor:
futures = {executor.submit(download, url): url for url in urls}
total_bytes = 0
completed = 0
for future in as_completed(futures):
url = futures[future]
_, size = future.result()
total_bytes += size
completed += 1
print(f"Downloaded {completed} files, total {total_bytes:,} bytes")
# map — simpler when order doesn't matter
with ThreadPoolExecutor(max_workers=4) as ex:
results = list(ex.map(download, urls[:4]))
print(f"map: {len(results)} results")
python3 concurrent_futures.py