Advanced Python: the machinery under the language

This is step three. Our beginner tutorial taught you to make things work; the intermediate one taught you to write them the way a Python programmer would. This tutorial is about the layer beneath that fluency — the protocols and machinery that make Python behave the way it does. You'll see how for loops really work, how @property is built from a lower-level primitive, what a metaclass actually is, and how to make code run concurrently. You rarely need all of this at once, but knowing it is the difference between using the language and understanding it.

The iterator protocol

Every for loop in Python is powered by one small contract, the iterator protocol: an object is iterable if it has __iter__(), which returns an iterator — an object with a __next__() that yields the next value and raises StopIteration when it's done.¹ Implement those and your own type works everywhere a list does: in for, in comprehensions, in zip().

class Countdown:
    def __init__(self, start):
        self.start = start
    def __iter__(self):
        n = self.start
        while n > 0:
            yield n          # a generator IS an iterator
            n -= 1

for x in Countdown(3):
    print(x)      # 3, 2, 1

Notice the shortcut: because a generator function is already an iterator, writing __iter__ as a generator (with yield) saves you from hand-writing __next__ and the StopIteration bookkeeping. Once you think in iterators, the itertools module becomes a toolbox of lazy, composable building blocks — chain, islice, groupby, product — that operate on any iterable without ever building the whole sequence in memory.²

Generators, deeper: delegation

You met generators in the last tutorial. The advanced move is delegation with yield from, which hands off to a sub-generator and transparently passes through its values (and return value).³⁴ It turns nested iteration into one flat, readable line:

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)   # recurse, streaming
        else:
            yield item

list(flatten([1, [2, [3, 4]], 5]))   # [1, 2, 3, 4, 5]

Because it's all lazy, chaining generators gives you memory-flat data pipelines: each stage pulls one item at a time from the one before it, so you can process a file larger than RAM through several transformations without ever materialising it.

Decorators that take arguments, and functools

A plain decorator wraps a function. A decorator with arguments is one level up — a function that returns a decorator — so you can configure the wrapping:

import functools

def retry(times):
    def decorator(fn):
        @functools.wraps(fn)          # keep fn's name/docstring
        def wrapper(*args, **kwargs):
            for attempt in range(times):
                try:
                    return fn(*args, **kwargs)
                except Exception:
                    if attempt == times - 1: raise
        return wrapper
    return decorator

@retry(times=3)
def fetch(): ...

Always apply @functools.wraps to your wrapper — it copies the original's name, docstring and signature so introspection and debuggers still see the real function.⁵ The rest of functools is a highlight reel of advanced tools: lru_cache memoises a function's results, partial pre-binds arguments to make a new callable, and singledispatch gives you type-based function overloading — pick an implementation based on the first argument's type.⁵⁶

Descriptors: how @property actually works

Here's a thing that surprises people: @property, methods, classmethod, and every ORM's typed field are all built on one protocol you can use yourself — the descriptor. A descriptor is any object that defines __get__ (and optionally __set__), placed as a class attribute; Python then routes attribute access on instances through it.⁷ The modern __set_name__ hook even tells the descriptor which attribute name it was assigned to.⁸

class Positive:
    def __set_name__(self, owner, name):
        self.attr = "_" + name
    def __get__(self, obj, objtype=None):
        return getattr(obj, self.attr)
    def __set__(self, obj, value):
        if value <= 0:
            raise ValueError("must be positive")
        setattr(obj, self.attr, value)

class Account:
    balance = Positive()      # validated on every assignment

Write the validation once as a descriptor and reuse it across dozens of fields. This is exactly the mechanism SQLAlchemy, Django models, and dataclasses' field machinery are built on — now you can see through it.

The data model up close: dunders and __slots__

Python's data model is the set of __dunder__ methods that hook your objects into language syntax.⁹ You've used __init__ and __repr__; the full set lets your objects respond to operators (__add__, __eq__, __lt__), to being called (__call__), to indexing (__getitem__), and to missing-attribute access (__getattr__). Implement the right dunder and your type behaves like a built-in.

When you create many instances of a small class, add __slots__. By default every instance carries a __dict__ for its attributes, which is flexible but memory-hungry; declaring __slots__ replaces that dict with a fixed, compact layout — often a large memory saving and a small speed win, at the cost of not being able to add new attributes on the fly.¹⁰

class Point:
    __slots__ = ("x", "y")   # no per-instance __dict__
    def __init__(self, x, y):
        self.x, self.y = x, y

Metaclasses — and when not to reach for them

Classes are objects too, and a metaclass is the class of a class: it controls how classes themselves are created, letting you inspect or rewrite a class at definition time.¹¹ They power frameworks — registering subclasses, validating APIs, injecting methods. But they're heavy machinery, and most goals that used to need a metaclass are now better served by __init_subclass__, a hook that runs whenever a class is subclassed, with none of the conceptual overhead.¹²

class Plugin:
    registry = []
    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        Plugin.registry.append(cls)   # auto-register every subclass

Context managers at scale

You know with for a single resource. At scale, contextlib gives you more: the @contextmanager decorator to write one from a generator, suppress() to swallow a specific exception cleanly, and ExitStack to manage a dynamic number of context managers — open an unknown list of files and have every one guaranteed to close, even if the third fails to open.¹³

from contextlib import ExitStack

with ExitStack() as stack:
    files = [stack.enter_context(open(p)) for p in paths]
    # all files close on exit, in reverse order, whatever happens

Concurrency: the GIL, threads, and processes

To run things at the same time you first have to know about the GIL (Global Interpreter Lock): in the standard CPython interpreter, a single lock allows only one thread to execute Python bytecode at a time.¹⁴ That single fact decides your tool:

• I/O-bound work (network calls, disk, waiting) → use threads. While one thread waits on I/O, the GIL is released and others run. The threading module and the friendlier concurrent.futures.ThreadPoolExecutor cover this.¹⁵¹⁷
• CPU-bound work (crunching numbers) → use processes. multiprocessing (or ProcessPoolExecutor) sidesteps the GIL by running separate interpreters across multiple cores, at the cost of copying data between them.¹⁶¹⁷

async / await

For high-concurrency I/O — thousands of simultaneous connections — threads get expensive. asyncio offers a different model: a single thread runs an event loop that juggles many tasks cooperatively.¹⁸ You mark a function async def and await the slow points; at each await, the task voluntarily yields control so the loop can run another until its result is ready.¹⁹

import asyncio

async def fetch(url):
    await asyncio.sleep(1)      # stand-in for a network call
    return url

async def main():
    # all three run concurrently, not one-after-another
    return await asyncio.gather(fetch("a"), fetch("b"), fetch("c"))

asyncio.run(main())

The mental model: await is a cooperative pause, not a preemptive one. Async is superb for I/O-bound concurrency and does nothing for CPU-bound work — the event loop is still one thread.

The type system, leveled up

The intermediate tutorial introduced type hints. Advanced code makes them precise. Use generics so a container's element type flows through: a TypeVar lets you say “this function returns the same type it was given.”²⁰²¹ And use Protocol for structural typing — “any object with a .read() method,” regardless of its base class — which types Python's natural duck-typing without forcing an inheritance hierarchy.²²

from typing import Protocol, TypeVar

T = TypeVar("T")
def first(items: list[T]) -> T:   # returns the element type
    return items[0]

class Readable(Protocol):
    def read(self) -> str: ...   # anything with read() qualifies

Before you optimise anything

Everything here is power, and power invites premature use. The professional habit is the opposite: measure first. Reach for the cProfile profiler to find where the time actually goes before you touch a line — intuition about performance is wrong more often than it's right, and the real hotspot is rarely the clever bit you were tempted to rewrite.²³ Descriptors, metaclasses, and async are the right tools exactly when a concrete problem demands them — and needless complexity the rest of the time.