<!-- Markdown version of https://ocxly.com/python-advanced.html · auto-generated, may lag the live page -->

# Advanced Python: the machinery under the language

This is step three. Our [beginner tutorial](python-for-beginners.html) taught you to make things work; the [intermediate one](python-intermediate.html) taught you to write them the way a Python programmer would. This tutorial is about the layer beneath that fluency, the protocols and machinery that *make* Python behave the way it does. You'll see how `for` loops really work, how `@property` is built from a lower-level primitive, what a metaclass actually is, and how to make code run concurrently. You rarely need all of this at once, but knowing it is the difference between using the language and understanding it.

## The iterator protocol

Every `for` loop in Python is powered by one small contract, the **iterator protocol**: an object is *iterable* if it has `__iter__()`, which returns an *iterator*, an object with a `__next__()` that yields the next value and raises `StopIteration` when it's done.[1](#ref-1) Implement those and your own type works everywhere a list does: in `for`, in comprehensions, in `zip()`.

```
class Countdown:
    def __init__(self, start):
        self.start = start
    def __iter__(self):
        n = self.start
        while n > 0:
            yield n          # a generator IS an iterator
            n -= 1

for x in Countdown(3):
    print(x)      # 3, 2, 1
```

Notice the shortcut: because a generator function is already an iterator, writing `__iter__` as a generator (with `yield`) saves you from hand-writing `__next__` and the `StopIteration` bookkeeping. Once you think in iterators, the **`itertools`** module becomes a toolbox of lazy, composable building blocks (`chain`, `islice`, `groupby`, `product`) that operate on any iterable without ever building the whole sequence in memory.[2](#ref-2)

## Generators, deeper: delegation

You met generators in the last tutorial. The advanced move is **delegation** with `yield from`, which hands off to a sub-generator and transparently passes through its values (and return value).[3](#ref-3)[4](#ref-4) It turns nested iteration into one flat, readable line:

```
def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)   # recurse, streaming
        else:
            yield item

list(flatten([1, [2, [3, 4]], 5]))   # [1, 2, 3, 4, 5]
```

Because it's all lazy, chaining generators gives you memory-flat data *pipelines*: each stage pulls one item at a time from the one before it, so you can process a file larger than RAM through several transformations without ever materialising it.

## Decorators that take arguments, and functools

A plain decorator wraps a function. A **decorator with arguments** is one level up, a function that *returns* a decorator, so you can configure the wrapping:

```
import functools

def retry(times):
    def decorator(fn):
        @functools.wraps(fn)          # keep fn's name/docstring
        def wrapper(*args, **kwargs):
            for attempt in range(times):
                try:
                    return fn(*args, **kwargs)
                except Exception:
                    if attempt == times - 1: raise
        return wrapper
    return decorator

@retry(times=3)
def fetch(): ...
```

Always apply `@functools.wraps` to your wrapper, it copies the original's name, docstring and signature so introspection and debuggers still see the real function.[5](#ref-5) The rest of **`functools`** is a highlight reel of advanced tools: `lru_cache` memoises a function's results, `partial` pre-binds arguments to make a new callable, and `singledispatch` gives you type-based function overloading, pick an implementation based on the first argument's type.[5](#ref-5)[6](#ref-6)

## Descriptors: how @property actually works

Here's a thing that surprises people: `@property`, methods, `classmethod`, and every ORM's typed field are all built on one protocol you can use yourself, the **descriptor**. A descriptor is any object that defines `__get__` (and optionally `__set__`), placed as a *class* attribute; Python then routes attribute access on instances through it.[7](#ref-7) The modern `__set_name__` hook even tells the descriptor which attribute name it was assigned to.[8](#ref-8)

```
class Positive:
    def __set_name__(self, owner, name):
        self.attr = "_" + name
    def __get__(self, obj, objtype=None):
        return getattr(obj, self.attr)
    def __set__(self, obj, value):
        if value <= 0:
            raise ValueError("must be positive")
        setattr(obj, self.attr, value)

class Account:
    balance = Positive()      # validated on every assignment
```

Write the validation once as a descriptor and reuse it across dozens of fields. This is exactly the mechanism SQLAlchemy, Django models, and dataclasses' field machinery are built on, now you can see through it.

## The data model up close: dunders and __slots__

Python's **data model** is the set of `__dunder__` methods that hook your objects into language syntax.[9](#ref-9) You've used `__init__` and `__repr__`; the full set lets your objects respond to operators (`__add__`, `__eq__`, `__lt__`), to being called (`__call__`), to indexing (`__getitem__`), and to missing-attribute access (`__getattr__`). Implement the right dunder and your type behaves like a built-in.

When you create many instances of a small class, add **`__slots__`**. By default every instance carries a `__dict__` for its attributes, which is flexible but memory-hungry; declaring `__slots__` replaces that dict with a fixed, compact layout, often a large memory saving and a small speed win, at the cost of not being able to add new attributes on the fly.[10](#ref-10)

```
class Point:
    __slots__ = ("x", "y")   # no per-instance __dict__
    def __init__(self, x, y):
        self.x, self.y = x, y
```

## Metaclasses: and when not to reach for them

Classes are objects too, and a **metaclass** is the class *of* a class: it controls how classes themselves are created, letting you inspect or rewrite a class at definition time.[11](#ref-11) They power frameworks, registering subclasses, validating APIs, injecting methods. But they're heavy machinery, and most goals that *used* to need a metaclass are now better served by **`__init_subclass__`**, a hook that runs whenever a class is subclassed, with none of the conceptual overhead.[12](#ref-12)

```
class Plugin:
    registry = []
    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        Plugin.registry.append(cls)   # auto-register every subclass
```

## Context managers at scale

You know `with` for a single resource. At scale, **`contextlib`** gives you more: the `@contextmanager` decorator to write one from a generator, `suppress()` to swallow a specific exception cleanly, and `ExitStack` to manage a *dynamic* number of context managers, open an unknown list of files and have every one guaranteed to close, even if the third fails to open.[13](#ref-13)

```
from contextlib import ExitStack

with ExitStack() as stack:
    files = [stack.enter_context(open(p)) for p in paths]
    # all files close on exit, in reverse order, whatever happens
```

## Concurrency: the GIL, threads, and processes

To run things at the same time you first have to know about the **GIL** (Global Interpreter Lock): in the standard CPython interpreter, a single lock allows only one thread to execute Python bytecode at a time.[14](#ref-14) That single fact decides your tool:

- **I/O-bound work** (network calls, disk, waiting) → use **threads**. While one thread waits on I/O, the GIL is released and others run. The `threading` module and the friendlier `concurrent.futures.ThreadPoolExecutor` cover this.[15](#ref-15)[17](#ref-17)
- **CPU-bound work** (crunching numbers) → use **processes**. `multiprocessing` (or `ProcessPoolExecutor`) sidesteps the GIL by running separate interpreters across multiple cores, at the cost of copying data between them.[16](#ref-16)[17](#ref-17)

## async / await

For high-concurrency I/O, thousands of simultaneous connections, threads get expensive. **`asyncio`** offers a different model: a single thread runs an *event loop* that juggles many tasks cooperatively.[18](#ref-18) You mark a function `async def` and `await` the slow points; at each `await`, the task voluntarily yields control so the loop can run another until its result is ready.[19](#ref-19)

```
import asyncio

async def fetch(url):
    await asyncio.sleep(1)      # stand-in for a network call
    return url

async def main():
    # all three run concurrently, not one-after-another
    return await asyncio.gather(fetch("a"), fetch("b"), fetch("c"))

asyncio.run(main())
```

The mental model: `await` is a *cooperative* pause, not a preemptive one. Async is superb for I/O-bound concurrency and does nothing for CPU-bound work, the event loop is still one thread.

## The type system, leveled up

The intermediate tutorial introduced type hints. Advanced code makes them precise. Use **generics** so a container's element type flows through: a `TypeVar` lets you say “this function returns the same type it was given.”[20](#ref-20)[21](#ref-21) And use **`Protocol`** for *structural* typing, “any object with a `.read()` method,” regardless of its base class, which types Python's natural duck-typing without forcing an inheritance hierarchy.[22](#ref-22)

```
from typing import Protocol, TypeVar

T = TypeVar("T")
def first(items: list[T]) -> T:   # returns the element type
    return items[0]

class Readable(Protocol):
    def read(self) -> str: ...   # anything with read() qualifies
```

## Before you optimise anything

Everything here is power, and power invites premature use. The professional habit is the opposite: **measure first**. Reach for the `cProfile` profiler to find where the time actually goes before you touch a line, intuition about performance is wrong more often than it's right, and the real hotspot is rarely the clever bit you were tempted to rewrite.[23](#ref-23) Descriptors, metaclasses, and async are the right tools exactly when a concrete problem demands them, and needless complexity the rest of the time.

### References

1. [Python documentation — Iterator Types (the __iter__ / __next__ protocol and StopIteration)](https://docs.python.org/3/library/stdtypes.html#iterator-types)
2. [Python documentation — itertools (lazy, composable building blocks for iterables)](https://docs.python.org/3/library/itertools.html)
3. [Python documentation — Yield expressions (generators; yield and yield from)](https://docs.python.org/3/reference/expressions.html#yield-expressions)
4. [PEP 380 — Syntax for Delegating to a Subgenerator (yield from)](https://peps.python.org/pep-0380/)
5. [Python documentation — functools (wraps, lru_cache, partial)](https://docs.python.org/3/library/functools.html)
6. [Python documentation — functools.singledispatch (type-based single dispatch)](https://docs.python.org/3/library/functools.html#functools.singledispatch)
7. [Python documentation — Descriptor HowTo Guide (__get__/__set__; how property and methods are built)](https://docs.python.org/3/howto/descriptor.html)
8. [Python documentation — Data model: __set_name__ (descriptor name binding)](https://docs.python.org/3/reference/datamodel.html#object.__set_name__)
9. [Python documentation — Data model: Special method names (the dunder protocol)](https://docs.python.org/3/reference/datamodel.html#special-method-names)
10. [Python documentation — Data model: __slots__ (compact instance layout)](https://docs.python.org/3/reference/datamodel.html#slots)
11. [Python documentation — Data model: Metaclasses (controlling class creation)](https://docs.python.org/3/reference/datamodel.html#metaclasses)
12. [PEP 487 — Simpler customisation of class creation (__init_subclass__)](https://peps.python.org/pep-0487/)
13. [Python documentation — contextlib (@contextmanager, suppress, ExitStack)](https://docs.python.org/3/library/contextlib.html)
14. [Python documentation — Glossary: global interpreter lock (GIL)](https://docs.python.org/3/glossary.html#term-global-interpreter-lock)
15. [Python documentation — threading (thread-based parallelism, best for I/O-bound work)](https://docs.python.org/3/library/threading.html)
16. [Python documentation — multiprocessing (process-based parallelism, sidesteps the GIL)](https://docs.python.org/3/library/multiprocessing.html)
17. [Python documentation — concurrent.futures (ThreadPoolExecutor / ProcessPoolExecutor)](https://docs.python.org/3/library/concurrent.futures.html)
18. [Python documentation — asyncio (asynchronous I/O and the event loop)](https://docs.python.org/3/library/asyncio.html)
19. [PEP 492 — Coroutines with async and await syntax](https://peps.python.org/pep-0492/)
20. [Python documentation — typing (generics, TypeVar, Protocol and more)](https://docs.python.org/3/library/typing.html)
21. [PEP 484 — Type Hints (generics and type variables)](https://peps.python.org/pep-0484/)
22. [PEP 544 — Protocols: Structural subtyping (static duck typing)](https://peps.python.org/pep-0544/)
23. [Python documentation — The Python Profilers (cProfile: measure before optimising)](https://docs.python.org/3/library/profile.html)

---
*Source: [ocxly.com/python-advanced.html](https://ocxly.com/python-advanced.html) — OCXLY, free 100% client-side privacy-first tools. The third step after our beginner and intermediate tutorials. An advanced Python tutorial on the machinery under the language: the iterator protocol, generator delegation, functools, descriptors, dunder methods and __slots__, metaclasses, context managers at scale, concurrency (the GIL, threading, multiprocessing, asyncio), and the advanced type system. Cited to the official docs and PEPs.*