The Python Standard Library

Intermediate⏱80 minutes📖25 min readLesson 7 of 10

The Python Standard Library

Use the modules that ship with Python: paths, regular expressions, containers, iterators, function tools, dates, decimals, statistics, configuration formats and dependency graphs

A small reporting service needed to read a TOML file, count requests by status, sort a build graph and store a few rows of results. The first draft pulled in four packages for those jobs. The second draft deleted all four, because every one of them already ships with Python. The service lost no functionality, the install stayed small and the build stopped depending on four release schedules.

That is the promise behind the phrase "batteries included." The standard library covers files, text, dates, containers, iterators, compression, networking, databases and testing. It is not always the fastest option and it is not always the most complete one. The skill is knowing which problems it solves well, and when a dependency earns its place.

This lesson moves through the modules that intermediate code reaches for most often. It starts with the process boundary, then paths and text, then containers and lazy pipelines, then function tools, numbers and structured data. The final section applies them to two small systems and ends with a decision checklist.

Key terms

Standard library

The collection of modules distributed with every Python installation, documented at docs.python.org and importable without installing a package.

Batteries included

The design philosophy that common programming tasks should be solvable with modules that ship alongside the interpreter.

Extension module

A module implemented in compiled code such as C rather than in Python source, often for speed or to reach an operating system interface.

Regular expression

A pattern language for describing sets of strings, used to search, extract, validate and replace text.

Greedy quantifier

A repetition such as .+ that consumes as much input as possible and then gives characters back when the rest of the pattern needs them.

Possessive quantifier

A repetition such as .++ that consumes as much input as possible and never gives characters back, so it can prevent a match that a greedy quantifier would find.

Iterator algebra

The practice of combining small lazy iterator functions into a pipeline that processes one item at a time without building intermediate lists.

Memoisation

Caching the result of a function by its arguments so a repeated call returns the stored value instead of recomputing it.

Naive datetime

A datetime with no time zone information, which cannot be safely compared or converted to another zone.

Topological order

An ordering of graph nodes in which every node appears after all of the nodes it depends on.

Time zone database

The IANA data set of zone names and offset rules that zoneinfo reads to convert aware datetimes.

Sentinel

A unique object used to mark a special condition such as missing data, distinguished by identity rather than by value.

1. What "Batteries Included" Actually Means

The interpreter can tell you which modules belong to the standard library. The sys.stdlib_module_names attribute is a frozenset of names that is the same on every platform, and it includes modules that a particular build may not provide.¹

python

import sys

names = sys.stdlib_module_names
print(len(names))
print(sorted(name for name in names if name.startswith("toml")))
print("numpy" in names, "sqlite3" in names)

On CPython 3.14.4 the output is 297, then ['tomllib'], then False True. The count changes between releases as modules are added, so treat the number as a snapshot rather than a constant. Membership answers a different question from availability: sqlite3 is in the set even when the interpreter was built without the SQLite extension. Check availability with an import or with importlib.util.find_spec before depending on an optional module.

The set also separates three kinds of module. Pure Python modules are source files under the library directory. Built-in modules are compiled into the interpreter and listed by sys.builtin_module_names. Extension modules are compiled separately and loaded on demand. The distinction matters when you debug an import, because the three kinds fail in different ways and appear in different places.²

The library has grown with the language. The timeline below marks additions that this lesson uses, which is also a reminder that a tutorial older than your interpreter may be missing the shortest solution.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A timeline of standard library additions from Python 3.9 to 3.15. Python 3.9 added zoneinfo and graphlib, 3.10 added pairwise and Counter.total, 3.11 added tomllib, datetime.UTC and possessive quantifiers, 3.12 added batched and Path.walk, 3.13 added statistics.kde and Path.from_uri, 3.14 added Path.copy and functools.Placeholder, and 3.15 added frozendict, sentinel and math.integer.

The first question is whether the task is stable and small. Reading a CSV file, formatting a date, walking a directory and parsing a query string are stable and small, so the standard library is usually the right answer. The second question is whether the task is dominated by performance, protocol coverage or a changing external specification. HTTP clients, image codecs and numerical arrays change often and benefit from dedicated maintainers, so a dependency is often the right answer there.

ℹ️Note

sys.stdlib_module_names describes the standard library, not the modules that are importable in the current build. A name in the set can still fail to import on a platform that lacks the underlying extension.

The standard library also has a compatibility promise that third-party packages do not. A module that was present in Python 3.9 is very likely present in 3.15 with the same core behaviour. That stability is worth more in a long lived script than a small speed advantage, which is why the default should be to look here first and to add a dependency only for a reason you can name.

2. os and sys: The Process Boundary

A script that works in a terminal can fail under cron, in a container or in a scheduler, because the working directory, the environment and the argument list are all different. os and sys expose those boundaries.

python

import os
import sys

print(os.getcwd())
print(os.environ.get("APP_MODE", "development"))
print(sys.argv)
print(sys.version_info[:2])

The output depends on where the script runs, which is the point. os.getcwd returns the directory the process started in, not the directory that contains the script. sys.argv[0] is the script name and the remaining items are the arguments as strings. Parse them with argparse rather than indexing by hand, because a missing argument should produce a usage message instead of an IndexError.

os.environ is a mutable mapping backed by the process environment. Reading a missing key raises KeyError, so get with a default is the safe form. Assigning to the mapping changes the environment that child processes inherit, but it cannot change the environment of the shell that started Python. A required variable should stop the program at startup, not fail on first use.

python

import os

required = ("APP_MODE", "DATABASE_URL")
missing = [name for name in required if not os.environ.get(name)]

if missing:
    raise SystemExit(f"missing environment variables: {', '.join(missing)}")

A SystemExit raised with a string prints the message to standard error and exits with status 1. That is the correct failure mode for a command line tool. Calling sys.exit(0) is the explicit success path, and it still runs finally blocks because it raises an exception rather than terminating the process immediately.³

os.scandir returns directory entries with cached type and stat information, which avoids a separate os.stat call for every name. os.replace renames a file atomically when both paths are on the same filesystem, which is the primitive behind a safe write to a temporary file and rename.³

⚠️Warning

Environment variables are inherited by every child process and are often visible in crash reports, container inspection and process listings. Keep secrets out of the environment when the platform offers a secret store, and never print the full environment in a log line.

3. pathlib: Paths as Objects

A path is not a string with slashes in it. It has a name, a suffix, a parent and a meaning that differs between operating systems. pathlib models those parts as attributes and operators, which removes a class of bugs built from os.path.join and manual splitting.⁴

python

from pathlib import Path

report = Path("/srv/reports/2026/q3/summary.csv")
print(report.name, report.stem, report.suffix)
print(report.parent / "archive" / "summary.csv")
print(report.with_suffix(".json"))

The output is summary.csv summary .csv, then /srv/reports/2026/q3/archive/summary.csv, then /srv/reports/2026/q3/summary.json. The / operator joins path components and returns a new Path, so the original object is never mutated. with_suffix replaces only the final suffix, which is usually what a report exporter wants. glob, rglob and iterdir list matching paths, and sorting the results keeps output reproducible.

The method that matters for security is resolve. It makes a path absolute and follows symbolic links, while absolute does not resolve links. is_relative_to then checks whether one path is inside another. Together they form the safe way to join untrusted input to a base directory.

python

from pathlib import Path

base = Path("/srv/uploads").resolve()
candidate = (base / "../etc/passwd").resolve()

if not candidate.is_relative_to(base):
    raise ValueError("path escapes the upload directory")

The check happens after resolution, so .. components and symbolic links are already expanded. Checking the raw string for .. is not enough, because an encoded separator or a link can reach the same place. Since Python 3.14, Path.copy, Path.copy_into, Path.move and Path.move_into cover common file operations that previously required shutil.⁴

🚨Danger

Path is a value type, not a security boundary. A path built from user input can point anywhere the process can reach. Resolve it, compare it with the allowed base, and reject anything outside before opening it.

The full path toolkit, including reading, writing and atomic replacement, is in lesson 4. This section keeps only the parts that differ from plain string handling.

4. re: Text That Must Be Exact

Regular expressions describe a set of strings. They are the right tool for validation, extraction and replacement when the shape of the text is stable. They are the wrong tool for parsing HTML, JSON or a programming language, because those formats have nested structure that a pattern cannot track. Use a parser for structured formats and a pattern for the flat text around them.

python

import re

pattern = re.compile(r"(?P<key>\w+)=(?P<value>[\w.]+)")
text = "a=1 b=2.5"

print(pattern.findall(text))
print([match.groupdict() for match in pattern.finditer(text)])

The output is [('a', '1'), ('b', '2.5')], then a list of two dictionaries with key and value. findall returns tuples when the pattern has more than one group, which is convenient for a quick extraction and easy to misread when the group order changes. finditer returns match objects with positions, named groups and the original text, so it is the better choice when the surrounding code needs more than the captured strings.⁵

The four matching functions answer different questions. search looks anywhere in the string, match requires the match to start at index zero, fullmatch requires the entire string to match, and findall returns every non-overlapping match. The distinction is a common source of silent bugs.

python

import re

print(re.search(r"\d+", "order=42").group())
print(re.match(r"\d+", "order=42"))
print(bool(re.fullmatch(r"\d{4}-\d{2}-\d{2}", "2026-11-07")))

The output is 42, then None, then True. The pattern is correct in all three calls; only the anchoring rule changed. A validation function should almost always use fullmatch, because search accepts a valid substring inside an invalid whole.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A decision flowchart for choosing a regular expression function. Start with the question of whether the whole string must match, which leads to fullmatch. If not, ask whether the match must start at index zero, which leads to match. Otherwise choose search for one match anywhere, findall for every match as strings, finditer for every match as objects, or sub when matched text must be replaced.

Replacement can take a function instead of a string. The function receives the match object and returns the replacement, which keeps the transformation close to the pattern and avoids a second pass over the text.

python

import re


def redact(match: re.Match[str]) -> str:
    digits = match.group(0)
    return digits[:2] + "*" * (len(digits) - 2)


print(re.sub(r"\b\d{16}\b", redact, "card 4111111111111111"))

The output is card 41**************. The pattern matches sixteen digits with word boundaries on both sides, and the function keeps the first two digits while masking the rest. A card number should not be logged in full, so the replacement function is both a formatting choice and a privacy control.

Greedy quantifiers consume as much as possible and then backtrack. On the text <a><b>, the pattern <.+> matches the whole string, while <.+?> matches <a> and then <b>. Possessive quantifiers and atomic groups, available since Python 3.11, consume without giving anything back, which can prevent a match that a greedy pattern would find.⁵

python

import re

print(re.findall(r"a+a", "aaa"))
print(re.findall(r"a++a", "aaa"))
print(re.findall(r"(?>a+)a", "aaa"))

The output is ['aaa'], then [], then []. The greedy pattern backtracks one character so the final a can match. The possessive pattern keeps all three and has nothing left for the final a, so it fails. The atomic group behaves the same way. The practical use is a pattern that should fail fast instead of exploring an exponential number of splits.

💡Tip

Compile a pattern once when it is used in a loop, and give every group a name when the pattern has more than two groups. A named group makes the extraction readable at the call site and survives a change in group order.

The fill in the blank exercise checks the three functions that beginners most often confuse.

Fill in the Blanks

How to answer

Fill-in-the-blank exercise with 3 blanks. Type the missing words, then check your answer.

Complete the regular expression workflow.

Call re.blank once to build a reusable pattern, use re.blank when the match may appear anywhere, and use re.blank when the entire string must match.

Word Bank, drag words into the blanks above:

compile

fullmatch

match

findall

sub

A regular expression is code, and it deserves the same care. Keep the pattern next to the data it describes, add a comment with one example, and test the failure cases as well as the success case. A pattern that accepts every input is usually a bug, and a pattern that rejects a valid edge case is usually a missing test.

5. collections: Containers With a Purpose

The built-in list, dict, set and tuple cover most needs. The collections module adds containers for specific access patterns: counting, grouping, queueing and named records. Choosing the right one removes code and makes the intent obvious.⁶

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A class diagram of the collections abstract base classes. Iterable is the root for iteration, Container supports the in operator and Sized supports len. Collection combines all three. Sequence, Set and Mapping extend Collection, and MutableSequence, MutableSet and MutableMapping add mutation to each.

Counter counts hashable items. It is a dictionary subclass, so missing keys return zero instead of raising KeyError, and it has methods for the questions that counting usually raises: the most common items, the total count and arithmetic between counters.

python

from collections import Counter

words = ["apple", "banana", "apple", "cherry", "banana", "apple"]
counts = Counter(words)

print(counts)
print(counts["orange"], counts.total())
print(counts.most_common(2))
print(counts - Counter(["apple"]))

The output is Counter({'apple': 3, 'banana': 2, 'cherry': 1}), then 0 6, then [('apple', 3), ('banana', 2)], then Counter({'apple': 2, 'banana': 2, 'cherry': 1}). Subtraction keeps only positive counts, so a counter cannot report a negative quantity unless you use the subtract method. Since Python 3.15 the ^ operator computes the symmetric difference of two multisets.⁶

defaultdict calls a factory when a key is missing and inserts the result. It turns a three line check into one line, which is why it appears in grouping code.

python

from collections import defaultdict

groups = defaultdict(list)
for category, item in [("fruit", "apple"), ("fruit", "pear"), ("colour", "red")]:
    groups[category].append(item)

print(dict(groups))
print(groups["missing"])

The output is {'fruit': ['apple', 'pear'], 'colour': ['red']}, then []. The second line also inserts a new empty list under missing, which is the behaviour to remember. Reading a key to test for its presence changes the dictionary. Use key in groups when the check must not create anything.

deque is a double ended queue with constant time appends and pops at both ends. With maxlen it also becomes a sliding window that discards from the opposite end when it is full.

python

from collections import deque

recent = deque([1, 2, 3], maxlen=3)
recent.append(4)
recent.appendleft(0)
print(recent)
print(recent.popleft(), recent.pop())

The output is deque([2], maxlen=3), then 0 3. The append on the right drops 1, the append on the left drops 4, and the two pops remove 0 and 3, leaving 2. That is exactly the behaviour of a fixed size recent items buffer, and it is easy to get backwards without a test.

namedtuple creates a tuple subclass with named fields. It keeps tuple unpacking and indexing while adding readable attribute access and a helpful repr.

python

from collections import namedtuple

Point = namedtuple("Point", "x y")
point = Point(3, 5)

print(point, point.x, point[0])
print(point._replace(x=10))
print(point._asdict())

The output is Point(x=3, y=5) 3 3, then Point(x=10, y=5), then {'x': 3, 'y': 5}. Instances are immutable, so _replace returns a new point. The underscore prefix marks the helper methods as part of the namedtuple API rather than fields, and the leading underscore is a convention that tools respect.

Memory is the other reason to choose a container. The chart below measures the traced memory of 100,000 coordinate pairs stored five different ways, on CPython 3.14.4. Each measurement starts a fresh tracemalloc session, so the numbers describe the objects themselves rather than the interpreter baseline.

Memory per record by container type

Description

Bar chart of traced memory per record for five ways to store 100,000 coordinate pairs. A class with slots uses about 120 bytes per record, a tuple about 136, a namedtuple about 144, a plain class about 160 and a dict about 256.

The exact values depend on the interpreter, the platform and the allocator. The shape is what transfers: a dict costs roughly twice a namedtuple, a class with __slots__ is the smallest of the object variants, and a plain class sits between the slots class and the dict. Use a dict when keys vary at runtime, a namedtuple when the field set is fixed and small, and a dataclass when the record also needs behaviour. The first lesson covers the class trade-offs in more detail.

💡Tip

Convert a groupby result or a defaultdict value to a plain list before returning it from a function. Callers should not need to know which container implementation produced the data, and a plain list is easier to serialize and test.

6. itertools: Iterator Algebra

An iterator produces one item at a time. itertools supplies small functions that combine, slice, group and repeat iterators without materialising the whole sequence. The result is a pipeline that uses constant memory even when the input is a file with millions of lines.⁷

The video below is a long tour of the module.⁸ Watch the sections on chain, islice, groupby and the combinatoric iterators, and skip the parts you already use.

Corey SchaferWatch on YouTube

Description

A video tutorial that demonstrates the itertools module, including infinite iterators, chain, islice, groupby, combinations and accumulate, and explains when a lazy iterator saves memory over a list.

The functions fall into families. Infinite generators such as count, cycle and repeat need a limit, usually islice. Combiners such as chain, chain.from_iterable, zip_longest and product join inputs. Selectors such as islice, takewhile, dropwhile, filterfalse and compress decide which items continue. Groupers such as groupby, pairwise and batched reshape a stream. Combinatoric functions such as permutations and combinations enumerate arrangements.

python

import itertools

print(list(itertools.islice(itertools.count(10, 2), 4)))
print(list(itertools.pairwise([1, 2, 3, 4])))
print(list(itertools.batched(range(7), 3)))
print(list(itertools.accumulate([1, 2, 3, 4], initial=100)))

The output is [10, 12, 14, 16], then [(1, 2), (2, 3), (3, 4)], then [(0, 1, 2), (3, 4, 5), (6,)], then [100, 101, 103, 106, 110]. The last batch is short rather than padded, which is the behaviour to expect from batched. pairwise and batched are recent additions, so a library that supports older interpreters needs a fallback.⁷

groupby is the function most often used incorrectly. It groups consecutive items with the same key; it does not collect equal keys from the whole input. The data must be sorted by the key first, or the same key appears in several groups.

python

import itertools

data = [("b", 1), ("a", 2), ("b", 3)]
key = lambda item: item[0]

print([(name, len(list(group))) for name, group in itertools.groupby(data, key)])
print([(name, len(list(group))) for name, group in itertools.groupby(sorted(data), key)])

The output is [('b', 1), ('a', 1), ('b', 1)], then [('a', 1), ('b', 2)]. The first result has three groups because the two b items are separated by an a. Sorting first gives the two groups the caller expected. The group iterator is also invalid after the loop advances, so convert it to a list immediately if it must outlive the iteration.

The matching exercise connects each function to the job it performs.

Matching Question

How to answer

Matching exercise with 6 pairs. Match each item on the left with its answer on the right.

Match each itertools function to the behaviour it provides.

chain

Drop here

islice

Drop here

groupby

Drop here

pairwise

Drop here

batched

Drop here

accumulate

Drop here

Drag answers from here:

Concatenates several iterables into one stream

Takes a range of items from an iterator without indexing

Groups consecutive items that share a key

Yields overlapping pairs from one iterable

Yields fixed size tuples, with a short final tuple

Yields running totals or running combinations

The memory advantage comes from never building the intermediate list. A chain of islice, map and filter processes one item at a time, so a ten million line file can be summarized with a few hundred bytes of state. The cost is that an iterator can be consumed only once and cannot be indexed or measured without materialising it.

📌Important

An iterator is a one pass stream. If two consumers need the same data, either tee it with itertools.tee or store it in a list. Calling list on a large stream to inspect it defeats the reason for using a lazy pipeline.

A useful pattern is a generator that reads a file and yields batches, with itertools.batched or islice controlling the batch size. The consumer then processes one batch at a time, which keeps memory flat and gives the database or API a natural transaction boundary.

7. functools: Functions That Configure Functions

functools contains tools that treat callables as values. partial freezes arguments, cache and lru_cache store results, singledispatch chooses an implementation by type, and reduce folds a sequence into one value. The decorator lesson covered wraps; this section covers the rest.⁹

partial returns a new callable with some arguments already supplied. The original function is unchanged, and the partial can be called many times with different remaining arguments.

python

from functools import partial


def power(base: float, exponent: float) -> float:
    return base ** exponent


square = partial(power, exponent=2)
print(square(5))
print(square(5, exponent=3))

The output is 25, then 125. A keyword supplied at call time overrides the keyword stored in the partial, which is useful for a default that the caller may replace. Since Python 3.14, functools.Placeholder marks a hole in the positional arguments so a later call can fill it.

python

from functools import Placeholder, partial


def power(base: float, exponent: float) -> float:
    return base ** exponent


print(partial(power, Placeholder, 2)(3))

The output is 9. The placeholder takes the first positional slot, so the call supplies base and the partial supplies exponent. A trailing placeholder is rejected at call time, because there is no later argument to fill it.

functools.cache is the unbounded form of lru_cache. Both store results by argument tuple, so arguments must be hashable and the function must be pure with respect to those arguments. A cached function that returns a mutable object returns the same object to every caller.

python

from functools import cache


@cache
def fib(n: int) -> int:
    return n if n < 2 else fib(n - 1) + fib(n - 2)


print(fib(100))
print(fib.cache_info())

The output is 354224848179261915075, then CacheInfo(hits=98, misses=101, maxsize=None, currsize=101). The maxsize field is None because cache is the unbounded form; lru_cache(maxsize=128) reports the bound instead. The exact hit count depends on the call order, so treat it as a diagnostic rather than a stable test value. The important fact is that the recursion becomes linear instead of exponential.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A sequence diagram of a memoised Fibonacci function. The caller asks for fib of 6, the cache reports a miss, the function recurses into fib of 5 and fib of 4. The fib of 5 lookup hits the cache and returns without recomputation, the result for 6 is stored, and the value returns to the caller.

singledispatch builds a function whose behaviour depends on the type of the first argument. The base implementation handles the general case, and each registration handles a more specific type. Dispatch happens at call time and falls back through the method resolution order.

python

from functools import singledispatch


@singledispatch
def render(value: object) -> str:
    return f"object:{value}"


@render.register
def _(value: int) -> str:
    return f"int:{value}"


@render.register
def _(value: list) -> str:
    return f"list:{len(value)}"


print(render(4), render([1, 2]), render("x"))

The output is int:4 list:2 object:x. A bool is a subclass of int, so it dispatches to the int implementation unless a bool registration exists. That is the method resolution order doing its job, and it is worth a test when the distinction matters.

reduce folds a sequence from left to right with a two argument function. It is the functional relative of a loop, and it is clearest when the operation is associative and short.

python

from functools import reduce

print(reduce(lambda a, b: a * b, [1, 2, 3, 4, 5]))
print(reduce(lambda a, b: a + b, ["a", "b"], ""))

The output is 120, then ab. The third argument is the initial value, which also determines the result type when the sequence is empty. Without it, an empty sequence raises TypeError, which is the correct behaviour for a reduction with no identity.

⚠️Warning

A cache keyed by arguments holds a strong reference to every argument and result until it is cleared or evicted. A cache on a method that receives a large object can keep that object alive for the life of the process. Bound the cache and measure its size, or clear it at a request boundary.

8. Dates, Decimals and Statistics

Three areas produce quiet, expensive bugs: time zones, money and averages. Python has a dedicated module for each. None of them requires a dependency for ordinary work, and all of them reward using the right type from the start.

datetime distinguishes naive from aware values. A naive value has no time zone, so it cannot be compared safely with an aware value or converted to another zone. Store timestamps as aware UTC values, and convert to a local zone only for display.¹⁰

python

from datetime import datetime, timedelta, timezone

start = datetime(2026, 11, 7, 9, 0, tzinfo=timezone.utc)
print(start + timedelta(hours=36))
print((start + timedelta(hours=36)).astimezone(timezone(timedelta(hours=-5))))

The output is 2026-11-08 21:00:00+00:00, then 2026-11-08 16:00:00-05:00. Since Python 3.11, datetime.UTC is a shorter alias for timezone.utc. Arithmetic on aware values keeps the offset unless a zone rule applies, which is exactly why named zones from zoneinfo are better than a fixed offset for anything involving daylight saving.

python

from datetime import datetime
from zoneinfo import ZoneInfo

meeting = datetime(2026, 11, 7, 14, 0, tzinfo=ZoneInfo("Europe/Berlin"))
print(meeting)
print(meeting.astimezone(ZoneInfo("America/New_York")))

The output is 2026-11-07 14:00:00+01:00, then 2026-11-07 08:00:00-05:00. zoneinfo reads the system time zone database when one is available and falls back to the first party tzdata package otherwise. Windows systems commonly need that package, so a cross platform deployment should declare it explicitly.¹¹

Decimal represents decimal fractions exactly, which is what money needs. Binary floating point cannot represent 0.1 exactly, so repeated addition and rounding drift away from the expected value.¹²

python

from decimal import Decimal, ROUND_HALF_UP

print(Decimal("0.1") + Decimal("0.2"))
print(Decimal("1.005").quantize(Decimal("0.01"), rounding=ROUND_HALF_UP))
print(round(1.005, 2))

The output is 0.3, then 1.01, then 1.0. The float result rounds down because the stored value is slightly below 1.005. That is not a Python bug; it is the same behaviour every IEEE 754 implementation has. Use Decimal with a string constructor, or store integer cents, and never use a float for a monetary total.

statistics provides pure Python functions for common summaries. fmean is the fast float mean, median handles even length sequences by averaging the two middle values, and linear_regression returns the slope and intercept of an ordinary least squares fit.¹³

python

import statistics

print(statistics.fmean([1, 2, 3, 4]))
print(statistics.median([1, 3, 2, 4]))
print(statistics.linear_regression([1, 2, 3], [2, 4, 6]))

The output is 2.5, 2.5, then LinearRegression(slope=2.0, intercept=0.0). mean is the slower general purpose version that preserves Decimal and Fraction inputs, while fmean always returns a float. Choose based on the input type, not on the name.

Randomness has two audiences. Simulations want random.Random with an explicit seed for reproducibility. Passwords, tokens and reset links want secrets, which reads from the operating system random source and cannot be predicted from earlier output.

ℹ️Note

statistics is for summaries, not for modelling. A regression with one predictor and a handful of points is fine; a model with interactions, regularization or uncertainty intervals belongs in a numerical library that documents those assumptions.

9. Structured Data and Dependency Order

Most programs eventually read a configuration file, exchange JSON, store rows or schedule work in dependency order. The standard library covers each of those jobs with a module that is small enough to learn in an afternoon.

JSON is the default interchange format. json.loads turns text into dictionaries, lists, strings, numbers, booleans and None. It accepts NaN and Infinity by default even though the JSON specification does not, so a strict producer should set allow_nan=False.

python

import json
from decimal import Decimal

payload = '{"total": 1.1, "items": [1, 2]}'
data = json.loads(payload, parse_float=Decimal)
print(data["total"] + Decimal("0.2"))
print(json.dumps({"total": 1}, separators=(",", ":")))

The output is 1.3, then {"total":1}. The parse_float hook keeps money exact instead of converting it to a float. separators removes the spaces that dumps adds by default, which matters when the output is a signature input or a compact log field.

CSV looks simpler than it is. Quoted fields can contain commas, newlines and escaped quotes, so splitting on a comma is not a parser. The csv module handles the dialect rules, DictReader maps the header row to keys, and newline="" prevents the file layer from translating line endings twice. Always pass an explicit encoding when opening the file, because the platform default is not portable.

TOML is a configuration format with comments, typed values and nested tables. tomllib parses it, and since Python 3.11 it implements TOML 1.0. The module is read only by design, so writing TOML requires a separate library.¹⁴

python

import tomllib

document = """
[server]
host = "localhost"
port = 8080

[[plugins]]
name = "auth"
"""

config = tomllib.loads(document)
print(config["server"]["port"], config["plugins"][0]["name"])

The output is 8080 auth. tomllib.load takes a binary file object and decodes UTF-8 internally, while loads takes a string. The result is a plain dictionary, so validation is still the caller's job. A missing key should produce a clear configuration error, not a KeyError from deep inside the application.

SQLite is an embedded database with the full SQL surface that small tools need. The sqlite3 module is a DB-API 2.0 driver, and it supports parameter binding, transactions and cursors without a server.¹⁵

python

import sqlite3

with sqlite3.connect(":memory:") as connection:
    connection.execute("create table orders (id integer primary key, total real)")
    connection.executemany("insert into orders (total) values (?)", [(10.5,), (3.25,)])
    print(connection.execute("select count(*), sum(total) from orders").fetchone())

The output is (2, 13.75). The ? placeholders bind values safely; string formatting an SQL statement invites injection and breaks on quotes. The with block commits on success and rolls back on failure, but it does not close the connection. Close it explicitly, or use contextlib.closing, when the connection is not process long.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A flowchart of a configuration pipeline. A TOML file is opened in binary mode, tomllib decodes it as UTF-8 and returns a dictionary, a validator checks required keys and types, the result becomes a typed configuration object, and a connection to SQLite stores the run record. A failure at validation returns a configuration error before any database work starts.

graphlib.TopologicalSorter orders tasks so that every dependency comes first. The input is a mapping from a node to the set of nodes it depends on. static_order returns a lazy iterator, and prepare with get_ready supports a scheduler that runs independent tasks in parallel.¹⁶

python

from graphlib import TopologicalSorter

graph = {"compile": {"test"}, "test": {"deploy"}, "deploy": set()}
print(list(TopologicalSorter(graph).static_order()))

The output is ['deploy', 'test', 'compile']. compile depends on test, which depends on deploy, so the deployment comes first in the returned order. A cycle raises CycleError, which is the correct place to fail rather than deadlocking a worker pool later.

The drag and drop exercise orders a configuration load from bytes to a typed object.

Ordering Question

How to answer

Ordering exercise with 5 items. Move them into the correct order, then check your answer.

Order the steps of loading a TOML configuration into a typed object.

Open the file in binary mode

Pass the file object to tomllib.load

Receive a dictionary of tables and values

Validate required keys and value types

Build the typed configuration object

📌Important

Parsing is not validation. A parser answers whether the text is well formed. A validator answers whether the values make sense for this application. Keep the two steps separate so a bad port number produces a configuration error, not a connection timeout an hour later.

10. Real-World Case Studies

The two cases below combine modules from the lesson. Each one is small enough to read in a sitting and realistic enough to show where the seams are.

Case 1: Triage a web server log

A support team needs a daily summary from an access log: total requests, status distribution and the busiest client addresses. The log is line based, so a compiled pattern extracts the fields and Counter and defaultdict aggregate them.

python

from collections import Counter, defaultdict
from pathlib import Path
import re

LINE = re.compile(
    r"(?P<ip>\d{1,3}(?:\.\d{1,3}){3}) "
    r"(?P<user>\S+) "
    r"\[(?P<time>[^\]]+)\] "
    r'"(?P<method>[A-Z]+) (?P<path>\S+) [^"]+" '
    r"(?P<status>\d{3}) (?P<size>\d+)"
)


def triage(path: Path) -> dict[str, object]:
    statuses: Counter[str] = Counter()
    paths_by_ip: defaultdict[str, list[str]] = defaultdict(list)

    for line in path.read_text(encoding="utf-8").splitlines():
        match = LINE.search(line)
        if match is None:
            continue
        record = match.groupdict()
        statuses[record["status"]] += 1
        paths_by_ip[record["ip"]].append(record["path"])

    busiest = Counter({ip: len(paths) for ip, paths in paths_by_ip.items()})
    return {
        "requests": statuses.total(),
        "statuses": statuses.most_common(),
        "top_ips": busiest.most_common(3),
    }


summary = triage(Path("/var/log/nginx/access.log"))
print(summary["requests"])
print(summary["statuses"])

The function reads the whole file into memory, which is fine for a daily log and wrong for a stream that never ends. The streaming version iterates over the file handle and keeps the same counters, so memory stays flat. The pattern is the part that deserves tests: a missing referrer field or a different log format should be skipped, not crash the report.

The Counter arithmetic is where the summary becomes useful. A status that never appears is simply absent, and most_common gives a stable order for equal counts only if the insertion order is stable. For a report, sort the final pairs explicitly by count and then by status code, so two runs over the same data produce the same output.

Case 2: Plan and record a release

A release tool reads a TOML manifest, orders the tasks by dependency and stores the planned run in SQLite. graphlib catches a cycle before any work starts, and the database gives the operator a record of what was scheduled.

python

import sqlite3
import tomllib
from graphlib import CycleError, TopologicalSorter

MANIFEST = """
[[task]]
name = "build"
needs = []

[[task]]
name = "test"
needs = ["build"]

[[task]]
name = "deploy"
needs = ["test"]
"""


def plan(text: str) -> list[str]:
    tasks = tomllib.loads(text)["task"]
    graph = {task["name"]: set(task["needs"]) for task in tasks}
    try:
        return list(TopologicalSorter(graph).static_order())
    except CycleError as error:
        raise ValueError("manifest contains a dependency cycle") from error


order = plan(MANIFEST)
print(order)

with sqlite3.connect("release.db") as connection:
    connection.execute("create table if not exists runs (task text primary key, status text)")
    connection.executemany(
        "insert or replace into runs (task, status) values (?, ?)",
        [(name, "pending") for name in order],
    )
    print(connection.execute("select count(*) from runs").fetchone())

The output is ['build', 'test', 'deploy'], then (3,). The TOML parser returns a list of tables, the graph maps each task to its prerequisites, and the sorter produces a safe order. The database write is a single transaction, so either every row is written or none is. If the manifest contains a cycle, plan raises before the connection opens, which keeps the failure close to the input.

The same shape scales to a real scheduler. Replace the eager static_order with prepare and get_ready, submit each ready task to a worker pool, and call done when it finishes. The graph module does not run the tasks; it only answers which ones are ready, which is the part that is easy to get wrong.

Choosing Modules With Two Questions

The first question is whether the standard library covers the task with an API you can keep stable. The second is whether the remaining gap is worth a dependency. Most projects answer yes to the first question far more often than they expect.

The quadrant below places the main container choices on two axes: how much the code needs named fields, and how much memory each record costs.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A quadrant chart of container choices by named field access and memory cost. The horizontal axis runs from positional access on the left to named access on the right, and the vertical axis runs from heavy memory at the bottom to compact memory at the top. A tuple sits top left, a namedtuple and a class with slots sit top right, a plain class sits mid right, and a dict sits bottom right.

For data formats, the same reasoning applies. JSON is the default for an API payload, CSV is for tabular exchange with a human audience, TOML is for configuration that people edit, and SQLite is for queries and transactions that outgrow a dictionary. Choosing the format before choosing the library usually makes the library decision obvious.

Pitfalls and a Review Checklist

The recurring mistakes are rarely about syntax. They are about using a tool outside its contract.

Reaching for a package before checking whether the standard library already covers the task.
Using re to parse HTML, JSON or source code instead of a real parser.
Using re.match when the intent was re.search, or search when the whole string had to match.
Reading a defaultdict key to test for presence and inserting an empty value.
Grouping an unsorted stream with groupby and getting repeated keys.
Caching a function that returns a mutable object, then mutating the cached result.
Using a float for money or a naive datetime for a timestamp.
Assuming with sqlite3.connect(...) closes the connection.
Unpickling data from an untrusted source, which can execute code.

📌Important

A short review before merging standard library code: is the module the right level of abstraction, is the encoding explicit, is the time zone explicit, is the money exact, and is every untrusted string either parameterized or parsed rather than formatted into a command?

2026 Updates (as of November 2026)

Python 3.15.2 is the current stable documentation set.¹⁷
Python 3.15 adds frozendict, an immutable dictionary that is hashable when its keys and values are hashable and that compares without regard to insertion order.¹⁸
Python 3.15 adds sentinel, a built-in type for unique marker values that survive copying and can be used in type expressions.¹⁹
Python 3.15 adds math.integer, a module of mathematical functions for integer arguments.²⁰
Counter supports the symmetric difference multiset operator ^ since Python 3.15, and Path gained copy, copy_into, move and move_into in Python 3.14.⁶⁴
functools.Placeholder since Python 3.14 leaves a positional hole in a partial, and itertools.batched since Python 3.12 chunks an iterable without padding the final batch.⁹⁷

The pattern across these releases is steady rather than dramatic. The library keeps absorbing small, well specified jobs that used to need a helper package: time zones, topological sorting, TOML parsing, batching and immutable dictionaries. Checking the current documentation before adding a dependency is now more valuable than it was five years ago.

📚 Did you know?

import this prints the Zen of Python, and import antigravity opens the relevant xkcd page in a browser.
sys.stdlib_module_names includes modules that are unavailable on some platforms, so membership is a statement about the library, not about the current build.
Counter is a dict subclass, which is why its insertion order follows the order in which keys were first seen.
The default decimal context carries 28 significant digits of precision, which is enough for most financial calculations and not a substitute for a documented rounding policy.