File Handling and I/O

Intermediate⏱75 minutes📖18 min readLesson 4 of 10

File Handling and I/O

Read and write text, binary and structured files safely: control encoding and newlines, stream large files, write atomically with a temporary file, and choose between CSV, JSON and JSON Lines

The nightly report writer opened its output file with open(path, "w") and wrote one row at a time. When the worker was restarted halfway through, the file on disk contained the first four thousand rows and no footer. The downstream loader read it happily, because a CSV file has no footer to miss. Finance spent a week reconciling a total that was short by a third.

The bug was not in the loop. It was in the choice to write directly over the only copy of the data. A file write is not a single operation, and a process that dies in the middle leaves a valid looking fragment. The same class of bug shows up with the wrong encoding, the wrong newline, and the wrong assumption about how much memory a file will need.

This lesson covers the four decisions that make file code reliable: text or bytes, which encoding, stream or load, and write in place or write then replace. It builds on the exception handling from lesson 3, because file operations are where most of those handlers live.

Key terms

File object

A Python object that wraps an operating system file and exposes read, write and seek methods.

Text mode

The mode that decodes bytes to str on read and encodes str to bytes on write.

Binary mode

The mode that returns and accepts bytes without any encoding or newline translation.

Encoding

The mapping between bytes and characters, such as UTF-8 or cp1252.

Newline translation

The translation of line endings between the platform convention and the \n character used in Python strings.

Buffer

A memory region that batches small reads and writes into larger system calls.

Streaming

Processing a file incrementally so memory use does not grow with file size.

Atomic replace

Renaming a complete temporary file onto the target so readers see either the old file or the new one.

Durability

The guarantee that written data survives a crash or power loss, which requires flushing and syncing.

JSON Lines

A text format with one complete JSON value per line, which allows streaming reads and appends.

BOM

A byte order mark at the start of a text file, handled by the utf-8-sig codec.

1. Text Mode and Binary Mode

open() returns one of two families of object. In text mode, which is the default, it returns a TextIOWrapper that decodes bytes into str on the way in and encodes str into bytes on the way out. In binary mode, it returns a buffered byte stream that reads and writes bytes with no translation at all.¹²

python

with open("notes.txt", "w", encoding="utf-8") as handle:
    handle.write("café\n")

with open("notes.txt", "rb") as handle:
    raw = handle.read()

print(raw)
print(raw.decode("utf-8"))

The first print shows b'caf\xc3\xa9\n', six bytes, because é is two bytes in UTF-8. The second prints café followed by a newline, because the decode step turns the bytes back into characters. The same file is text or bytes depending on how it is opened, and the two views are not interchangeable.

The diagram below follows a string through the encode and write path, then back through the read and decode path.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A flowchart of text and binary input output. A Python string is encoded to bytes with the chosen encoding, written through a buffer to the file, then read back as bytes and decoded to a string. A binary path skips encoding and decoding and returns bytes directly.

The rule is simple. Use text mode for text, and binary mode for images, audio, archives, executables and anything whose bytes are not a character encoding. Opening a PNG in text mode will usually fail with a UnicodeDecodeError, and on a permissive encoding it can silently corrupt the data.

⚠️Warning

Never open a binary format in text mode. The decoder may reject the bytes, or it may replace them and return a string that cannot be written back without changing the file.

The buffering argument controls how many small operations are batched. The default is fine for most code. Pass buffering=0 only in binary mode when every write must reach the operating system immediately, and pass buffering=1 in text mode when line buffering is required for a log.

2. Encoding and Newlines

For most of Python's history the default text encoding depended on the platform, which meant a file written on a Windows desktop could fail to read on a Linux server. Since Python 3.15, UTF-8 mode is enabled by default, so open() uses UTF-8 whether or not the locale agrees.³ The documentation still recommends passing encoding explicitly when the file format defines one, because a library or a future interpreter setting can change the default.¹

python

raw = "café".encode("utf-8")
print(raw)

try:
    raw.decode("ascii")
except UnicodeDecodeError as exc:
    print(exc.encoding, exc.start, exc.end, exc.reason)

The output is b'caf\xc3\xa9', then ascii 3 4 ordinal not in range(128). The position is a byte offset, not a character index, which matters when a single character occupies several bytes.

Two codecs deserve a special mention. utf-8-sig reads and writes a byte order mark, which some Windows tools add at the start of a UTF-8 file. cp1252 is the legacy Western European codec that older Windows software still produces. If a file looks like cafÃ© when it should say café, the bytes are UTF-8 and the reader used cp1252, or the reverse.

python

with open("legacy.csv", "w", encoding="cp1252", newline="") as handle:
    handle.write("name,city\n")
    handle.write("Ana,São Paulo\n")

The write succeeds because cp1252 can represent ã. It would fail on a character outside that codec, which is one more reason to standardise on UTF-8 for new files.

Newline translation is the second platform difference. In text mode Python translates \n to the platform line ending on write and translates the platform line ending back to \n on read. Passing newline="" disables that translation. Passing newline="\n" on write makes the output byte identical on every platform.

python

with open("output.txt", "w", encoding="utf-8", newline="\n") as handle:
    handle.write("first\nsecond\n")

The file now contains two \n bytes on Windows, Linux and macOS. That matters for checksums, for generated source code and for any file that another program will compare byte for byte.

📌Important

The csv module performs its own newline handling, so files passed to csv.reader or csv.writer must be opened with newline="". Without it, quoted fields that contain a newline are read incorrectly and Windows output gains an extra carriage return.⁴

3. Modes and the with Statement

The mode string answers two questions at once: what the program may do with the file, and what happens to existing content. The table below is the short version that covers almost every real call.

Mode	Reads	Writes	Creates	Truncates	Starts at
`"r"`	Yes	No	No	No	Start
`"w"`	No	Yes	Yes	Yes	Start
`"a"`	No	Yes	Yes	No	End
`"x"`	No	Yes	Yes, fails if present	No	Start
`"r+"`	Yes	Yes	No	No	Start
`"b"`	Modifier	Binary	n/a	n/a	n/a

The "x" mode is the safe way to create a new file. It raises FileExistsError rather than overwriting a file that appeared between the check and the open, which is the race that if path.exists(): open(path, "w") cannot close. The "a" mode is the safe way to append a log line, because the operating system positions every write at the end of the file.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A flowchart for choosing an open mode. Reading an existing file uses r, replacing a file uses w, appending uses a, and creating a file that must not already exist uses x. Adding b selects binary mode, and opening with a with statement guarantees the file closes.

The with statement guarantees that the file is closed when the block exits, including when the block raises. That guarantee is implemented by the context manager protocol, which lesson 6 covers in detail. For now the important part is that with replaces the manual try/finally pattern and removes the most common file descriptor leak.

python

with open("orders.csv", "w", newline="", encoding="utf-8") as handle:
    handle.write("order,total\n")
    handle.write("1001,4999\n")

The file is closed at the end of the block, and the data is flushed by the close. A program that opens thousands of files without closing them will eventually hit the operating system limit, which is usually 1024 descriptors on a Linux process. The with statement makes that failure almost impossible.

The video below demonstrates the same mechanics in a REPL, including what happens when a file is read twice without seeking back to the start.

Corey SchaferWatch on YouTube

Description

A video lesson that opens a file for reading and writing, shows how the file position advances as content is read, and demonstrates the with statement closing the handle automatically.

The matching exercise below connects each mode to its observable behaviour.

Matching Question

How to answer

Matching exercise with 5 pairs. Match each item on the left with its answer on the right.

Match each file mode to what happens when the file already exists.

Drop here

Drag answers from here:

Opens the existing file for reading and fails if it is missing

Truncates the existing file to zero bytes

Keeps the content and writes new data at the end

Raises FileExistsError instead of touching the file

Opens for reading and writing without truncating

🚨Danger

"w" truncates the file as soon as it is opened, before the first write. A crash between open and write leaves an empty file, not the previous version. Section 6 shows the atomic pattern that avoids this.

4. Reading Without Loading Everything

There are four common ways to read a text file, and they differ in memory use. read() returns the whole file as one string. readlines() returns a list of line strings. readline() returns one line. Iterating over the file object returns one line at a time and is the only option whose memory use stays flat as the file grows.

python

with open("access.log", encoding="utf-8") as handle:
    for line in handle:
        if " 500 " in line:
            print(line.rstrip())

The loop reads through an internal buffer and yields one line at a time. The rstrip() removes the trailing newline, which is present in every line except possibly the last one. For a file with ten million lines, the loop uses a few kilobytes of memory rather than several gigabytes.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A flowchart that chooses a reading strategy. If the whole file is needed as one value, read returns a string. If every line is needed at once, readlines returns a list. If the file is large or processed line by line, iteration streams one line at a time with constant memory.

Binary files use the same idea with a fixed size buffer. The walrus operator keeps the read and the test in one expression, and the loop stops when read returns an empty bytes object.

python

BUFFER_SIZE = 8192

with open("archive.tar", "rb") as source, open("copy.tar", "wb") as target:
    while chunk := source.read(BUFFER_SIZE):
        target.write(chunk)

The copy uses at most 8 kilobytes of memory regardless of the archive size. The shutil.copyfileobj function implements the same loop and also accepts a length, which is a better choice when the destination is an open file object.⁵ For a complete file copy that also preserves metadata, shutil.copy2 is the higher level option.

ℹ️Note

Iterating over a text file works because the file object is an iterator of lines. Mixing iteration with read() on the same handle is a common source of confusion, because the two share one file position.

The fill in the blank exercise checks the memory behaviour.

Fill in the Blanks

How to answer

Fill-in-the-blank exercise with 4 blanks. Type the missing words, then check your answer.

Complete the description of reading strategies.

Calling blank loads the entire file into one string. Calling blank loads every line into a list. Iterating with blank yields one line at a time and keeps memory flat. For binary files, a fixed size buffer and the blank operator keep the copy loop compact.

Word Bank, drag words into the blanks above:

read()

readlines()

for line in handle

walrus

readline()

seek()

with

yield

5. Writing Text and Structured Data

A write call accepts only a string in text mode and only bytes in binary mode. The write method returns the number of characters or bytes written, which is rarely needed because Python writes the whole buffer or raises. The writelines method takes an iterable of strings and writes them without adding separators, which surprises people who expect it to insert newlines.

python

rows = ["1001,4999", "1002,12900"]
with open("orders.txt", "w", encoding="utf-8", newline="\n") as handle:
    handle.writelines(line + "\n" for line in rows)

The generator adds the newline that writelines does not add. A plain list of strings without newlines would produce a single concatenated line.

For tabular data, use the csv module rather than joining strings. It handles quoting, embedded delimiters and embedded newlines, which hand written code almost always gets wrong.⁴

python

import csv

rows = [
    {"order": 1001, "total": 4999},
    {"order": 1002, "total": 12900},
]

with open("orders.csv", "w", newline="", encoding="utf-8") as handle:
    writer = csv.DictWriter(handle, fieldnames=["order", "total"])
    writer.writeheader()
    writer.writerows(rows)

The file contains order,total\r\n1001,4999\r\n1002,12900\r\n. The \r\n is the csv default line terminator, chosen for Excel compatibility. It is correct CSV, and a reader opened with newline="" handles it on every platform. Pass lineterminator="\n" when the consumer expects Unix endings.

Reading uses the matching class. csv.reader returns each row as a list, while csv.DictReader uses the header row as keys and returns a dictionary per row.⁴

python

with open("orders.csv", newline="", encoding="utf-8") as handle:
    reader = csv.DictReader(handle)
    for row in reader:
        print(row["order"], row["total"])

The output is 1001 4999 and 1002 12900. Every value is a string, because the CSV format carries no type information. Convert with int or Decimal where the value is used, and decide explicitly what to do with a row that fails to convert.

⚠️Warning

DictReader fills a short row with None and puts extra fields in a list under the key None. A missing column does not raise. Validate the header and the row width when the input comes from another organisation.

JSON is the better fit when the data is nested or when another program will parse it. json.dump writes to a file object and json.dumps returns a string.⁶

python

import json

config = {"theme": "dark", "columns": ["order", "total"]}
with open("config.json", "w", encoding="utf-8") as handle:
    json.dump(config, handle, indent=2, ensure_ascii=False)

The file contains two space indentation and a real é rather than an escape sequence when the data contains one. The indent argument is for humans and for diffs, and it costs bytes. The default compact form is better for a machine that will parse the file immediately.

JSON has one structural limitation: a single document must be parsed as a whole. When the producer appends one record at a time, use JSON Lines instead, with one complete value per line. A reader can then process the file line by line and keep memory flat.

python

import json

with open("events.jsonl", encoding="utf-8") as handle:
    for line in handle:
        event = json.loads(line)
        print(event["order"])

The file remains appendable: each new record is one more line. The cost is that the file is not a single valid JSON document, so tools that expect one array will not read it without a conversion step.

The quadrant below places the four common formats by structure and by how easily they stream.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A quadrant chart of file formats. CSV and JSONL sit in the structured and easy to stream corner, JSON sits in the structured but hard to stream corner, and raw binary sits in the unstructured but easy to stream corner.

The positions are a judgement, not a measurement. CSV is structured but flat, JSON is structured and nested, JSON Lines keeps the structure and gains streaming, and raw binary has no schema at all. The next section shows why the choice affects failure recovery as much as it affects parsing.

6. pathlib and Safe Paths

pathlib.Path replaces most string manipulation of file names. The / operator joins path components, and the properties name, stem, suffix, parent and parts read them back without manual splitting.⁷

python

from pathlib import Path

path = Path("data/reports/summary.csv")
print(path.name, path.stem, path.suffix)
print(path.parent, path.with_suffix(".json").name)
print(path.parts)

The output is summary.csv summary .csv, then data/reports summary.json, then ('data', 'reports', 'summary.csv'). The with_suffix method replaces only the final suffix, which is safer than slicing the string at the last dot.

Creating a directory is explicit. mkdir(parents=True, exist_ok=True) creates missing parents and does not raise when the directory already exists. The exist_ok flag is the difference between an idempotent setup step and a crash on the second run.

python

output = Path("data/reports")
output.mkdir(parents=True, exist_ok=True)

Reading and writing small text files has a one line form. These methods accept the same encoding and newline arguments as open, and they close the file for you.⁷

python

text = path.read_text(encoding="utf-8")
Path("output.txt").write_text("hello\n", encoding="utf-8", newline="\n")

Finding files uses glob for one directory and rglob for the whole tree. The pattern language is the shell one, so *.csv matches every CSV file in the directory and **/*.csv is the recursive equivalent.

python

for report in Path("data").rglob("*.csv"):
    print(report, report.stat().st_size)

The loop prints every CSV file under data, with its size in bytes. The stat() call returns a result object with st_size, st_mtime and the mode bits, which is enough for a report or a freshness check.

⚠️Warning

Never join a user supplied string onto a base directory with + or os.path.join and assume the result stays inside it. A value such as ../../etc/passwd escapes the base. Resolve the candidate and check that the base is one of its parents before opening it.

The drag and drop exercise orders the steps of an atomic write, which the next section explains.

Ordering Question

How to answer

Ordering exercise with 5 items. Move them into the correct order, then check your answer.

Order the steps of an atomic file write from first to last.

Create a temporary file in the target directory

Write the complete new content

Flush the buffer and fsync the descriptor

Replace the target with os.replace

Delete the temporary file if any step failed

7. Atomic Writes and Durability

Opening the target with "w" truncates it immediately. Every crash between that open and the final close leaves a partial file, and every reader in that window sees the partial version. The fix is to write a complete temporary file and rename it over the target in one step.

python

import os
import tempfile
from pathlib import Path


def write_atomic(path: Path, text: str, encoding: str = "utf-8") -> None:
    fd, temp_name = tempfile.mkstemp(
        dir=path.parent, prefix=f".{path.name}.", suffix=".tmp"
    )
    try:
        with os.fdopen(fd, "w", encoding=encoding, newline="\n") as handle:
            handle.write(text)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temp_name, path)
    except BaseException:
        os.unlink(temp_name)
        raise

The temporary file is created in the same directory as the target, because os.replace is only guaranteed to be atomic within one filesystem. The flush pushes the Python buffer to the operating system, and fsync pushes it to the storage device. The final os.replace renames the complete file onto the target in one atomic operation.⁸

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A sequence diagram of an atomic write. The writer creates a temporary file in the target directory, writes and flushes the content, syncs the descriptor, then calls os.replace to swap it onto the target. Readers see either the old file or the new one, never a partial write.

The durability guarantee has a limit. fsync makes the file content durable, but the rename itself is a directory operation, and some filesystems need a second sync on the directory to make the new name durable. For a configuration file the simpler pattern is enough. For a database or an audit log, use the storage engine that already solves this problem.

The demo below shows the failure path. The first write succeeds, the second raises before the replace, and the original content survives.

python

settings = Path("settings.json")
write_atomic(settings, '{"theme": "dark"}')
print(settings.read_text(encoding="utf-8"))

try:
    write_atomic(settings, None)
except TypeError as exc:
    print(type(exc).__name__)

print(settings.read_text(encoding="utf-8"))

The output is {"theme": "dark"}, then TypeError, then the same JSON again. The temporary file is removed by the except branch, so the directory does not fill with orphans after repeated failures.

📌Important

Atomic replace protects readers from partial content. It does not protect against two writers. When several processes may write the same file, use a lock file or a database, because the last replace wins and the earlier writer's work disappears without an error.

8. Real-World Case Studies

Two shapes of file work appear in almost every service: a streaming transform that must not load the input, and a small state file that must never be read half written.

Streaming a large CSV into JSON Lines

The pipeline below reads a CSV of orders and writes one JSON object per line. Memory stays flat because both sides stream, and a malformed row is reported with its line number instead of aborting the whole run.

python

import csv
import json
from pathlib import Path


def convert(source: Path, target: Path) -> tuple[int, int]:
    written = 0
    skipped = 0
    with open(source, newline="", encoding="utf-8") as infile, open(
        target, "w", encoding="utf-8", newline="\n"
    ) as outfile:
        reader = csv.DictReader(infile)
        for row in reader:
            try:
                record = {
                    "order": int(row["order"]),
                    "total": int(row["total"]),
                }
            except (TypeError, ValueError):
                skipped += 1
                continue
            outfile.write(json.dumps(record) + "\n")
            written += 1
    return written, skipped

The reader exposes line_num for diagnostics, and the writer emits one compact JSON object per line. The function returns two counts so the caller can decide whether a high skip rate is a warning or a failure. Notice that the output file is opened once, not once per row, because reopening a file in "a" mode inside a loop is slow and can interleave badly with other writers.

The same pattern works for logs, exports and migrations. The important properties are a bounded buffer, a per row error policy, and a final count that makes silent data loss visible.

Replacing a configuration file safely

A command line tool stores its settings in a JSON file under the user's home directory. A crash during a save must not leave the user with an unreadable configuration, and a reader must never see a half written object.

python

import json
from pathlib import Path


def save_settings(path: Path, settings: dict) -> None:
    text = json.dumps(settings, indent=2, ensure_ascii=False) + "\n"
    write_atomic(path, text)


def load_settings(path: Path) -> dict:
    try:
        return json.loads(path.read_text(encoding="utf-8"))
    except FileNotFoundError:
        return {}
    except json.JSONDecodeError as exc:
        raise ValueError(f"{path} is not valid JSON") from exc

The save builds the complete text in memory before touching the disk, then uses the atomic helper from section 7. The load distinguishes a missing file, which is a normal first run, from a corrupt file, which is a real error. That distinction is the difference between a tool that starts with defaults and a tool that silently discards the user's settings.

write_atomic is deliberately small. If the project needs more, the standard library already has tempfile.NamedTemporaryFile with delete_on_close=False for the case where the temporary name must be reopened, and os.replace for the final step.⁹⁸

9. Failure Modes and a Review Checklist

Most file bugs are variations on a short list. The table below names each one, the symptom it produces, and the fix.

Failure	Symptom	Fix
Missing encoding	`UnicodeDecodeError` or mojibake on another machine	Pass `encoding="utf-8"` explicitly
Missing `newline=""` on CSV	Extra blank lines, broken quoted fields	Open CSV files with `newline=""`
Reading a large file with `read()`	Memory grows with file size	Iterate over the handle or read chunks
Writing over the only copy	Truncated file after a crash	Write a temp file and `os.replace`
Forgetting to close	Too many open files, unflushed data	Use a `with` block
Text mode for binary data	Corrupt image or archive	Use `"rb"` and `"wb"`
Checking `exists()` then opening	Race between check and use	Use `"x"` or handle the error
Joining user input onto a path	Path traversal outside the base	Resolve and check the parent

The checklist below is short enough to apply in review. Each question maps to a failure that a reader would notice.

Is the file opened in the mode that matches the data, text or binary?
Is the encoding explicit, and does it match the producer?
Is newline="" set for every CSV handle?
Does the read path stream when the file can be large?
Does every write either append safely or replace atomically?
Is every handle closed by a with block?
Does the code validate a path that came from outside the process?

ℹ️Note

A file that is written correctly but never flushed is still a file that can lose its last records. Closing the handle flushes the Python buffer, and fsync moves the data from the operating system cache to the device.

2026 Updates (as of November 2026)

Python 3.15.2 is the current stable documentation set.¹⁰
Since Python 3.15, UTF-8 mode is enabled by default, so open() uses UTF-8 for text files even when the locale says otherwise. Passing encoding="locale" restores the locale encoding, and PYTHONUTF8=0 or -X utf8=0 disables the mode for a whole process.³
Python 3.15 added the parent_mode parameter to Path.mkdir, which sets the permissions of intermediate directories when parents=True.¹¹
Python 3.15 added an array_hook parameter to json.load and json.loads, so a decoder can map JSON arrays to a custom type, including an immutable one when combined with object_pairs_hook.¹¹
tempfile.NamedTemporaryFile accepts delete_on_close since Python 3.12, which is the supported way to close a temporary file and reopen it by name before the context manager deletes it.⁹

📚 Did you know?

open() accepts an integer file descriptor as its first argument, and closefd=False leaves that descriptor open when the file object closes. That is how a library can wrap a socket or a pipe without taking ownership.
The csv writer defaults to \r\n because Excel expects it, which is why a CSV file produced on Linux still contains carriage returns.
Path.read_text() and Path.write_text() are convenience wrappers around open, so they accept the same encoding, errors and newline arguments as the built in function.⁷

📌Important

Comparative Verdict. Use text mode with an explicit UTF-8 encoding for human readable files, and binary mode for every other format. Stream when the file can exceed memory, and load whole only when the format requires it. Use CSV for flat tables that other tools must open, JSON for nested data that is written once, and JSON Lines for append friendly streams. Write in place only for append only logs, and use a temporary file plus os.replace for every file that a reader depends on.