Data Formats and Compression: CSV, JSON, Avro, Parquet, and ORC

Amazon Athena prices one thing: "Querying compressed data is faster and also cheaper because you pay for the number of bytes scanned before decompression" (Amazon Athena User Guide, Compression formats, accessed 2026-10-11). Everything else in a lakehouse design, partitions, workgroups, cluster sizes, lifecycle rules, is downstream of a decision you make in the first week and then live with for years: which file format, which compression codec, and how big each object should be.

AWS publishes the same decision as a three-step cost story on its own pricing page: one 3 TB, four-column table, one single-column query, billed at $5.00 per TB scanned as of Oct 2026. Uncompressed text = $15.00. Gzip the text 3:1 = $5.00. Gzip and convert to Parquet, so only one of four columns is read = $1.25. Same query, same answer, same data: a 12× spread created entirely by format and compression choices.

That is why this lesson comes before table formats, before ETL, before the query layer. Get the bytes wrong and no amount of engine tuning buys the money back.

In this lesson you will:

• read row versus columnar anatomy properly, row groups, column chunks, data pages, footers, min/max statistics, the optional page index, and see exactly where predicate pushdown happens;
• use the format matrix for CSV, JSON, Avro, Parquet, ORC and Protobuf: orientation, in-file schema, splittability, nested types, and which AWS engine will actually read them;
• apply the schema-evolution matrix per format, including why Parquet reads by name while ORC reads by index and what that costs you;
• explain encodings, dictionary + RLE, run-length encoding, the delta family, BYTE_STREAM_SPLIT, and why they run before compression;
• choose between Snappy, Gzip, Zstd and LZ4 with dated, third-party benchmark numbers in hand rather than folklore;
• size files into the 128 MB: 1 GB band and diagnose (then fix) the small-files problem;
• walk a when-to-use-which decision flow from raw landing zone to gold;
• work seven worked examples with real numbers as of Oct 2026: the Athena arithmetic, codec ratios, the micro-batch file-count disaster, a 420 GB CSV → Parquet CTAS, the JSON expansion factor, the Iceberg + ZSTD recommendation, and a hot-table codec decision;
• study named AWS customer cases, Vanguard's "data swamp" of Parquet files and Nasdaq's columnar lake, plus Woot.com and ZS;
• read the October 2026 update box (Glue 6.0 Iceberg v3, S3 Metadata, S3 Tables compaction), and answer twelve exam-style questions with four interactive checks.

---

1. Row versus columnar: the anatomy that sets your bill

1.1 A row format stores records, whole and repeated

CSV, TSV, NDJSON, Avro and Protobuf are row formats: a record's fields are kept together. The advantage is exactly what a producer needs: write the whole record, read the whole record, append the next record. The disadvantage appears the moment a query touches one field out of forty.

SELECT amount FROM events WHERE date >= '2026-01-01' against row-oriented text has to walk every byte of every row and skip the thirty-nine fields it does not want. There is no way to jump to column amount, because column amount does not exist as a contiguous region anywhere in the file.

JSON adds a second, quieter cost: keys repeat for every single record. The digest's format matrix puts it plainly: JSON has "no in-file schema (keys repeat per record)". Avro solves that half of the problem by embedding a JSON schema in the file (and keeping the writer's schema alongside the data for schema resolution), but it remains row-oriented.

1.2 A columnar format stores fields, whole and repeated

Parquet and ORC are columnar: a column's values are kept together, so a 2-of-40-column query reads roughly 2/40 of the data plus metadata. That is not a marketing ratio; it is the mechanism Athena bills for. Its own columnar-storage page says predicate pushdown "enables Athena queries to fetch only the blocks it needs … it uses statistics from data block predicates, such as max/min values, to determine whether to read or skip the block" (accessed 2026-10-11).

1.3 …and both are actually hybrids (PAX)

The honest description is neither "row" nor "columnar" but PAX: a hybrid that partitions rows horizontally first, then stores the columns of each partition separately. As the 2023 academic survey puts it: "both Parquet and ORC employ the PAX format. The DBMS first partitions a table horizontally into row groups. It then stores tuples column-by-column within each row group" (arXiv 2304.05028, 2023-11-07: third-party academic source).

Two consequences follow, and both are examinable:

1. "Row vs columnar" is a spectrum, not a switch. Reconstructing a full row still touches one locality unit (one row group, one stripe): just a much smaller one than an entire text file.
2. Unit size is a tuning knob, not a constant. Athena documents the defaults: "The default size for row groups is 128 MB, and for stripes, 64 MB … Decreasing the row group or stripe size … is not recommended" (accessed 2026-10-11).

1.4 Parquet file anatomy, end to end

Parquet's own format documentation describes a file that begins and ends with a 4-byte magic number PAR1, holds one or more row groups, and writes file metadata last so the writer makes a single pass. The reader's contract is the mirror image: "Readers are expected to first read the file metadata to find all the column chunks they are interested in" (parquet.apache.org/docs/file-format/, accessed 2026-10-11).

The parquet-format README adds two facts worth memorising: "A file consists of one or more row groups. A row group contains exactly one column chunk per column. Column chunks contain one or more pages", and nested data uses "the Dremel encoding with definition and repetition levels".

Sizing guidance differs by workload, and both numbers are official:

1.5 Statistics, the page index, and predicate pushdown

The footer carries Statistics {min_value, max_value, null_count, distinct_count} per column chunk (parquet.thrift, accessed 2026-10-11). A query predicate is evaluated against those ranges before any data page is read:

Parquet's page-index documentation describes the payoff precisely: "a page index is optional metadata for a ColumnChunk, containing statistics for DataPages that can be used to skip those pages", and "A single-row lookup in a row group … will only read one data page per retrieved column" (accessed 2026-10-11).

ORC's equivalent is not optional and is finer-grained by default. ORC provides "three level of indexes within each file: file level … stripe level … row level: statistics … for each set of 10,000 rows within a stripe" (orc.apache.org/docs/indexes.html, accessed 2026-10-11), and "As of Hive 1.2, the indexes can include bloom filters". The 10,000-row stride is Hive's orc.row.index.stride=10,000. ORC also sizes its compression blocks at 256 KiB (orc.compress.size=262,144) so an unneeded block is never decompressed at all.

• 📚 Did you know? Glue crawlers discover a file's schema by reading different parts of different formats: Avro "Reads the schema at the beginning of the file", Parquet "Reads the schema at the end of the file", ORC "Reads the file metadata", and JSON reads the beginning (AWS Glue classifiers, accessed 2026-10-11). That is why a truncated Parquet object with a missing footer is undecipherable to a crawler, while a truncated Avro object is not.

1.6 Worked example E1: the $15 →$5 → $1.25 arithmetic (as of Oct 2026)

AWS's own published example (aws.amazon.com/athena/pricing/, accessed 2026-10-11): a 3 TB table with four columns, one query that reads a single column, Athena on-demand at $5.00 per TB scanned.

Scale it: at 600 queries per month that is $9,000.00 versus $3,000.00 versus $750.00: a $8,250.00 monthly delta between the worst and the best design on identical data. Compression bought the first 3×; columnar layout plus predicate pushdown bought the next 4×.

Note the ordering. Compression helps in proportion to bytes scanned; column pruning multiplies it. You cannot get the $1.25 with gzip'd CSV alone, and you cannot get it from Parquet alone if every query selects all four columns.

---

2. The format matrix: CSV, JSON, Avro, Parquet, ORC and Protobuf

2.1 The core matrix

Six formats, four axes. Everything in the table is drawn from the AWS and Apache documentation cited in the digest (all accessed 2026-10-11).

Three rows deserve commentary.

Protobuf is not a queryable lakehouse format on Athena. Athena's supported-SerDe table lists Ion, Avro, Parquet, CSV, JSON, ORC, TSV and log formats, Protobuf is absent. What Protobuf does have is two legitimate AWS positions: as a Glue Schema Registry input ("Protocol Buffers (Protobuf) versions proto2 and proto3 without support for extensions or groups", accessed 2026-10-11), and as an Athena MSK connector input alongside JSON, CSV and AVRO. Land it, convert it, query it, in that order.

Firehose converts JSON and nothing else. "Amazon Data Firehose can convert the format of your input data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3 … If you want to convert an input format other than JSON, such as comma-separated values (CSV) … use AWS Lambda to transform it to JSON first" (Firehose docs, accessed 2026-10-11). With conversion enabled, Amazon S3 is the only destination, S3-side compression is disabled, and "Snappy compression happens automatically" using Hadoop-compatible framing.

NDJSON is the only Athena-friendly JSON shape. Athena's JSON SerDe is line-oriented, so one JSON object per line (JSON Lines) works and pretty-printed or array-wrapped JSON does not. Land NDJSON, not pretty JSON.

2.2 Splittability: the property people get backwards

The single most common misconception in this lesson: "Parquet is compressed, so it is not splittable." Wrong. AWS's own Top-10 Athena tuning blog (2017-03-24) states: "Parquet and ORC files are always splittable because these formats compress sections of the files separately, and have metadata that contains the locations within the files."

What is not splittable is compressed text:

• "most compressed text files are not splittable": a 2 GB gzip.csv is one split and therefore one worker;
• bzip2 and LZO are the exceptions that are splittable among text codecs, but "not recommended if you want performance and compatibility";
• the same blog's sizing rule: "aim for splits that are around 128 MB".

Athena reinforces the parallelism ceiling for compressed data: the number of files is the worker-count ceiling. One huge file means one worker regardless of cluster size.

2.3 Schema evolution: the matrix nobody teaches until it breaks

Athena publishes a per-format matrix of what survives which schema change (handling-schema-updates-chapter.md, accessed 2026-10-11). The mechanical reason behind the table is one line: "By default, Parquet will access columns by name and ORC by index (ordinal value)".

Reading the table as a designer rather than memorising it:

• Parquet read-by-name buys you add / remove / reorder and costs you rename. Flip parquet.column.index.access=true and you get rename at the price of add/remove/reorder. That is a schema-design decision, not a performance setting.
• ORC is the mirror image: rename works, structural change does not, until you flip orc.column.index.access=false.
• Avro goes furthest because its spec defines formal Schema Resolution: writer's schema plus reader's schema, promotions such as "int … long, float, double", "string is promotable to bytes", and defaults for fields the writer lacks: with the rule that "files or systems that store Avro data should always include the writer's schema."
• Iceberg largely escapes this table, because it stores field IDs and its own schema metadata (lesson 04). Raw files on S3 still obey every row of the matrix above.

2.4 Interactive: match the format to its traits

• 📚 Did you know? Athena will silently misread some compressed files because it keys on the file extension: "uppercase file extensions such as .GZ … are not recognized by Athena" (accessed 2026-10-11). A CLI script that writes part-00000.csv.GZ instead of .gz produces a table that lists files but scans nothing. Keep every extension lowercase.

---

3. Encodings: what happens before compression

Compression is the second pass. The first pass is encoding: converting typed values into a byte representation that exploits the shape of the column. Both Parquet and ORC apply this layer, and it is where most of the size win actually comes from.

3.1 Dictionary encoding + RLE: the low-cardinality workhorse

For a column with few distinct values (country codes, status flags, event names), the writer builds a dictionary of unique values and replaces each occurrence with a small integer ID. Those IDs are then run-length-encoded. The Parquet encoding registry calls the combination RLE_DICTIONARY (enum value 8).

The fallback rule matters more than the happy path: "If the dictionary grows too big … the encoding will fall back to the plain encoding" (parquet.apache.org, modified 2026-02-24). High-cardinality columns, user IDs, UUIDs, timestamps at microsecond precision, blow past the dictionary budget and silently drop to PLAIN, which is then compressed by the codec alone.

A measured example of that cliff: disabling dictionary encoding on a single column of a 10 M-row Parquet file added 18.79 MiB (aicodeinvest.com, 2026-07-19: third-party benchmark). Iceberg's default dictionary budget is write.parquet.dict-size-bytes = 2097152 (2 MB).

3.2 Run-length encoding and bit packing

RLE pays off only when values actually repeat: booleans, sorted status columns, and the definition/repetition levels that Dremel uses for nested data. Bit packing stores small integers in the minimum number of bits, and is SIMD-friendly.

The 2023 academic survey draws the practical line: Parquet switches to RLE only at run length ≥ 8, while ORC ships four separate integer encodings. In their tests that helped Parquet scan faster while ORC's finer zone maps helped selective point lookups (arXiv 2304.05028, 2023-11-07: third-party).

3.3 The delta family and float encodings

• DELTA_BINARY_PACKED (enum 5): deltas between sorted integers, very effective on sorted timestamps and sequence numbers.
• DELTA_LENGTH_BYTE_ARRAY (enum 6) and DELTA_BYTE_ARRAY (enum 7): for strings with similar prefixes and lengths.
• BYTE_STREAM_SPLIT (enum 9) and ALP (enum 10): for floating-point columns.

ORC's equivalent kit, from its own specification: "type specific readers and writers that provide light weight compression techniques such as dictionary encoding, bit packing, delta encoding, and run length encoding … Additionally, ORC can apply generic compression using zlib, or Snappy on top of the lightweight compression."

The full Parquet encoding registry as of Oct 2026: PLAIN=0, PLAIN_DICTIONARY=2 (deprecated), RLE=3, BIT_PACKED=4, DELTA_BINARY_PACKED=5, DELTA_LENGTH_BYTE_ARRAY=6, DELTA_BYTE_ARRAY=7, RLE_DICTIONARY=8, BYTE_STREAM_SPLIT=9, ALP=10.

---

4. Compression codecs: Snappy, Gzip, Zstd, LZ4

4.1 The trade, in one axis

Parquet's own compression page frames it as a spectrum: "The Parquet format supports several compression codecs covering different areas in the compression ratio / processing cost spectrum": SNAPPY, GZIP, LZO, BROTLI, LZ4 (deprecated Hadoop framing), ZSTD, LZ4_RAW (accessed 2026-10-11).

4.2 What AWS recommends, in AWS's own words

• Zstd for Iceberg. "We recommend using the ZSTD compression codec because it strikes a balance between GZIP and Snappy … Snappy might provide the best overall read and write performance but has a lower compression ratio than GZIP and ZSTD" (AWS Prescriptive Guidance, Apache Iceberg on AWS: general best practices, accessed 2026-10-11).
• Zstd levels. "levels from 1 to 22 … Athena uses ZSTD compression level 3 by default … We recommend using the default level of 3 or a level in the range from 6 to 9 … Use levels greater than 19 with caution." The compression_level property is supported only in Athena engine version 3 (accessed 2026-10-11).
• Gzip is broad and reliable. "GZIP … is the default write compression format for files in the Parquet and text file storage formats" for Hive tables engine v2/v3, and "provides good compression ratios and has wide range support" (accessed 2026-10-11).
• Zstd is the newer balanced one. "a newer compression format with a good balance between performance and compression ratio" (accessed 2026-10-11).

4.3 Dated third-party benchmarks (labelled as such)

None of the numbers below are AWS-published; all are dated and reproducible in principle.

10 M-row Parquet codec benchmark (aicodeinvest.com, 2026-07-19):

Two readings from that table: gzip was ~47× slower to write than zstd level 3 (44.363 ÷ 0.942) yet produced a larger file, and zstd level 9 bought 0.09 MiB for 2.5× the write time of level 3. Level choice is not free; that is exactly why Athena recommends 3 or 6 to 9 and cautions above 19.

Two public datasets, Parquet with and without codecs (jeronimo.dev, 2025-01-16, third-party): source CSVs of 1,761 MB and 2,983 MB became: uncompressed Parquet 564 / 760 MB; snappy 220 / 542; gzip 146 / 448; zstd 148 / 430; lz4_raw 209 / 547; lzo 215 / 518. Write seconds on the same two datasets: none 25.0 / 57.9; snappy 25.2 / 56.4; gzip 39.3 / 91.1 (roughly +50%); zstd 27.3 / 64.1; lz4_raw 24.9 / 56.5; lzo 26.0 / 56.1. Decompression overhead measured 10 to 20%.

Codec micro-benchmark (mamonas.dev, 2025-08-18, third-party): Snappy and LZ4 decompress at ">3.5 GB/s"; Zstd holds roughly 1 GB/s across levels; gzip is slowest; Zstd at level 19 writes extremely slowly.

Uber's production migration (Uber engineering blog, 2022-01-24, third-party): moving its top-10 tables "From GZIP to ZSTD … ~7% drops in storage size, while from SNAPPY to ZSTD, we see ~39% reduction", and "~~79% size reduction" against uncompressed. Compute fell too: 870,472 → 639,184 → 213,914 vCoreSeconds across Snappy → Gzip → Zstd in one measured quarter: a 75.4% reduction from Snappy to Zstd.

4.4 Interactive: the codec defaults, fill them in

• 📚 Did you know? "Snappy" appears twice in this lesson with two different jobs: it is a codec you choose for Parquet or ORC, and it is what Firehose applies automatically whenever record-format conversion is enabled ("However, Snappy compression happens automatically … The framing format for Snappy … is compatible with Hadoop"). That is why Firehose's docs tell you to set CompressionFormat to UNCOMPRESSED when conversion is on: otherwise you compress twice and pay for both.

4.5 Worked example E2: codec arithmetic on a 10 GB hourly staging table (as of Oct 2026)

A pipeline stages 10 GB per hour and the write path is CPU-bound. Using the third-party 10 M-row benchmark above (aicodeinvest.com, 2026-07-19) as a scaling proxy:

Reading of the numbers: Zstd level 3 beats Gzip on both axes in this run: smaller file and fifty times faster to write. There is no scenario in this table where Gzip wins; it is beaten on ratio by Zstd and beaten on speed by everything. Snappy is 2.7 GB/hour larger than Zstd-3 (≈ 65 GB/day, ≈ 23.7 TB/year across 365 days of continuous staging) for a write-path saving nobody can measure.

The defensible rules:

• write-bound staging → Snappy (or LZ4), because Firehose and Spark already default there and the ratio loss is the price of throughput;
• storage- or scan-bound curated tables → Zstd level 3, which is Athena's Iceberg default and AWS's Prescriptive Guidance recommendation;
• never choose Gzip for a Parquet write on write-speed grounds: Gzip exists in the wild because it is Athena's CTAS default, not because it is the best codec.

---

5. File sizing and the small-files problem

5.1 The band

Below roughly 128 MB per file, per-object overhead dominates: S3 GET/LIST cost, listing time, footer and dictionary setup, Glue partition metadata, and, the one Athena states directly, "For small files, the overhead of the columnar format outweighs the benefits." Above roughly 1 GB you start losing split parallelism and raising retry cost on a failed task.

5.2 Worked example E3: the micro-batch file-count disaster (as of Oct 2026)

A team writes 1 TB per day with 5-second micro-batches, believing micro-batches give freshness.

• Files per day: 86,400 s ÷ 5 s = 17,280 files/day
• Bytes per file: 1,000 GB ÷ 17,280 ≈ 58 MB: already below the band
• Over a month: ≈ 518,400 objects, each with its own footer, its own S3 GET, its own listing entry

Now coalesce to the course band:

• At 1 GB per file: 1,000 GB ÷ 1 GB = 1,000 files/day (a 17× reduction)
• At 128 MB per file: 1,000 ÷ 0.128 = 7,813 files/day (a 2.2× reduction)

The symptom you will actually see is not "slow" but planning time becoming a meaningful share of total query runtime: Athena enumerating hundreds of thousands of objects, building a plan, then scanning a tiny amount of data. The fix is compaction (lesson 04) or write sizing (lesson 08), never a return to CSV.

5.3 The small-files funnel, measured

AWS's own EMR Iceberg case (AWS Big Data Blog, 2023-10-03) is the cleanest number set in this course:

Every step of that funnel is a file-format-and-size fix, not a compute fix: no cluster was resized, no query was rewritten, no partition was added. The objects simply became the right format at the right size.

• 📚 Did you know? Columnar formats make small files worse, not better. Athena's performance-tuning page states it outright: "For small files, the overhead of the columnar format outweighs the benefits." A 40 KB Parquet file still carries a PAR1 magic pair, a footer with a full schema, statistics and dictionary state: overhead that can exceed the data itself. Plain 40 KB text at least has no footer to write.

---

6. When to use which: the decision flow

6.1 The flow

6.2 The layer-by-layer rule

6.3 Worked example E4: CSV → Parquet with Athena CTAS (as of Oct 2026)

Input: 420 GB of newline-delimited CSV at s3://lake/raw/ghcn/, 60 columns.

Stage 0: DDL the raw table and take a baseline.

Stage 1: CTAS. This is the official AWS pattern (Athena CTAS INSERT INTO ETL, plus the AWS Big Data Blog walkthrough of 2019-11-27):

One statement sets format, codec, partitioning and location, and auto-registers the table in the Glue Data Catalog. (CTAS accepted formats: PARQUET, ORC, AVRO, JSON, TEXTFILE: Parquet if you omit format.)

Stage 2: incremental. INSERT INTO ghcn_parquet SELECT … WHERE year_col = '2026';

Stage 3, the arithmetic. Applying the third-party two-dataset result (Parquet + codec ≈ 9 to 15% of CSV; jeronimo.dev, 2025-01-16, third-party, not an AWS guarantee):

• 420 GB × 0.09 → 420 GB × 0.15 = ≈ 38 GB: 63 GB on S3
• one-column query reads 63 GB ÷ 60 columns ≈ 1.05 GB
• cost: 1.05 GB at $5.00/TB ≈ $0.005 versus 420 GB at $5.00/TB ≈ $2.10: a ≈400× reduction per query

Stage 4, verify. Check Data Scanned in the Athena console, aws s3 ls --summarize on the silver prefix, and confirm objects land in the 128 MB, 1 GB band rather than in kilobytes.

Stage 5: guardrails. Workgroup data-usage limit, partition projection on year_col, and a Glue crawler schema-change policy of MergeNewColumns.

The published AWS CTAS walkthrough reached "Size of dataset after parquet with snappy compression: 1.2 GB", and its own note is the lesson: "the file sizes are pretty small. Merging them into larger files … would lead to faster query execution."

6.4 Worked example E5: the JSON expansion factor (as of Oct 2026)

Take one six-field sensor record and compare representations. This is worked arithmetic on an illustrative record, not a published benchmark: the fact it demonstrates (JSON repeats keys per record) is from the format matrix in §2.1.

CSV (52 bytes):

NDJSON (107 bytes):

• Expansion: 107 ÷ 52 = 2.06×: the same information, 106% larger
• Of the 107 bytes, 45 bytes (42%) are repeated field-name tokens ("ts":, "device":, "region":, "temp":, "status":, "seq":) plus braces: structure that never changes between records
• At 1,000,000 records/day: CSV ≈ 52 MB/day (≈ 19.0 GB/year), JSON ≈ 107 MB/day (≈ 39.1 GB/year): ≈ 20.1 GB/year of pure repetition, before any compression
• And the compression codecs treat both as opaque text: Gzip and Zstd compress the repetition somewhat, but only a columnar format removes it structurally by storing each field once per row group

The design consequence: JSON's repetition is not a storage problem you solve with a better codec, it is a representation problem you solve with a conversion at the silver boundary: exactly what Firehose's record-format conversion or a CTAS does.

6.5 Worked example E6: the Iceberg + ZSTD recommendation (as of Oct 2026)

AWS Prescriptive Guidance, Apache Iceberg on AWS: general best practices (accessed 2026-10-11), states the recommendation: "we recommend using the ZSTD compression codec because it strikes a balance between GZIP and Snappy … Snappy might provide the best overall read and write performance but has a lower compression ratio than GZIP and ZSTD." Athena's own gloss on the same pair: Zstd is "a newer compression format with a good balance between performance and compression ratio" (accessed 2026-10-11).

Iceberg's own configuration confirms it as the shipped default (iceberg.apache.org/docs/1.10.2/configuration/, accessed 2026-10-11):

And Athena already writes it: "ZSTD is the default compression for Iceberg tables … level 3 by default."

The arithmetic. Take a 10 TB Iceberg table and compare codecs using Uber's production measurements (Uber engineering blog, 2022-01-24: third-party):

At the $5.00/TB-scanned Athena rate, the codec win is a scan win, because Athena bills the bytes it reads off S3, "you pay for the number of bytes scanned before decompression" (accessed 2026-10-11). So the same 10 TB table scanned monthly costs about $50.00 uncompressed, about $46.50 if it had been Gzip (−7% to Zstd implies Zstd is 7% under Gzip), about $30.50 if it had been Snappy (−39%), and about $10.50 if it had been uncompressed and was written as Zstd (−79%). Those are third-party storage ratios applied to an AWS list rate, label them that way, and keep the two effects separate: the codec shrinks the bytes you scan; column layout plus predicate pushdown shrinks the number of columns and row groups you scan at all. E1's $1.25 figure came from both; this table only shows the first.

The level discipline: level 3 default, 6 to 9 if you have measured the trade, >19 with caution, and compression_level only on Athena engine version 3. Uber itself used level 19 for cold backfills and low levels on streaming paths: reads are constant across levels, writes are not (third-party, 2022-01-24).

6.6 Worked example E7: choosing a codec for a hot staging table (as of Oct 2026)

A team must choose one codec for a staging table that is written continuously and read by dashboards every five minutes. Decision inputs from the digest's numbers:

Verdict: Zstd level 3. It is 33% smaller than Snappy for a 7.5% write-time penalty, and Gzip is eliminated on both axes. If the table were a high-volume write-only ingest buffer with no analytical reads, Snappy would be the correct answer instead: the decompression-throughput figure is the only argument in Snappy's favour, and it binds only when reads dominate CPU.

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

Case A, Vanguard: a "data swamp" of Parquet files is still a data swamp

Vanguard (investment management; Financial Advisor Services "FAS 360"). In the AWS Big Data Blog dated 2026-03-18, Vanguard describes the starting point of its modernization in one phrase: "migrating from a fragmented 'data swamp' of Parquet files on Amazon Simple Storage Service (Amazon S3) to a structured, unified system." Two years after the work, AWS reports 20 TB in Amazon Redshift and another 150 TB in the S3 data lake, 600+ AWS Glue ETL jobs (a 30× increase), 300+ tables, 250+ Redshift materialized views, 500+ Tableau dashboards, and 500,000+ user queries per month. Vanguard is now "curating data products on the data lake using Apache Iceberg format and leveraging AWS Glue for metrics computation and data lake integration."

The lesson angle for this lesson is uncomfortable and important: the files were already Parquet. Format alone did not save them. Parquet without sizing discipline, without a table format providing ACID and snapshots, and without ownership turns a lake into a swamp just as effectively as raw CSV would: the format matrix in §2 is a necessary input, not a sufficient one. Vanguard's fix was to add the missing layer (Iceberg data products, owned by domains), not to change the file format.

Customer-reported figures from an AWS blog published 2026-03-18; not an AWS commitment.

Case B, Nasdaq: columnar lake files under an exchange's load

Nasdaq (exchange operator, financial services). Per AWS's case study (page year 2020, accessed Oct 2026), Nasdaq built "the foundation of a new data lake on Amazon Simple Storage Service (Amazon S3)" in 2018 and adopted Redshift Spectrum to "query data both in the data warehouse and in the Amazon S3 data lake": what AWS calls "a lake house architecture". The nightly ingest profile was "30 billion to 55 billion records and surpassing 4 terabytes" from thousands of sources; the lake held 15 terabytes of data on Amazon S3 queryable in place; outcomes reported were Redshift queries "32 percent faster" and reaching the 90% data-load mark 5 hours sooner. Robert Hunt, VP Software Engineering, is quoted: "Amazon S3 gave us a solution that enables zero contention between data loading and querying processes."

Why it belongs here: 4 TB per night, queried the next morning, is precisely the workload the format matrix exists to serve. Column layout plus predicate pushdown is what lets an exchange scan a night of records for a surveillance question without reading every column of every record, and Nasdaq's own "zero contention" quote is the storage/compute separation lesson from lesson 01 showing up at the byte level.

Customer-reported figures from an undated AWS case-study page (year tag 2020, accessed Oct 2026); not an AWS commitment.

Case C, Woot.com: why CSV broke and Parquet did not

Woot.com (retail). In the AWS Big Data Blog dated 2019-01-22, the team reports that exporting text columns to CSV "conflicted with the commas present in the text data. We opted to use AWS Glue to export data from RDS to S3 in Parquet format": Parquet is "unaffected by commas because it encodes columns directly." Two lessons in one line: CSV's delimiter fragility is a correctness risk, not just a performance one, and AWS Glue ETL writing format="parquet" is the second of the two sanctioned conversion routes alongside Athena CTAS.

AWS blog, 2019-01-22; dated, treat as historical evidence of the pattern rather than current pricing or performance.

---

Practice Questions