Glue Data Catalog: Crawlers, Partitions, and Schema Management

Lessons 02 through 06 put bytes on Amazon S3: storage classes, formats, table formats, batch and streaming ingestion. None of that is queryable by a SQL user until something else exists: a description of where the files are, what they contain, how to deserialize them, and which of them are worth reading. That description is the AWS Glue Data Catalog, and this lesson is about owning it: what a catalog object actually is, how a crawler writes one, how partitions and partition indexes keep GetPartitions from loading the world, how table versions let a schema change be audited and rolled forward, how the Schema Registry extends the same discipline to streams, and where Lake Formation takes the permissions away from you and gives them to a grant table.

In this lesson you will:

• read the Data Catalog object model, catalog, database, table, partitions, StorageDescriptor, SerDeInfo, SchemaReference, Parameters, TableType, and the byte limits that shape it;
• run a crawler in your head: custom classifiers in the order you specify, first success wins, certainty scoring, folder grouping heuristics, and the three policy knobs (RecrawlPolicy, SchemaChangePolicy, Configuration);
• price a crawler correctly, $0.44 per DPU-hour, per second, with a 10-minute floor, and know that the API is a free alternative;
• separate incremental crawls from S3-event crawls, including what each one is blind to;
• defend an exclusive-ownership design pattern for crawlers without pretending AWS ships an "exclusive crawler" toggle;
• use partition indexes the way the engine actually uses them: first-key rule, AND only, max three per table, and the Athena opt-in flag;
• operate table versions: what UpdateTable archives, why there is no rollback, and how a blue/green metadata deployment is assembled from a shadow table plus a resource-link swap;
• govern streaming schema with the Glue Schema Registry: hierarchy, quotas, the eight compatibility modes, and the producer/consumer SerDe path;
• hand off to Lake Formation cleanly: who owns objects, who owns grants, and what a resource link does and does not give you;
• wire Glue as the Iceberg catalog and read Athena's Iceberg metadata tables: including where Lake Formation blocks them;
• execute a crawl → register → Athena flow end to end, then classify a schema change as safe or breaking;
• study the BMW Group case and a sourced October 2026 update box;
• practise with 12 exam-style questions plus three interactive checks.

---

1. The Data Catalog object model

1.1 The tree, and the one sentence everything hangs on

AWS describes the Data Catalog as a service that "automatically captures and manages the schema of your data sources, including schema inference, evolution, and versioning", and lists its consumers as Amazon Athena ("Store and query table metadata in the Data Catalog for the Amazon S3 data using SQL"), Amazon Lake Formation ("Centrally define and manage fine-grained data access policies") and Amazon EMR ("Access data sources defined in the Data Catalog for big data processing") (AWS Glue Data Catalog docs, accessed 2026-10). One metadata layer, several engines; that is the entire architectural bet.

The load-bearing property is this: a catalog table is not data. It is a record of how to find and interpret data that lives somewhere else, almost always an S3 prefix. Deleting the table does not delete the objects; two databases can point at the same prefix; and a table whose Location is wrong still "exists" and still returns zero rows.

1.2 The table object field by field

Inside the StorageDescriptor the pieces you will actually touch are few:

• Columns: the inline column list. It can legally be empty, because…

• SchemaReference: "An object that references a schema stored in the AWS Glue Schema Registry. When creating a table, you can pass an empty list of columns for the schema, and instead use a schema reference." This is the hinge between the batch catalog and the streaming registry (Section 5).

• Location: the S3 URI (or JDBC/DynamoDB handle) the table reads.

• InputFormat / OutputFormat: Hive class names such as TextInputFormat / IgnoreKeyTextOutputFormat. The catalog is Hive-shaped on purpose: lowercase names, Hive I/O classes, Hive SerDe names.

• SerdeInfo: Name, Parameters, and SerializationLibrary, for example org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe, or LazySimpleSerDe (the built-in CSV classifier's default) and OpenCSVSerde (the one you switch to when CSV fields are quoted).

• Compressed, BucketColumns, SortColumns, SkewedInfo, StoredAsSubDirectories: Hive physical hints the engine may or may not exploit.

• 📚 Did you know? Table Parameters is where a crawler's findings land, and AWS tells you exactly which of them are meant for you: "We expect users to consume the classification and compressionType properties. Other properties, including table size estimates, are used for internal calculations, and we do not guarantee their accuracy." So recordCount, sizeKey, objectCount and averageRecordSize are hints for the crawler's own arithmetic, never inputs to a capacity plan. The two you should build on are classification (how to parse the files) and compressionType (whether an entire file was compressed, Parquet's internal compression deliberately does not populate it) (AWS Glue crawler table-properties docs, accessed 2026-10).

---

2. Crawlers: classification, classifiers, and the run order

2.1 The five steps AWS documents

AWS is blunt about the crawler's status: "You can use an AWS Glue crawler to populate the AWS Glue Data Catalog with databases and tables. This is the primary method used by most AWS Glue users." The documented workflow is fixed: custom classifiers run "in the order that you specify"; then built-ins; then the crawler connects to the store, infers the schema, and writes a table carrying the winning classifier's classification label. The rule that governs everything else: "The first custom classifier to successfully recognize the structure of your data is used to create a schema." Classifiers below the winner are skipped (AWS Glue crawler docs, accessed 2026-10).

2.2 Classifiers, certainty, and the CSV trap

A classifier "reads the data in a data store. If it recognizes the format of the data, it generates a schema. The classifier also returns a certainty number." The scoring is deterministic and worth memorising:

Built-in classifiers are tried in a documented order that runs through Avro, ORC, Parquet, JSON, BSON, XML, Amazon Ion, log formats, CSV, Redshift, MySQL, PostgreSQL, Oracle, SQL Server and DynamoDB. Compressed formats that can still be classified are ZIP (single-file archives only), BZIP, GZIP, LZ4 and Snappy.

CSV has three documented gates that catch people out: the schema must have at least two columns and two rows; header heuristics must parse (all-header columns read as STRING, under 150 characters except the last, regex-valid names, and the header must differ from the data rows) or the columns degrade to col1, col2, …; and the built-in CSV classifier emits LazySimpleSerDe, which mangles quoted strings: "if the CSV data contains quoted strings, edit the table definition and change the SerDe library to OpenCSVSerde."

And the trap with a long shelf life: "If you change a classifier definition, any data that was previously crawled using the classifier is not reclassified." Editing the rule does not rewrite history; you must point a crawler at the old data again.

2.3 Naming, grouping, and the empty-result trap

Crawler-generated names follow hard rules: "Only alphanumeric characters and underscore (_) are allowed. Any custom prefix cannot be longer than 64 characters. The maximum length of the name cannot be longer than 128 characters… If duplicate table names are encountered, the crawler adds a hash string suffix."

Grouping is a majority vote, not a rule: "When the majority of schemas at a folder level are similar, the crawler creates partitions of a table instead of separate tables. To influence the crawler to create separate tables, add each table's root folder as a separate data store." Get that wrong and you inherit AWS's explicit warning: "If objects have different schemas, Athena does not recognize different objects within the same prefix as separate tables… This might lead to queries in Athena that return zero results."

• 📚 Did you know? The crawler is the default path, not the only one. The pricing page says so in as many words: "Use of Amazon Glue crawlers is optional, and you can populate the Amazon Glue Data Catalog directly through the API." That matters for cost and for control: CreateTable writes exactly the schema you intend, with no classifier lottery, no folder majority vote, and no 10-minute billing floor. Teams that curate a handful of high-value tables should hand-write them and reserve crawlers for the long tail (AWS Glue pricing page and China-partition mirror, accessed 2026-10-11).

2.4 Recrawl policies: incremental versus S3-event

These are two different optimizations and they fail in two different directions.

The CLI shape for the incremental mode is --recrawl-policy RecrawlBehavior=CRAWL_NEW_FOLDERS_ONLY --schema-change-policy UpdateBehavior=LOG,DeleteBehavior=LOG.

2.5 Schema-change policy: three behaviours, one JSON document

By default "crawlers updates the schema in the Data Catalog to match the data source being crawled"; which means a retype in one new file can silently rewrite your table and mint a new version. The API knobs are SchemaChangePolicy.UpdateBehavior ∈ {LOG, UPDATE_IN_DATABASE} and DeleteBehavior ∈ {LOG, DELETE_FROM_DATABASE, DEPRECATE_IN_DATABASE}. The console's "Add new columns only" and "Ignore the change and don't update the table in the Data Catalog" are UI faces of the same fields, expressed in the Configuration document:

MergeNewColumns is the additive, safe behaviour; LOG is the freeze-and-let-humans-decide behaviour; the default UPDATE_IN_DATABASE is the one that can retype columns behind your back.

2.6 "Exclusive" crawlers: a design pattern, not an AWS feature

An important honesty note: AWS does not ship a crawler option called "exclusive" or "non-exclusive." If you see that framing, it is a team convention, not a console checkbox. What AWS does document are three hard, single-writer/single-target constraints:

1. Iceberg: "Use an Iceberg crawler only when it is the sole writer to a Data Catalog table. Do not use it alongside concurrent writers such as table optimizers, Apache Spark, or other engines that commit directly to the same table… If another writer modifies the Data Catalog table while the crawler is running, the crawler fails to update the table."
2. S3-event crawlers: "Only a single target is supported by the crawler, whether for Amazon S3 or Data Catalog targets."
3. Lake Formation-credential crawlers: "Currently, only a single catalog target with a single catalog table is allowed when using a Data Catalog table as a crawler's target."

Everything broader, "one crawler owns one table", "one table has exactly one writer", "the bronze prefix is crawled only by nightly_sales_crawler", is the exclusive-ownership pattern: a convention where you give each table a single crawler, a single TablePrefix, and a single Configuration, so version churn and classification races cannot occur. Adopt it deliberately; do not cite it as an AWS feature.

2.7 Worked example E1: what a crawler actually costs

The second bar assumes 30 event-triggered crawls a month, one per day of actual change, at the same $0.147 floor. Read it as a shape, not a forecast: you are paying for schedules, not for discovery.

---

3. Partitioning, partition indexes, and the projection lane

3.1 How partitions get into the catalog

Hive-style key=value/ folder segments are discovered by the crawler when schemas at a level are similar; the folder becomes a partition row keyed by those values, and the keys become the table's PartitionKeys (primitive types only). The catalog's stated job includes exactly this: it "is designed to compute statistics and register partitions to help make queries against your data efficient and effective. It is also designed to maintain a schema version history so you can understand how your data has changed over time."

Crawlers also build the performance layer for you: "A crawler creates partition indexes for Amazon S3 and Delta Lake targets by default", disable-able with --configuration '{"Version": 1.0, "CreatePartitionIndex": false}'.

3.2 What a partition index does, and when it does nothing

Without an index, "AWS Glue loads all the partitions of the table, and then filters the loaded partitions using the query expression." With one, the catalog fetches a subset. The rules are narrow and they are all testable:

And a lock-in warning: crawler-created tables "do not have the variable partition_filtering.enabled by default", so a crawler can build the index and Athena will still ignore it until you flip the property.

• 📚 Did you know? Partition indexes are billable metadata objects, and they multiply. A table with 50,000 partitions and 3 indexes consumes 50,000 partitions plus 150,000 index items: plus the table, its versions, its database. The Data Catalog's free tier counts "a table, table version, partition, partition indexes, statistics, database, or catalog" as separate objects, so index design is a cost decision as much as a performance one (AWS Glue pricing and partition-index docs, accessed 2026-10).

3.3 Partition projection: the lane that bypasses the catalog

There is a third way to get partition values into a query, and it is deliberately out of scope here: Amazon Athena partition projection, which computes partition values and locations at query time instead of reading registered metadata. AWS Glue's own feature list pairs it with partition filtering under the Athena heading and points at the Athena documentation for it. This lesson owns the catalog side: registered partitions and the indexes over them; the projection configuration itself belongs to the query-layer lesson (see Lesson 09). For now, hold one distinction: an index makes GetPartitions cheaper, projection makes GetPartitions unnecessary.

3.4 Worked example E2: the metadata-object bill

---

4. Table versioning and blue/green schema deployment

4.1 Versioning is automatic and forward-only

Every UpdateTable archives the previous row and increments VersionId: "A VersionId is a string representation of an integer. Each version is incremented by 1." Retrieval is GetTableVersion / GetTableVersions ("Retrieves a list of strings that identify available versions of a specified table", MaxResults 1 to 100), each entry being a full Table plus its VersionId. The service is documented to "maintain a schema version history so you can understand how your data has changed over time."

4.2 There is no rollback: only publish-forward

This is the fact teams discover in an incident. An AWS-answered re:Post thread (2023-11-22, reviewed by AWS) states: "It is not possible to 'rollback' to a previous version per se and also you cannot delete the latest version of the table (neither on the console nor via API/CLI) so the only option is to publish a new schema version that is identical to the one you want to restore back to." The documented route is console-side: navigate to the version you want, Edit Schema as JSON, then Save as new table version.

So "rollback" is a misnomer. Metadata deployment is append-only in practice: you can read every historical version, diff two of them, and publish a new one, but you cannot remove the head.

4.3 Two blue/green patterns, both assembled from documented parts

For data blue/green there is a cheaper primitive than either: Iceberg snapshots let the catalog entry keep its shape while the data rolls forward and back (Lesson 04).

4.4 Worked example E3: a metadata blue/green cutover with a resource link

---

5. Glue Schema Registry: compatibility for streams and batches

5.1 The object hierarchy and what it costs

The registry is the streaming counterpart to the catalog: "The AWS Glue Schema registry allows you to centrally discover, control, and evolve data stream schemas", with integrations to Apache Kafka, Amazon MSK, Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and AWS Lambda. The hierarchy is Registry → Schema → SchemaVersion → SchemaVersionMetadata, where a Schema carries a DataFormat, a Compatibility mode and a SchemaCheckpoint.

Compatibility is enforced at registration: for every mode except DISABLED, "validation of compatibility settings will be applied only from the second version onwards when the RegisterSchemaVersion API is used", and DISABLED "restricts any additional schema versions from being added after the first schema version."

5.2 The SerDe path, producer to consumer

Two properties of that diagram are easy to miss. First, the registry is not streaming-only: "you can create or update AWS Glue tables and partitions using schemas stored within the registry", and a catalog table may delegate its column list entirely to a SchemaReference. Second, cross-account access has a hard shape: "the AWS Glue Schema Registry doesn't support resource-based policies", so a cross-account registry is reached by assuming an identity-based role into the owning account, not by a resource policy.

5.3 Worked example E4: a compatibility drill on one Avro schema

• 📚 Did you know? The registry's free tier is not a promotion, it is the architecture: the Schema Registry is "serverless and free to use", while the Data Catalog next to it charges for objects and the crawler that fills it charges by the DPU-second. So the cheapest schema governance on AWS is the streaming one: a team can register thousands of Avro versions under BACKWARD_ALL for nothing and still spend six figures on crawlers scanning the same conceptual data into partitions (AWS Glue Schema Registry and pricing docs, accessed 2026-10).

---

6. Lake Formation handoff: who owns what

6.1 One catalog, two owners

The single sentence that resolves most confusion: "AWS Lake Formation metadata tables contain information about data in the data lake, including schema information, partition information, and data location. These tables are stored in the AWS Glue Data Catalog." Lake Formation does not have a second copy of your metadata; it has a permission layer over the same objects.

LF tables can be born five ways: "Run a crawler in AWS Glue", "Create and run a workflow", manual creation via the Lake Formation console, Glue API or CLI, "Create a table using Amazon Athena", or "Create a resource link to a table in an external account." And ownership starts at creation: "When a principal creates a table in the Data Catalog, all Lake Formation permissions on the table are granted to the principal, with the ability to grant these permissions to others." Price is a non-factor: "There are no separate charges when creating Lake Formation permissions or using Lake Formation permissions with integrated AWS services."

6.2 Resource links: the two-hop rule

A resource link is "a Data Catalog object that is a link to a local or shared database or table… you can use the resource link name wherever you would use the database or table name." Four rules decide whether your share works:

1. Link permissions are DESCRIBE and DROP only.
2. "Granting permissions on a resource link doesn't grant permissions on the target (linked) database or table. You must grant permissions on the target separately."
3. Creating one needs Lake Formation CREATE_TABLE/CREATE_DATABASE plus IAM glue:CreateTable/glue:CreateDatabase.
4. "When you create a resource link, no check is performed to see if the target shared resource exists… If you delete a resource link, the linked shared resource is not dropped."

Cross-account flow runs through AWS RAM: "If your account is in the same AWS organization as the granting account, the shared resource is available to you immediately. However, if your account is not in the same organization, AWS RAM sends an invitation." And a small API asymmetry worth knowing: DESCRIBE is implicitly granted with any other permission, but "You can't grant DESCRIBE to a user who has partial select on a table."

6.3 Crawling as an LF principal

A crawler can run with Lake Formation permissions rather than raw S3 IAM. The documented setup is: register the S3 location in Lake Formation; grant Data location permission to the crawler role; grant the role Create on the output database; add lakeformation:GetDataAccess to the role; then enable "Use Lake Formation credentials for crawling Amazon S3 data source" (LakeFormationConfiguration.UseLakeFormationCredentials=true). Remember the constraint from Section 2.6: only a single catalog target with a single catalog table is allowed in this mode.

---

7. Iceberg catalog wiring

7.1 Glue is the Iceberg metastore

There are two documented ways to get an Iceberg table into the catalog:

• Glue API: OpenTableFormatInput.IcebergInput with MetadataOperation=CREATE (Version defaults to 2). The Table object then carries IcebergTableMetadata: format version, schemas, partition specifications, sort orders.
• Amazon Athena: "Athena creates Iceberg v2 tables", and the tell is the property: set 'table_type' = 'ICEBERG' in TBLPROPERTIES. Attempting the external-table route fails: "Running CREATE EXTERNAL TABLE results in the error message External keyword not supported for table type ICEBERG."
• Glue crawler, point an IcebergTargets crawler at "an Amazon S3 path that contains the metadata for your Iceberg tables"; it "registers it in the Data Catalog" and "supports schema merging across snapshots and updates the latest metadata file location in the Amazon Glue Data Catalog." Support is Apache Iceberg 1.10, both v2 and v3 tables including VARIANT, UNKNOWN, Geography and Geometry, subject to the sole-writer rule from Section 2.6.

7.2 Athena metadata tables, and where Lake Formation cuts them off

Athena exposes Iceberg internals as pseudo-tables appended to the real name: $files, $manifests, $history, $partitions, $snapshots, $refs: SELECT * FROM "dbname"."tablename$files". Engine version 3 supports Parquet, ORC and Avro for Iceberg.

The catch is a governance boundary: "Lake Formation does not evaluate the Iceberg metadata tables. Hence, the Iceberg metadata tables are restricted if there are any Lake Formation row or cell filters present on the base table or if you do not have permissions to view all columns in the base table. For such cases, when you query the $partitions, $files, $manifests, and $snapshots Iceberg metadata tables, it fails and you get an AccessDeniedException error. Additionally, the metadata column $path has the same Lake Formation restrictions."

7.3 Worked example E5: promoting bronze to an Iceberg silver

---

8. Worked example E6: crawl → register → Athena, end to end

---

9. Worked example E7: schema evolution, safe versus breaking

9.1 The audit table

Start from a Hive-style external table events(id string, ts string, payload string) partitioned by dt, catalogued by a crawler.

9.2 The crawler-policy overlay

The same three-way split exists on the crawler side, and it decides whether a change arrives as a PR or as a surprise:

For Iceberg silvers, ALTER TABLE ADD/DROP/CHANGE COLUMN is metadata-only and rewrites nothing; which is why evolution belongs there rather than on a crawler-owned bronze.

• 📚 Did you know? The safe/breaking distinction is not cosmetic for cost: on a Hive-style external table a "safe" ADD COLUMNS costs nothing because no data file is rewritten, while on formats without schema-in-the-metadata the same change may force a rewrite. Iceberg sits at the cheap end, ALTER TABLE there is a metadata operation, which is why the recommended shape is evolve on the silver, freeze the bronze (AWS Athena ALTER TABLE and Iceberg schema-evolution docs, accessed 2026-10).

---

10. Glue version state: the platform under the catalog

The catalog itself is version-agnostic, but every crawler, job and SerDe library you attach to it runs on a Glue version, and as of October 2026 the ladder has a hard bottom rung.

Worked example E8: the upgrade path a catalog team actually runs

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

Case A, BMW Group: the catalog is the product, not the byproduct

BMW Group (connected vehicles). AWS's case study opens with the scale: "BMW Group uses AWS to process 10 TB of data daily from 1.2 million vehicles." The prior art was a 2015 attempt, "in 2015 the BMW Group created a centralized, on-premises data lake that collects and combines anonymized data from sensors in vehicles, operational systems, and data warehouses", which could not carry multi-tenant demand.

The line that makes this a Lesson 07 case rather than an architecture case: "The individual data layers are stored in Amazon S3 buckets, and their schemas are registered in the AWS Glue Data Catalog." Providers and consumers are separated on purpose: "Data providers ingest and transform data with AWS services such as Amazon Kinesis Data Firehose, AWS Lambda, AWS Glue, and Amazon EMR. Data consumers can then use services such as Amazon Athena, Amazon SageMaker, AWS Glue, and Amazon EMR." Note what is shared between the two sides: the catalog, not the accounts.

And the adoption layer on top of it: a Data Portal "offering a 'popularity index' based on data usage patterns for more than 500 users across the organization." Popularity is computed from usage of the catalogued data, which is the detail that turns a table list into a product.

Lesson angle: one Glue-cataloged S3 lake plus a human-readable portal is what actually democratizes a lake. Providers register schemas once; consumers in separate accounts discover them through the same metadata layer; the portal's popularity index answers the question a catalog alone cannot; which of these 300 tables is anyone using? BMW's 10 TB/day from 1.2 million vehicles is the volume; the 500-user portal is the adoption mechanism. Rebuild that shape with E6 (crawl and register) and E3 (share without copying).

Customer-reported figures from an AWS case-study page (undated, accessed 2026-10-11); not an AWS commitment.

---

Case B, Vanguard FAS 360: from data swamp to catalogued data products

Vanguard (Financial Advisor Services, "FAS 360"). AWS's Big Data Blog (2026-03-18) opens on the before-state: "Vanguard's first wave of modernization established FAS 360 as a centralized enterprise data warehouse, migrating from a fragmented 'data swamp' of Parquet files on Amazon Simple Storage Service(Amazon S3) to a structured, unified system." The catalog is what made the after-state survivable. Two years later the published numbers are: "20 TB in data volume in Amazon Redshift and another 150 TB in S3 data lake · 600+ AWS Glue ETL jobs (a 30x increase)… · 300+ tables (3x growth)… · 250+ Amazon Redshift materialized views optimizing query performance… 500,000+ user queries/months", and the curation layer on top: "Vanguard is curating data products on the data lake using Apache Iceberg format and leveraging AWS Glue for metrics computation and data lake integration."

Lesson angle: a 30× jump in ETL jobs and a 3× jump in tables is only safe when every job reads through the catalog, never through ad-hoc paths. BMW shows the catalog as the adoption surface for 500 humans; Vanguard shows the catalog as the contract for 600 automated jobs. Same Glue Data Catalog, two different reasons it exists: discovery for people, stability for pipelines. The Iceberg-curated data products are the endpoint of that contract: schema evolution becomes metadata-only (Section 9), and Glue computes the metrics that the Redshift multi-warehouse layer then serves.

AWS-reported figures from the AWS Big Data Blog, published 2026-03-18 (accessed 2026-10-11); customer-reported, not an AWS commitment.

---

Practice Questions