Data Ingestion: Batch and Streaming Patterns

Domain 1 of DEA-C01 carries 34% of scored content, and its first task: 1.1, ingestion: is where most candidates lose easy marks. The reason is that ingestion questions almost never ask "what is a stream". They ask you to choose: batch or streaming, full load or CDC, PutRecord or PutRecords, shared reads or enhanced fan-out, Firehose or Kinesis Data Streams, Transfer Family or DataSync, Snowball Edge or something else entirely. Every one of those choices has exactly one answer that fits the stated requirement, and three that sound right.

In this lesson you will:

• separate batch from streaming with the four-question decision every Task 1.1 stem hides;
• design an S3 landing zone: prefixes, SSE-KMS, multipart rules and the four event-trigger destinations;
• run AWS DMS correctly, full load, full load + CDC, CDC only, and size the replication instance;
• choose between AppFlow, Transfer Family, the Snow Family and data APIs for non-stream sources;
• model Kinesis Data Streams: shards, partition keys, MD5 placement, 24-hour to 365-day retention, ordering and at-least-once delivery;
• do shard-count and throughput math by hand, including hot-partition diagnosis and KPL aggregation;
• buffer near-real-time delivery with Amazon Data Firehose: size or interval, whichever hits first;
• stand up Amazon MSK (Standard, Express, Serverless) and pick the right authentication mode;
• apply the Streams vs Firehose vs MSK decision matrix and decision tree;
• trace CDC patterns end to end, and reason about replayability, fan-in/fan-out and state;
• study four AWS customer case studies: Hearst, AGCO, FINRA and Baqend;
• read the October 2026 update box, and practise with 14 exam-style questions plus four interactive checks.

---

1. Batch or streaming? The Task 1.1 decision

1.1 Two lanes, one requirement

Everything in Domain 1 hangs off a single distinction. The exam states it through Task 1.1's split between batch sources (1.1.2) and streaming sources (1.1.1):

1.2 The four questions that settle every stem

1. Is the data set bounded? Yes → batch lane. No → streaming lane.
2. Must a consumer process each record as it arrives? Yes → a real stream with a consumer API (Kinesis Data Streams or MSK), not Firehose.
3. Must a consumer be able to replay? Yes → you need a durable log with retention (Kinesis 24 to 8,760 h, or a Kafka topic). Firehose has no consumer API, so it offers no consumer-side replay.
4. How many independent consumers read the same data? Many, each with its own pace → fan-out design (shared reads, enhanced fan-out, or MSK consumer groups: never Firehose).

Worked pattern: a nightly customer-export job at 02:00 that writes one CSV to S3 is question 1 = batch; a clickstream pipeline where an analyst must replay yesterday's events is questions 1, 2 and 3 = streaming.

• 📚 Did you know? Amazon Kinesis Data Streams launched in 2013 and AWS describes it as the first cloud-native serverless streaming data service (AWS Big Data Blog, "Amazon Kinesis Data Streams: Celebrating a Decade of Real-Time Data Innovation", 2023-11-14, accessed Oct 2026). That is why the exam's streaming vocabulary, shard, partition key, shard iterator, is Kinesis vocabulary first, and why Kafka-style "topic/partition/offset" vocabulary belongs to MSK.

---

2. The S3 landing zone: prefixes, multipart, triggers

2.1 Landing-zone hygiene

Almost every batch pipeline in Domain 1 starts at the same place: producers land raw data in Amazon S3, and everything downstream reads from there. The landing zone is examinable through Task 1.1.3 (configuration options for batch ingestion) and Task 1.1.6 (event triggers).

Rules the exam reuses: a single notification configuration supports one destination type; SQS FIFO queues are not supported as a destination; a Lambda function must live in the same Region as the bucket; a new trigger takes about 5 minutes to take effect and the service sends an s3:TestEvent you should handle.

2.2 Worked example E1: multipart part count

Published S3 multipart rules (as of Oct 2026): use multipart at 100 MB or more; each part 5 MiB: 5 GiB (the last part is exempt from the minimum); at most 10,000 parts; maximum object size 48.8 TiB; a single PUT can carry at most 5 GB.

The examinable point is the chain: 5 GB single-PUT ceiling → multipart required → part-count ceiling → object ceiling. Miss one link and you pick the wrong part size.

• 📚 Did you know? S3 Transfer Acceleration is documented to have cut average ingest time for 300 MB files to ap-southeast-2 by 50%, with an improvement of over 500% for 250 MB uploads sent as 50 MB parts to us-east-1 (Amazon S3 FAQ, accessed Oct 2026). Those are AWS-published observations for specific Region pairs: never a guarantee for your workload.

---

3. AWS DMS: full load, full load + CDC, CDC only

3.1 Three task types, one service

AWS Database Migration Service is the exam's default answer for database-to-database and database-to-stream/file movement, and Task 1.1.1 explicitly names DMS as a streaming source. Its documentation defines exactly three task types:

CDC is log-based: DMS reads the source's native transaction log through its log API. It is not a diff job, and it is not a snapshot: a stem that describes "polling the table every night for changed rows" is describing a batch reconciliation, not CDC.

3.2 The full load + CDC timeline

Task start points for CDC are: a custom CDC time (SCR/SCN or timestamp: requires openTransactionWindow), the native log position, or a checkpoint stored in awsdms_txn_state.

3.3 Sizing, apply mode and topology

Heterogeneous migrations (Oracle → Aurora PostgreSQL, for example) are a two-step answer: convert the schema and code first, then move the data. The conversion tool is now DMS Schema Conversion, built on the engine of the former AWS Schema Conversion Tool: AWS SCT was removed from the DEA-C01 in-scope list at exam-guide v1.1 (2025-12-12).

• 📚 Did you know? DMS applies changes through a sorter that preserves the source's commit order on the target, which is why large multi-table CDC streams can still race ahead of the full load and must be cached on the replication instance's disk (AWS DMS User Guide, accessed Oct 2026). That is also why storage is a 50/100 GB choice rather than a "small default"; it is the spill area for that cache.

---

4. Managed ingestion lanes: AppFlow, Transfer Family, Snow Family, data APIs

Not every source is a database or a stream. Task 1.1 also names SaaS APIs (1.1.2), data APIs (1.1.4), event triggers (1.1.6) and throttling (1.1.9).

4.1 Amazon AppFlow: SaaS into S3 on demand, on event, on schedule

Cadence is capped by the vendor's API quota, not by AWS: Salesforce, ServiceNow and Zendesk at 1 run/minute, Marketo at 1/hour, Google Analytics at 1/day; the account is limited to 1 scheduled run per minute in total (as of Oct 2026). Other published quotas: 100 GB per flow run, 1,000 flows, 10 million runs/month, maximum run duration 48 hours, 100 connector profiles per account, and every call consumes the vendor's API quota. Destinations: S3 (the usual landing zone), Redshift, Snowflake, Salesforce, Slack, with PrivateLink where supported.

4.2 AWS Transfer Family: protocol-preserving partner ingestion

Transfer Family gives partners SFTP, FTPS, FTP and AS2 endpoints backed by Amazon S3, EFS or FSx, so the partner changes nothing. Authentication is service-managed, Active Directory, or a custom API (your identity provider).

The three-way exam split is fixed: Transfer Family = protocol-preserving ingest (partner keeps using SFTP), AWS DataSync = online bulk copy between on-premises file systems and S3/EFS/FSx, Snow Family = offline physical transfer.

4.3 Snow Family: check the calendar before you choose it

4.4 Consuming data APIs (Task 1.1.4, 1.1.9)

Throttling responses you must recognise by name: HTTP 429, and DynamoDB's ProvisionedThroughputExceededException. Handle them with backoff, not with bigger limits: AWS's own guidance is "do not use resource-level limits and quotas as a way to control your usage" (Kinesis Data Firehose quotas, accessed Oct 2026).

---

5. Amazon Kinesis Data Streams: shards, partition keys, retention

5.1 The shard is the unit of everything

A shard is the throughput contract, the ordering boundary and the price unit of a stream. Every limit below is as of Oct 2026:

Extended retention is what turns a stream into a replay window (Task 1.1.11): replaying means opening a new shard iterator at an earlier sequence number: a new read, not a re-write of the log.

5.2 Partition key → MD5 → shard

Three consequences the exam tests relentlessly:

• Order is per partition key, never global. Records with the same key are ordered; records with different keys interleave across shards.
• A hot key is a hot shard. 40,000 records/s spread over 20 shards is fine; 1,500 records/s on one key throttles at 1,000 records/s no matter how many shards you add.
• Keys should outnumber shards widely. Even distribution needs many more distinct keys than shards.

5.3 Produce and consume paths

5.4 Capacity modes

A new on-demand stream starts at 4 MB/s write and 8 MB/s read (Kinesis sizing guidance, as of Oct 2026), and capacity-mode switches are limited to 2 per stream per 24 hours, so "flip modes when traffic changes" is not a per-minute tactic.

• 📚 Did you know? Since 28 August 2026 a Kinesis stream can deliver natively: without Amazon Data Firehose: into Iceberg tables on Amazon S3 Tables and, since 29 August 2026, into general-purpose S3 buckets, in on-demand capacity modes only (AWS What's New, 2026-08-28 and 2026-08-29). Published us-east-1 delivery prices are $14/TB to S3 Tables and $11/TB to S3 (AWS Big Data Blog, 2026-08-31, as of Oct 2026). Firehose is no longer the only stream-to-S3 path, but it still owns transforms, format conversion and dynamic partitioning.

---

6. Worked math: shard counts, throughput and cost

The sizing formula for a provisioned stream (AWS Kinesis sizing guidance, as of Oct 2026):

You must evaluate both terms and take the maximum: stems are written so that one term dominates and the other misleads.

6.1 Worked example E2: records-bound

6.2 Worked example E3: bytes-bound

Same answer as E2 for the opposite reason. That coincidence is the trap: check which term dominated before you trust your arithmetic.

6.3 Worked example E4: small records and KPL aggregation

6.4 Worked example E5: the hot partition

Chart the two worked terms side by side and the "take the maximum" rule stops being abstract, E2 and E3 both land on 12 shards, but from opposite terms:

---

7. Amazon Data Firehose: buffers, destinations, near-real-time

7.1 A delivery service, not a stream

Amazon Data Firehose has no shards and no consumer API. You write records in (Direct PUT, or from Kinesis Data Streams, Amazon MSK, or AWS service logs such as CloudTrail, WAF and VPC flow logs), and Firehose delivers them to a destination: buffered, optionally transformed, automatically retried.

7.2 Worked example E6; which buffer condition fires first

7.3 Retries, and what "replay" means here

Because there is no consumer API, there is no consumer-side replay: your only recovery artefacts are the S3 backup prefix and the error prefix. Need replay? Read from Kinesis Data Streams directly.

• 📚 Did you know? The product was renamed Amazon Data Firehose on 9 February 2024, and AWS states the rename changed nothing about endpoints, APIs, CLI commands, IAM policy names or CloudWatch metrics: IAM actions are still firehose:*. Meanwhile the DEA-C01 in-scope list still prints "Amazon Kinesis Data Firehose" (exam guide, accessed Oct 2026). Both names are correct; never "fix" an option that uses the older one.
• 📚 Did you know? AWS does feature NerdWallet on its Amazon Data Firehose customer page, with a Firehose → Lambda → SNS architecture but no published performance or cost numbers (accessed Oct 2026). So any option that quotes "NerdWallet saved X%" or "NerdWallet ingests Y GB/day" is fabricated: a named customer on an AWS page does not automatically make a number examinable. Verified figures for this lesson's cases are Hearst's 30 TB/day, AGCO's −78%, FINRA's 6 TB/37 billion records and Baqend's sub-minute latency, each attributed to its own source.

---

8. Amazon MSK: brokers, MSK Serverless, IAM authentication

8.1 Three ways to run Kafka

Amazon MSK runs Apache Kafka: the broker/partition/topic model, plus the Kafka ecosystem (Kafka Connect, ksqlDB, Schema Registry).

MSK Serverless bills by cluster-hour, partition-hour and GB I/O, with published limits (as of Oct 2026) of 200/400 MBps per cluster, 8 MiB per message, 5/10 MBps per partition, 2,400 non-compacted partitions and 10 clusters per account. AWS publishes a 99.9% SLA for MSK, including MSK Serverless and MSK Connect.

8.2 Authentication: the highest-yield fact in this section

Cluster policies are what let an IAM principal in another account produce or consume: the cross-account mechanism for MSK. Private connectivity is available through AWS PrivateLink.

8.3 MSK Connect and the rest of the toolkit

• 📚 Did you know? MSK expanded aggressively in 2026: +3 Regions announced 2026-02-20, +13 Regions on 2026-04-21, and China on 2026-07-08 (AWS What's New, accessed Oct 2026). On 30 July 2026 MSK added streaming tables that materialize Kafka topics into Apache Iceberg: the same move Kinesis made in August 2026. Regional availability is a live fact: always re-check before you memorise a Region count.

---

9. Streams vs Firehose vs MSK: the decision matrix

9.1 The official comparison, compressed

AWS publishes head-to-head compare pages; this is their "Best for" column distilled (accessed Oct 2026):

---

10. CDC patterns end to end

Change data capture is the seam between the batch and streaming lanes, which is why DMS appears under both Task 1.1.1 and Task 1.1.2. The examinable patterns:

Design rules that show up as distractors:

• CDC reads the transaction log, so the source needs backups plus roughly 24 hours of log retention: a source whose logs have been truncated cannot start CDC.
• DMS CDC is asynchronous and has no latency SLA: never "real time".
• Do not build on the withdrawn Firehose database-source preview (removed 2025-09-24); use DMS or a Kafka-native connector.
• Cross-account change movement: a Kinesis resource-based policy shares a stream (or an enhanced fan-out consumer) with no data copy; MSK uses cluster policies; DMS → S3 uses an IAM role trusted by dms.amazonaws.com plus a bucket policy.
• Cross-region options: S3 cross-Region replication (Replication Time Control gives a 15-minute SLA), MSK Replicator, or run DMS near the source and replicate the target.

---

11. Replayability, fan-in/fan-out and state (Tasks 1.1.10 to 1.1.12)

Fan-in is a partition-key design problem: many producers, one stream, keys spread across shards. Fan-out is a read-path problem: one stream, many independent consumers at different speeds, and the answer is never Firehose.

Stateful vs stateless (Task 1.1.12): stateless means each record is independent and idempotent; stateful means results are checkpointed: KCL checkpoints, Lambda tumbling windows, Flink windows. A Lambda event source mapping retries a failed batch until it succeeds and pauses the affected shard meanwhile: ordering is preserved, and lag appears as rising IteratorAge (ParallelizationFactor 1 to 10 keeps ordering per key when you raise it).

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

AWS publishes what these patterns look like in production. Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it: the examinable point is the pattern, not the marketing.

Case A, Hearst: clickstream at 30 TB per day (streaming ingestion)

"I don't know how we could have made our clickstream data pipeline work without Amazon Kinesis services. It would have involved many weeks of engineering. Kinesis Data Streams and Firehose make the entire process extremely simple and reliable.": Peter Jaffe, Data Scientist, Hearst

Note the architecture: Streams is the log, Firehose is the delivery. That pairing is exactly the decision matrix in Section 9: replayable log for the producers, buffered no-code delivery for the S3 landing zone.

Case B, AGCO: machinery telemetry fan-out (streaming + analytics)

Derived arithmetic (labelled as such, not an AWS figure): 1,200 data points/minute ÷ 60 ≈ 20/second, and the tested ceiling of 10,000/minute ≈ 167/second, both sit comfortably inside a single shard's 1,000 records/s write limit (as of Oct 2026). AGCO's design challenge was therefore fan-out and enrichment, not shard capacity, which is why the same stream feeds an S3 archive, a Flink analytics path and a Lambda → DynamoDB/OpenSearch lookup path in parallel.

Case C, FINRA: 37 billion records a day into a replayable lake (high-volume ingestion)

"FINRA processes approximately 6 terabytes of data and 37 billion records on an average day … On busy days, the stock markets can generate 75 billion+ records.": John Brady, VP Cyber Security/CISO (AWS Big Data Blog, 2017-10-03)

Derived arithmetic (labelled as such, not an AWS figure): 37,000,000,000 ÷ 86,400 s ≈ 428,000 records/s, while 6 TB ÷ 86,400 s ≈ 69 MB/s. Feeding those into the Section 6 formula, the records term wins outright: max(ceil(69), ceil(428,000 / 1,000)) = max(70, 428) = **428 shards**. AWS publishes no FINRA shard count: the teaching point is that a modest 6 TB/day feed can still be records-bound, exactly like worked example E2, so a stem quoting only gigabytes per day is not telling you which term dominates.

• 📚 Did you know? FINRA's Consolidated Audit Trail collects over 100 billion events per day from 22 exchanges and 1,500 broker-dealers (AWS press release, 2019-12-04, accessed Oct 2026). That is why this lesson never asks you to memorise a shard count: at that scale the examinable decisions are where the data lands (S3 prefixes, SSE-KMS), how it is replayed (versioning + retention), and which read engine sits on top (EMR, Redshift): not one number in a sizing spreadsheet.

Case D, Baqend: from nightly batch to sub-minute dashboards (stream processing)

"Our architecture is based on fully managed AWS services that made it easy to scale from 1 to over 100 million users." · "sub-minute latency based on Kinesis Data Analytics for Apache Flink.": Baqend (AWS Big Data Blog, 2021-02-05)

Read the stem words: "sub-minute", "windowed aggregation over a live event feed", "a small team that must not operate Kafka or Flink clusters" → Kinesis Data Streams + Amazon Managed Service for Apache Flink. Two naming traps live in this one story: the service was renamed from Kinesis Data Analytics on 2023-08-30 (API unchanged), and Kinesis Data Analytics for SQL streams was discontinued on 2026-01-27, so an option offering a "Kinesis Data Analytics SQL application" describes a retired product, not a lighter alternative.

• 📚 Did you know? AGCO's published figure of 1 person running the platform instead of 3 to 5 is the operational half of the streaming argument: managed services move shard-splitting, broker patching and buffer tuning to AWS. Read it as a customer-reported outcome for their workload (accessed Oct 2026), never as a staffing formula.

---

Practice Questions