Programming Concepts for Data Operations

Every pipeline you have studied so far moves and reshapes data. This lesson asks the question underneath: what code runs, and why does it behave the way it does? DEA-C01 answers that in Domain 1, Task 1.4, and the task statement is deliberately narrow. The exam guide lists "language-agnostic programming concepts" as in scope and "programming language-specific syntax" as out of scope, so no item will ask you to recall a decorator, a list comprehension or a package import. Instead the stem hands you a pipeline problem (rank these rows, deduplicate this replay, parse this nested field, pick a runtime) and expects the right construct or service back (DEA-C01 exam guide, accessed Oct 2026).

In this lesson you will:

• separate language-agnostic concepts from language-specific syntax, and map each concept to the construct the exam expects;
• read a Lambda handler and a Glue job configuration as programs, not as trivia;
• choose between pandas on one node and PySpark across partitions, and name the crossover honestly;
• apply lazy evaluation: transformations build a plan, an action triggers compute, and a shuffle is where the cost lands;
• size partitions, pick coalesce or repartition, and explain why gzip files hurt;
• climb the UDF cost ladder from built-in SQL down to plain Python, and know when a UDF is justified;
• write window functions in Spark SQL, Redshift and pandas, and know why PARTITION BY changes parallelism;
• use CTEs, joins, PIVOT and JSON parsing as the SQL patterns the exam re-asks;
• configure AWS Lambda for data work: event sources versus event source mappings, the concurrency pool, the 900-second ceiling, memory-coupled CPU and layers;
• apply the Lambda vs Glue vs Batch decision matrix on size, duration, frequency and runtime limits;
• distinguish at-least-once delivery from an exactly-once effect, and build idempotent writes;
• use watermarks to bound late data, and climb the poison-pill handling ladder;
• flatten nested JSON and choose between schema-on-read and schema-on-write;
• read the October 2026 update box, study two AWS customer case studies, and practise with 12 exam-style questions plus interactive checks.
• expanded scope of this edition: the lesson now carries five AWS customer case studies (EMX, AppsFlyer, Hearst, Nasdaq and PayU), fourteen practice questions (dea-06-q1 through dea-06-q14) and three interactive checks: matching, fill-in-the-blank and drag-and-drop.

---

1. Programming concepts the exam actually tests

1.1 The scope boundary, in the guide's own words

Two phrases do all the work:

In scope: "language-agnostic programming concepts" · Out of scope: "programming language-specific syntax"

Everything in this section therefore has to survive a test: could I ask this without naming a language? If yes, it is examinable. "What does def handler(event, context) mean as an entry point?" is examinable, the shape is the same in Python and the concept of an entry point is universal. "What is the exact argument order of pd.merge()?" is not, that is syntax.

Task 1.4's skill list makes the same point from the other side. The named skills are optimise runtime performance, Lambda concurrency and performance, languages and frameworks (Python, SQL, Scala, R, Java, Bash, PowerShell), version control, testing, logging and monitoring, infrastructure as code, AWS SAM, mounting storage volumes in Lambda, CloudFormation/CDK, CI/CD, distributed computing and data structures and algorithms: a list of capabilities, with Git questions kept generic after AWS removed CodeCommit from the in-scope list at exam-guide v1.1 (2025-12-12) (DEA-C01 exam guide revisions page, accessed Oct 2026).

1.2 The concept → construct map

This table is the whole of Section 1 compressed into what a question can actually test:

1.3 Worked example E1: reading a handler, not memorising syntax

The exam gives you a shape like this and asks what it does or what to change. Read it as a control-flow diagram:

Three examinable facts fall out of it: the entry point is a function with a fixed signature; conditionals route records rather than retry them; and the strategy change at a size threshold is exactly the "optimise runtime performance" skill: small sets go row-wise, large sets go bulk.

1.4 Conditionals and loops prefer set-based SQL

The single most common Domain 1 pattern is a stem describing a loop over rows, with the correct answer being an aggregation or a window function:

1.5 Object orientation shows up as layers, not as classes

Class hierarchies are not examinable. Reuse is. AWS's answer to "my team has three Lambda functions that all need the same helper library" is a layer: a zip archive mounted under /opt, with a hard maximum of 5 layers per function, and, this is the detail questions reach for, layers are zip-only, so a container image function cannot use them at all (the image bakes its dependencies instead, up to 10 GB) (AWS Lambda limits and layers documentation, accessed Oct 2026).

1.6 APIs show up as pagination and throttling

A stem describing "call a service that returns 100 records at a time until exhausted" is testing pagination; a stem describing "the pipeline fails every few minutes with HTTP 429" is testing backoff with jitter, not a specific retry library. Both are language-agnostic, and both map to the same service-side levers you met in Lesson 05: retry policies, DLQs and idempotent writes.

1.7 Serialization is a decision, not a syntax

The rule AWS's own guidance reduces to: Avro/JSON to move it, Parquet/ORC to read it. Because Athena bills bytes scanned, the format is the cost: a columnar file that lets the engine skip untouched columns is cheaper than a row format that cannot (AWS Athena and AWS Glue documentation, accessed Oct 2026).

• 📚 Did you know? AWS's DEA-C01 guide changed how it presents this topic rather than what it contains: at v1.1 (2025-12-12) the separate "Knowledge of" and "Skills in" lists were consolidated into one skills list per task, adding 8 skills and removing none. That is why current study material quotes a flat numbered list like "1.4.1 optimise runtime performance" while older material quotes a Knowledge/Skills pair for the same content (DEA-C01 revisions page, accessed Oct 2026).

---

2. Python and PySpark patterns for data operations

2.1 Single node versus distributed: pick before you code

The first fork in every data task is where the loop runs. Getting it wrong is expensive in both directions: a distributed cluster for a 40 MB file wastes startup and idle capacity, while a single node for a 200 GB join runs out of memory.

Crossover rule of thumb: roughly 10 million rows / 5 to 10 GB is where people stop reaching for pandas on one node, and you must state its provenance honestly. It is a 2017 third-party rule of thumb, never an AWS-published figure; the exam-grade facts are the service limits above, which are AWS-documented (as of Oct 2026).

2.2 Lazy evaluation: plan first, compute once

Apache Spark transformations are lazy: reading, filtering, mapping and joining build a plan, and nothing runs until an action (write, count, collect, toPandas) forces execution (AWS Prescriptive Guidance, Tuning AWS Glue for Apache Spark, accessed Oct 2026).

Two consequences the exam builds questions on:

1. A transformed dataset is recomputed on every action unless you cache()/persist() it. Two actions on the same lineage pay for the lineage twice.
2. Chain your transformations, then act once. Ten cheap actions beat one expensive action only if they share no recomputed lineage; which they usually do.

2.3 Partitions: the unit of parallelism and of cost

Worked example E2: partition math

You have 120 GB of Parquet on a 32-core Glue job:

The two functions are not interchangeable: coalesce merges partitions without a full shuffle (use it to go down), while repartition performs a full shuffle to reach an exact count (use it to go up, or to spread skew across a key).

• 📚 Did you know? "More partitions = faster" is the most expensive misconception in Spark. Past about 2 to 3× cores, extra partitions do not add parallelism; they add scheduler overhead, small-file writes and GC pressure. The opposite error is just as bad: one 60 GB gzip partition is one task on one core, and because gzip is unsplittable the engine cannot break it up. The habit worth building is to repartition by the partition columns before the write, so the file layout you land is the layout the reader will prune on (AWS Glue parallelize-tasks guidance, accessed Oct 2026).

2.4 The UDF cost ladder

A user-defined function is the escape hatch for third-party parsers, model scoring and stateful row logic. Everything else should stay in built-in SQL. AWS's own performance guidance places the options on a ladder, and the top rung is roughly five times faster than the bottom one:

Exam rule: reach for a UDF only when no built-in can do it. On a 1-billion-row job, rung ③ versus rung ① is the difference between minutes and hours, and it is a performance optimisation question, which is Task 1.4 skill "optimise runtime performance" (AWS Glue Optimize user-defined functions, accessed Oct 2026).

2.5 Window functions: three stems, three answers

Parallelism hinge (a favourite distractor): in Amazon Redshift, PARTITION BY is what lets the window function run across slices in parallel; without it the window is computed serially on a single slice. And window functions are legal only in a SELECT list or in the final ORDER BY: not in a WHERE clause (Redshift window-function documentation, accessed Oct 2026).

In pandas, the same job is groupby().transform() or groupby().cumcount(): correct, but single-node. Past the crossover size it collects everything onto one process, which is why df.toPandas() / collect() inside a Glue Spark job is the canonical out-of-memory bug: it pulls the whole distributed frame to the driver.

---

3. SQL patterns the exam keeps re-asking

3.1 CTEs stage the logic so each step is readable

A common table expression (WITH) is how you express "first deduplicate, then filter, then rank" without three separate statements, and how you make a query reviewable. It is also the natural home for the dedupe step in an idempotent pipeline (Section 6).

3.2 Joins: qualify names, broadcast the small side

3.3 PIVOT, and the portable fallback

Amazon Redshift supports PIVOT, with hard limits worth memorising:

The portable, engine-independent form; which is the answer whenever a question asks for a pivot that PIVOT cannot express:

3.4 JSON parsing: two engines, two families

Amazon Athena (Presto/Trino dialect) separates extracting a JSON value from extracting a scalar:

Amazon Redshift parses JSON once into the SUPER type and then queries it with PartiQL-style paths: JSON_PARSE(payload) → SUPER, dot and bracket access, UNNEST, UNPIVOT. AWS is explicit that JSON is not a good fit for large datasets because it is not columnar; parse it into SUPER once, or land it as Parquet/SUPER columns (Redshift JSON and SUPER documentation, accessed Oct 2026).

3.5 Bulk load: one COPY, not a thousand INSERTs

Amazon Redshift's own guidance is unambiguous: use COPY for large loads: individual INSERTs "might be prohibitively slow". Sizing rules: one COPY per table/prefix (concurrent COPYs to the same table serialize), non-splittable gzip/JSON files between 1 MB and 1 GB, count ≈ number of slices, and CSV/Parquet files ≥128 MB are auto-split (Redshift COPY best practices, accessed Oct 2026).

Worked example E3: COPY sizing

---

4. AWS Lambda for data tasks

4.1 Two ways an event arrives: push (trigger) versus pull (event source mapping)

This distinction is the spine of every Lambda data question. A trigger means the service pushes the event to your function; an event source mapping (ESM) means Lambda polls the stream or queue on your behalf and batches records for you (AWS Lambda event-source-mapping documentation, accessed Oct 2026).

4.2 Concurrency: reserved versus provisioned

AWS gives every account a pool of 1,000 concurrent executions per Region, shared across all functions, with an unreserved floor of 100, and a sustainable ingestion rate of roughly 10 requests/second per unit of concurrency (as of Oct 2026).

Sizing arithmetic: required concurrency ≈ requests/second × duration in seconds, and AWS's provisioning guidance adds ~10% headroom. Provisioned concurrency is not the answer to "my async pipeline should start quickly": AWS states async pipelines usually do not need it (AWS Lambda concurrency documentation, accessed Oct 2026).

Worked example E4: concurrency sizing

4.3 Timeout and the memory → CPU coupling

There is no vCPU knob. CPU, network and GPU scale with the memory setting, which is the single most testable Lambda performance fact:

Worked example E5: memory, CPU and unit cost

4.4 Layers, environment variables and /tmp

• Layers are the object-oriented reuse story (Section 1.5): ≤5, zip-only, mounted at /opt. A container-image function cannot use layers.
• Environment variables are capped at 4 KB in aggregate and are readable through the API, so they are configuration, not a secret store. Secrets belong in AWS Secrets Manager or Systems Manager Parameter Store, encrypted with KMS.
• /tmp is transient and per-execution-environment: it survives warm invocations of the same environment and disappears when the environment is recycled. It is a cache, never durable state, and the first 512 MB is free (as of Oct 2026).
• Mounting storage: Task 1.4 explicitly names mounting storage volumes in Lambda: Amazon EFS is the durable, multi-environment answer; /tmp is the ephemeral one.

Worked example E6: is Lambda the right runtime at all?

---

5. The Lambda vs Glue vs Batch decision matrix

5.1 The matrix

This table is the highest-yield single artefact in Task 1.4. Every row is a sourced limit as of Oct 2026:

5.2 Worked example E7: the same job on two runtimes

The lesson is not "Lambda is always cheaper"; it is that pricing follows the shape of the work. A 40-minute nightly merge cannot run on Lambda at any price (the 900 s ceiling), and a 2-second job cannot justify AWS Batch, where scheduling overhead can exceed the runtime.

• 📚 Did you know? AWS's own Glue guidance puts the startup cost in writing: Glue Spark jobs commonly spend 2 to 3 minutes before the first task runs, which is more runtime than the job itself for sub-100 MB payloads. That is why the exam's "small, frequent, event-driven" stems point at Lambda and its "big, scheduled, join-heavy" stems point at Glue, and why a stem that says "frequent, small files arriving every few seconds" is usually telling you to reject both Glue and Batch (AWS Glue and AWS Batch best-practice documentation, accessed Oct 2026).

---

6. Idempotency, watermarks and poison pills

6.1 At-least-once delivery, exactly-once effect

Streams and queues, Kinesis, SQS, DynamoDB Streams, MSK, deliver at-least-once by default. There is no end-to-end exactly-once transport. What pipelines build instead is an exactly-once effect: at-least-once delivery + an idempotent side effect (AWS Big Data Blog, Build a fault-tolerant serverless data aggregation pipeline with exactly-once processing, 2021-11-01).

6.2 Watermarking: how long you wait for late data

A watermark is max event time − allowed lateness. It bounds how long a streaming job waits for late records before it closes a window and releases the state behind it. Move it forward only: too generous and state grows without bound; too tight and late records are dropped silently (AWS and community streaming guidance, accessed Oct 2026).

In an incremental Glue job the same idea appears as a stored high-water mark plus a sliding safe window:

Worked example E8: the safe window

6.3 Poison-pill handling: the ladder

A poison pill is a record that always throws. Because an ESM retries the whole batch and pauses the shard, one bad record can stop an entire stream, and the defaults are infinite:

Worked example E9: from a one-week stall to a one-hour stall

• 📚 Did you know? The three retry owners on this exam are different, and questions routinely mix them up: for synchronous invocations the caller retries; for asynchronous invocations Lambda retries the function twice (at 1 and 2 minutes) before an on-failure destination; and for event source mappings the mapping itself retries the batch and pauses the shard. The same 429 in a stem therefore has three different answers depending on how the function was invoked (AWS Lambda asynchronous and ESM documentation, accessed Oct 2026).

---

7. Nested data: flattening, and schema-on-read versus schema-on-write

7.1 Two schema philosophies

7.2 Flattening nested data, per engine

Two consequences worth memorising: a NULL array produces no rows under a cross-unnest (silently shrinking your result), and an explicit schema at read time beats schema inference: inference is convenient once and unpredictable forever.

7.3 Format follows the access pattern, and format is cost

Worked example E10: the format decision, priced

• 📚 Did you know? The lineage question loves the four-format contrast because it is purely shape: CSV has no schema and no pruning, JSON nests but must be parsed whole, Avro carries its schema in the header and is splittable, which is exactly why streaming and schema-evolution jobs prefer it, and Parquet/ORC are columnar, which is why analytics engines do. The one-line memory is Avro/JSON to move it, Parquet/ORC to read it (AWS Glue and Athena format guidance, accessed Oct 2026).

---

2026 Updates (as of October 2026)

• 📚 Did you know? Nothing in the runtime ranking changed across 2024 to 2026 even though almost every number around it did. Lambda still stops at 900 s with CPU tied to memory, Glue Python shell still runs single-node at 0.0625 or 1 DPU far beyond 15 minutes, and Glue Spark still starts at 2 DPU with ~2 to 3 minutes of startup; while what moved is everything around those ceilings (Glue 6.0 on 2026-08-21, EMR Serverless workers up to 32 vCPU / 244 GB, Athena Capacity Reservations down to 4 DPU for 1 minute). That is why the Section 5 decision matrix survives release-note churn: it is keyed on the shape of the work: duration, size, frequency, distribution: not on a version number (AWS What's New and service documentation, accessed Oct 2026).

---

Real-World Case Studies

AWS publishes what these abstractions look like in production. Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it: the examinable point is the pattern (which programming decision was applied, which service did the work, which number moved), not the marketing.

Case A, EMX: a programmatic exchange rebuilt on S3 and Athena

Read EMX as Section 7.3 in production: the saving came from where the compute happens and how much of the file gets read, not from a discount. Athena bills bytes scanned, so landing columnar files and pruning them with partitions is a programming decision: the same decision as choosing a window function over a Python loop.

Case B, AppsFlyer: an interactive workload moved off HBase onto Athena

The examinable point is the selection logic, not the percentage: a low-latency, key-oriented store (HBase) was the wrong shape for an analytical, scan-and-filter workload, and the answer was to move the read path to a columnar lake plus a serverless SQL engine. That is the same fork as Section 5, pick the runtime from the shape of the work, and it is why "we already have a database, so query it" is a distractor in a Domain 1 cost stem.

Case C, Hearst: a 30 TB/day clickstream pipeline built on Kinesis

Read Hearst as Section 4.1 in production. Choosing a stream over a nightly batch is a programming decision about event shape and latency, and Firehose's managed buffering and delivery is precisely what removes the "weeks of engineering" the customer describes. When a stem says "frequent, small files arriving every few seconds", Hearst is the shape it is describing, and the answer is a stream plus a Lambda or Kinesis consumer (an event source mapping that polls and batches), never a scheduled Glue job whose 2 to 3 minute startup dwarfs the work.

Case D, Nasdaq: an exchange decouples its write path from its read path

The engineering quote behind the case is about contention: with S3 holding the bytes and Redshift holding the compute, "data loading and querying" stop competing for the same resources. That is the Section 2.3 partition lesson at scale: the lake is written in a layout the reader can prune, and the query engine scales independently of the ingest window. Note also what the case is not: it is not a claim that Redshift was replaced. Spectrum keeps the warehouse as the SQL engine while S3 becomes the storage tier, which is exactly the lake house pattern the exam tests.

Case E, PayU: consolidation, data sharing, then demand rationalisation

Two levers must be kept apart here, and the exam will try to blur them. The platform move (40 databases → one warehouse plus S3, with Data Sharing isolating consumers without copying data) bought freshness and the $20,000/month saving; the demand move (150,000 → 35,000 queries per month) is fewer queries, not cheaper queries. AWS quotes the customer: "In 1 month, we cut down queries by 77 percent, which would have been a 6-month exercise in the previous environment." The lesson for Task 1.4: rationalise the workload before you buy capacity: a rewritten CTE or a cached aggregate removes cost permanently, whereas right-sizing only moves the bill.

• 📚 Did you know? Neither AWS case study names a single function, class or library: AWS publishes the outcomes and leaves the code to you. That mirrors the exam's own boundary: DEA-C01 asks you to recognise that 85% storage savings on an ad-hoc query workload implies a columnar format plus a bytes-scanned engine, and that 80% off an interactive workload implies a runtime re-selection, never to recall a customer's code (AWS Big Data Blog case studies, accessed Oct 2026).

• 📚 Did you know? Amazon Kinesis Data Streams launched in 2013 and AWS calls it "the first cloud-native serverless streaming data service", which is why every customer story written before 2024 uses old service names. Kinesis Data Analytics became the Amazon Managed Service for Apache Flink on 2023-08-30, and Kinesis Data Firehose became Amazon Data Firehose on 2024-02-09 with no API change, the IAM action is still firehose:*, and the DEA in-scope list still prints the old Firehose name. Older case studies (Hearst, AGCO, Baqend) therefore read like a different stack from current documentation: map the names, never treat them as two services (AWS Kinesis retrospective, AWS What's New and DEA-C01 in-scope services page, accessed Oct 2026).

---

Practice Questions