Generative AI Fundamentals: Tokens, Transformers, Prompting and Foundation Models

This is the highest-leverage lesson in the course. Domain 2 (Fundamentals of Generative AI) carries 24% and Domain 3 (Applications of Foundation Models) carries 28% of the scored content, 52% in total, and almost every question inside those two domains is built from the same small set of objects: a token, a context window, a prompt, an inference parameter, an embedding and a foundation model (FM). Learn the objects and the numbers attached to them, and half the exam becomes arithmetic plus vocabulary rather than guesswork.

By the end of this lesson you will be able to:

• explain self-attention in query/key/value terms and say why transformers beat RNNs on long sequences;
• estimate tokens with both AWS-published rules (6 characters per token, and 4.7 characters per token for English embeddings) and know which one the question is using;
• budget a context window as system prompt + client input + RAG context + output, and predict what a Context Window Overflow does to behaviour;
• choose between zero-shot, few-shot, chain-of-thought, self-consistency and prompt chaining and state each one's cost;
• set temperature, top-p, top-k, max tokens, stop sequences and penalties for a workload and defend the determinism answer;
• read a modality table and pick the right FM family for text, embeddings, image, video or image+text inputs;
• walk the customization spectrum from prompt engineering to a custom model and name the numbers AWS attaches to distillation and reinforcement fine-tuning;
• evaluate an FM with ROUGE, BLEU, BERTScore, LLM-as-a-judge and human evaluation;
• apply the six Guardrails policies and the three-step hallucination mitigation ladder;
• defend the comparative verdict between generative AI, traditional ML and rules-based systems.

---

1. What generative AI is, and what it is not

1.1 The definition the exam actually uses

Generative AI is the class of systems that creates new content: text, code, images, video, audio and embeddings. That single word, new, is the whole distinction. A traditional ML model answers "which bucket does this row belong to?"; a generative model answers "write me something that belongs in this bucket." Foundation models are the engines underneath: large models trained on broad corpora that you adapt through prompting, retrieval-augmented generation (RAG) or fine-tuning instead of retraining.

AWS is explicit about the trade. Foundation models are non-deterministic: "identical inputs can produce different responses". That property is what makes them useful for drafting and brainstorming and what makes them dangerous for extraction, scoring and anything an auditor will read twice.

1.2 The advantages and the risks AWS names

The exam guide lists four advantages and four disadvantages. They are not decoration: each advantage pairs with a control, and each risk pairs with a mitigation you will be asked to name.

1.3 The foundation model lifecycle

Task 2.1 requires the lifecycle in order. It is also the shape of most sequencing questions on the exam.

• 🔢 📚 Did you know? The exam guide's own wording for the lifecycle is a cycle, not a line: feedback loops back into data selection. That is why "monitor the deployed model and feed production data back into the corpus" reads as correct on the exam while "deployment is the final step" reads as a distractor, and it is the same loop that makes drift a generative-AI problem and not only a classical-ML problem.

---

2. Tokens: the unit everything is measured in

2.1 What a token is

A token is the chunk of text a model actually reads and writes. A tokenizer splits raw text into tokens before the model sees it, and generation happens one token at a time. The critical property is that tokenization is model-specific: the same sentence can be 12 tokens on one model and 17 on another, which is why a fixed "words ÷ X = tokens" formula is unreliable and why Amazon Bedrock exposes a CountTokens API that counts with the selected model's tokenizer on the exact input you are about to invoke.

Chunking is the companion concept: when a document is longer than a model's input limit, you split it into segments (paragraphs, sections, pages) so each segment fits. Chunking quality is the single biggest lever in a RAG system: AWS published a case where better chunking alone raised context relevance from 67% to 93% and removed hallucinations without changing the prompt.

2.2 The two AWS-published estimation rules

2.3 Worked example 1: fine-tuning quota in characters

A team wants to fine-tune Amazon Titan Text G1 Express and has a 30,000-character sample prompt. The batch-size-1 fine-tuning quota for that model is 4,096 input+output tokens.

1. Convert the quota to characters: $4{,}096 \times 6 = 24{,}576$ characters allowed.
2. Convert the sample to tokens: $30{,}000 \div 6 = 5{,}000$ tokens: 904 tokens over the 4,096 quota.
3. Convert the overage back to characters: $30{,}000 - 24{,}576 = 5{,}424$ characters too many.

Answer: the sample exceeds the quota by 5,424 characters; split it into two records or truncate. The distractor "50,000 characters" is the Titan Embeddings V2 cap, a different model and a different limit.

2.4 Worked example 2; which embedding limit binds first?

Titan Text Embeddings V2 accepts 8,192 tokens or 50,000 characters. Which one stops a long English document first?

1. English averages 4.7 characters per token, so the token bound binds at $8{,}192 \times 4.7 \approx 38{,}502$ characters.
2. A 50,000-character article therefore converts to $50{,}000 \div 4.7 \approx 10{,}638$ tokens.
3. Overage: $10{,}638 - 8{,}192 = 2{,}446$ tokens.

Answer: the token bound binds first, at roughly 38,502 characters for English: 11,498 characters before the 50,000-character ceiling. AWS's documented remedy is to segment the document into paragraphs or sections and embed each one.

• 🔢 📚 Did you know? Bedrock's CountTokens endpoint answers a question no formula can: it runs the invoked model's own tokenizer over your exact string. The same 47,000-character article can be over budget on one model and comfortably under budget on another, which is why AWS documents CountTokens as the planning step before any quota-sensitive call rather than an optional convenience.

---

3. Transformers and self-attention: how a model reads a sequence

3.1 Query, key and value

Self-attention lets every token in a sequence look at every other token and decide what matters. Each token embedding is projected into three vectors, and AWS's descriptions of them are the exact phrasing the exam uses:

Attention scores are computed from Q matched against K, then used to weight the V vectors. The result is a representation of each token that already contains its context; which is why "bank" in river bank and "bank" in bank account end up as different vectors even though they were the same input string.

3.2 The transformer block and positional encoding

A transformer block is, in AWS's definition, multi-head self-attention plus a position-wise feed-forward network. Multiple heads attend in parallel so different heads can specialise: one tracking syntax, another tracking coreference, another tracking distance. Because attention itself is order-blind, positional encoding injects the order of the sequence so the model knows which token came first.

3.3 Why transformers beat RNNs, and why GPT-style models are decoder stacks

Transformers process the whole sequence in parallel, where an RNN must walk it one position at a time. The consequences are the two things worth memorising: faster training and the ability to handle much longer sequences. GPT-style models are stacks of decoder blocks and are autoregressive; they produce one token, append it to the input, and run again.

---

4. Embeddings, vectors and vector databases

4.1 From one-hot tables to embedding space

Before embeddings, categorical data lived in one-hot tables: a tall, sparse vector where every word was an independent column and no notion of meaning existed between them. Embeddings replace that with a dense vector in which semantically similar words sit near each other. The payoff is arithmetic that one-hot encoding could never do: AWS's documented example is Paris − France + Germany ≈ Berlin, where the direction between two embeddings encodes a relationship.

4.2 Who computes similarity: the model or the database?

A detail that decides many marks: the model produces the vector; the vector database computes similarity, using cosine similarity or k-nearest neighbours (k-NN). Retrieval in a RAG pipeline is therefore a database operation, and the exam guide names the embedding stores it expects you to recognise.

4.3 The embedding models and their verified limits

4.4 Worked example 3: what a vector store actually costs you in bytes

Titan Text Embeddings V2 returns 1,024 dimensions. Assuming 32-bit floats (4 bytes per value: an assumption, not an AWS-published figure), one vector is:

$$1{,}024 \times 4\ \text{bytes} = 4{,}096\ \text{bytes} \approx 4\ \text{KB}$$

For a knowledge base of 100,000 passages:

$$100{,}000 \times 4\ \text{KB} = 400{,}000\ \text{KB} \approx 400\ \text{MB}$$

Answer: roughly 400 MB of raw vector payload before index overhead, metadata and sharding. That is the number you carry into the choice between OpenSearch, Aurora (pgvector), Neptune, DocumentDB and RDS for PostgreSQL, and it is why dimensionality reduction to 512 or 256 (both supported by V2) is a legitimate cost lever: halving the dimensions halves the raw payload to about 200 MB.

• 🔢 📚 Did you know? Embedding arithmetic is not a party trick; it is evidence that the vector space encodes relationships as directions. Paris − France + Germany ≈ Berlin works because the "capital of" displacement is consistent across countries. On the exam this shows up indirectly: when a question says vectors place semantically similar items near each other, the intended answer is almost always about cosine similarity in a vector database, not about the model re-reading the text.

---

5. Prompt engineering: the first lever you should always pull

5.1 The vocabulary of shots

5.2 The technique table

5.3 Chain-of-thought: what it buys and what it costs

Amazon Bedrock documents that chain-of-thought, branded model reasoning in the console, "can often improve model accuracy by giving the model a chance to think before it responds". The cost is equally explicit: increased latency and increased output tokens. And the exclusion matters as much as the benefit: AWS states that simple tasks are not good candidates for chain-of-thought. Asking a model to reason at length to classify a support ticket into one of four queues is paying output tokens for no accuracy.

5.4 The prompt-engineering decision tree

5.5 Worked example 4: what few-shot prompting really costs

An instruction costs 80 tokens; one labelled example costs 60 tokens.

Answer: a 5-shot prompt costs 4.75× the zero-shot input tokens for the same single answer. Since AWS says 3 to 5 examples suffice for simple text classification, 3 shots is the sweet spot; you get format control at 3.25× instead of 4.75×. And if you then add chain-of-thought, the cost moves to the output side, where Titan Text's default maxTokenCount of 512 starts to bite; raise it before enabling reasoning.

• 🔢 📚 Did you know? AWS's prompt-engineering guidance hides two habits that separate a working prompt from a demo: put the question or instruction at the end of the prompt (so the most relevant tokens sit closest to where generation starts), and keep a held-out set of prompts you never show the team while iterating. Without the held-out set, "the prompt got better" is an anecdote: the same discipline as a held-out test set in classical ML, applied to text.

---

6. Inference parameters: steering the next token

6.1 The full parameter table

Every number here is verified against the Bedrock Inference response generation and Titan Text pages.

6.2 The defaults you must know cold

6.3 Worked example 5: three workloads, three parameter sets

A team runs three jobs on the same Titan Text Express model. Assume the same prompt and the same 512-token default output cap.

1. Contract clause extraction (must be auditable): temperature 0.0, top-p lowered from 0.9 toward 0.1 to 0.3. Effect: the distribution steepens, the candidate pool shrinks, and re-runs converge on the same extraction.
2. Campaign headline brainstorm: temperature 1.0, top-p left at 0.9. Effect: the distribution flattens, more of the tail is reachable, and diversity rises: at the cost of reproducibility.
3. Chain-of-thought reasoning over a 400-token answer: temperature 0.2, but maxTokenCount raised from 512 to at least 1,024, because reasoning tokens are output tokens. Leaving the default in place truncates the chain mid-derivation.

Answer: determinism is bought with temperature and top-p; creativity with temperature; and reasoning capacity with max tokens: the parameter that most often gets forgotten, because the failure looks like a bug rather than a setting.

---

7. Context windows: the budget that silently fails

7.1 What actually fits

The context window is the maximum number of tokens a model considers at once, and it is a budget across everything in the call:

$$\text{context window} \ge \text{system prompt} + \text{client input} + \text{RAG context} + \text{output}$$

Set maxTokenCount by subtraction: window − input tokens − safety margin. Never by intuition.

7.2 The FIFO ring buffer and Context Window Overflow

AWS's security blog describes the context window as a FIFO ring buffer. Once it is full, each new token silently evicts the oldest token, no error, no warning, just drifting behaviour. The phenomenon has a name on AWS: Context Window Overflow (CWO). The practical horror story is that the system prompt is usually the oldest content, so under CWO the instructions that define your agent's behaviour are the first thing to disappear, and the application keeps returning 200 OK while ignoring its own rules.

7.3 Worked example 6: a context-window budget

System prompt 400 + RAG passages 1,600 + conversation history 900 + question 100:

$$400 + 1{,}600 + 900 + 100 = 3{,}000\ \text{input tokens}$$

With maxTokenCount at Titan Text Premier's maximum of 3,072, the total becomes $3{,}000 + 3{,}072 = 6{,}072$ tokens of demand. If the model's window is smaller than that, the overflow does not raise an exception: the oldest 3,000-or-so tokens (starting with the system prompt) get evicted as generation proceeds, and behaviour drifts instead of failing.

Fix, in order: run CountTokens on the exact prompt → set maxTokenCount = window − input − margin → shorten or re-rank RAG passages → summarise older turns of history.

7.4 The end-to-end generation flow

• 🔢 📚 Did you know? The context window is not published as one fixed number per model family on the pages AWS has verified for this digest, so "which model has the biggest window?" items are announcement-driven and should be re-checked close to exam day. What is verified and stable: Titan Text maxTokenCount maximums (4,096 / 8,192 / 3,072), Titan Embeddings 8K tokens, Embeddings V2 8,192 tokens, Llama fine-tuning sum of input + output 16,000 (10,000 for the 90B), and Claude 3 Haiku fine-tuning 32,000 tokens. Budget from those numbers, not from a blog headline.

---

8. Foundation models by modality, and how images are made

8.1 The modality table

8.2 How a diffusion model produces an image

Diffusion training teaches a model to remove noise that was added to real images. Generation runs that process backwards: start from random noise and denoise iteratively, each step conditioned on the embedding of the text prompt. Latent diffusion does the work in a compressed latent space (a U-Net, or a transformer denoiser) and only then decodes to pixels.

Amazon Nova Canvas and Nova Reel are latent diffusion models with transformer backbones, often called diffusion transformers, built from three parts: a VAE that converts between pixels and visual tokens, a text encoder for the prompt, and a transformer denoiser. Amazon Titan Image Generator is a text-conditioned diffusion FM: it encodes the prompt, matches vectors in a joint text/image embedding space, and turns a low-resolution random encoding into the final image.

8.3 Worked example 7: vision inputs are token math too

AWS's published formula for image tokens on Llama vision fine-tuning is:

$$\text{tokens} = \min(2, \max(H // 560,\ 1)) \times \min(2, \max(W // 560,\ 1)) \times 1601$$

Against the 16,000 sum-of-input-plus-output fine-tuning quota for Meta Llama 3.2: one 1,120 × 1,120 image = 6,404 ✓, two = 12,808 ✓, three = 19,212 ✗: over quota.

Answer: two large images fit, three do not, and the clamp behaviour means a small image still costs at least 1,601 tokens.

• 🔢 📚 Did you know? Amazon Titan Image Generator ships with an invisible watermark enabled by default and accepts at most 5 reference images: both are exam-shaped facts because they answer two different questions: "how does AWS make generated content identifiable?" (watermarking, a transparency control) and "how much personalisation does the API accept?" (5 images, not 50). Pair either with the Responsible AI Lens in Section 11 when a question asks for a transparency artifact.

8.4 2025 to 2026 Updates

The modality table in 8.1 is the exam-guide baseline; the roster behind it moved twice inside the window AWS verified for this lesson. Everything below comes from AWS What's New, the Amazon Bedrock user guide and AWS's own re:Invent 2025 coverage: only features and dates AWS published itself.

Guardrails grew alongside the model roster: AWS now documents code safeguards, a standalone ApplyGuardrail API, filter a prompt for any model without invoking an FM, and cross-account safeguards that enforce one guardrail across an AWS Organization. The six-policy set and the published claims (88% harmful-content blocking, 99% Automated Reasoning accuracy) are unchanged, so Section 11 still answers every policy question; what changed is where those policies can be applied.

• 🔢 📚 Did you know? Exam-guide v1.1 made token-based pricing an explicit objective (2.1.4): the exam now expects you to reason about how token counts and pricing shapes move cost and performance, which is precisely the arithmetic already practised in Sections 2, 5 and 7. AWS states that guide changes appear on the exam about one month after publication, so anything v1.1 added is live: format is unchanged at 65 questions (50 scored + 15 unscored), 90 minutes, 700/1000 to pass.

---

9. The customization spectrum: prompt first, custom model last

9.1 The ladder AWS publishes

AWS's own summary: most workloads never go past Step 3. That sentence is worth more on the exam than any individual technique, because it converts "which customization should I use?" into "what is the cheapest rung that fixes the problem?", and every cheaper rung you skip is cost and latency you pay for nothing.

9.2 The fine-tuning vocabulary of Task 3.3

9.3 Reinforcement fine-tuning and distillation: the numbers

Reinforcement fine-tuning (RFT) uses reward functions, an AWS Lambda function implementing rule-based verification (RLVR), or a model-as-judge (RLAIF), with the GRPO optimisation method. AWS claims up to 66% accuracy gain. Guardrails on the feature itself: minimum 100 records, maximum 20,000 prompts for Nova models, and AWS's own advice to skip RFT when the baseline reward already exceeds 95%.

Model Distillation (GA May 2025) trains a smaller student on the teacher's outputs. AWS claims the student can be up to 500% faster, 75% cheaper, with less than 2% accuracy loss.

9.4 Worked example 8: should this team fine-tune?

A support team wants their assistant to adopt house style. Baseline evaluation reward on their held-out prompt set: 91%.

1. Baseline 91% < 95%, so RFT is technically available, but it needs ≥ 100 labelled records and adds training and reward-function maintenance.
2. AWS's ladder says try steps 1 to 3 first: prompt engineering (house style rules in the system prompt) and RAG (the knowledge base of past answers).
3. If style still fails after steps 1 to 3, fine-tuning (step 6) is the correct rung, and the team needs a labelled dataset, not more temperature tweaking.

Answer: do not start at step 6. If the baseline reward ever reaches above 95%, AWS explicitly says to skip RFT entirely.

• 🔢 📚 Did you know? The customization spectrum is the direct answer to the exam's recurring "the team wants to fine-tune, what should they do first?" shape, and AWS's 66% accuracy gain for RFT, 500% faster / 75% cheaper / <2% accuracy loss for distillation and "most workloads never go past step 3" are all AWS claims reproduced as published. Use them as directions (distillation is cheaper, RFT can help hard tasks, prompting and RAG come first) rather than as promises your architecture must deliver.

---

10. Evaluating generative AI

10.1 Three evaluation families in Amazon Bedrock Model Evaluation

Built-in datasets include Real Toxicity, BOLD, TREX, WikiText-2, Gigaword, BoolQ, Natural Questions and TriviaQA.

10.2 The judge rules that appear verbatim as answers

• AWS claims LLM-as-a-judge delivers up to 98% cost savings versus human review.
• Best practice: use an evaluator model from a different family than the generator: a judge from the same family inherits the same blind spots.
• Faithfulness is the hallucination metric: does the answer stay inside the retrieved context?
• Human review is not replaced; it is layered: judge metrics for scale, human workflows for sign-off.

10.3 The classic text-generation metrics of Task 3.4

10.4 RAG-specific metrics

The RAG triad is context relevance, faithfulness and answer relevance; AWS added citation precision and citation coverage in 2025, both scored 0 to 1 (coverage is N/A when the response is entirely unfaithful). AWS's published case: better chunking raised context relevance from 67% to 93% and removed hallucinations without changing the prompt: the clearest available evidence that in RAG, retrieval quality is usually the bug and the prompt is not.

10.5 Worked example 9: judge plus human, priced

A team must evaluate 2,000 generations:

1. LLM-as-a-judge scores all 2,000 on correctness, completeness, faithfulness and safety: AWS claims up to 98% cost savings versus doing it by hand.
2. Human review on a 10% sample of 200 tasks at the documented $0.21 per completed task: $200 \times 0.21 = $42.00$.
3. The judge model comes from a different family than the generator, so shared failure modes do not cancel out.

Answer: judge at scale for coverage, humans at $42.00 for a 200-task sample for authority: the pattern AWS documents rather than judge-only or human-only evaluation.

---

11. Guardrails, hallucination mitigation and responsible AI

11.1 The six Guardrails policies

AWS claims: 88% harmful-content blocking, 99% Automated Reasoning validation accuracy, and more than 75% of hallucinated responses filtered by grounding checks.

11.2 Automated Reasoning checks: the limits worth memorising

11.3 The hallucination mitigation ladder: in AWS's order

AWS documents exactly three routes, in this sequence:

1. Refine or optimise the prompt: cheapest, no infrastructure.
2. Use RAG to supply relevant data: grounds the answer in your documents.
3. Try a different model: some models hallucinate less on your workload.

Then, on top of the ladder, contextual grounding checks in Guardrails detect answers that are not supported by the source (AWS claims >75% filtered).

11.4 Worked example 10: how much does 75% filtering actually leave you?

A production chatbot emits 1,000 answers per day, and the team enables only contextual grounding checks:

$$1{,}000 \times (1 - 0.75) = 250$$

Answer: roughly 250 ungrounded answers still pass per 1,000 when grounding checks are the only control, and that is why AWS layers controls: content filters at 88%, Automated Reasoning at 99% on policy violations, evaluation jobs before release, and human review for the cases that remain. No single percentage in this section is a substitute for the stack.

11.5 The responsible-AI dimensions and their artifacts

AWS assesses generative AI systems across eight dimensions, fairness, explainability, veracity/robustness, governance, transparency, privacy & security, safety and controllability, which are codified in the AWS Well-Architected Responsible AI Lens. The transparency artifact AWS publishes for its own services is the AI Service Card, and the same lens vocabulary is what a "which dimension does this concern?" item is quoting.

• 🔢 📚 Did you know? AWS's published guardrail percentages (88% / 99% / >75%), the 66% RFT gain, the 98% LLM-as-a-judge savings, the 500% faster / 75% cheaper distillation claim and the 67% → 93% chunking case are all AWS claims with their own methodologies; they are reproduced here as AWS publishes them. On the exam, they function as ordering cues (which control is stronger, which step comes first), not as guarantees; treat any option that turns a claim into an absolute as suspicious.

---

12. Comparative verdict

---

13. Exam traps for this lesson

---

14. Real-World Case Studies

Everything to this point is mechanism, how tokens, retrieval, inference parameters and Guardrails behave. This section is evidence: AWS-published generative-AI deployments, each with the services used, the numbers AWS and the customer printed, and the source. Read them as pattern examples for Domain 3 (Applications of Foundation Models, 28%): the exam rarely asks for a company name, but it constantly asks which architecture shape delivers which outcome, and these are the shapes AWS itself publishes.

14.1 Six generative-AI deployments, with services and numbers

14.2 The three cases the exam will borrow from

Sun Finance: OCR and reasoning are separate jobs. The first prototype sent ID photographs straight to an LLM and was rejected at 61.8% overall (43% on ID numbers); AWS attributes the failure to privacy protections that block direct PII extraction. Separating the concerns: Textract reads, Claude structures, rules validate: produced 85% on the second attempt and 90.80% in the published pipeline, with −91% cost per document and 20 hours → under 5 seconds per case. That is Section 11's mitigation ladder applied before the parameters are touched: fix the pipeline, not the temperature.

Nippon India: retrieval beats parameter tuning. Naive RAG degraded as document volume grew and produced hallucinations, so the team engineered retrieval: FM-as-parser, query reformulation, multi-query RAG, reranker models, GraphRAG, metadata filtering, and layered Bedrock Guardrails and citations on top: +>95% accuracy, −90 to 95% hallucination, reports falling from 2 days to about 10 minutes. AWS flags all of these as GA product features, not customer-only code. The exam-shaped reading: when a grounded assistant hallucinates, the ladder is still prompt → RAG → different model, and retrieval quality is usually the bug (Section 10.4).

Adobe: chunking is a measurable hyperparameter. Four chunking strategies were benchmarked on Adobe's own test set; 400-token chunks with 20% overlap were both simplest and most accurate, giving +20% retrieval accuracy over incumbent search when embedded with Titan Text Embeddings V2 into Amazon OpenSearch Service. It is the live confirmation of Section 2.1: chunking quality, not prompt wording, is the biggest lever in a RAG system.

14.3 Before and after: the numbers worth carrying into the exam

• 🔢 📚 Did you know? The documented failures are the most exam-shaped part of these cases: Sun Finance's LLM-only extraction (61.8%), Prime Focus Technologies' first attempt on external LLM APIs whose latency was too high for live tagging, and Chronomics' 4 months of in-house computer vision that never hit target before Rekognition Custom Labels shipped in 3 to 4 weeks at 96.5% accuracy / 97.9% F1. All three restate Section 12 in production form: choose the tier deliberately (prebuilt → managed custom → generative), and evaluate before you launch: Epilot chose its model with Bedrock Evaluations rather than in a notebook.

---

Practice Questions

---