Session 08🧩

Question Answering & Prompt Engineering

QA task design and evaluation, BERT span extraction, retrieval-augmented generation, prompting strategies, and learned prompts for VLMs.

Learning Objectives

✓Classify QA systems by answer form, context source, and retrieval setting
✓Compute Exact Match and token-level F1 for extractive QA
✓Explain why RAG helps and why retrieval quality and context placement still limit it
✓Choose between zero-shot, few-shot, decomposition, sampling, and search-based prompting
✓Connect manual prompt engineering to learnable prompts through the BiomedCoOp case study

Question Answering Is a Family of Tasks

A QA system maps a natural-language question and some source of knowledge to an answer. The answer may be a text span, paragraph, yes/no decision, database entry, or list. The context may be one passage, a document collection, knowledge graph, table, image, or the web. Reading comprehension assumes a supplied passage; open-domain QA must locate evidence; conversational QA carries dialogue state; long-form QA must synthesize rather than extract. Define the answer and evidence contract before selecting a model.

SQuAD, Exact Match & Token F1

SQuAD pairs a passage with a question and a short answer span. Exact Match gives one only when the normalized prediction exactly equals a gold answer. Token-level F1 gives partial credit from precision and recall over overlapping answer tokens. Because several wordings may be plausible, evaluation compares against multiple human answers and takes the best score before averaging across questions. These metrics fit extractive spans; they are insufficient for factuality, citation quality, and completeness in long-form answers.

P=\frac{|tokens_{pred}\cap tokens_{gold}|}{|tokens_{pred}|},\quad R=\frac{|tokens_{pred}\cap tokens_{gold}|}{|tokens_{gold}|},\quad F_1=\frac{2PR}{P+R}

Normalize articles and punctuation consistently before computing EM or token overlap.

BERT for Extractive Reading Comprehension

An extractive BERT reader encodes the question and passage together. Two learned vectors score every contextual token as a possible answer start and end. Training minimizes the start and end cross-entropies. Inference searches valid spans and returns the highest joint score, subject to constraints such as end after start and a maximum answer length. This formulation is powerful when the answer appears verbatim in the passage, but it cannot generate an answer that requires wording absent from the context.

p_{start}(i)=\operatorname{softmax}(w_s^{\top}h_i) \qquad p_{end}(i)=\operatorname{softmax}(w_e^{\top}h_i)

Each token representation

h_i

receives independent start and end scores.

Retrieval-Augmented Generation

A language model cannot memorize every changing fact or private document, and an unsupported answer is difficult to verify. RAG retrieves relevant chunks just in time, places them in the model context, asks for an answer grounded in those chunks, and can return citations. The knowledge base becomes updateable without retraining model parameters. Retrieval recall is a hard ceiling: if the evidence is missing, the generator cannot ground itself in it. Chunking, metadata filters, reranking, deduplication, and citation verification are therefore core model-quality work.

RAG pipeline as explicit stagesPython

def answer(question, index, reranker, generate):
    candidates = index.search(question, top_k=40)
    evidence = reranker(question, candidates)[:8]
    prompt = build_grounded_prompt(
        question=question,
        passages=evidence,
        require_citations=True,
    )
    response = generate(prompt)
    return verify_citations(response, evidence)

Long Context Is Not Unlimited Attention

Passing more documents is not always better. Irrelevant chunks increase cost, distract generation, and can push evidence into positions the model uses poorly. In the lecture's retrieval experiments, retriever recall continued rising while end-to-end RAG performance saturated after a modest number of documents. Treat context length as a budget: retrieve broadly, rerank precisely, include a small diverse evidence set, and test robustness by moving the relevant passage to different positions. Measure retrieval and answer quality separately.

Prompt Anatomy & Shot Prompting

A useful prompt states the task, supplies necessary context, defines constraints, gives an output schema, and specifies what to do when evidence is insufficient. Zero-shot prompting gives instructions only. One-shot gives one demonstration. Few-shot provides several representative input-output pairs and is especially useful for format, label meaning, and edge cases. Demonstrations should match the target distribution and should not leak the test answer. Prompt engineering is iterative experimental design: version prompts, freeze an evaluation set, and compare results.

A testable prompt templatePython

prompt = f'''Task: classify the sentiment as POSITIVE, NEGATIVE, or NEUTRAL.

Rules:
- Use only one of the three labels.
- If positive and negative evidence balance, choose NEUTRAL.

Examples:
Text: The battery lasts all day.
Label: POSITIVE
Text: It arrived broken.
Label: NEGATIVE

Text: {text}
Label:'''

Reasoning, Sampling & Decomposition Strategies

Chain-of-Thought prompting asks for intermediate reasoning and historically improves some arithmetic, commonsense, and symbolic tasks, especially with sufficiently capable models. Self-consistency samples multiple reasoning paths and selects the most consistent answer. Generated-knowledge prompting produces useful facts before answering. Least-to-Most decomposes a hard problem into ordered subproblems. Tree of Thoughts searches across partial solutions and can backtrack. These methods trade additional calls and tokens for reliability; score the final answer with task-specific tests instead of assuming longer reasoning is better.

Automatic, Active & Directional Prompting

Automatic Prompt Engineer generates candidate instructions, executes them on examples, and selects the best with an evaluation score. Auto-CoT clusters questions and samples demonstrations from different clusters. Active prompting samples multiple answers, uses disagreement as uncertainty, and asks humans to annotate the most uncertain examples. Directional Stimulus Prompting supplies a small hint or policy-trained stimulus that guides generation. The common pattern is a closed loop: generate candidates, measure behavior, select informative feedback, and update the prompt or lightweight parameters.

Case Study: BiomedCoOp Learns the Prompt

Manual prompt engineering is not the endpoint. BiomedCoOp adapts BiomedCLIP for few-shot biomedical image classification while preserving the pretrained encoders. It learns context tokens and anchors them to ensembles of domain descriptions generated by an LLM. Semantic Consistency by Contextual Mapping pulls learned prompts toward the average class knowledge. Knowledge Distillation with Selective Prompting removes outlier descriptions using a median-absolute-deviation rule, then distills the selected prompt distribution. The combined cross-entropy, semantic, and distillation objective improved accuracy and base-to-novel generalization across 11 datasets, showing how prompt quality can become a trainable, evaluated component.

\mathcal{L}=\mathcal{L}_{CE}+\lambda_1\mathcal{L}_{SCCM}+\lambda_2\mathcal{L}_{KDSP}

BiomedCoOp balances task labels, semantic consistency, and selective prompt distillation.

Evaluation Is Part of the Prompt

Create a held-out set before tuning. Include common cases, edge cases, ambiguous inputs, adversarial wording, and examples with insufficient evidence. Track task accuracy or F1, schema validity, citation support, refusal correctness, latency, and cost. Change one prompt component at a time and keep a baseline. For QA, test retrieval misses separately from generation failures. For prompts used in sensitive domains, add expert review and uncertainty handling; fluent output is not evidence of correctness.

←PreviousAttention & Transformers