Active Example Selection
Not all demonstration sets are created equal. In-context learning is unstable: the same prompt with a different example set can swing accuracy sharply. Active Example Selection treats choosing demonstrations as a sequential decision problem and trains a reinforcement-learning policy to assemble sets that generalize.
*These are learning tools, not prompts. They teach you to write your own, think of them as training wheels that guide you while you learn, then fall away once you can ride on your own.
Treat Example Choice as Something You Optimize
Active Example Selection starts from an uncomfortable finding: in-context learning is highly sensitive to which examples you show and in what order. The same model, the same task, and the same prompt template can produce very different accuracy depending only on the demonstration set. Instead of guessing, this technique frames example selection as a problem you can optimize, searching for the set that performs best and generalizes to new inputs.
The original method (Zhang, Feng, and Tan, EMNLP 2022) formalizes this as a sequential decision process: an agent builds the prompt one demonstration at a time, and a reinforcement-learning policy learns which examples to add. The reward is simple, how well the assembled prompt performs on held-out data, so the policy is trained to chase real accuracy rather than surface similarity.
The payoff is generalization. Because the policy learns a strategy for selecting examples rather than memorizing one fixed set, it transfers to tasks it never saw during training and tames the instability that makes naive few-shot prompting unreliable.
Picking few-shot examples by hand is a guess, and the guess matters more than it looks. Studies of in-context learning show accuracy can swing by tens of points purely from which examples you include and how you order them. Active Example Selection replaces the guess with a learned policy, one trained against measured performance so the demonstrations you ship are the ones proven to help, not the ones that merely look representative.
The Active Example Selection Process
Four stages from candidate pool to a trained selection policy
Build a Candidate Pool and a Feedback Signal
Curate a bank of labeled input-output pairs, the candidate demonstrations the policy is allowed to choose from. Then set aside a held-out validation set that will score how well any assembled prompt actually performs. The pool defines the action space; the validation set defines success.
A customer support classification system curates 500 labeled tickets spanning billing inquiries, technical issues, cancellation requests, feature requests, and account recovery, plus a separate held-out set of tickets used only to score candidate prompts.
Frame Selection as Sequential Decisions
Model the act of building a prompt as a sequence of choices. The state is the set of demonstrations chosen so far; each action adds one more example from the pool. Building the prompt step by step lets the method weigh how each new example interacts with the ones already chosen, rather than scoring examples in isolation.
Starting from an empty prompt, the agent picks a first ticket to include, then a second conditioned on the first, then a third, assembling a demonstration set one decision at a time.
Reward What Works, and Learn From It
Score each assembled prompt by its accuracy on the held-out set, that score is the reward. A reinforcement-learning algorithm (the original paper uses Q-learning) updates the policy so that decisions leading to higher accuracy become more likely. Over many episodes, the policy learns which kinds of examples, in which combinations, actually raise performance.
The policy discovers that prompts pairing a clear billing example with a borderline billing-versus-cancellation case score higher than prompts of three near-identical tickets, so it learns to favor that combination.
Deploy a Policy That Generalizes
Apply the trained policy to assemble demonstrations for new tasks, including ones it never saw during training. Because it learned a selection strategy rather than memorizing a single set, it outperforms fixed or random selection and sharply reduces the run-to-run instability of naive few-shot prompting.
On a brand-new ticket type the policy was never trained on, its selected example set still lifts classification accuracy above a hand-picked baseline, the strategy transfers even when the specific examples do not.
See the Difference
Why a learned selection policy outperforms a fixed example set
Static Example Selection
Three examples chosen once by hand: a clear billing ticket, a clear technical issue, and a clear feature request. They were picked because they look representative, not because they were measured against held-out performance.
Accuracy is unstable: swap in a different “representative” trio and the score moves. On the mixed billing-and-cancellation ticket the model lands on “cancellation request,” the hand-picked set gave no signal for distinguishing overlapping categories.
Active Example Selection
The trained policy assembles the set that scored highest on held-out data: (1) a payment-method-change ticket labeled billing, (2) a conditional cancellation threat labeled cancellation, (3) a borderline case where billing was primary despite cancellation language, a combination it learned distinguishes the two classes.
The model classifies it correctly as “billing inquiry,” and the same policy holds up across the test set instead of swinging run to run. The learned set encodes the distinction the task actually needs.
Practice Responsible AI
Always verify AI-generated content before use. AI systems can produce confident but incorrect responses. When using AI professionally, transparent disclosure is both best practice and increasingly a legal requirement.
Most US states are actively legislating AI transparency and accountability. Critical thinking remains your strongest tool against misinformation.
Active Example Selection in Action
See how a learned selection policy beats hand-picked examples on hard cases
A customer support system receives the message: “My subscription renewed but I thought I cancelled it last month. Can you check what happened and reverse the charge? If this keeps happening I’m done with the service.” The case is genuinely ambiguous between a billing dispute, a cancellation request, and an account inquiry.
Why example choice matters here: with a random trio of demonstrations the label flips between “billing dispute” and “cancellation request” from run to run, the prompt, not the model, is deciding the answer.
Policy-selected set: the trained policy assembles three demonstrations it learned are discriminating: (1) a customer who disputed an unexpected charge but explicitly wanted to keep the service, labeled “billing dispute,” (2) a customer who cited a past cancellation attempt and demanded closure, labeled “cancellation request,” (3) a renewal-error complaint with frustration but no firm exit intent, labeled “billing dispute.”
Result: with this set the classification is stable and correct, “billing dispute,” because the customer’s core request is to reverse a charge and investigate, while the cancellation language is conditional (“if this keeps happening”) rather than definitive.
A data extraction pipeline encounters the date string “3rd Quarter FY2024 (Oct-Dec 2023)” in a financial document. The model needs to extract a standardized date range, but the fiscal year offset and informal quarter notation create ambiguity that standard date-parsing examples do not cover.
Why example choice matters here: depending on which date-parsing demonstrations are in the prompt, the model returns “2024-07 to 2024-09,” “2023-10 to 2023-12,” or “2024-10 to 2024-12”, the example set, not the input, drives the inconsistency.
Policy-selected set: the policy favors demonstrations that pin down the fiscal-versus-calendar rule: (1) a fiscal-year date whose correct output used the parenthetical calendar dates, (2) a “Q3 FY2025” case mapped to January through March 2025, (3) a case where parenthetical clarification overrode the fiscal-year label as the authoritative range.
Result: with this set the model reliably extracts “2023-10-01 to 2023-12-31”, when explicit calendar dates appear in parentheses, they take precedence over fiscal-year quarter calculations.
An automated code review system needs to categorize a pull request that renames a function from processData to validateAndTransformInput, updates its internal logic to add input validation, and fixes a null-pointer exception in the process. It is genuinely ambiguous whether this is a “refactor,” a “bug fix,” or a “feature enhancement.”
Why example choice matters here: with an arbitrary set of demonstrations the label splits three ways (“refactor,” “bug fix,” “feature enhancement”) and shifts depending on which examples are shown.
Policy-selected set: the policy assembles discriminating cases: (1) a PR that only renamed functions and reorganized code with no behavior change, labeled “refactor,” (2) a PR that renamed a function but also fixed an edge-case crash in the same function, labeled “bug fix” because the behavioral fix was the primary motivation, (3) a PR that added input validation without fixing any known bug, labeled “feature enhancement.”
Result: with this set the model reliably categorizes the PR as “bug fix”, when a rename and logic change accompany a fix for a known defect (the null-pointer exception), the defect resolution is the motivating change.
When to Use Active Example Selection
Best for systems where query diversity demands adaptive demonstrations
Perfect For
When the same system handles billing questions, technical issues, and account management, each query type benefits from fundamentally different demonstrations.
Domains where unusual inputs appear frequently, date formats, code patterns, legal language, and a fixed example set cannot anticipate every variant.
When generic examples leave performance gaps that matter, medical triage, financial classification, or content moderation where errors have real consequences.
Systems with hundreds or thousands of labeled examples available for retrieval, the larger the pool, the more precisely the system can match demonstrations to each query.
Skip It When
When the task is straightforward enough that the same fixed examples work consistently across all queries, sentiment analysis on product reviews, for instance.
When you cannot train or run a selection policy, the method requires a labeled candidate pool, a held-out scoring signal, and a reinforcement-learning loop to learn the policy.
When fewer than 50 labeled examples are available, dynamic selection offers little advantage over simply including all of them or manually curating a representative subset.
Use Cases
Where Active Example Selection delivers the most value
Production Classification Systems
Route incoming tickets, documents, or messages to the correct category by deploying a selection policy trained to assemble the demonstration sets that classify reliably, rather than relying on generic hand-picked examples.
Adaptive Tutoring
Train a policy to select the worked examples that most improve outcomes for each topic, assembling demonstration sets that teach a concept more reliably than a hand-picked set.
Real-Time Content Moderation
For borderline content where the policy line is subtle, a learned selection policy supplies the demonstrations that best sharpen the distinction between acceptable and prohibited content.
Dynamic Code Review
Categorize pull requests by retrieving examples of similar code changes that were correctly labeled, especially for borderline cases that blend refactoring, bug fixes, and feature additions.
Medical Triage
When symptom descriptions are ambiguous, retrieve case examples with similar presentations but different diagnoses to help the model distinguish between conditions that share overlapping symptoms.
Multilingual Query Handling
When processing queries that mix languages or use culturally specific idioms, retrieve demonstrations in the relevant language combination to help the model handle code-switching and cultural context accurately.
Where Active Example Selection Fits
Active Example Selection bridges static curation and full knowledge-base integration
The reason Active Example Selection matters is not that it finds one perfect prompt, it is that it learns a strategy for choosing examples. Because the policy is trained against measured accuracy rather than surface similarity, it learns which examples actually change the model’s behavior in the desired direction, and that strategy transfers to new tasks the policy never saw in training. The result is a principled answer to a question few-shot prompting usually leaves to guesswork: which demonstrations should you actually show?
Related Techniques
Explore complementary in-context learning techniques
Target Your Demonstrations
Build a learned example-selection policy or explore our prompting tools to design more effective in-context learning strategies.