examples

Examples and use cases

Real tests you can run with your agent — what to compare, how to score it, and what your agent reports back once it finds a winner.

example 01

Finding the best prompt wording

A team wants to know which of three system-prompt versions produces the best summaries, judged by the same scoring rules every time.

you, to your agent

"Use the improve-fast skill to test three summarization prompts: concise, detailed, and bulleted. Score each summary 0–1 like this: 1.0 if it needs no edits, 0.5 if it needs small edits, 0.0 if it needs a rewrite. Let me know when there's a winner."

your agent

Tries a version, generates a summary with it, scores that summary against the rules (or has you confirm the score), and repeats — sending more tries to whichever version is doing best while still checking the others. After about 60 scored tries, it's confident enough to report: detailed wins, averaging 0.81 across 21 tries versus 0.68 and 0.61 for the other two, and updates the production prompt.

example 02

Comparing model and temperature settings

An engineering team compares two model and temperature settings for a code-review agent, judged by whether a human reviewer accepts the suggestion.

you, to your agent

"Compare a cautious, low-temperature setting against a more exploratory, high-temperature one for our code-review agent, using improve-fast. Score 1.0 if the human reviewer accepts the suggestion outright, 0.5 if they change it, 0.0 if they reject it. Run it across our next batch of pull requests and tell me which setting wins."

your agent

Tries each setting across incoming pull requests, recording the reviewer's response as the score each time and leaning into whichever setting is scoring better. Partway through, it can tell you the current standings if you ask; once the leader has enough tries to be confident, it reports a winner with the number of tries and the average score, and recommends switching the default setting.

example 03

Testing support agent strategies

A support team compares "resolve directly" against "ask a clarifying question first" strategies, judged by customer satisfaction.

you, to your agent

"For our support agent, use improve-fast to test two response strategies: resolving the ticket directly versus asking a clarifying question first. Score each conversation using the customer's post-chat satisfaction rating, on a 0–1 scale. Run it across incoming tickets and tell me which strategy does better."

your agent

Picks a strategy for each incoming ticket, uses it, and records the satisfaction rating as the score. After roughly 50 tickets, it reports a winner — clarify-first, averaging 0.79 versus 0.63 for resolving directly — and switches the agent's default strategy to match.

Need more help?

Check out the full documentation for more details.

Read the docs