How to test a prompt, step by step

Suppose you ask an AI assistant to make a seven-day study plan. The answer looks organized, but Tuesday contains an hour of work even though you only have 30 minutes. The table is formatted correctly; the plan is still wrong.

Prompt testing means checking the answer against the task, rather than judging it by how convincing it sounds. We'll use this study planner throughout the guide.

1. Write down the task before the prompt

Describe the inputs and the result you need. Our inputs are the subject, topics, number of days, and time available each day. Our output is a schedule with a topic, an activity, a duration, and a self-test for each day.

A requirement is something you can inspect. “Make a useful plan” is vague. “No day exceeds 30 minutes” can be checked directly.

Inputs: biology; cells, genetics, ecology; 7 days; 30 minutes/day
Output: a daily table with an activity and a self-test
Constraints: stay within the time limit; cover the supplied topics
If full coverage cannot fit: explain the shortfall

2. Build a small evaluation set

An evaluation set is a collection of inputs you run through the prompt. Each input should reveal a different possible problem.

Start with an ordinary request, then change one important condition. Try ten minutes per day, a missing topic list, and a deadline that cannot fit the requested work. Include a case that supplies unrelated text so you can check whether the assistant stays on task.

Keep a few cases aside while revising the prompt. These are your holdout cases: inputs you did not use to tune the instructions. They help reveal whether a revision only fits the examples you kept showing it.

3. Define the expected behavior for each case

You do not need to write the exact answer in advance. You do need to know what would count as acceptable.

For missing topics, expect a clarification or an explicitly labeled proposal. Do not accept invented topics presented as user-supplied facts. For an unrealistic time budget, expect reduced scope or a clear limitation instead of a promise to cover everything.

Case: a 30-minute daily limit
Pass: every day's activities total 30 minutes or less
Fail: any day exceeds the limit without asking to change it

Case: no topic list supplied
Pass: ask for topics, or label suggested topics as a proposal
Fail: claim the user supplied a topic list they never gave

4. Separate format checks from content checks

Format checks examine structure: required columns, seven distinct days, and numeric durations. You can often automate them.

Content checks examine meaning. Does a self-test relate to the day's topic? Does the plan actually cover the supplied material? These require more judgment, and factual statements may require external sources.

A response can pass the first group and fail the second. Keep those results separate so a well-formed table does not conceal an unsuitable answer.

5. Run the original prompt and save the outputs

Save the prompt, inputs, model identifier, relevant settings, and date of the run. Preserve the actual outputs. Do not replace an awkward result with a cleaned-up example and then call it a test result.

Run some cases more than once when consistency matters. One successful answer shows that the model can produce that answer; it does not establish that it reliably follows the requirement.

6. Review using a short rubric

A rubric is a set of criteria used to assess the response. Keep each criterion specific enough that a second reviewer can understand your decision.

For this planner, check the daily time total, topic coverage, self-tests, and treatment of missing information. Mark each criterion pass, fail, or unclear, and point to the part of the answer that supports the verdict.

If you use another model as a reviewer, compare its verdict with your own on representative cases. An AI grader is another component to evaluate, not an independent guarantee of correctness.

7. Revise the instruction that caused the failure

Suppose the model keeps exceeding the daily limit. Add an explicit instruction to total the activities and reduce scope when necessary. Do not simultaneously change the model, add five examples, and rewrite the whole task unless you are willing to lose track of which change mattered.

Before: Create a seven-day study plan.

Revision: Create a seven-day study plan with no more than 30 minutes
of activities per day. Include a duration for each activity.
If the topics cannot fit, flag the uncovered material rather than
quietly increasing the daily time.

8. Rerun the same cases and look for regressions

A regression is a behavior that used to work but fails after a change. The revised planner may respect time limits while dropping a supplied topic without saying so.

Compare the original and revised outputs against the same rubric. Then run the holdout cases. Keep the failures visible even if an overall score improves.

Your evaluation record should answer four questions: what changed, which cases improved, which cases became worse, and what remains unresolved. Only report scores or improvements after you have measured them.

Try the process with the study-planning project. For a shorter exercise on writing requirements, use Check a response.