Evaluation and Regression Testing for AI Workflows: Why Testing Prompts Isn't Enough
How we built repeatable evaluation pipelines for PHHM by testing complete workflows instead of individual prompts.
Focus
Evaluation
Read time
12 min
Series
PHHM Journal
Theme
Production AI
The first prompt we wrote for PHHM looked great. It generated clean summaries. The formatting was consistent. The responses made sense. Every manual test passed. So we deployed it. Within a day, workflows started failing. Not because the prompt was wrong.
Because the workflow was. A small prompt change caused the Analyst Agent to return recommendations in a different structure. The response still looked correct to a human. But the Care Agent expected a different contract. Validation failed. Retries increased. Workflow latency doubled. Nothing was technically broken.
Yet the platform became less reliable. That was the moment we realized something important.
We weren't deploying prompts. We were deploying workflows.
From that point on, we stopped evaluating prompts in isolation. We started evaluating the entire system.
Part 1
Why Prompt Testing Doesn't Scale
Most AI tutorials demonstrate prompt testing like this.
Input
↓
Prompt
↓
Output
If the answer looks reasonable... The prompt passes. That works for experiments. It doesn't work for platforms. PHHM isn't one prompt. It's a sequence of coordinated decisions.
User Request
│
▼
Overseer
│
┌────┼───────────┐
▼ ▼ ▼
Analyst Care Communications
│ │ │
▼ ▼ ▼
Validation Validation Validation
│ │ │
└──────┼───────────┘
▼
Final Response
Testing one prompt tells us almost nothing about whether the workflow still works.
A Prompt Can Improve While the Platform Gets Worse
One of the hardest lessons we learned was that prompt quality and workflow quality aren't the same thing. Imagine improving the Analyst prompt. It now generates more detailed recommendations. Great. Except those recommendations increase token usage by 60%. Validation takes longer. The Care Agent retries more often. Overall workflow latency increases by two seconds.
The prompt improved. The platform regressed. That's why workflow evaluation became our primary metric.
Think Like a Software Engineer
Traditional software isn't tested one function at a time. We also test:
integration
end-to-end behavior
contracts
regression
performance
AI systems deserve exactly the same discipline. Instead of asking:
"Did the prompt produce a good answer?"
Ask:
"Did the workflow behave correctly?"
That small wording change completely transformed how we approached testing.
Part 2
Defining Workflow Quality
Before writing tests, we had to answer a more important question.
What does a successful workflow actually look like?
Interestingly, the answer had very little to do with wording. A workflow is successful if:
the correct agents were selected
routing followed the expected path
validation passed
contracts remained compatible
retries stayed within acceptable limits
execution completed successfully
latency remained acceptable
Notice what's missing. Perfect prose. The platform cares about behavior. Not adjectives.
The Biggest Mindset Shift
One sentence summarizes almost the entire testing strategy.
We're not testing language. We're testing system behavior.
That distinction became the foundation of every evaluation pipeline in PHHM.
The Architecture We Wanted
By the time evaluation became a first-class concern, this was the flow we were aiming for.
Nothing reaches production because a prompt "looks better." It reaches production because the workflow proves it behaves correctly.
Why This Matters
Looking back, evaluation wasn't really about prompts. It was about confidence. Confidence that changing one component wouldn't quietly degrade everything else. That's exactly what regression testing provides. It doesn't guarantee perfection. It dramatically reduces surprises.
The Lesson That Changed Everything
Example
If there's one idea I'd want readers to remember from this article, it's this:
Prompt engineering is a development activity. Evaluation is an engineering discipline.
Production AI systems need both.
Part 3
Building Golden Datasets: Testing Workflows Instead of Prompts
Once we stopped evaluating prompts individually, another question appeared.
What exactly should we test?
At first, we collected examples of good prompts. That didn't help much. Changing one prompt rarely broke the prompt itself. It broke something downstream. Eventually we realized our test cases shouldn't represent prompts. They should represent real workflows. Every test became a complete user journey from request to final response.
What Is a Golden Dataset?
A golden dataset is a curated collection of representative workflows that should continue behaving correctly as the platform evolves. Think of it as the AI equivalent of a regression test suite. Instead of checking whether one response "looks good," we verify that the entire orchestration behaves as expected.
The workflow itself becomes the unit under test. That's much closer to how traditional software is evaluated.
Good Tests Describe Behaviour
One temptation is to compare the entire AI response against a stored answer. That rarely works. Language models naturally produce variation. Instead, PHHM focuses on behavioural assertions. For example:
Was the correct workflow selected?
Did the expected agents execute?
Did validation succeed?
Was the output contract satisfied?
Were retries within acceptable limits?
Did the workflow complete?
These assertions remain stable even if the wording changes.
Avoid Exact String Matching
Suppose yesterday the Analyst generated:
Example
"Schedule a follow-up within seven days."
Today it generates:
Example
"Arrange a follow-up appointment next week."
Both responses satisfy the same intent. Exact string comparison would incorrectly report a failure. Behavioural evaluation accepts both because the recommendation still satisfies the business objective. That's why we evaluate meaning through workflow behaviour rather than literal text.
Designing Representative Scenarios
One lesson became obvious very quickly. Tiny datasets produce misleading confidence. We deliberately built scenarios that reflected real production diversity. Examples included:
straightforward onboarding
incomplete member information
conflicting care recommendations
invalid user requests
unusually long conversations
high-risk member cases
multilingual requests
edge-case routing decisions
The goal wasn't to cover every possible input. It was to cover every important category of behaviour.
Happy Paths Aren't Enough
Early in development, almost every test represented a successful workflow. Production quickly proved that unrealistic. Good regression suites deliberately include failures. For example:
Testing recovery paths proved just as valuable as testing successful ones.
Regression Tests Protect Architecture
One unexpected benefit of workflow testing was architectural confidence. Suppose we update:
an agent prompt
a routing rule
a validation schema
a configuration file
a model provider
The regression suite immediately tells us whether the platform still behaves correctly. Notice what we're protecting. Not individual prompts. The architecture itself.
Version the Dataset Too
Prompt versions aren't the only assets that evolve. Evaluation datasets evolve as well. Every scenario should be version-controlled. For example:
As new workflows are introduced, new scenarios are added. The dataset grows alongside the platform. That's exactly how software test suites mature.
Every Bug Becomes a Test
One engineering habit paid off more than almost anything else. Whenever production exposed a bug... We added it to the golden dataset. That means every incident permanently improves the platform. The workflow that failed yesterday becomes tomorrow's regression test. Over time, the evaluation suite becomes a living record of everything the platform has learned.
Synthetic Data Complements Real Data
Production scenarios are invaluable. They're also limited. To explore edge cases, we supplemented them with synthetic workflows. Examples include:
extremely large member profiles
ambiguous requests
conflicting instructions
intentionally malformed inputs
unusual routing combinations
Synthetic data helped us explore situations that rarely occurred in production but could still reveal weaknesses. The goal wasn't realism. It was resilience.
Evaluation Is Continuous
One mistake we avoided was treating evaluation as something that happens only before deployment. Instead, evaluation became part of the development cycle.
Every meaningful change passes through the same process. That consistency builds confidence over time.
The Biggest Lesson
Looking back, one principle shaped our entire evaluation strategy.
Every production incident should become a permanent test case.
That's how the platform improves. Not by hoping the same mistake never happens again. By ensuring it can't happen unnoticed.
Part 4
Automating Evaluation: Bringing Regression Testing into CI/CD
One of the biggest mistakes we made early was treating evaluation like a checklist. Someone changed a prompt. Someone manually tested a few examples. Everything looked fine. The change was deployed. A few days later, an unrelated workflow started failing. Nothing had been intentionally broken. The problem was that manual testing doesn't scale.
As PHHM grew, prompts, routing rules, validation schemas, configuration files, and models changed constantly. No engineer could reliably verify every workflow by hand. That's when evaluation became part of the deployment pipeline.
Every Change Should Prove Itself
One architectural rule guided the entire release process.
Every meaningful change must demonstrate that it hasn't made the platform worse.
Notice the wording. Not better. Not smarter. Not faster. Simply not worse. That's exactly what regression testing is designed to verify.
Evaluation Becomes a Quality Gate
Instead of deploying immediately after a prompt change, every update passes through an evaluation pipeline.
Nothing is mocked unnecessarily. The goal is to evaluate the orchestration exactly as it will behave in production.
Behavioural Assertions
Earlier, we avoided comparing exact wording. Automation follows the same philosophy. Instead of asking: "Did the response match this paragraph?" We ask questions such as:
Was the correct workflow selected?
Were the expected agents executed?
Did validation succeed?
Were contracts preserved?
Did retries remain below the threshold?
Did execution finish successfully?
Those assertions stay stable even as prompts evolve.
Define Success Before Running Tests
One lesson became surprisingly important. Every scenario should define success before execution. For example: scenario: member_report
Now evaluation becomes objective. The workflow either satisfies the contract or it doesn't.
Regression Isn't Just Functional
One misconception is that regression only checks correctness. Production systems also monitor operational characteristics. For every workflow we track metrics like:
execution time
retry count
validation failures
token consumption
model latency
total workflow duration
Suppose a prompt update still passes validation but doubles token usage. That's still a regression. Quality includes operational efficiency.
Baselines Matter
Every workflow establishes a historical baseline. For example:
Metric
Previous
Current
Result
Success Rate
99.1%
99.0%
✅ Pass
Validation Failures
1.2%
1.3%
✅ Pass
Average Latency
2.1 s
2.0 s
✅ Pass
Tokens
2,380
3,940
❌ Regression
The output is still correct. The deployment should still be reviewed. Evaluation isn't only about correctness. It's also about efficiency.
Prompt Changes Need Evidence
One engineering habit dramatically reduced deployment risk. Every prompt modification answers one question.
Show the evaluation results.
Not opinions. Not screenshots. Evidence. If the regression suite improves or maintains workflow quality, deployment proceeds. If not, the change goes back for revision.
CI/CD Makes Evaluation Automatic
Eventually evaluation became another automated stage in the delivery pipeline.
Nobody remembers to run tests. The pipeline does it automatically. That's how software engineering has worked for years. AI systems deserve the same discipline.
Evaluation Reports Should Explain Decisions
One lesson from observability carried directly into testing. Reports shouldn't simply say: PASS They should explain why. A useful evaluation summary includes:
workflows executed
success rate
failed scenarios
validation results
retry statistics
token changes
latency differences
prompt version
model version
That makes reviews much more meaningful than a simple pass/fail indicator.
Human Evaluation Still Matters
Automation doesn't eliminate human judgement. It changes where humans spend their time. Instead of manually checking hundreds of routine scenarios, reviewers focus on:
nuanced language quality
empathy
clarity
tone
domain-specific correctness
Automation verifies predictable behaviour. Humans evaluate subjective quality. Together they create a much stronger review process.
Every Deployment Teaches the Platform
One unexpected benefit emerged over time. Every production issue eventually became:
a new golden dataset scenario
a new regression test
a stronger evaluation suite
The testing framework evolved alongside the platform. Instead of remaining static, it continuously learned from production experience. That's exactly how mature software test suites grow.
The Evaluation Lifecycle
Looking back, evaluation became another continuous engineering loop.
Code or Prompt Change
│
▼
Automated Evaluation
│
▼
Regression Analysis
│
▼
Deployment Decision
│
▼
Production Monitoring
│
▼
New Test Cases
Notice the feedback loop. Production informs testing. Testing protects production. Each cycle strengthens the platform.
The Biggest Lesson
Example
If there's one engineering principle I'd carry into every future AI project, it's this:
Don't deploy because the output looks good. Deploy because the evidence says the workflow is still reliable.
That shift—from subjective confidence to measurable evidence—is what turns AI development into AI engineering.
Part 5
Evaluation Is an Engineering Discipline, Not an AI Feature
When people first start building AI applications, evaluation usually happens informally. A prompt is updated. A few example questions are tried. The responses look better. The change is deployed. That approach works surprisingly well. Until the platform grows. Once multiple agents collaborate, prompts evolve independently, workflows become more sophisticated, and multiple engineers contribute changes, informal testing stops being sufficient.
At that point, evaluation becomes infrastructure. Not because AI is special. Because production software requires confidence.
Confidence Comes From Evidence
One realization shaped almost every deployment in PHHM.
Confidence should come from measurements, not intuition.
A developer may believe a prompt is better. An evaluation pipeline demonstrates whether the workflow actually improved. Those aren't the same thing. Engineering should always prefer evidence over opinion.
Every Deployment Becomes an Experiment
Instead of treating deployments as final decisions, we began treating them as controlled experiments. Every release answers questions such as:
Did workflow completion improve?
Did validation failures increase?
Did retries decrease?
Did latency remain acceptable?
Did token consumption change?
Did routing behaviour remain stable?
Every deployment generates new operational evidence. That evidence guides the next improvement.
Production Closes the Feedback Loop
One lesson from previous articles becomes incredibly important here. Evaluation doesn't end when deployment succeeds. Production continues evaluating every workflow.
Development
│
▼
Regression Tests
│
▼
Deployment
│
▼
Production Metrics
│
▼
Observability
│
▼
New Test Cases
The platform continuously teaches the evaluation pipeline. Every production incident becomes another opportunity to strengthen future releases.
Testing Protects Innovation
One unexpected benefit was psychological rather than technical. Engineers became more willing to improve the platform. Why? Because they trusted the safety net. Without regression testing:
Example
"I'm not sure this change is safe."
With regression testing:
Example
"Let's evaluate it."
That confidence accelerated development while reducing operational risk. Ironically, better testing allowed us to move faster.
Evaluation Is About Behaviour
Looking back, the biggest mistake we made early was focusing on responses. Eventually we realized we should have been measuring behaviour instead. For every workflow we asked questions like:
Did the correct specialist execute?
Was the routing decision correct?
Were contracts preserved?
Did validation succeed?
Was recovery required?
Did the workflow complete successfully?
Those questions remain meaningful regardless of which model generated the text. Behaviour survives model upgrades. Prompt wording changes. Framework migrations. That's exactly why behavioural evaluation ages so well.
The Complete Quality Pipeline
By the end of the project, every significant change followed the same lifecycle.
The important point is that evaluation doesn't replace observability. It complements it. Observability explains what happened. Evaluation helps prevent the same issue from happening again.
The Five Principles of AI Evaluation
If I were building another multi-agent AI platform tomorrow, these are the principles I'd follow from day one.
1. Test workflows—not prompts
Users experience complete workflows. That's the unit that deserves evaluation.
2. Measure behaviour instead of wording
Routing, validation, contracts, retries, and successful completion are more stable indicators than exact text.
3. Automate every regression
Manual testing doesn't scale. Regression testing should be part of every deployment pipeline.
4. Learn from production
Every incident should become another regression test. Production continuously improves the evaluation suite.
5. Let evidence drive deployments
Ship changes because the workflow demonstrated reliability—not because the output looked impressive.
Final Thoughts
When I started building PHHM, I assumed the biggest challenge would be writing effective prompts. It wasn't. The harder challenge was creating an engineering process that allowed those prompts to evolve safely. That required a shift in perspective. Instead of asking:
"Is this prompt better?"
We started asking:
"Is the platform still behaving correctly?"
That single question influenced every evaluation strategy we adopted. It led us to:
test workflows instead of prompts
measure behaviour instead of wording
automate regression testing
integrate evaluation into CI/CD
treat production incidents as future test cases
The prompts improved over time. The evaluation pipeline made those improvements safe to deploy. That's the difference between experimenting with AI and operating an AI platform.
Key Takeaways
Example
If you're building production AI systems, I'd recommend adopting these practices from the beginning:
Build golden datasets around real workflows.
Assert behaviour rather than exact wording.
Version evaluation datasets alongside prompts and configuration.
Include successful paths and recovery paths in every regression suite.
Measure latency, retries, validation, and token usage—not just output quality.
Automate workflow evaluation inside your deployment pipeline.
Turn every production incident into a permanent regression test.
Combine automated evaluation with targeted human review.
Use observability to continuously strengthen your evaluation suite.
Treat workflow reliability as the primary deployment metric.