How PHHM runs thousands of regression tests in CI/CD without spending money on API calls—and why most AI behaviour can be verified before a model is ever invoked.
Focus
Testing
Read time
10 min
Series
PHHM Journal
Theme
Production AI
One question came up repeatedly as PHHM grew.
"How do you run regression tests without spending hundreds of dollars on API calls?"
At first, our answer was simple. We didn't. Every test invoked a live language model. Every pull request generated real completions. Every workflow consumed real tokens. It worked. Until it didn't. As the platform expanded, so did the test suite.
A single change to the orchestration layer triggered hundreds of AI calls. CI pipelines slowed down. Costs increased. Test results became inconsistent because language models don't produce identical outputs every time. The problem wasn't our code. It was our testing strategy. That's when we realized something important.
Most of our platform wasn't AI.
It was orchestration. Routing. Validation. State management. Permissions. Retries. Observability. Workflow execution.
None of those components require a language model to verify. Once we separated platform behaviour from model behaviour, our testing strategy changed completely. Today, PHHM runs well over a thousand regression tests during CI without making a single production LLM call. The result?
Faster feedback.
Zero API cost.
Deterministic test results.
Higher confidence before deployment.
Ironically, removing the language model from most tests made us significantly better at testing AI systems.
The Biggest Misconception in AI Testing
Early on, we assumed every AI feature required a real AI model during testing. That assumption felt reasonable. If an application depends on GPT-4, surely every test should call GPT-4. The longer we worked on PHHM, the more that assumption broke down. Consider a simple workflow.
User Request
↓
Overseer
↓
Analyst
↓
Validation
↓
Care
↓
Response
How much of that flow actually depends on a language model? Surprisingly little. Most of the workflow is ordinary software engineering. The LLM contributes one step. The platform contributes everything else.
AI Platforms Are Mostly Software
This realization completely changed how we thought about testing. Instead of viewing PHHM as "an AI application," we started viewing it as a software platform that happened to use language models. That distinction matters. Because software platforms have decades of testing practices we can reuse. We don't need to reinvent testing. We need to decide which parts genuinely require intelligence and which parts simply require correctness.
Two Different Kinds of Tests
Eventually, our entire testing strategy split into two categories.
Platform Tests
↓
No LLM Required
──────────────
Model Tests
↓
Live LLM Required
That separation immediately reduced both cost and complexity.
What Doesn't Need an LLM?
More than most teams expect. For example, we can fully test:
workflow routing
YAML loading
configuration parsing
permission checks
validation rules
retry logic
workflow state transitions
execution IDs
correlation IDs
observability
audit logging
deployment behaviour
None of those require generated text. They're deterministic engineering problems.
What Does Need an LLM?
Some behaviours genuinely require live models. For example:
reasoning quality
summarization
tone
empathy
hallucination resistance
prompt effectiveness
instruction following
Those tests remain important. They're simply a much smaller part of the overall test suite.
The Principle That Changed Everything
Looking back, one engineering principle reshaped our entire CI pipeline.
Don't test the model when you're trying to test the platform.
That single sentence reduced costs, improved reliability, and made every deployment pipeline dramatically faster.
Part 1
Designing a Mockable AI Platform
The biggest architectural change wasn't in our tests. It was in our application. Early versions of PHHM called the language model directly from individual agents.
Example
response = client.responses.create(...)
It worked. Until we wanted to test the workflow. Now every unit test needed:
API credentials
network access
available quota
a live model
deterministic luck
None of those belong in a fast test suite. The problem wasn't OpenAI. The problem was our architecture.
Separate the Interface from the Implementation
Instead of allowing agents to call models directly, every interaction now goes through a common interface.
Agent
↓
LLM Interface
├── Production Model
└── Mock Model
The agent no longer knows whether it's talking to GPT-4, another provider, or a test double. It simply requests a completion. That separation became the foundation of our entire testing strategy.
Dependency Injection Makes Testing Easy
Once we introduced an abstraction layer, swapping implementations became trivial. In production:
Example
llm = OpenAIProvider()
During testing:
Example
llm = MockProvider()
Nothing inside the workflow changes. The orchestrator executes exactly the same logic. Only the implementation behind the interface is different. That's a classic software engineering technique—and it works just as well for AI.
What Does the Mock Actually Return?
One misconception is that a mock has to generate realistic AI responses. It doesn't. Its job is to return predictable responses that allow the workflow to execute. For example:
Example
{
"summary": "Mock member summary",
"risk": "medium",
"recommendation": "Follow up within seven days."
}
The wording doesn't matter. The structure does. We're testing orchestration—not creativity.
Part 2
Deterministic Inputs Create Deterministic Tests
One of the biggest advantages of mocks is consistency. A live model may produce slightly different wording every time. A mock always produces the same response.
Input
↓
Mock Response
↓
Validation
↓
PASS
That predictability eliminates flaky tests. If a regression appears, it's because the platform changed—not because the model happened to phrase something differently.
Testing the Workflow, Not the Model
Once the model became replaceable, our tests became much more focused. Instead of asking:
Example
"Did GPT produce a good answer?"
We asked:
Did the Overseer choose the correct workflow?
Did the expected agents execute?
Was workflow state updated correctly?
Did validation pass?
Were retries triggered appropriately?
Was the final response assembled correctly?
Those questions remain valuable regardless of which model powers the platform.
Golden Responses Replace Live Completions
Earlier in the series we discussed golden datasets. Here they become incredibly powerful. Instead of calling a model, the mock loads a predefined response.
Test Scenario
↓
Golden Response
↓
Workflow Execution
↓
Assertions
The workflow behaves exactly as though an LLM had responded.
Example
Except:
there's no network latency
there are no API costs
there is no output variation
That's ideal for regression testing.
Behaviour Matters More Than Text
One temptation is to compare generated text character by character. We deliberately avoid that. Instead, every test focuses on behaviour. For example:
Was the correct workflow selected?
Did the right validation schema execute?
Was the response accepted?
Did downstream agents receive the expected structure?
Was the workflow completed?
The exact wording is irrelevant. The workflow behaviour is what matters.
CI Pipelines Became Dramatically Faster
Removing live model calls had an immediate operational impact. Our continuous integration pipeline no longer waited for external APIs.
Git Push
↓
Unit Tests
↓
Workflow Tests
↓
Regression Suite
↓
Deploy
Everything runs locally. Everything is deterministic. Most importantly, every engineer gets rapid feedback without consuming AI credits. That encourages frequent testing rather than avoiding it because it's slow or expensive.
Faster Tests Encourage Better Engineering
One unexpected benefit had nothing to do with AI. When the test suite became fast, engineers started writing more tests. A workflow that takes milliseconds to verify gets tested often. A workflow that depends on dozens of external API calls usually doesn't. Reducing friction improved engineering discipline across the project.
The Mock Isn't Pretending to Be Smart
This is an important distinction. Our mock doesn't attempt to simulate a language model. It doesn't invent new responses. It doesn't generate creative text. It simply provides known outputs so the rest of the platform can be exercised. That's exactly what a good test double should do. Its purpose isn't realism. Its purpose is isolation.
The Bigger Lesson
Looking back, one principle reshaped our entire testing strategy.
Mock the dependency. Test the behaviour.
The language model is a dependency. The workflow is the behaviour. Separating those two ideas allowed us to build a regression suite that was:
deterministic
inexpensive
fast
reliable
easy to run on every pull request
That's a much stronger foundation than relying on live model calls for every test.
The AI Testing Pyramid: Balancing Speed, Cost, and Confidence
By the time PHHM matured, one thing had become obvious. Neither extreme worked. Testing everything with mocks missed model-specific problems. Testing everything with live LLMs made CI painfully slow, expensive, and inconsistent. The answer wasn't choosing one approach. It was combining both. The goal became simple:
Use the cheapest test that still gives you meaningful confidence.
That philosophy shaped our entire testing strategy.
Not Every Test Needs Intelligence
One mistake many teams make is assuming every AI-related feature requires a live model. In reality, most engineering questions have nothing to do with language generation. For example:
Did the correct workflow execute?
Was the right agent selected?
Was validation successful?
Did retries behave correctly?
Was workflow state updated?
Were audit records created?
Did observability capture the execution?
Those questions don't require GPT-4. They require deterministic software tests.
Build Confidence in Layers
Instead of relying on one giant test suite, PHHM evaluates the platform in layers.
Manual Evaluation
▲
Live LLM Testing
▲
Workflow Regression Tests
▲
Unit & Integration Tests
Each layer answers different questions. Together they provide confidence without unnecessary cost.
Layer 1: Unit Tests
The foundation is exactly what you'd expect in any Python application. We test:
routing logic
YAML parsing
validation rules
state management
configuration loading
retry policies
utility functions
These tests run in milliseconds. No models. No APIs. No network. Thousands can execute on every pull request.
Layer 2: Workflow Regression
Above that sits the workflow regression suite. These tests replace live models with deterministic mocks.
Golden Dataset
↓
Mock Provider
↓
Workflow Execution
↓
Assertions
Now we're testing the complete orchestration layer. Exactly the same routing. Exactly the same validation. Exactly the same workflow state. Without paying for AI inference. This became the largest layer in our testing strategy.
Layer 3: Live Model Evaluation
Eventually, some questions genuinely require intelligence. Examples include:
reasoning quality
summarization accuracy
instruction following
prompt effectiveness
tone
empathy
hallucination resistance
Those evaluations still use live language models. They're simply executed far less frequently. Instead of every commit, they typically run:
before releases
after prompt changes
during scheduled evaluations
when introducing new models
That dramatically reduces cost while preserving confidence.
Layer 4: Human Review
Automation eventually reaches its limits. Some qualities remain subjective. For example:
clarity
empathy
readability
usefulness
domain-specific nuance
Those are evaluated by humans. Importantly, humans no longer review every workflow. Automation filters out routine cases. Human attention focuses on the areas where judgement genuinely adds value.
CI/CD Becomes Predictable
Separating these layers transformed our deployment pipeline.
Git Push
↓
Unit Tests
↓
Workflow Regression
↓
Quality Gates
↓
Merge
↓
Scheduled Live Evaluation
↓
Production
Most pull requests now complete quickly because they never wait for external AI services. That encourages engineers to run the pipeline often. Fast feedback creates better engineering habits.
Live Models Become Validation, Not Infrastructure
One mindset shift changed everything. We stopped treating language models as part of our testing infrastructure. We started treating them as another dependency to validate. That distinction matters. Our CI pipeline shouldn't fail because an external API is slow. It should fail because our platform regressed. Separating those concerns made the entire delivery process much more reliable.
Confidence Comes from Diversity
No single testing technique catches every problem. Mocks verify deterministic behaviour. Regression suites protect workflows. Live evaluations verify reasoning. Humans assess quality. Observability monitors production. Together they create overlapping layers of confidence. That's far stronger than depending on one expensive end-to-end test.
The Architecture We Ended Up With
By the end of the project, testing had become another first-class architectural capability.
Developer
│
▼
Git Commit
│
▼
Unit & Integration Tests
│
▼
Mock-Based Workflow Tests
│
▼
Regression Suite
│
▼
Quality Gates
│
▼
Deployment
│
▼
Live Model Evaluation
│
▼
Production Monitoring
│
▼
New Golden Test Cases
Notice the feedback loop. Production continuously strengthens the test suite. Every incident becomes another deterministic regression test. That's how confidence compounds over time.
The Five Principles of AI Testing
If I were building another production AI platform tomorrow, these are the principles I'd adopt from day one.
1. Test the platform separately from the model
Most regressions occur in orchestration, validation, state management, and workflow logic. Those don't require an LLM.
2. Mock external dependencies
Language models are external services. Treat them like any other dependency during automated testing.
3. Reserve live evaluations for behavioural quality
Use real models to verify reasoning, instruction following, and prompt effectiveness—not routing logic.
4. Make deterministic tests your default
The faster and more predictable the test suite, the more frequently engineers will use it.
5. Let production improve the test suite
Every bug. Every incident. Every regression. Should become another permanent test. Over time, your testing strategy becomes smarter than any single engineer's memory.
Final Thoughts
When we started PHHM, we assumed AI systems required AI-powered testing. Experience taught us something different. Most of the platform wasn't artificial intelligence. It was software engineering. That meant decades of proven testing practices still applied. Dependency injection. Test doubles. Golden datasets.
Behavioural assertions. Regression suites. Continuous integration. Observability. The language model changed what our application could do. It didn't change how disciplined engineering teams build reliable systems. Today, our CI pipeline executes thousands of regression tests without making a single production LLM call. Not because we don't trust language models.
Because we understand exactly which parts of the platform actually require one. That's the difference between testing an AI model... ...and testing an AI platform.
Key Takeaways
Example
If you're building production AI systems, I'd recommend adopting these practices from the beginning:
Separate platform testing from model evaluation.
Introduce an LLM interface so providers can be mocked during testing.
Use dependency injection to swap production and mock implementations.
Build golden datasets around workflows rather than individual prompts.
Assert behaviour instead of exact wording.
Run deterministic regression suites on every pull request.
Reserve live model evaluations for reasoning, quality, and prompt validation.
Keep human review focused on subjective qualities that automation can't reliably assess.
Feed production incidents back into your regression suite.
Build an AI testing pyramid that balances speed, cost, and confidence.