The Cost of Production AI: How We Engineered PHHM to Scale Without Exploding Our LLM Bill

One of the biggest misconceptions about production AI is that your cloud bill is determined by model pricing. It isn't. Pricing matters. Architecture matters far more. Early in PHHM's development, we tracked token usage almost as an afterthought. The platform worked. Users were happy. Workflows completed reliably. Then we started looking at execution telemetry. Two workflows serving similar user requests consumed dramatically different numbers of tokens. Some specialist agents consistently generated far more text than they actually needed. The same context was being transmitted repeatedly between agents. Identical requests triggered identical model calls. Nothing was technically broken. The platform was simply doing far more work than necessary. That realization completely changed how we approached AI cost.

Instead of asking:

"Which model should we switch to?" We started asking: "Why is the platform spending these tokens in the first place?" That single question led to architectural changes that reduced operational cost without sacrificing workflow quality. More importantly, it changed how we think about AI economics. Today, cost isn't something we review at the end of the month. It's another engineering signal—just like latency, reliability, and validation failures.

Cost Doesn't Come From One Place

One mistake we made early was treating every AI request as though it had roughly the same cost. Production quickly proved otherwise. A simple welcome workflow might invoke two lightweight agents. A care-planning workflow could involve:

• the Overseer
• the Analyst
• the Care Agent
• validation
• downstream communications

Each additional step consumed more:

• prompt tokens
• completion tokens
• orchestration overhead
• validation time

By the time a workflow finished, the total cost reflected dozens of architectural decisions—not just one API call.

Every Workflow Has Its Own Economics

Eventually, every execution produced a detailed cost profile.

Care Planning Workflow

│

▼

Overseer .............. 280 tokens
Analyst ............. 1,520 tokens
Care ................. 940 tokens
Validation ............ 90 tokens
Communications ....... 610 tokens
──────────────────────────────
Total .............. 3,440 tokens

Looking at costs this way immediately changed the conversation. Instead of debating model prices, we could identify which parts of the workflow were genuinely expensive. That's where meaningful optimisation begins.

Cost Is an Architectural Property

By the end of the project, one engineering principle guided almost every optimisation. AI costs are an emergent property of architecture. The model only generates tokens. The platform decides:

• how many agents execute
• how much context each receives
• whether work is repeated
• whether retries occur
• whether responses are cached
• which model performs each task

Those decisions determine the majority of operational cost. The rest of the article explores the architectural changes that had the greatest impact.

The first optimization didn't involve changing models. It didn't involve changing providers. It didn't even involve changing our architecture. We started by questioning something we'd never measured before.

How much of every prompt was actually useful?

As PHHM evolved, prompts naturally became longer. Every new feature added another instruction. Another edge case. Another formatting rule. Another example. Another reminder. Individually, none of these additions seemed significant. Collectively, they became one of the largest contributors to token usage.

Prompts Naturally Accumulate Technical Debt

Prompt growth happens gradually. Version 1 might look like this.

System Role
↓
Instructions
↓
User Input

Six months later, the same prompt often looks more like this.

System Role
↓
Formatting Rules
↓
Business Rules
↓
Edge Cases
↓
Examples
↓
Reminders
↓
Historical Notes
↓
User Input

Every section was added for a good reason. Very few were ever removed.

We Started Measuring Prompt Efficiency

Instead of asking whether a prompt was "good," we asked better engineering questions.

• Which instructions are never exercised?
• Which examples no longer improve outputs?
• Which constraints duplicate validation rules?
• Which formatting instructions belong in code instead?
• Which context can be supplied dynamically?

Those questions revealed something surprising. Many prompts contained instructions that had become obsolete as the platform evolved.

Every Word Has a Cost

One mindset shift changed how we wrote prompts. Every token has three costs.

• It increases API usage.
• It increases latency.
• It increases cognitive load for the model.

More instructions don't automatically produce better reasoning. Sometimes they simply produce more noise.

Shorter Prompts Often Performed Better

This was one of the most surprising discoveries. Several prompts actually improved after becoming shorter. Why? Because removing irrelevant instructions made the core objective clearer. Instead of telling the model everything it should avoid... ...we focused on what it needed to accomplish. Less ambiguity. Less distraction.

Better outputs. Lower cost.

Static Instructions Moved into Software

Another optimization came from asking a simple question.

"Does the model actually need to know this?"

Many prompt instructions described deterministic behaviour. For example:

• output valid JSON
• include required fields
• reject invalid values
• follow schema

Those aren't reasoning tasks. They're validation tasks. Instead of reminding the model repeatedly, we enforced those rules after generation.

Model Output
↓
Schema Validation
↓
Business Validation
↓
Accepted Response

The prompt became smaller. The validation became stronger. Both improved simultaneously.

Examples Were Treated Like Code

Few things increase prompt size faster than examples. They're valuable. They're also expensive. Rather than continuously adding examples, we reviewed them like production code. Every example had to justify its existence. If removing one didn't reduce output quality, it stayed out. The result wasn't fewer examples. It was better examples.

Dynamic Context Replaced Static Context

Another common source of prompt bloat was static information. Early versions included large sections describing workflows that rarely changed. Eventually we replaced those with dynamic context. Base Prompt + Workflow Context + User Request

Each agent received only the information relevant to its current task. The prompt stayed compact. The context stayed relevant.

The second major improvement had almost nothing to do with AI. It was a classic software engineering technique. Caching. One pattern appeared repeatedly in our telemetry. The platform kept asking the same questions. Generate this member summary. Summarize this profile. Analyse this document.

Nothing had changed. Yet we were paying for another model invocation every time.

The Orchestrator Became the Cache Manager

Instead of allowing every agent to call a model independently, the orchestrator checked whether an existing result could be reused.

Workflow Request
↓
Cache Lookup
├── Cache Hit
│      │
│      ▼
│ Reuse Response
│
└── Cache Miss
       │
       ▼
   Call Model
       │
       ▼
   Store Result

This kept caching completely transparent to individual agents. Specialists focused on reasoning. The orchestrator handled efficiency.

Cache Stable Knowledge, Not Conversations

One lesson became clear very quickly. Not everything should be cached. We avoided caching:

• active conversations
• evolving workflow state
• personalised recommendations
• time-sensitive decisions

Those depend on fresh context. Instead, we cached information that changes infrequently. For example:

• member profile summaries
• analysed documents
• organisation metadata
• workflow templates
• reusable reference material

The result was a high cache hit rate without sacrificing accuracy.

Cache Invalidation Matters

Caching introduces its own engineering challenges. The hardest question isn't:

"When should we cache?"

It's:

"When should we stop trusting the cache?"

Every cached artifact includes clear invalidation rules.

Profile Updated
↓
Cache Invalidated
↓
Next Request
↓
Fresh AI Generation

Freshness always takes priority over savings. A stale recommendation costs far more than another API call.

The Results Were Larger Than Expected

Prompt compression reduced the number of tokens sent to the model. Caching reduced the number of model calls altogether. Together they changed the economics of the platform. Not by making the AI less capable. By ensuring we only paid for work that genuinely needed to happen.

The Bigger Lesson

Looking back, the biggest savings didn't come from clever prompt engineering. They came from applying decades of software engineering principles to AI systems. Remove duplication. Reduce unnecessary work. Cache expensive operations. Keep responsibilities separate. Those ideas existed long before large language models. They remain just as valuable today.

Early in development, we made a very common mistake. Every agent used exactly the same language model. It simplified configuration. It simplified deployment. It also meant we were paying premium prices for tasks that didn't require premium reasoning. The Welcome Agent doesn't solve complex analytical problems. The Communications Agent often formats existing information. The Overseer primarily routes workflows.

Treating every task as equally difficult was convenient. It wasn't economical.

Intelligence Should Match Complexity

Eventually, model selection became another orchestration responsibility. Instead of allowing every specialist to choose independently, the Overseer selected the most appropriate model for each task.

Workflow
↓
Overseer
├── Welcome → Lightweight Model
├── Communications → Standard Model
├── Analyst → Advanced Model
└── Care → Advanced Model

The agents never needed to know which provider or model they were using. They simply requested reasoning. The orchestration layer supplied the most appropriate capability.

Not Every Problem Needs Maximum Intelligence

One framework guided our routing decisions.

Notice the pattern. We reserved our most capable—and most expensive—models for the work that genuinely benefited from deeper reasoning. Everything else used simpler alternatives without affecting user experience.

Earlier in this engineering series, we explored how asynchronous execution improved workflow performance. It also influenced cost. Not by reducing token usage. By reducing idle time. Originally, many workflows executed sequentially.

Overseer
↓
Analyst
↓
Care
↓
Communications

Each agent waited for the previous one to finish. As workflows grew, those delays became increasingly noticeable.

Independent Work Should Execute Together

Many specialists don't depend on one another. Once we recognized those boundaries, the orchestrator began executing them concurrently.

Overseer
        │
 ├──────────────┬──────────────┐
 ▼              ▼              ▼

Analyst Care Communications The number of model calls stayed the same. The total execution time dropped significantly. That improved both user experience and infrastructure efficiency.

Faster Workflows Reduce Hidden Costs

Parallel execution doesn't reduce token consumption directly. It reduces another important cost. Time. Shorter workflows mean:

• fewer concurrent requests waiting in queues
• lower infrastructure utilisation
• improved throughput
• better user responsiveness

Cost engineering isn't only about API pricing. It's about overall operational efficiency.

Optimisation only works when engineers can see what's happening. That's why cost became another first-class metric inside PHHM. Every workflow automatically records:

• total tokens
• input tokens
• output tokens
• estimated execution cost
• model used
• execution duration
• cache hits
• retry count

Instead of guessing where money was being spent, we could see it.

Every Workflow Has a Financial Profile

Our internal dashboards evolved beyond simple token counts. Each workflow produced an operational summary.

Workflow
↓
Execution Time
↓
Token Usage
↓
Model Selection
↓
Cache Status
↓
Estimated Cost

One execution tells a useful story. Thousands reveal architectural trends.

Looking Beyond Individual Users

As adoption increased, another perspective became valuable. Instead of asking:

"How much did this request cost?"

We started asking:

• What does this workflow cost?
• What does an average user cost?
• What does each organisation consume?
• Which features generate the highest operational expense?

That changed optimisation discussions completely. Architecture decisions became data-driven instead of intuitive.

The Dashboard That Changed Our Thinking

Eventually, every deployment review included the same operational dashboard.

No single metric tells the whole story. Together they describe the economic health of the platform.

Cost Became an Engineering Signal

Perhaps the biggest mindset shift was this. We stopped treating cost as something the finance team monitored. Instead, engineers reviewed cost alongside:

• latency
• reliability
• validation failures
• deployment success
• workflow completion

Unexpected cost increases often indicated architectural inefficiencies long before users noticed any impact. In that sense, cost became another form of observability.

The Architecture We Ended Up With

By the end of the project, cost optimisation wasn't a separate activity. It was embedded throughout the platform.

                 User Request
                      │
                      ▼
                Orchestrator
                      │
     ├───────────────┼───────────────┐
     ▼               ▼               ▼
Context Builder  Cache Layer   Model Router
     │               │               │
     └───────────────┼───────────────┘
                     ▼
            Specialist AI Agents
                     │
                     ▼
           Validation & Monitoring
                     │
                     ▼
              Cost Dashboard
                     │
                     ▼
              Final Response

Cost wasn't optimized at the end. It was considered at every stage of execution.

The Five Principles of Cost Engineering

If I were designing another production AI platform tomorrow, these are the principles I'd adopt from day one.

1. Measure workflows—not requests

Users experience complete workflows. That's where meaningful optimisation begins.

2. Remove unnecessary work before changing models

The largest savings usually come from eliminating redundant computation—not replacing providers.

3. Match model capability to task complexity

Reserve advanced reasoning for problems that genuinely require it.

4. Make caching part of orchestration

Repeated work is one of the easiest sources of avoidable AI spend. The orchestrator is the ideal place to eliminate it.

5. Treat cost as operational telemetry

Unexpected spending often reveals architectural inefficiencies before other monitoring systems do.

Final Thoughts

When people ask how to reduce the cost of production AI, they often expect recommendations about model pricing. That's certainly one lever. It just wasn't the most important one for PHHM. The biggest improvements came from changing the architecture around the model. We compressed prompts so the model received only what it needed. We cached stable results instead of regenerating them. We matched models to the complexity of the task. We executed independent work in parallel.

We measured every workflow instead of estimating monthly spend. None of those changes reduced the platform's capabilities. They simply removed unnecessary work. That's why I now think about AI costs the same way I think about latency or reliability. They're not finance metrics. They're engineering metrics. When your architecture is efficient, lower costs become a natural consequence—not the primary objective.

Key Takeaways

If you're building production AI systems, I'd recommend adopting these practices from the beginning:

• Measure cost at the workflow level rather than per API request.
• Continuously review prompts and remove instructions that no longer add value.
• Move deterministic rules into validation code instead of repeating them in prompts.
• Cache stable AI outputs wherever freshness isn't critical.
• Route tasks to models based on reasoning complexity.
• Execute independent agents concurrently to improve throughput.
• Track token usage, cache hits, retries, and latency together.
• Build dashboards that explain why costs change, not just how much they changed.
• Treat cost as another form of engineering observability.
• Design systems that spend tokens intentionally rather than automatically.