PHHM Journal • Production AI Series
Engineering essays on production AI
A fifteen-part series on building multi-agent AI systems that survive production — from architecture and APIs through guardrails, observability, cost engineering, and the war stories in between. Best read in order.
Building a Multi-Agent AI Orchestration Layer in Python
Lessons from six collaborating agents — how PHHM reduced workflow time by 85% and errors by 73% with specialist agents, central orchestration, and shared workflow state.
Building Production AI APIs with FastAPI
Why AI APIs are orchestration, not CRUD — a single workflow endpoint, Pydantic validation at the boundary, and keeping FastAPI thin while the orchestrator does the thinking.
Configuration-Driven AI: Why YAML Beats Hardcoded Prompts in Production
How PHHM adds new agents, models, prompts, and workflows without rewriting application code — configuration changes behavior, not code structure.
State Management for Multi-Agent AI
Workflow state is not shared memory. Every execution owns structured context that survives retries and agent handoffs, so the platform behaves like one system.
Prompt Versioning at Scale
Prompts are deployable software assets — semantic versions, workflow mapping, regression tests, and rollbacks that make prompt changes boring instead of risky.
Guardrails for Multi-Agent AI Systems
Treat every AI response as untrusted input. Schema validation protects the software, business rules protect the meaning, and recovery paths keep failures contained.
Observability for Multi-Agent AI Systems
You can't debug what you can't see — correlation IDs, structured logs, distributed traces, workflow metrics, and audit trails that make AI systems explain themselves.
Evaluation and Regression Testing for AI Workflows
Stop testing prompts in isolation. Golden datasets, behavioral assertions, and CI/CD evaluation gates that prove the workflow still works before every deploy.
Securing Multi-Agent AI Systems
Prompt injection isn't the root problem — trust boundaries are. Why the model sits outside the trusted core and never decides what the platform does.
Why We Didn't Use LangChain in Production (And What We Built Instead)
Evaluating LangChain, CrewAI, and AutoGen taught us an important lesson — use frameworks for capabilities, but own the architecture that differentiates your platform.
From Monolith to Multi-Agent: The Refactoring We Should Have Done Earlier
Simplicity has a hidden expiration date. How PHHM outgrew its single-agent design and rebuilt into six specialists without breaking production.
The Production Incident That Changed How We Deploy AI Prompts
No alert fired. Nothing crashed. A post-mortem on the single sentence that quietly degraded our Care Agent — and how the telemetry found it.
Testing AI Systems Without Calling a Single LLM
Thousands of regression tests in CI/CD with a zero-pound API bill — mockable interfaces, dependency injection, golden responses, and deterministic workflows.
Cost Engineering for Production AI — Part 1: Measure
You can't optimize costs you don't measure. Cost per workflow, token accounting, and why context — not model pricing — is usually the biggest expense.
The Cost of Production AI — Part 2: Optimize
The cheapest AI call is the one you never make — prompt compression, intelligent routing, caching, and parallel execution that made PHHM economically sustainable.