Two tracks: policybazaar.ai, and shifting to open source models.
Each with its own four steps. They share the evals, the models and the inference.
Track 1 · policybazaar.ai
One context layer and your own models, starting on policybazaar.ai.- 01Harness on pb.ai
- Baseline: quality, cost, speed
- Context layer, router, skills
- Every chat logged and scored
- 02Train PB Flash and PB Pro
- With the router, on step 1's logs
- Ships only if it beats baseline
- Scored on your evals
- 03More channels
- WhatsApp first: shared backend
- Then PB Buddy, web and app
- One context, one set of models
- 04Improve from usage
- Every chat in production scored
- Failures become training data
- Each release beats the last
Track 2 · Shifting to open source models
Move Policybazaar's busiest model work onto cheaper models that pass your evals.- 01Find high-usage workflows
- Map where tokens go today
- Rank by volume and cost
- Pick the first to move
- 02Build evals, pick the model
- Evals from your own cases
- Open and closed, same tests
- Cheapest model that passes
- 03Inference for these models
- One API, hosted in India
- Guardrails and audit built in
- Cost and quality watched live
- 04Scale across workflows
- Next workflow, same playbook
- Evals rerun on every release
- Savings tracked every month