The plan

Two tracks: policybazaar.ai, and shifting to open source models.

Each with its own four steps. They share the evals, the models and the inference.

Track 1 · policybazaar.ai

One context layer and your own models, starting on policybazaar.ai.
  1. 01
    Harness on pb.ai
    • Baseline: quality, cost, speed
    • Context layer, router, skills
    • Every chat logged and scored
  2. 02
    Train PB Flash and PB Pro
    • With the router, on step 1's logs
    • Ships only if it beats baseline
    • Scored on your evals
  3. 03
    More channels
    • WhatsApp first: shared backend
    • Then PB Buddy, web and app
    • One context, one set of models
  4. 04
    Improve from usage
    • Every chat in production scored
    • Failures become training data
    • Each release beats the last

Track 2 · Shifting to open source models

Move Policybazaar's busiest model work onto cheaper models that pass your evals.
  1. 01
    Find high-usage workflows
    • Map where tokens go today
    • Rank by volume and cost
    • Pick the first to move
  2. 02
    Build evals, pick the model
    • Evals from your own cases
    • Open and closed, same tests
    • Cheapest model that passes
  3. 03
    Inference for these models
    • One API, hosted in India
    • Guardrails and audit built in
    • Cost and quality watched live
  4. 04
    Scale across workflows
    • Next workflow, same playbook
    • Evals rerun on every release
    • Savings tracked every month