Next steps
Both tracks start this week. Dates are proposed, to agree together.
Track 1 · policybazaar.ai
| When | Step | What happens | Owner |
|---|---|---|---|
| By 18 Nov | 1Data sync and KPIs* | Data systems, SOPs and past pb.ai chats shared; KPIs and today's baseline agreed | Policybazaar, with Dvar |
| By 25 Nov | 2Post-train PB Flash and PB Pro | Router, PB Flash and PB Pro trained on pb.ai's chats; scored on your evals against today's Gemini stack | Dvar |
| By 5 Dec | 3Pilot | 1.5 weeks on a share of live pb.ai traffic, with a control group; quality, cost and latency tracked daily | Both |
| 15–30 Dec | 4Iterate and full rollout | Fixes from the pilot, then all pb.ai traffic; evals and observability stay on | Both |
* Dates assume Policybazaar's data and access are ready by 18 Nov. Data sync can take up to twice as long; each later date moves with it.
Track 2 · Shifting to open source models
Target: Start saving 50% on 1–2 high-volume workflows within 2–3 weeks
| When | Step | What happens | Owner |
|---|---|---|---|
| By 18 Nov | 1Pick 1–2 high-volume workflows | Where tokens go today, by workflow: volume, model, cost and owner; the 1–2 with the biggest saving go first | Policybazaar, with Dvar |
| By 24 Nov | 2Evals and the open model | Evals on the owner's own cases; the cheapest open model that matches today's quality and speed | Dvar, with each owner |
| By 1 Dec | 3Live and saving 50% | Traffic moves over with evals and observability on; spend tracked against today's bill every week | Both |
| From Dec | 4Next workflows, same playbook | Ranked by saving; each moves only when it passes its evals | Dvar, with each owner |