In 2023 I built a small Colab notebook that used PaLM's chat-bison to role-play a customer. It was fun and not much more. The customer was a random adjective attached to a product area, and the feedback was one freeform prompt that mostly said "good job."

Customer Scenario Simulator II is the rebuild. This piece walks through the three decisions that made it useful: giving the customer something to hide, grading with a rubric instead of a vibe, and testing the grader before trusting it. The code is on GitHub.

The shape of the system

There are two agents, one web service, and one warehouse.

A session has three calls. /session/new creates a persona. /chat sends your message and returns the customer's reply. /grade hands the whole conversation to the coach and returns a scorecard.

The two agents never talk to each other. They have separate system prompts and separate jobs, and the only thing that passes between them is the transcript. That separation is the whole design. The persona must not know how it will be graded, or it starts performing for the rubric. The coach must not be in the conversation, or it stops being a judge.

Give the customer something to hide

The first version failed because the customer had no interior. Ask it what it needed and it told you. Real customers do not do that. The stated problem ("we want to cut our inference bill") is rarely the real one ("our CTO promised the board a launch date and we do not have the capacity").

So each session starts by generating a structured persona card. Here is a simplified illustration of the idea, not the exact schema in the repository:

{
  "company": "Series B logistics startup, 60 engineers",
  "role": "VP of Engineering",
  "personality": "skeptical, short on time",
  "product_area": "data analytics",
  "stated_goal": "Cut warehouse spend before the next board meeting",
  "hidden_objectives": [
    "A migration last year went badly and the team distrusts lift-and-shift",
    "The real deadline is a customer audit in six weeks"
  ],
  "reveal_rules": "Share a hidden objective only when asked a specific, open question about it"
}

The persona agent receives the card and the running conversation state as context on every turn. The person practicing never sees the hidden objectives. They have to earn them, the same way they would in a real meeting: by asking better questions.

Two things made this work in practice.

Structure beats adjectives. "Be a difficult customer" produces theater. A card with a stated goal, specific hidden objectives and explicit rules for when to reveal them produces a customer who is consistent across twenty turns.

State is context, not memory. The service keeps session state itself and injects it on each call. The model is never asked to remember what it already revealed. It is told.

Grade with a rubric, and demand evidence

When you finish, the coach agent gets three things: the full transcript, the hidden objectives, and a rubric. The rubric lives in a YAML file and is the single source of truth for what "good" means. Changing how Customer Engineers are assessed is a change to that file, not to a prompt buried in code.

The coach returns structured JSON: a score for each rubric dimension, and for each score, the turns in the transcript that justify it. Then the scorecard reveals what the customer was hiding, so you can see which objective you found and which one walked out of the room with them.

The evidence requirement matters more than the score. A grader that says "discovery: 3 out of 5" teaches you nothing. A grader that says "3 out of 5, because in turn 4 you asked about timeline but accepted the first answer, and in turn 9 you pitched before confirming the constraint" gives you something to do differently tomorrow.

Structured output from a model fails occasionally, so the service validates the response against a schema and allows exactly one repair attempt if parsing fails. One retry catches almost everything. Unlimited retries hide a prompt problem you should fix instead.

Grade the grader

This is the part most LLM-as-judge projects skip, and it is the part I would keep if I had to throw the rest away.

A judge is only useful if it is consistent and calibrated. So the repository includes a small harness with hand-labeled golden transcripts: one strong, one mixed, one weak. Each has an expected score range written down in advance. The harness runs every transcript through the judge several times and checks three things:

1. Consistency. Does the same transcript get roughly the same score on every run?
2. Calibration. Do the scores land inside the range a human said they should?
3. Ranking. Does strong beat mixed, and mixed beat weak, every time?

python -m evals.judge_consistency --runs 3

If you change the rubric, the coach prompt or the model, you rerun this before you believe a single scorecard. It is the same discipline as a unit test, pointed at the one component whose output everyone will take on faith.

Keep the history, but never block on it

Every session, turn and score is written to BigQuery, with a leaderboard view on top. That turns a toy into a training tool. You can ask whether your discovery scores improved over a month, or which rubric dimension a whole team is weakest on.

Persistence is fail-soft. If BigQuery is unreachable, the conversation carries on and the app says nothing dramatic. Session state lives in memory and BigQuery is the durable record. For a practice tool that is the right trade: losing a row of analytics is cheap, and killing someone's session in the middle of a hard conversation is not. You can also switch BigQuery off entirely with one environment variable and run the full loop on a laptop.

Deployment notes that cost me time

• Scale to zero. The service is a source-based Cloud Run deploy. A practice tool is idle most of the day and should cost nothing while it is.
• No keys in code. Authentication is Application Default Credentials end to end.
• Use /health, not only /healthz. On Cloud Run the Google Front End intercepts /healthz before it reaches your container. I registered both paths and point checks at /health.
• One script. A single deploy script enables the APIs, creates the dataset, tables and view, and ships the service.

What I would tell a startup building something similar

The pattern here is not specific to sales training. Any product where one model plays a role and another evaluates the result has the same three risks.

Keep the actor and the judge apart. Shared context is how a judge ends up grading intentions instead of outcomes.

Put the rubric in data. The people who own the standard should be able to read it and change it without reading Python.

Test the judge first. If the grader cannot rank three hand-labeled examples in the right order, nothing built on top of it means anything.

No leaderboard would have told me that. It took a small, representative evaluation on the task I cared about, and that one finding said more about whether the model fit my workload than any benchmark score.

The full article builds a working app twice with Gemma 4, once against a hosted endpoint and once on a laptop. This page is the argument without the code.

The trade is generality against fit

The usual framing is that big models are smart and small models are not. That stopped being useful a while ago. Small models have improved along with the frontier, and on bounded tasks a current small model can beat a much larger old one.

A frontier model is a generalist. You pay for that breadth in compute, latency, money and a network trip to someone else's data center. A small model deployed well is a specialist. It does fewer things, and in exchange it can give you properties a hosted API cannot.

Four properties worth paying attention to

• Privacy. When inference runs inside a boundary you control, prompts and outputs can stay there. I have watched deals stall for months on data residency questions that get easier once that is true. You still owe the access controls, logging and governance.
• Cost. Self-hosting swaps per-token spend for capacity you pay for whether or not it is busy. For high-volume, low-complexity work such as classification or extraction, that can be cheaper to sustain. It is never free.
• Latency. No network round trip. Measure time to first token on the device you will ship on, not the one on your desk.
• Availability. This is the one I care about most. A tutoring app that needs a stable connection to a US-hosted API is not a product for a student in rural Ghana. A model that runs offline on a $200 device is a different product, and it reaches people the cloud version leaves out by design.

A rule you can use on Monday

Reach for a frontier model when the task needs deep multi-step reasoning, very broad knowledge, or the best possible open-ended writing.

Reach for a small one when the task is bounded, repetitive, sensitive, latency-critical, or has to run somewhere with poor connectivity. A lot of the production features I see teams build sit in that second list, and they are running on a frontier model because it was the default.

So: use the smallest model that passes a representative evaluation, and escalate only when the workload makes you. The same app can have two right answers. In the article, the hosted demo uses a 26B mixture-of-experts model and the laptop build uses the 2B edge model, on purpose.

One note on terms

Open source, open weights and local are three different things. Open-source software publishes code you can study and change. An open-weight model publishes trained parameters, often without the full training recipe. A local model is one you run on infrastructure you control, whatever its license. Gemma 4 is an Apache 2.0, open-weight family. Keeping those apart saves a lot of confused meetings.

1. Decide what kind of build this is before writing any code

There are three honest options: use an agent someone already built, adopt a partner's, or write your own. We wrote our own, code-first, because the workflow encoded logic specific to the team and we needed control over orchestration.

Most startups skip this decision and discover it later, halfway through fighting a tool that was never meant for their case. Ten minutes on it up front is cheap.

2. Let the model talk and let code do the work

The design that held up was a hybrid. A language model handles the conversational part, such as onboarding the user and collecting what it needs. Then it hands off to deterministic workflow steps that loop over the real work, with each step wrapped as a tool.

The instinct to make everything an agent is strong and usually wrong. If a step has one correct way to run, it should be code. Save the model for the parts that need judgment.

3. Isolate state, or your agent gets worse as it goes

Our agent processed a list of companies in a loop. Left alone, context from the first company bled into the fifth, and quality dropped the longer it ran. The fix was to wrap the research pipeline as its own agent, called as a tool, so every item started with a clean context.

If your agent is good on the first task and strange by the tenth, look here first.

4. Treat the path to production as part of the build

A demo on localhost proves the idea. It does not prove you can ship it. We started from a production-grade project template, set up CI/CD with a manual approval before production, and deployed to a managed runtime that scales to zero when idle.

None of that is exciting. All of it is the difference between an agent your team shows in a meeting and one your customers use.

Why I keep coming back to this project

Every one of these lessons is about structure, not model quality. The model was never the hard part. Knowing where to put the boundaries was.

A warning before the summary. The original compares specific features as they stood in July 2024, and both products have moved a long way since. Check the current documentation for what each one supports today. The decision underneath the feature table is the part that lasted.

Two tools for two moments

Google AI Studio is where you find out if the idea works. It is a web-based sandbox built for speed. You sign in, try prompts, and have a prototype the same afternoon.

Vertex AI Studio is where you make it something customers can rely on. It sits inside Google Cloud, next to your data, your identity controls and your billing. It exists for the things a prototype does not need and a product cannot live without.

The questions that decide it

When a startup asks me which to use, I ask four things back.

1. Who will use what you build? If the answer is "me, this week," start in the sandbox. If it is "thousands of customers," you need the production platform.
2. What data goes through it? The moment real customer data is involved, enterprise security and compliance controls stop being optional.
3. Will you need to adapt the model? Tuning, distillation and the MLOps tooling around them live on the platform side.
4. How much traffic, and how predictable? Sandbox quotas are tight by design. Platform quotas are set per project and region, and can be raised.

What I tell founders now

Start in the sandbox. Move when one of those four answers changes, and not before. The mistake I see is rarely choosing the wrong tool. It is staying in the prototype tool six months after the product became real, or building production infrastructure for an idea nobody has validated yet.

Both tools let you export your work as code, so the move is a step and not a rewrite.