Nov 2 · 5 min read
How we measured "40% faster" without fooling ourselves
Anyone can say a redesign is faster. Here's how we tested a Salesforce chatbot prototype against the live one, and the one detail that kept the data honest.
Our Salesforce chatbot redesign turned walls of text into cards, questions, and choices. It felt faster. But feeling faster is not a result. People are bad at judging their own speed: in one 2025 study, developers using AI were 19% slower while believing they'd been 20% faster (METR). If we wanted to say our design saved time, we had to time it.
So we set up a head-to-head. We picked a realistic goal: understand what a product does, then choose between two options. We used Sales Cloud, so the comparison stayed on one real product.
On one side was the chatbot live on Salesforce's website. On the other was our prototype. We programmed the prototype with one scripted conversation that answered the same set of questions, so both versions were asked exactly the same things.
Participants, people who could realistically use the chatbot, asked the same questions and went through the same conversation on both.
The detail that mattered most was order. If everyone tries the live chatbot first, they've already learned the product by the time they reach the prototype, and of course the second one feels faster. So we flipped it: some people did the prototype first and the website second, others did the website first and the prototype second. That way, learning from the first round couldn't make either version look better.
On the live chatbot, it took people much longer to understand what a product was about and to choose between two. With the prototype, the cards let them scroll faster. Their understanding came out the same, it just came sooner.
That's what our number means. "40% less browsing effort" is 40% less time to reach the same understanding and make a decision. Not more clicks, not a happier rating. Time to the same answer.
It's a prototype result, not a production one. A scripted conversation is cleaner than real traffic, and I'd want to see it hold up live. But it's a number I can explain, and that's the kind I want to bring to any team.