A SaaS dashboard's onboarding flow scored well in five usability sessions and then lost close to a third of real signups at the same step within the first two weeks of launch. The gap between the test and reality was the test's participants, not the flow.
What the test sessions had in common
Five participants, recruited and briefed, sitting with a facilitator, motivated to finish because that's the nature of a scheduled session. None of them were interrupted by a phone call, none of them were doing this at the end of a long day, and none of them had a second browser tab open to something more urgent.
Where real users actually dropped off
At a step asking for a company size and industry before showing any value. In the test sessions, participants answered it because they were mid-session and the facilitator was watching. In the wild, it was the first point where a user could reasonably ask "why do you need this before showing me anything", and enough of them left rather than answer.
What we changed
- Moved value before commitment. Users now see a working, if generic, version of the dashboard before being asked for any company detail, and the detail is requested to personalise something they've already seen work.
- Made the company-size question optional, with a sensible default, since it wasn't actually blocking anything downstream — it had been asked because it seemed reasonable to ask, not because anything needed it immediately.
- Instrumented drop-off by step in production, not just recorded in usability sessions, so a future regression shows up as a number rather than a hunch.
A usability session tells you whether people can complete a flow when someone is watching them try. It doesn't tell you whether they want to.
Why we still run usability sessions
They're excellent at catching confusion — genuinely unclear labels, broken expectations, dead ends. They're weak at measuring motivation, because motivation in a test session is partly manufactured by the act of being tested. We now pair sessions with production funnel data before calling an onboarding flow finished, rather than treating the session as the final word.
The number that mattered
Signup completion rate by day, split by whether the user had the browser tab in focus for the whole flow — a proxy for how much attention they were actually giving it, which the usability sessions couldn't reproduce by design.