I observed my first lab-based user testing session 18 years ago. The facility had observation rooms hidden behind one-way mirrors, multiple cameras, microphones, eye-tracking kit and two co-facilitators. Think mini cinemas: tiered seating, clear sightlines into the test room, monitors showing exactly what was happening on screen.
We were testing a website concept for a well-known car brand. Was it intuitive? Could users find key information? Could they complete core journeys without issue? We wanted to validate our solutions, and this lab felt like the perfect, professional setting in which to do it.
The exercise cost a hefty five-figure sum. We’d spent several weeks planning the tests, building the prototype and recruiting the right candidates.
I remember our first user arriving. He was greeted in reception, handed a form and taken into the test room. The facilitators did their best to put him at ease. Even so, there was a distinct air of nervousness about him, and understandably so. He was in an unfamiliar place, surrounded by unfamiliar people and equipment. He was in the 50+ demographic and didn’t appear comfortable with the computer setup in front of him.
The test began. We all leaned forward.
He struggled with the most basic tasks. A similar pattern played out throughout the day. Candidates sought reassurance after almost every step. “Do you want me to click this?” “Did I do that right?” “I’ve done it wrong again, haven’t I?” The facilitator tried hard not to lead them, but caved on several occasions. It felt wrong.
That first user, the one who seemed uncomfortable from the start, moved the mouse to the left side of the keyboard towards the end of the session. He’d been using it the wrong way round for most of the test.
By lunchtime, I was ready to call the concept a failure.
A week or so later, a glossy formal report arrived, complete with heat maps and recommendations. To my surprise, the agency had deemed the test a success. Our solution had been validated. The client was delighted. There was backslapping all round.
It didn’t sit right with me.
Why lab testing can mislead
Several things troubled me about that session, and they’ve troubled me in varying degrees ever since.
The test wasn’t truly impartial. The agency I was working for was the actual client of the test agency, not the brand whose site we were evaluating. The test agency had pitched for this work and knew how much we wanted a positive outcome. That’s not a neutral starting point.
The candidates had been paid to participate. Payment changes the dynamic. People who feel compensated may feel pressure to perform well or tell you what they think you want to hear.
The environment was unnatural. A test lab is unfamiliar territory: new people, new equipment, a new social setting. That alone is enough to make people anxious and eager to please.
And the candidates knew they were being observed and recorded. Self-consciousness skews behaviour. What you see in the lab isn’t always what you’d see at home.
The project pushed ahead with the validated solution. The site evolved over time, bugs were fixed and features improved, but I was never fully convinced we’d launched with the best version. A lot of money could have been saved by addressing the real issues at prototype stage.
What 500 sessions have taught me
In the years since, I’ve planned or conducted close to 500 user testing sessions. After each one, I reflect on what I could have done better. These aren’t hard rules; every project is different and requires flexibility. But they represent my most consistent learnings.
Go to the user. Always observe contextually. Travel to their home, their workplace, somewhere they feel comfortable and can speak their mind. The lab doesn’t show you that someone checks the fridge three times during an online grocery order, or calls their partner to confirm plans before completing a booking. That context matters.
Keep it one-to-one. Arriving mob-handed doesn’t put anyone at ease. A relaxed, one-to-one session consistently yields better insights.
Don’t pay upfront. Where possible, find users in your target demographic who are happy to give their time if you make the effort to come to them. A small gesture at the end of the session is fine; a payment upfront changes the dynamic.
Make recording discreet. I’ve learned to take notes with pen and paper. Where recording is necessary, make it easy to forget. An iPad left on the side is far less intrusive than a microphone pointed at someone’s face.
Test earlier and more often. Rather than spending weeks refining what you hope is the perfect solution, sketch, discuss and get rough ideas in front of people quickly. Corridor testing early in the process is faster and cheaper than validating late.
Separate what people say from what they do. I’ve watched candidates fail to complete a task and then describe the experience as easy in the post-test interview. What matters is what you observe, not what people report.
Use their own device. A user knows their own computer or phone. Hand them an unfamiliar device and you introduce distractions: shortcuts that don’t work, settings that differ, an interface they haven’t personalised. Remove those variables where you can.
Informal, contextual sessions are quicker to plan, cheaper to run and faster to yield results than anything I’ve seen come out of a formal lab.
The honest trade-offs
That said, contextual testing isn’t without its challenges.
Some clients feel that a lab setting sounds more scientific and therefore more credible. The informal alternative, watching someone attempt an online order from their sofa, can feel less rigorous on paper. But it’s precisely that sofa, with the fridge visits and the phone calls and the dog wandering in, that gives you the real picture.
There’s also the question of trust. Many clients want to observe users themselves and worry that findings might be filtered or that a facilitator might inadvertently lead the session. That’s a legitimate concern. What’s worth knowing is that good researchers actively want to find the problems. The pain points, the confusion, the moments of abandonment: those are the insights that drive better design. Finding them is the whole point.
