I was pressure-testing a filled-in Design Criteria Canvas last week; you know, the one I sent out in the last issue. I tagged a line in the Must band with a "U," for user-truth, and added a real-ish quote to it. One that I figured was pretty clean, specific, a little blunt, and the kind of thing that sounds like a real person said it because they were annoyed enough to say it exactly that way.
And then…after continuing to build out a lesson covering this, it dawned on me that this super-relatable, super-believable quote could just as easily come from a synthetic persona or even a synthetic panel, rather than a real human. Ummm…yikes (again)!
As AI becomes the norm for quickly prototyping and validating ideas with synthetic personas, we MUST find ways to treat them differently than the quotes we gather from real people.
— Justin Lokitz
Design Deep Dive
Two issues ago the argument I made (hopefully clearly) was that generating options got cheap, so judgment became the scarce skill. Last issue was that judgment needs a mechanism, not a vibe, so I created a new set of criteria: falsifiable, written before the options exist, able to kill your favorite idea (one of my favorite things to do, BTW). Both of those still hold. But…in doing that I built an entire tool around a "U" line for user-truth and never asked the obvious next question: user-truth sourced from where?
A criterion is only as honest as what you feed it. This issue is about exactly that: the feed.
The Mom Test, a book that I have recommended to designers and innovators for decades, got one thing right that still holds completely (at least in most cultures, but not all). The author, Rob Fitzpatrick's move was simple and it was correct: people are nice (in most cultures), and nice people will tell you your idea is great to your face because they don't want to watch you deflate in front of them (again, in most cultures). So don't ask about the idea. Ask about their life, their last purchase, the thing they actually did last Tuesday. Behavior doesn't flatter you. Opinions do.
That advice assumed something specific about the liar, though. It assumed the liar was a person, being kind, at some social cost to themselves. Maybe just a little awkward, a little tired, and even eventually out of patience, out of things to say, out of polite ways to avoid the truth.
What’s changed is NOT a small update. The liar in your research process now might not be a person at all (holy sh-t!). If you open your favorite AI tool and run a synthetic panel, you’ll get respondents with infinite patience, zero social discomfort, and a bottomless supply of plausible-sounding reasons. As you probably already know, they're not being nice to spare your feelings. Nope! They're being helpful, which is a different failure mode entirely, because helpful is the actual objective they were built and tuned to hit. A model that's good at its job will hand you a coherent answer that fits your framing, because coherent-and-on-topic is what good means to it. That's the whole feature working exactly as intended, aimed at the wrong target. And…it's not a bug you can patch.
Image courtesy of Lakmoos
The Mom Test never had to plan for a liar that doesn't get tired and doesn't need your validation. Understandably, it planned for a person (and mostly someone living in the US). This (AI that is) is a different animal, and pretending it's the same problem with a faster interface is how a fabricated quote ends up stapled to your Must band with nobody in the room feeling like they did anything wrong.
A few honest signs the research has gone circular, if you want to check your own:
Every respondent has a tidy reason ready, with no hedging and no "I'd have to think about it."
Nobody contradicts anybody else. Real humans, put in a room together, disagree constantly. Silence on that front is a tell.
Nobody says "I don't know" or "I don't care." Real research is full of both.
The quotes never surprise the person who ran the session.
The finding lines up with the hypothesis almost exactly as it was originally written.
Any one of those, fine, could be a coincidence (hahah). Two or three together and you're not doing research anymore. You're doing an elaborate improv exercise where the model plays back your own belief wearing a trench coat.
There's a second problem sitting right next to this one that just got sharper. In Design a Better Business we told readers to act like a fly on the wall during research and not tell subjects what you were actually trying to learn. That was fine advice for a 2016 reader. It runs straight into where consent norms actually sit today, after a decade of GDPR, data scandals, and a public that's learned to ask what's being done with what they say. I'm not just walking that back because the world changed around it. Rather, I'm updating it because it has to survive contact with 2026, the same as everything else in this series.
Put those two next to each other and you get the actual shape of the integrity problem now. On one end: deceiving the people you're supposed to be learning from. On the other: being deceived by something standing in for them, dressed up as evidence. These are different mechanisms with the same failure. Both leave you with a canvas full of confident answers to a question nobody real actually weighed in on.
So here's the practical version that you can hopefully use. First and foremost, synthetic panels aren't the enemy. They're genuinely useful for exploration, for stress-testing a framing before you spend a real human's time on it, for widening a Could band you haven't earned the right to narrow yet. I use them in my own work and teach students how to use them. What they absolutely, unequivocally CANNOT do is defend a Must. If a line in your Must band is tagged "U," it needs a transcript from an actual person attached to it, full stop, no exceptions for deadline pressure. That's 100% not me being precious about methodology. That's the whole canvas from two issues back only working if the inputs feeding it are real, because a falsifiable test run against a fake fact just returns a confident, false result.
One more check: ask what a synthetic finding would have to say to change your mind. If the honest answer is nothing, because it's just going to keep agreeing with whatever you ask it, then it was never evidence. It was a mirror with better manners.
I'll admit I've been tempted by the shortcut myself. A synthetic panel whilst sipping coffee feels exactly like the five-hundred-note wall from the first issue in this series: fast, confident, and hollow the second you press on it. Cheap confidence is still cheap. It just wears a nicer outfit now.
Send this to colleagues and friends who are actively looking at or using a synthetic panel as part of a stakeholder review, or to the researcher on your team quietly drowning in AI note-taking tools trying to figure out where the line actually is. My hope is that they'll recognize the moment on the second paragraph. And…as I teach my students: hope is not a strategy.
That closes out this run for now, but it leaves one thread hanging that I'm not done with. Options are cheap, criteria are the mechanism, and the inputs feeding those criteria have to be real. None of that answers what happens once an entire org is moving at this speed at once, with dozens of teams doing all three of these things simultaneously and nobody watching whether judgment survives the pace. That's a different problem that I'm not quite ready to write about yet, but I'm getting there.
Subscribe to Design Shift for more conversations that help creative professionals grow into strategic leaders.
What did you think of this week's issue?
