{"id":267,"date":"2025-03-28T08:00:00","date_gmt":"2025-03-28T09:00:00","guid":{"rendered":"https:\/\/jibblemuffzoodle.com\/?p=267"},"modified":"2025-05-20T14:51:39","modified_gmt":"2025-05-20T14:51:39","slug":"how-to-argue-against-ai-first-research","status":"publish","type":"post","link":"https:\/\/jibblemuffzoodle.com\/index.php\/2025\/03\/28\/how-to-argue-against-ai-first-research\/","title":{"rendered":"How To Argue Against AI-First Research"},"content":{"rendered":"<p>              <title>How To Argue Against AI-First Research<\/title><\/p>\n<article>\n<header>\n<h1>How To Argue Against AI-First Research<\/h1>\n<address>Vitaly Friedman<\/address>\n<p>                  2025-03-28T09:00:00+00:00<br \/>\n                  2025-05-20T14:32:37+00:00<br \/>\n                <\/header>\n<p>With AI upon us, companies have recently been turning their attention to <strong>\u201csynthetic\u201d user testing<\/strong> \u2014 AI-driven research that replaces UX research. There, questions are answered by AI-generated \u201ccustomers,\u201d human tasks \u201cperformed\u201d by AI agents.<\/p>\n<p>However, it\u2019s not just for desk research or discovery that AI is used for; it\u2019s an <em>actual<\/em> usability testing with \u201cAI personas\u201d that <strong>mimic human behavior<\/strong> of actual customers within the actual product. It\u2019s like UX research, just&hellip; well, without the users.<\/p>\n<p>.course-intro{&#8211;shadow-color:206deg 31% 60%;background-color:#eaf6ff;border:1px solid #ecf4ff;box-shadow:0 .5px .6px hsl(var(&#8211;shadow-color) \/ .36),0 1.7px 1.9px -.8px hsl(var(&#8211;shadow-color) \/ .36),0 4.2px 4.7px -1.7px hsl(var(&#8211;shadow-color) \/ .36),.1px 10.3px 11.6px -2.5px hsl(var(&#8211;shadow-color) \/ .36);border-radius:11px;padding:1.35rem 1.65rem}@media (prefers-color-scheme:dark){.course-intro{&#8211;shadow-color:199deg 63% 6%;border-color:var(&#8211;block-separator-color,#244654);background-color:var(&#8211;accent-box-color,#19313c)}}<\/p>\n<figure class=\"\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/www.syntheticusers.com\/\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"499\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/user-research-without-users-opt.png\" alt=\"One of the tools to conduct synthetic testing, or AI-generated UX research, without users.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      One of the tools to conduct \u201csynthetic testing,\u201d or AI-generated UX research, without users. (Source: <a href=\"https:\/\/www.syntheticusers.com\/\">Synthetic Users<\/a>) (<a href=\"https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/user-research-without-users-opt.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>If this sounds worrying, confusing, and outlandish, it is \u2014 but this doesn\u2019t stop companies from adopting AI \u201cresearch\u201d to drive business decisions. Although, unsurprisingly, the undertaking can be <strong>dangerous, risky<\/strong>, and expensive and usually diminishes user value.<\/p>\n<p class=\"course-intro\">This article is <strong>part of our ongoing series<\/strong> on <a href=\"\/category\/ux\">UX<\/a>. You can find more details on <strong>design patterns and UX strategy<\/strong> in <a style=\"font-weight:700\" href=\"https:\/\/smart-interface-design-patterns.com\/\">Smart Interface Design Patterns<\/a>&nbsp;\ud83c\udf63 &mdash; with live UX training coming up soon. <a href=\"https:\/\/www.youtube.com\/watch?v=jhZ3el3n-u0\">Free preview<\/a>.<\/p>\n<h2>Fast, Cheap, Easy&hellip; And Imaginary<\/h2>\n<p>Erika Hall <a href=\"https:\/\/medium.com\/mule-design\/a-three-part-plan-to-save-the-world-98653a20a12f\">famously noted<\/a> that \u201cdesign is only as \u2018human-centered\u2019 as the business model allows.\u201d If a company is heavily driven by <strong>hunches, assumptions, and strong opinions<\/strong>, there will be little to no interest in properly-done UX research in the first place.<\/p>\n<figure class=\"\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/medium.com\/mule-design\/a-three-part-plan-to-save-the-world-98653a20a12f\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"450\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/value-exchange-opt.png\" alt=\"The opportunity for business value is in delivering user value when users struggle.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      The opportunity for business value is in delivering user value when users struggle. By <a href=\"https:\/\/medium.com\/mule-design\/a-three-part-plan-to-save-the-world-98653a20a12f\">Erika Hall<\/a>. (<a href=\"https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/value-exchange-opt.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>But unlike UX research, AI research (conveniently called <em>synthetic testing<\/em>) is <strong>fast, cheap, and easy<\/strong> to re-run. It doesn\u2019t raise uncomfortable questions, and it doesn\u2019t flag wrong assumptions. It doesn\u2019t require user recruitment, much time, or long-winded debates.<\/p>\n<p>And: it can manage <strong>thousands of AI personas<\/strong> at once. By studying AI-generated output, we can discover common journeys, navigation patterns, and common expectations. We can anticipate how people behave and what they would do.<\/p>\n<p>Well, that\u2019s the <strong>big promise<\/strong>. And that\u2019s where we start running into big problems.<\/p>\n<h2>LLMs Are People Pleasers<\/h2>\n<p>Good UX research has roots in <strong>what actually happened<\/strong>, not what <em>might<\/em> have happened or what <em>might<\/em> happen in the future.<\/p>\n<p>By nature, LLMs are trained to provide the most \u201c<strong>plausible<\/strong>\u201d or most likely output based on patterns captured in its training data. These patterns, however, emerge from expected behaviors by statistically \u201caverage\u201d profiles extracted from content on the web. But these people don\u2019t exist, they never have.<\/p>\n<p>By default, user segments are <strong>not scoped and not curated<\/strong>. They don\u2019t represent the customer base of any product. So to be useful, we must eloquently prompt AI by explaining who users are, what they do, and how they behave. Otherwise, the output won\u2019t match user needs and won\u2019t apply to our users.<\/p>\n<figure class=\"\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/www.nature.com\/articles\/d41586-025-00068-5\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"597\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/hallucination-opt.png\" alt=\"Every LLM hallucinates, but newer models perform better at some tasks, such as summarizing.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      Every LLM hallucinates, but newer models perform better at some tasks, such as summarizing. By <a href=\"https:\/\/www.nature.com\/articles\/d41586-025-00068-5\">Nature.com<\/a>. (<a href=\"https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/hallucination-opt.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>When \u201cproducing\u201d user insights, LLMs <a href=\"https:\/\/www.ideo.com\/journal\/the-case-against-ai-generated-users\">can\u2019t generate unexpected things<\/a> beyond what we\u2019re already asking about.<\/p>\n<p>In comparison, researchers are only able to define what\u2019s relevant as the process unfolds. In actual user testing, insights can help <strong>shift priorities<\/strong> or radically reimagine the problem we\u2019re trying to solve, as well as potential business outcomes.<\/p>\n<p>Real insights come from <strong>unexpected behavior<\/strong>, from reading behavioral clues and emotions, from observing a person doing the opposite of what they said. We can\u2019t replicate it with LLMs.<\/p>\n<h2>AI User Research Isn\u2019t \u201cBetter Than Nothing\u201d<\/h2>\n<p>Pavel Samsonov <a href=\"https:\/\/uxdesign.cc\/no-ai-user-research-is-not-better-than-nothing-its-much-worse-5add678ab9e7\">articulates<\/a> that things that sound like customers <em>might<\/em> say them are <strong>worthless<\/strong>. But things that customers <em>actually<\/em> have said, done, or experienced carry inherent value (although they could be exaggerated). We just need to interpret them correctly.<\/p>\n<p>AI user research isn\u2019t \u201cbetter than nothing\u201d or \u201cmore effective.\u201d It creates an <strong>illusion of customer experiences<\/strong> that never happened and are at best good guesses but at worst misleading and non-applicable. Relying on AI-generated \u201cinsights\u201d alone isn\u2019t much different than reading tea leaves.<\/p>\n<h2>The Cost Of Mechanical Decisions<\/h2>\n<p>We often hear about the breakthrough of automation and knowledge generation with AI. Yet we often forget that automation often comes at a cost: the cost of mechanical decisions that are typically <strong>indiscriminate<\/strong>, favor uniformity, and erode quality.<\/p>\n<figure class=\"\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/www.nngroup.com\/articles\/plan-research-ai\/\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"874\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/ai-prompt-research-questions-opt.png\" alt=\"Some research questions generated by AI could be useful, others useless.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      Some research questions generated by AI could be useful, others useless. By <a href=\"https:\/\/www.nngroup.com\/articles\/plan-research-ai\/\">Maria Rosala<\/a>. (<a href=\"https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/ai-prompt-research-questions-opt.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>As Maria Rosala and Kate Moran <a href=\"https:\/\/www.nngroup.com\/articles\/synthetic-users\/\">write<\/a>, the problem with AI research is that it most certainly will be <strong>misrepresentative<\/strong>, and without real research, you won&#8217;t catch and correct those inaccuracies. Making decisions without talking to real customers is dangerous, harmful, and expensive.<\/p>\n<p>Beyond that, synthetic testing assumes that people fit in well-defined boxes, which is rarely true. Human behavior is shaped by our experiences, situations, habits that can\u2019t be replicated by text generation alone. AI <strong>strengthens biases, supports hunches<\/strong>, and amplifies stereotypes.<\/p>\n<h2>Triangulate Insights Instead Of Verifying Them<\/h2>\n<p>Of course AI can provide <strong>useful starting points<\/strong> to explore early in the process. But inherently it also invites false impressions and unverified conclusions \u2014 presented with an incredible level of confidence and certainty.<\/p>\n<p><strong>Starting with human research<\/strong> conducted with real customers using a real product is just much more reliable. After doing so, we can still apply AI to see if we perhaps missed something critical in user interviews. AI can enhance but not replace UX research.<\/p>\n<figure class=\"\n  \n  \n  \"><\/p>\n<p>    <a href=\"https:\/\/www.linkedin.com\/posts\/johnpcutler_whenever-i-see-a-linear-customer-journey-activity-7288764384380624896-P_j6\/\"><\/p>\n<p>    <img loading=\"lazy\" width=\"800\" height=\"451\" src=\"https:\/\/res.cloudinary.com\/indysigner\/image\/fetch\/f_auto,q_80\/w_400\/https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/layering-customer-journeys-opt.png\" alt=\"Triangluate linear customer journeys by layering them on top of each other to identify the most frequent areas of use.\" \/><\/p>\n<p>    <\/a><figcaption class=\"op-vertical-bottom\">\n      Triangluate linear customer journeys by layering them on top of each other to identify the most frequent areas of use. By <a href=\"https:\/\/www.linkedin.com\/posts\/johnpcutler_whenever-i-see-a-linear-customer-journey-activity-7288764384380624896-P_j6\/\">John Cutler<\/a>. (<a href=\"https:\/\/files.smashing.media\/articles\/how-to-argue-against-ai-first-research\/layering-customer-journeys-opt.png\">Large preview<\/a>)<br \/>\n    <\/figcaption><\/figure>\n<p>Also, when we do use AI for desk research, it can be tempting to try to \u201c<strong>validate<\/strong>\u201d AI \u201cinsights\u201d with actual user testing. However, once we plant a seed of insight in our head, it\u2019s easy to recognize its signs everywhere \u2014 even if it really isn\u2019t there.<\/p>\n<p>Instead, we study actual customers, then <strong>triangulate data<\/strong>: track clusters or most heavily trafficked parts of the product. It might be that analytics and AI desk research confirm your hypothesis. That would give you a much stronger standing to move forward in the process.<\/p>\n<h2>Wrapping Up<\/h2>\n<p>I might sound like a broken record, but I keep wondering why we feel the urgency to replace UX work with automated AI tools. Good design requires a good amount of <strong>critical thinking<\/strong>, observation, and planning.<\/p>\n<p>To me personally, cleaning up after AI-generated output takes way more time than doing the actual work. There is an <strong>incredible value<\/strong> in talking to people who actually use your product.<\/p>\n<p>I would always choose one day with a real customer instead of one hour with 1,000 synthetic users pretending to be humans.<\/p>\n<h2>Useful Resources<\/h2>\n<ul>\n<li><a href=\"https:\/\/www.nngroup.com\/articles\/synthetic-users\/\">Synthetic Users<\/a>, by Maria Rosala, Kate Moran<\/li>\n<li><a href=\"https:\/\/uxdesign.cc\/synthetic-users-the-next-revolution-in-ux-research-0d43f7111e7f?sk=v2%2F05a1bb3c-4312-4665-bc5f-06339bbb609e\">Synthetic Users: The Next Revolution in UX Research?<\/a>, by Carolina Guimar\u00e3es<\/li>\n<li><a href=\"https:\/\/rbefored.com\/ai-users-are-neither-ai-nor-users-c701f32fbbc4\">AI Users Are Neither AI Nor Users<\/a>, by Debbie Levitt<\/li>\n<li><a href=\"https:\/\/www.nngroup.com\/articles\/plan-research-ai\/\">Planning Research with Generative AI<\/a>, by Maria Rosala<\/li>\n<li><a href=\"https:\/\/blog.uxtweak.com\/artificial-participants-and-ai-generated-responses\/\">Synthetic Testing<\/a>, by St\u00e9phanie Walter, Nikki Anderson, MA<\/li>\n<li><a href=\"https:\/\/www.uxforai.com\/p\/navigating-the-abyss-the-dark-side-of-synthetic-ai-user-research-tools\">The Dark Side of Synthetic AI Research<\/a>, by Greg Nudelman<\/li>\n<\/ul>\n<h2>New: How To Measure UX And Design Impact<\/h2>\n<p>Meet Measure UX &amp; Design Impact (8h), a new practical guide for designers and UX leads to measure and show your UX impact on business. Use the code \ud83c\udf9f <code>IMPACT<\/code> to save 20% off today. <a href=\"https:\/\/measure-ux.com\/\">Jump to the details<\/a>.<\/p>\n<figure style=\"margin-bottom:0;padding-bottom:0\" class=\"article__image\">\n    <a href=\"https:\/\/measure-ux.com\/\" title=\"How To Measure UX and Design Impact, with Vitaly Friedman\"><br \/>\n    <img loading=\"lazy\" width=\"900\" height=\"466\" style=\"border-radius: 11px\" src=\"https:\/\/jibblemuffzoodle.com\/wp-content\/uploads\/2025\/05\/measure-ux-and-design-impact-course.png\" alt=\"How to Measure UX and Design Impact, with Vitaly Friedman.\" \/><br \/>\n    <\/a><br \/>\n<\/figure>\n<div class=\"book-cta__inverted\">\n<div class=\"book-cta\" data-handler=\"ContentTabs\" data-mq=\"(max-width: 480px)\">\n<nav class=\"content-tabs content-tabs--books\">\n<ul>\n<li class=\"content-tab\"><a href=\"#\"><button class=\"btn btn--small btn--white btn--white--bordered\"><br \/>\nVideo + UX Training<\/button><\/a><\/li>\n<li class=\"content-tab\"><a href=\"#\"><button class=\"btn btn--small btn--white btn--white--bordered\">Video only<\/button><\/a><\/li>\n<\/ul>\n<\/nav>\n<div class=\"book-cta__col book-cta__hardcover content-tab--content\">\n<h3 class=\"book-cta__title\"><span>Video + UX Training<\/span><\/h3>\n<p><span class=\"book-cta__price\"><span><span class=\"\"><span class=\"currency-sign\">$<\/span>&nbsp;<span>495<span class=\"sup\">.00<\/span><\/span><\/span> <span class=\"book-cta__price--old\"><span class=\"currency-sign\">$<\/span>&nbsp;<span>799<span class=\"sup\">.00<\/span><\/span><\/span><\/span><\/span><br \/>\n<a href=\"https:\/\/smart-interface-design-patterns.thinkific.com\/enroll\/3081832?price_id=3951439\" class=\"btn btn--full btn--medium btn--text-shadow\"><br \/>\nGet Video + UX Training<\/p>\n<div><\/div>\n<p><\/a><\/p>\n<p class=\"book-cta__desc\">25 video lessons (8h) + <a href=\"https:\/\/smashingconf.com\/online-workshops\/workshops\/vitaly-friedman-impact-design\/\">Live UX Training<\/a>.<br \/>100 days money-back-guarantee.<\/p>\n<\/div>\n<div class=\"book-cta__col book-cta__ebook content-tab--content\">\n<h3 class=\"book-cta__title\"><span>Video only<\/span><\/h3>\n<div data-audience=\"anonymous free supporter\" data-remove=\"true\"><span class=\"book-cta__price\" data-handler=\"PriceTag\"><span><span class=\"\"><span class=\"currency-sign\">$<\/span>&nbsp;<span>250<span class=\"sup\">.00<\/span><\/span><\/span><span class=\"book-cta__price--old\"><span class=\"currency-sign\">$<\/span>&nbsp;<span>395<span class=\"sup\">.00<\/span><\/span><\/span><\/span><\/span><\/div>\n<p><a href=\"https:\/\/smart-interface-design-patterns.thinkific.com\/enroll\/3081832?price_id=3950630\" class=\"btn btn--full btn--medium btn--text-shadow\"><br \/>\nGet the video course<\/p>\n<div><\/div>\n<p><\/a><\/p>\n<p class=\"book-cta__desc\" data-audience=\"anonymous free supporter\" data-remove=\"true\">25 video lessons (8h). Updated yearly.<br \/>Also available as a <a href=\"https:\/\/smart-interface-design-patterns.thinkific.com\/enroll\/3082557?price_id=3951421\">UX Bundle with 2 video courses.<\/a><\/p>\n<\/div>\n<p><span><\/span><\/div>\n<\/div>\n<div class=\"signature\">\n  <img src=\"https:\/\/www.smashingmagazine.com\/images\/logo\/logo--red.png\" alt=\"Smashing Editorial\" width=\"35\" height=\"46\" loading=\"lazy\" \/><br \/>\n  <span>(cm)<\/span>\n<\/div>\n<\/article>\n","protected":false},"excerpt":{"rendered":"<p>How To Argue Against AI-First Research How To Argue Against AI-First Research Vitaly Friedman 2025-03-28T09:00:00+00:00 2025-05-20T14:32:37+00:00 With AI upon us, companies have recently been turning their attention to \u201csynthetic\u201d user testing \u2014 AI-driven research that replaces UX research. There, questions are answered by AI-generated \u201ccustomers,\u201d human tasks \u201cperformed\u201d by AI agents. However, it\u2019s not just&hellip;<\/p>\n","protected":false},"author":1,"featured_media":247,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[9],"tags":[],"_links":{"self":[{"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/posts\/267"}],"collection":[{"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/comments?post=267"}],"version-history":[{"count":2,"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/posts\/267\/revisions"}],"predecessor-version":[{"id":269,"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/posts\/267\/revisions\/269"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/media\/247"}],"wp:attachment":[{"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/media?parent=267"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/categories?post=267"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jibblemuffzoodle.com\/index.php\/wp-json\/wp\/v2\/tags?post=267"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}