Why A/B Testing Can't Replace Traditional Marketing Research (and What Skipping It Really Costs)

In marketing strategy conversations, there's a lot of buzz around "move fast and break things." Startups, including those in healthcare, often rush to market with rapid-fire digital tactics and endless A/B tests. The appeal is understandable: why spend time on surveys or focus groups when you can launch now and figure it out on the fly? But too often, this trial-and-error approach backfires. As a fractional CMO who's seen the aftermath, I can tell you that skipping foundational marketing research is usually far more costly than doing it right from the start.

Here is the short answer. A/B testing can only choose between the options you give it, so if your positioning or message is wrong, testing just finds the best version of the wrong idea. Traditional marketing research, meaning customer interviews, surveys, focus groups, and concept tests, tells you which idea to test in the first place. Do a small amount of research first, then use testing to refine what you learned.

1. Why do so many companies skip research and go straight to testing?

Digital platforms make it easy to run A/B tests on ads, landing pages, and emails overnight. Leaders under pressure to show results think, "Why spend weeks on interviews when we can get live data by launching?" Fast answers feel productive, and testing does have a place.

The problem starts when testing replaces research entirely. Without a foundation, teams end up in what I call "digital trial-and-error hell": constantly tweaking ads, rewriting website copy, chasing new segments at random, and burning through budget. It's like trying to crack a safe by spinning the dial at random. You might get lucky, but more likely you'll lose a lot of time, while your product sits in the market with the wrong message.

This post is for companies already in market and stuck in that loop. If you're still before launch, my guide to market research before you launch covers the basics.

2. What can A/B testing tell you, and what can't it?

A/B tests are excellent at answering narrow questions. Which headline gets more clicks? Which button color, offer, or subject line performs better?

What a test can't do is tell you why. It can't tell you that buyers misunderstand your category, that a missing feature is the real deal-breaker, or that the benefit you lead with isn't the one they care about. If the concept is off-base, you're testing variations of the wrong thing. You might see small lifts here and there, and still miss the bigger insight that the product or positioning needs to change.

3. What does skipping research look like in practice?

Two of my clients learned this the expensive way.

A digital health startup built a B2C wellness app to help chronic disease patients track their symptoms. The founders were passionate, assumed they knew what patients needed, skipped formal research, and launched with a flashy digital campaign. They poured over $100,000 into Facebook and Google Ads, trying different headlines and creatives in rapid sprints. Sign-ups trickled in, but engagement was abysmal: people downloaded the app and didn't stick with it. The team kept A/B testing the onboarding flow and tweaking the UX to improve retention. Nothing moved the needle meaningfully, and after months of struggling, they called us.

When I joined as their fractional CMO, I paused the expensive campaigns and went back to basics: talk to the users. We ran in-depth interviews with a dozen people from the target demographic and a few moderated focus groups. The app's core functionality was fine, but the positioning was way off. The marketing emphasized tracking symptoms, and users told us what they really needed was guidance on managing those symptoms day to day. The company was selling a diary to patients who wanted a coach. No amount of ad optimization would have revealed that nuance. We helped them shift the product messaging to personalized care tips and community support, and the research also pointed to new partnership opportunities, like integrating with popular patient support forums and healthcare influencers in that disease space. Within a few months of relaunching with research-driven positioning, user engagement and retention improved dramatically. But that came only after a hefty sum and precious time had been wasted. Most of the $100,000+ in ad spend could have been saved by investing maybe 10% of that upfront in traditional marketing research to get the concept and messaging right.

A healthcare SaaS company built a platform for medical practices to manage patient referrals. They assumed their selling point, an AI-driven referral matching system, was enough to win clinics over. Eager to scale, they skipped formal market validation and started selling aggressively with a small sales team and online ads, testing pitches on the fly. Clinics showed initial interest but rarely moved forward to purchase, and conversions were far below expectations. By the time I was called in, the company had cycled through dozens of pitch deck iterations and website copy A/B tests looking for a message that clicked with clinic administrators. Nothing worked consistently, and they were burning cash.

To find the problem, my team set up phone interviews with potential buyers, clinic managers and doctors, plus a couple of roundtable discussions with people in the industry. The real issue was a product feature gap that no one internally had fully appreciated. Clinics told us the AI matching was nice, but what they really needed was better integration with their existing Electronic Health Records (EHR) systems. That was the deal-breaker. The marketing and A/B tests kept highlighting speed and AI while buyers kept thinking, "We can't use this if it doesn't talk to our patient records system." We advised the company to temporarily halt the broad marketing push and work on an integration solution, or at least adjust the marketing to communicate a roadmap for integration. That insight came from plain old conversations, and no multivariate test would have surfaced it. Addressing the real need eventually revived the sales pipeline, and the company repositioned the product as a seamless add-on to EHR systems, a much stronger pitch. But consider the cost of learning this so late: nearly a year of underwhelming sales, thousands spent on ads and sales salaries, and growing skepticism among the sales team and investors. Some upfront interviews or concept testing with doctors could have caught this requirement early and saved a lot of headache.

4. What does skipping research really cost?

Here's a simple example to make the math concrete. These are illustrative numbers. Say a team spends $10,000 a month on ads and testing tools while it searches for a message that works, and it takes five months to get there. That's $10,000 times 5, or $50,000, before counting the team's time. If interviews and a survey could have pointed them to the right message before the spending started, most of that $50,000 would have gone toward a campaign that worked instead of the search for one.

The money is only part of it. Every month of experimenting is a month a better-informed competitor can use to win the market.

There's also the cost of misalignment. If you rush a product to market on gut feel and it doesn't meet a real need, you may have to rebuild features later, which costs far more than a round of concept testing would have. Messaging that hasn't been checked with real customers can come across as tone-deaf or confusing, and in healthcare it can use the wrong terms or miss the trust factors your audience cares about.

Finally, consider morale. Launching with fanfare and hearing crickets is rough on any team, and it often leads to panicked "random acts of marketing." Skipping research trades an upfront bill for a much larger one later.

5. What does traditional research give you that testing can't?

Clarity on the big decisions. Research informs four areas in particular.

Positioning and value proposition: you learn from customers what they actually care about, so you position on what the market values instead of the features you assume matter.

Messaging: research lets you use the words your audience uses. That matters most in fields like healthcare, where the wrong terminology or missing emotional context can push people away.

Budget allocation: when you know where your audience spends time and whom they trust, you can concentrate spend on the right channels instead of spreading it across every platform to see what sticks.

Brand and trust: research shows what makes your audience trust one brand over another, whether that's credentials, endorsements from respected clinicians, or tone, and what skepticism you need to overcome.

6. Can AI tools like ChatGPT replace customer research?

AI can help. It can summarize hundreds of open-ended survey responses, organize interview notes, and brainstorm questions. But it can't replace talking to real customers.

AI models predict likely answers from patterns in existing text. Ask one what patients want from a healthcare app and you'll get a sensible, generic list: privacy, ease of use, accuracy. It won't surface the unexpected insight that is specific to your product and audience. And it can't hear hesitation in a customer's voice and ask a follow-up question, which is often where the best insight comes from.

AI-generated personas and simulated customer feedback can look convincing, and that's the danger. They give you a false sense of security built on what's already widely known. Use AI to process research data, then have an experienced person interpret it and decide what it means for strategy.

7. How do you add research when you're already in market?

You don't need to stop everything. Build a short research loop into your plan.

Start with exploratory interviews. Talk to 5 to 10 people who look like your target customers, using a short discussion guide with open questions like "What's your biggest challenge with this today?" and "How do you solve it now?" In healthcare, talk to both end users and the other stakeholders, such as clinicians or IT leaders, who influence the decision.

Then survey to check how widespread those themes are. If interviews suggest data security is a major concern, a survey shows how many potential customers share it and how it ranks against cost or convenience.

Run concept tests or small focus groups with a mock-up or concept description. It's much better to discover in a one-hour session that people are confused by your pricing than after a major campaign has launched.

Pilot the new message with a small budget, such as one tightly targeted campaign or an email to part of your list, before you scale it. Measure clicks, sign-ups, and inquiries, and if it falls flat, go back to your notes.

Last, document and share what you learned. A short summary of pain points, winning language, and the real target keeps the team aligned and keeps the highest-paid person's opinion from overriding what customers told you. My market research services page explains how I run this kind of work for clients.

8. Who should own a research-driven strategy?

Many startups and growth-stage companies don't have senior marketing leadership to drive this work. An experienced marketing leader asks the uncomfortable question early, "What do we actually know, and what are we assuming?", and then designs research that answers it quickly.

A fractional CMO gives you that leadership without a full-time executive salary, often for a defined project or a few days a month. If you're weighing that option, my post on why your business needs a fractional CMO walks through when it makes sense.

9. How do you know it's time to stop testing and start listening?

Watch for these signs: you've cycled through several messages without a clear winner, conversion rates won't move no matter what you change, sales hears objections marketing never planned for, or nobody on the team can describe the best customer in that customer's own words. Any one of those is a sign to pause and talk to customers. If you aren't sure whether the problem is research, messaging, or something else in the system, a marketing audit will sort it out.

If you'd like help putting research at the center of your marketing before more budget goes into guesswork, you can book a free 30-minute introductory call here.

Dr. Linda Orr, PhD, is the founder of Orr Consulting, a fractional CMO and marketing analytics practice serving brands from $5M to $500M in revenue. She built Upwork's first enterprise marketing mix model, covering an $88M annual media portfolio, and spent about 20 years as a marketing professor before moving fully into consulting.