Customer research in the age of agent-led development

When I was at Bolt, we spent weeks and weeks optimizing our login screen. It was one of the most important screens for the business, because a higher login rate for identified users meant higher conversion for our customers and more revenue for Bolt. Because of that, it was also one of the more challenging places to make updates. There were a lot of downstream dependencies, and testing needed to be robust to make sure we didn't introduce any regressions. Despite all the effort we put in, we kept hitting a dead end. Nothing we did, big or small, seemed to significantly move the needle on getting more customers to log in when prompted.

Then our design team suggested doing a usability study. We had originally dismissed the idea, because it seemed obvious how to streamline a login flow, and because studies take time and a fair amount of effort. You have to design the study, make sure all the stakeholders agree on the goals, recruit and screen the candidates, schedule the interviews, hold the interviews, and then summarize the findings. Given how stuck we were, though, we pulled the trigger and went through the process.

It did end up taking a lot of time and energy, but we came out of it with a core insight. The language we used to tell customers we had recognized them as having an account was extremely off-putting to a segment of our audience. We made some changes to the copy and to how it was presented when prompting the user to log in, and pretty quickly saw a very significant lift in login rate across all of our customers, which translated directly into higher impact and more revenue.

What stuck with me afterward was how long we had been working on the wrong problem. We had the data to tell us that customers weren't logging in, and we had the testing infrastructure to tell us which version of the screen performed better. What we didn't have was the why. Every experiment we ran was a guess about what was holding people back, and every guess was built on our own assumptions about how customers felt when they saw that screen. A few conversations with real people answered a question that weeks of optimization couldn't. That's not unusual, either. When Ronny Kohavi looked at experiments at Bing, fewer than a third of the ideas that were tested actually moved the metrics they were designed to improve.¹ Most of what teams build on a hunch doesn't work, and the only way to find out why is to ask.

With agent-led software development, the speed at which teams, or even individuals, can ship code has exploded. Coding assistants and agents are now the most common way developers use AI, and most of the people who use them do so every day.² The head of product for Claude Code has described going from an idea to something demo-able in a couple of hours, where getting something tangible in front of customers used to take weeks.³ A single engineer with a few agents can now produce what a whole team used to.

At the same time, while some parts of customer research have improved, like analyzing interviews, the core process has stayed the same. You still design the study, align the stakeholders, recruit, schedule, interview, and summarize. That mismatch in speed has led more and more teams to skip research entirely. When it takes weeks to get signal back from customer research, even if the research brings a novel insight, there's a good chance that insight is about a version of the product that's no longer relevant. By the time the findings deck is ready, the team has already shipped two more iterations.

The skipping was a problem before agents, too. McKinsey found that more than 40% of companies weren't talking to their end users during development at all.⁴ Now research teams are being asked to do more with less. In one recent survey, two-thirds of research, design, and product professionals said demand for research had increased,⁵ while in another, about one in five researchers said their company had laid off UX researchers.⁶ The work is growing, the people doing it are shrinking, and the build cycle it's supposed to inform keeps getting faster.

However, the core problem with skipping customer research hasn't changed. It has only compounded. At the speed agent-first teams can develop, an early mistake based on a misunderstanding of the customer can cascade in a way that becomes harder and harder to unravel as more gets built on top of it. A simple misunderstanding can end up as a core tenet of how your agents shape the user experience, the information architecture, or the data model, and then get repeated over and over in rapid succession across individual developers and teams.

Think about how an agent actually works. It takes the context it's given and runs with it, quickly and confidently. If the brief says customers want more control over their settings, the agent will build more settings. It won't stop to ask whether that's true. Anthropic makes the point in its own guidance on building agents: they need to get "ground truth" from their environment at each step, and pause for human feedback at checkpoints.⁷ For code, ground truth is tests and errors. For a product, ground truth is the people who use it. Without that, agents just turn your assumptions into software faster than ever.

It's tempting to close that gap by having an AI play the customer, but that doesn't really solve it. When Nielsen Norman Group compared synthetic users to real ones, the synthetic users were overly optimistic and seemed to care about everything.⁸ They told researchers what sounded reasonable, not what was true. If the goal is to stop building on guesses, a more convincing guess isn't the answer.

At Woven, we believe customer research needs to go through the same transformation that software development has. With automated, agent-led customer research, we can go from a step backward, where fewer and fewer teams research the products they build, to a leap forward, where research is a core part of the "software factory" workflow. If research becomes fast and cheap enough, it will become a critical part of nearly every major feature release.

That matters not just for making sure you're developing the right products and user journeys, but also for how you spend your tokens. Agents make it cheap to explore, which also makes it easy to explore in the wrong direction. Without a clear understanding of the customer, teams end up with a "wandering in the wilderness" approach: generating version after version, reviewing each one, debating which is best, and throwing most of them away. Just like my login example, research leads to much more focused development. Instead of asking an agent to try everything, you can ask it to build the thing your customers actually told you they need.

We envision a world where teams can build fast but listen and learn at the same speed. Say a team has an idea for a major new product. As they start defining it, they spin up a generative, exploratory study to interview prospective customers about the problem, and get results back the same day to help refine their thinking. Sometimes those interviews confirm the idea. Sometimes they show the real pain is somewhere next to it, which is exactly what you want to learn before anything gets built.

Next, they use agents to build a prototype of the end-to-end experience and launch a second study focused on usability. Real people try the prototype in their own browser while an AI interviewer watches where they hesitate, get confused, or get excited, and asks about it in the moment. The results come back almost immediately and help the team refine the prototype before they build the first version and release it. After launch, they run a third study with customers who are actually using the product, to understand how it holds up in the real world, where the use cases are messier than anyone planned for.

None of those steps has to slow the team down. These "extra" steps can run in parallel with the rest of the development cycle. A study can be in the field while the agents are building, and the findings can arrive in time to shape the next iteration, rather than weeks after it's shipped. Even more valuable, they remove unnecessary loops and internal arguments. Every team has had the meeting where the designer, the PM, and the engineer each have a different theory about what customers want, and nobody can prove theirs. When everyone can work from the same base truth about customers, that meeting gets a lot shorter, and so does the list of things you build and throw away.

None of this replaces judgment. Research tells you what's true for your customers. Deciding what to build is still the team's job. It just gets to happen with much better inputs, and much more often.

As Steve Blank put it, "There are no facts inside the building so get the heck outside."⁹ For a long time, getting outside was slow and expensive, so most teams did it rarely, if at all. Now, more than ever, there's no excuse not to get in the weeds with real people every time you're developing a product.