Investing in AI For AI’s Sake
My Conversation With LaunchDarkly CPO Claire Vo
It’s been a while since my last post! For the past several months, I’ve been busy in my new job heading up product and technology at Glean, a role that’s proving to be both fast-paced and fun. As a part of my job, I get to connect with enterprise leaders about how they’re using AI and learn both what’s working and…what isn’t. I also have the opportunity to meet lots of interesting people working on cool AI products.
One of these interesting people is Clarie Vo, the Chief Product Officer at LaunchDarkly, who I recently met at an AI meetup in San Francisco. (What everyone says about the AI scene in San Francisco is true, by the way: the city is AI’s epicenter. Just about every night there’s some sort of AI event to attend.)
What stands out to me about Claire is how she’s both spearheading new AI tools at LaunchDarkly and spinning out her own AI product ChatPRD, an AI-backed tool that helps product managers write more effective product requirements documents. I love her gung ho embrace of AI. She’s a big proponent of the idea that companies should invest in AI for AI’s sake — an idea that’s slightly controversial, at least among CFOs.
Claire also believes that in order to make GenAI models work well, you need to “pair builders with non-builders to get the wheels turning.” Her words inspired me to incorporate this concept into Glean’s upcoming Prompt-A-Thon, where we’ll be pairing engineers and product managers with people working in non-technical roles. I’m excited to try out this strategy; like Claire, I see how developers might help the entire company — from sales to marketing to support — come up with prompts that will make their roles more effective.
Here’s my conversation with Claire, edited and condensed for clarity:
Tamar: How did you get into working in product?
Claire: I’ve had a pretty nontraditional journey. To start off, I have a liberal arts degree and my first job was working at a startup as a copywriter. I soon realized that I’d be a good fit for a product role, because product is all about using words to get somebody from step A to step B. Ultimately, I got my first product job because I could use words to get people to where they needed to go, which made it easy to optimize conversion.
Tamar: It's so interesting that words were the foundation that helped you become a product manager because LLMs are obviously all about words and natural language.
Claire: It’s really interesting that the programming language of the future may just become English. The fact that I spent my academic and professional career working on my ability to communicate clearly via language has been a tangible benefit to me as we've seen this AI shift happen. It’s a skill that will differentiate product managers and engineers in the future.
Tamar: You are leading all of product development at LaunchDarkly. What exactly does the product do and how do you describe feature management?
Claire: Feature management is an engineering practice that decouples the release of features from the deployment of code. It’s a way to release code for a new feature in incremental, progressive, or targeted ways. This is a really powerful tool for engineering teams to roll out access gradually or by region, and LaunchDarkly provides the platforming capability to do this.
Tamar: While LaunchDarkly wasn’t initially considered an AI company, the team has embraced AI tools and built in pipelines for developing AI. How did LaunchDarkly handle this pivot to AI?
Claire: One of the reasons I joined LaunchDarkly six months ago was because I saw companies investing in AI initiatives and AI features and products in their portfolio, and I knew this was a tailwind for LaunchDarkly. Behind those AI products are either commercial, open source, or internally built models that are highly configurable and have a lot of parameters you can tune and prompts to customize. Whether it's the model itself or the parameters of the model, there's lots of ways to get this non-deterministic algorithm tuned to output something for your business.
I had a hypothesis that LaunchDarkly had a really big role to play as they already had the best in market capabilities to configure features.
When I got access to the internal data, I saw that our customers were doing all of this anyway and with hockey stick growth. We looked at all the ways our customers were using feature flags that had a model name in it, had a prompt in it, had a temperature in it, and we saw a steep increase in application of feature management against the AI use case. That is when we decided to build these use cases into the product.
Tamar: It’s great when you realize, “Wow, I didn't know you could do that with our product. We should build this!”
Claire: One of my favorite ways to build product is to find funny and interesting ways that customers use our product to tease out real value. And then I just follow them down that path.
This is something I’ve been seeing in prompt management. LaunchDarkly still doesn’t have the best developer experience for this yet, but customers will do it anyway. And that’s often a sign of product market fit, when your customers will do something that’s sort of painful because it delivers core value. But we plan to make it better for them — it’s an easy user experience and technical problem to solve on our end.
Tamar: What is the biggest challenge for your customers when building AI products?
Claire: Quality is one of them. As a smaller company, you can handle more variability. Early buyers tend to be more forgiving when it comes to using startups’ products, so long as they deliver on their core innovation. But in the enterprise, the bar is higher: not only do customers expect consistency but they want accuracy and performance.
The fundamental challenge is that buyers want the benefits of these AI-backed products with the consistency and predictability of more deterministic systems. To add another wrinkle, the fundamental technologies are changing so fast that you have to continually adapt — oftentimes, every few days — to keep up with the market. So the pursuit of quality and consistency is incredibly challenging.
Tamar: How does this risk aversion come into play with AI-backed products?
Claire: Just about everybody's risk averse when it comes to AI. We have some AI-backed products in beta at the moment and scaling up from a beta product with early-access to a generally available product has been really challenging. For every customer that asks about our AI strategy we get another customer who says, “Never, ever, ever, ever turn on AI-backed features in my account. Never.”
Tamar: So it’s not that they don’t want to pay for AI-backed features. These are customers that are telling you, “Even if these AI-backed features are free, I don’t want them.”
Claire: Right. They’re saying, “I don't want my team to have it. I don't want them to see it. I don't trust them using it. And I don't want my data in it.”
Tamar: Do you think they’re afraid of information leaking? Is that what it is?
Claire: Probably. In reality, LaunchDarkly is not vacuuming up tons of customer data into any of these products. But because these large companies don’t yet have scalable policies for their vendors, the simplest policy is banning all AI-backed features.
Tamar: Some CISOs say they will never, ever use AI, and maybe they won’t. But they’ll likely be in the minority as these tools continue to see increased adoption. Which brings me to my next question: What AI products and tools is LaunchDarkly finding most effective?
Claire: We’ve actually found Glean to be extremely helpful in terms of solving this ongoing debate about where information goes internally. We’ve also been using a GitHub Copilot implementation which we love.
Copilot has come up a lot lately in recruiting as we’re assessing AI-assisted engineering skills in interviews. Recently, we had a spirited debate about whether we should allow Copilot in code interviews. I was very far on the side of yes, absolutely, we should allow them to use Copilot because we all use it and we want to assess if candidates can develop code in the same way that we ourselves develop code. But there were some very interesting arguments against this, especially about what role Copilot plays in live code.
Tamar: I’d love to talk about your personal experiences building in AI. You recently built an AI tool for product managers that’s had a lot of success. Can you tell me more about that?
Claire: It’s called ChatPRD and it’s an AI co-pilot that helps product managers work more efficiently and at higher quality. It’s a creation tool as well as a coaching tool and it’s mostly used by individual product managers or teams without product managers.
I started this nine months ago as something I built for my own private use that I eventually launched publicly. Today, multi-billion dollar publicly traded companies are using it.
One reason for its success is that it writes well. People find its voice to be shockingly human. Usually, you can spot a paragraph written by AI from 100 yards away, which is why I put a lot of work into the tone, voice, and structure of the language outputs. I wanted ChatPRD to deliver clear and believable language, which is exactly how a product manager would want to communicate.
Tamar: How do you encourage engineers and product managers to use AI tools like Copilot for ChatPRD?
Claire: Some of the practical things I’ve done is to assign a budget for it. I see things differently than the majority of CFOs who often take the perspective that they don’t want to invest in AI for AI's sake. I do want to invest in AI for AI's sake because I see that it’s kind of inevitable. The only way we can learn to use these tools is to get them into the hands of our team. So carving out a budget can be high impact and pretty obvious from a business value perspective especially when it comes to Copilot or a general OpenAI license or something like Glean.
The second thing is loudly normalizing the use of these tools and ensuring that everyone knows the policies when it comes to using them. That means creating transparency. At my previous company I documented any AI assistance I used. I’d link every AI tool I used in a reference guide that showed how I generated that document. Even today, so many leaders hide the fact that they're using these tools because there's still a stigma around relying on AI. To offset this, I ripped off the bandaid and made it as clear as possible about what tools I was using to make things.
Tamar: CIOs often tell me that behavior change is really hard. If I was a CIO and I came to you and I said, “Claire, I'm at a big 50,000 person company and I've just bought all these AI products but no one’s using them.” What would you tell me to do to usher in a behavior change?
Claire: You have to have change agents inside the company that can make the application tangible to an individual. This is the best accelerant for adoption.
People say GenAI is going to make everybody a software engineer and maybe that’s true. But at this point, you still have to think like a software engineer to get software-like outputs from these tools. I would not expect someone without a software engineering background to have the mental models necessary to get the most out of these tools. You have to make it practical. And I think that means you have to pair builders with non builders to get the wheels turning. I haven't figured out any other way.