Lessons in Experimentation: Building a Culture of Testing at Scale
When I joined Hearst Magazines, I thought I had a decent handle on A/B testing—after all, I’d tinkered with experiments at two startups. Spoiler alert: I didn’t. The scale, complexity, and sheer rigor of running experiments at a large organization hit me hard. It’s not just about tweaking variables and waiting for stat sig results. It’s about navigating bureaucracy, aligning stakeholders, and, ultimately, moving the needle for a business that thrives on scale.
Here’s the paradox: small startups, fueled by the adrenaline of finding product-market fit, thrive on risk and iteration. Big corporations? They’re risk-averse juggernauts where a 0.1% improvement in the user journey can mean millions in revenue. Running experiments here isn’t just a task; it’s an art. Over the past few years, I’ve helped foster an experimentation culture across Hearst’s 20+ digital magazines, working with a team of more than 100 talented peers with strong opinions. Along the way, I’ve learned some lessons that might be worth sharing.
Lesson 1: Define the Words Before the Work
Every organization needs a shared vocabulary. Without it, you’re playing a game of telephone, and the message gets garbled. Here is my dearest one: “engagement.” Ask three product managers what it means, and you’ll get five answers. For the video team, it’s video viewability. For subscriptions, it’s intent to subscribe. For ad tech, it’s impressions. The result? Engineers scratching their heads, writing code that measures something nobody asked for.
The solution? Be a language cop. Define your terms, and enforce those definitions everywhere: experiment briefs, engineering tickets, dashboards. It’s not the most desirable part of my work, but clarity is contagious and critical.
Lesson 2: Stakeholders Are Your First Customers
Experimentation isn’t just a technical exercise, it’s a cultural shift. People fear what they don’t understand, and experimentation can feel like a threat. Editors worry their carefully crafted pages will look unrecognizable. Designers get defensive about their layouts. Stakeholders wonder if their OKRs will implode.
Communication is your superpower. Frame experiments as temporary, not permanent. Show how they align with business goals. Speak to the fears before they speak to you. When you win over the skeptics, you’re halfway to success.
Lesson 3: Love the Sequel
The first version of any experiment is rarely the best. When the results come in, the insights pile up. Stakeholders chime in with ideas you hadn’t considered. That’s your cue: iterate.
Launching a follow-up test is easier because the groundwork is already done. You don’t need to resell the idea or reeducate your team. Pro tip: include a “follow-up test” section in your analysis reports. It sets the stage for iteration and helps people see the experiment as a journey, not a one-off.
Lesson 4: Templates Are the Swiss Army Knife of Product Management
Product managers make mistakes—it’s part of the job. But when I found myself asking the same questions over and over in experiment briefs, I knew it was time for a system. I came up with the experiment brief template.
The goal was balance—thorough enough to avoid back-and-forths, but not so detailed that nobody uses it. The result? A template that covers 80% of use cases (thanks, Pareto principle) and drastically reduces errors. Post-template, QA issues were less about logic and more about platform quirks. That’s I can call a progress.
Lesson 5: Be the Gatekeeper
In the early days of experimentation, enthusiasm often outruns expertise. People launch tests with weak hypotheses, the wrong metrics, or no real purpose. It’s like trying to fly a plane looking only at the altitude meter.
Here’s where you come in: QA isn’t just for engineers. Product managers need to play gatekeeper, ensuring experiments are worth the time and resources. Ordinary QA specialists are amazing at catching bugs, but they might miss the bigger picture. Your job is to keep the forest in sight, even when others are lost in the trees.
Final Thoughts
Experimentation is more than just a tool for optimization—it’s a way of thinking. It challenges us to embrace curiosity, question assumptions, and keep learning. The best (worst) part? It’s never finished. There’s always one more variable to test, one more insight to discover, and one more way to improve. So, how do you foster experimentation in your team or organization? Drop your thoughts below—I’d love to hear your take.