361: Experimenting for product success – with Alex Mitchell

How small, focused tests can help product managers create better products Today we are talking about using experimenting and testing to create products customers love. To help us with this topic, Alex Mitchell is our guest. He is Director of Product at Kin Insurance and founder of The Modern Product Manager, which provides resources and courses for aspiring product managers. He also is involved in other organizations, lending his product expertise. Summary of some concepts discussed for product managers [3:47] Tell us about experimentation and testing. We need to have a broad definition of experimentation; it’s more than controlled experiments like A/B testing. Experimentation is a foundational and continuous element of product management. Don’t wait until you’ve built a feature to experiment. Test continuously throughout your product development lifecycle. Do small experiments to learn more and continually develop hypotheses you can test to help you make better decisions. [6:19] Product managers often become experts in their industry, and that can lead to assumptions. How do you test assumptions? Assumptions slow down innovation. One way to counter that is to involve more people in experimentation. Including your team and people outside your team will help keep innovation going and prevent you from getting complacent. Second, move around to different domains in your company if you can. This will help keep your ideas fresh and keep you coming up with testing and experimentation ideas. [8:25] What’s an example of using experimentation at Kin Insurance? At Kin Insurance, we sell home insurance to catastrophe regions like Florida and Louisiana. Through research, we observed that many customers were not making changes to their insurance after their initial purchase, like adding additional coverage, even if changes were in their best interest. We hypothesized they didn’t know they could make changes and if we let them know about additional products soon after their initial purchase, a meaningful percentage of people would at least be interested in learning more. If we were not taking an experimental approach to this, we might design a full-blown, intricate add-on system. We wanted the opposite since we were trying to validate whether our customers were interested. We took a simple approach and found a subset of customers eligible for flood insurance. A week after their initial purchase, we sent them an email conveying the benefits of flood insurance and asking if they were interested, and we found that many people were. We tested our hypothesis in a very short period, with a very small amount of effort. With testing like this, you can validate or invalidate your hypothesis, and you’ll learn other things along the way too. We did all this with no development effort and didn’t invest any of our most precious resources until we had validated our hypothesis and understood the customer experience well. You can use lightweight, scrappy tests to get information quickly and inexpensively. However, this may be different from how some of your colleagues are used to working. They might want a fully-developed plan, while you’re trying to do something you’ve never done before. Be prepared for skepticism, questions, and discomfort—that may be an indication you’re on the right track. [17:46] Have you seen an experiment produce unhelpful data? Why was that? One reason an experiment can produce unhelpful data is if it doesn’t line up with the hypothesis you’re trying to test or if you don’t have a way to make changes to prove or disprove your hypothesis. Another reason is if you’re looking at data that’s irrelevant to your experiment or attributing changes to your experiment that were actually caused by something else. Make sure you have someone on your team who understands statis...