TEI 281: The right way to use experiments to create better products more quickly – with Stefan Thomke

How to make experimentation work for product managers If you work in a medium to large company, your CEO wants the organization to be more innovative. All the surveys about such things tell us this is the case. If you are in a smaller organization or on your own, you still care about innovation. But, how can you bring about more innovation, or let’s face it, in some cases, anything that looks like innovation? In some form, this is what company leaders have talked with me about the most in the last year — incorporating more innovation into their work. It is something we especially need now. I believe a key paving stone on the path to better innovation – aka, more products customers love — is adopting an experimentation mindset and conducting fast low-cost experiments. And that is what our guest, Dr. Stefan Thomke, Harvard Business School professor and researcher, is discussing with us. He is an authority on the management of innovation and the use of experimentation. Summary of some concepts discussed for product managers [2:29] Why is it important for innovative organizations to run experiments? One of the big challenges in innovation is uncertainty. When we rely on our experience, we’re wrong 9 out of 10 times. Big data only shows correlations and not causations. The answer to uncertainty is experiments. Experimentation is the engine of innovation. [6:53] How should experimentation be used in innovation? It’s not experimentation versus data or experimentation versus experience. It’s experience plus data plus experiments. Companies that experiment still do normal research activities, but they don’t run to market with the research before it’s tested. Instead, they use research to generate hypotheses which they then test with experiments. [10:53] What is the structure of a good experiment? In an experiment, the tester wants to separate an independent variable from a dependent variable while holding everything else constant. The independent variable is the presumed costs, such as a bonus for the sales team, and the dependent variable is the observed effect, such as the revenue of the sales team. You change the independent variable, observe changes in the dependent variable, and make a conclusion about cause and effect. To minimize bias, ideally an experiment should be blind and randomized. A blind experiment in which the person doesn’t know they’re being experimented on ensures that their knowledge of the experiment doesn’t change their behavior. Randomization distributes random variables among all the people so that bias is evened out. [14:21] What’s an example of using experimentation? Imagine you have a restaurant and want to find out if a new menu will increase your revenue. The wrong way to set up an experiment is to run the normal menu for a month and then the new menu for a month and compare your revenue. There are too many variables that could have changed from one month to the other. The right way is to create a randomized experiment by running both menus at the same time. As customers come into your restaurant, they’re randomly assigned to one menu or the other. Other variables, such as the chefs in the kitchen, are also randomized. [18:55] How do we identify the hypotheses that need to be tested? Hypotheses can come from many places, such as qualitative research, customer insights, or data mining, or even intuition. Good hypotheses should be grounded in facts, identify possible causes and effects through clearly measurable variables, and be able to be falsified. [23:04] How does experimentation give us confidence? Experimentation gives us confidence to be bold. Without experiment, you’re just gambling. Many times your intuition can get in the way. For example, Ron Johnson, who created the Apple store,