TEI 296: Better product testing – with Luke Freiler

What product managers need to know about customer validation and alpha, beta, and delta testing Product testing is about more than determining if a product functions properly or not. A larger perspective, and one that our guests shares is Customer Validation. We discuss how to use the various types of product tests, including alpha, beta, and delta tests, to judge product performance, customer satisfaction, and areas for improvement. Our guest is Luke Freiler, CEO and co-founder of Centercode. Luke has spent most of his career improving product testing. Centercode is a Customer Validation solutions provider that helps tech companies bring products to market. Summary of some concepts discussed for product managers [1:46] How did you become a testing expert? Early in my career in software, I became very passionate about the new field of usability—making technology easy to use. I was asked to run a beta test for one of my company’s products, and although we were a large, established company, we had no process for running testing. As I looked for solutions, I realized I’d found a hole—everyone had the problem of testing but nobody had solved it. I realized that testing with customers aligned with my passion for usability; customers can help you make technology more accessible. At age 21, I started a company to do tests, and I’ve learned a lot and have been doing it ever since. [6:55] How do you relate customer validation to testing? We realized that there is no single standard term that people use to refer to testing. We wanted to establish a better vocabulary and methodology that could scale and be adaptable to any company. We chose customer validation as an umbrella term for the various ways we engage with customers to develop a product. Customer validation includes three forms of testing: Alpha testing looks for quality. Beta testing looks for satisfaction. Delta testing, where we’re seeing a lot of innovation, is a continuous test throughout the life of the product to gather feedback about specific details. [15:49] Tell us more about alpha testing. The goal of alpha testing is to make sure the product works. We focus on technographics—the technology that surrounds people and products. Alpha testing is about targeting diverse ecosystems rather than your target market. Alpha testers can be internal employees or strangers. [23:50] Tell us more about beta testing. A beta tester should be someone who matches the target market, is enthusiastic enough to provide feedback, and is a stranger rather than an employee. The goal is measuring satisfaction. We start a beta test with a test plan, which is a list of features we want tested. Each feature has a basic description. Using a 1-5 scale, we rate the effort or time we want to put in and the value of the feedback to us. Then we design activities that tell the tester where the features are but are not overly directive. We use these activities to take testers on a tour of the product and engage them over a period of time. We want the beta testers to collaborate and communicate with each other about the product as they complete activities to explore features. We look for actionable, prioritized feedback. Out of each test, you want to discover: * issues—what needs to be fixed * ideas—what needs to be improved * praise—what needs to be promoted We ask testers to rate their satisfaction with each feature on a 1-5 scale. Then we ask why they gave that rating. We prioritize and act on the results. [31:27] What is the timeline of alpha and beta tests? Our average alpha test takes two weeks, and our average beta test takes three weeks. This is not very time-consuming. [33:57] Tell us more about delta testing. Delta testing is concerned with the next version of the product. We want to maximize small data to find quality issues.