How to Design Smart Business Experiments
Managers now have the tools to conduct small-scale tests and gain real insight. But too many “experiments” don’t prove much of anything.
Smart Business Experiments
by Thomas H. Davenport
EVERY DAY, managers in your organization take steps to implement new ideas without having any real evidence to back them up. They ﬁddle with offerings, try out distribution approaches, and alter how work gets done, usually acting on little more than gut feel or seeming common sense – “I’ll bet this” or “I think that.” Even more disturbing, some wrap their decisions in the language of science, creating ...view middle of the document...
ing it difficult if not impossible to detraining, readily available software, Consumer-facing companies rich in termine what was really driving any and the right encouragement, an transaction data are already routinely improved results. Branches undergoing organization can build a “test and testing innovations well outside the interventions weren’t matched to conlearn” capability. realm of product R&D. They include trol sites for the most part, so no one » Companies that equip managbanks such as PNC, Toronto-Dominion, could say for sure that the outcomes ers to perform small-scale yet and Wells Fargo; retailers such as CKE noted wouldn’t have happened anyway. rigorous experiments don’t only Restaurants, Famous Footwear, Food Anxious to head off criticism, managers save themselves from expensive Lion, Sears, and Subway; and online did provide a control in one test, which mistakes – they also make it more ﬁrms such as Amazon, eBay, and Google. was designed to see if placing video likely that great ideas will see the As randomized testing becomes standard screens showing television news over light of day. procedure in certain settings – website waiting lines would shorten customers’ analysis, for instance – ﬁrms build the perceived waiting time. But rather than capabilities to apply it in other circumlooking at control and test groups, they stances as well. (See the sidebar “Stop compared just one control site with one Wondering” for a sampling of tests conducted recently.) To test site. That wasn’t enough to ensure statistically valid results. be sure, there remain many business situations where it is not Perceived waiting time did drop in the test branch, but it went easy or practical to structure a scientiﬁcally valid experiment. up substantially in the control branch, despite no changes But while the “test and learn” approach might not always be there. Those confounding data kept the test from being at all appropriate (no management method is), it will doubtless conclusive – but that’s not how the ﬁndings were presented to gain ground over time. Will it do so in your organization? If top management. it’s like many companies I have studied, an investment in softIt doesn’t have to be this way. Thanks to new, broadly availware and training will yield quick returns of the low-hangingable software and given some straightforward investments fruit variety. The real payoff, however, will happen when the to build capabilities, managers can now base consequential organization as a whole shifts to a test-and-learn mind-set. decisions on scientiﬁcally valid experiments. Of course, the scientiﬁc method is not new, nor is its application in business. The R&D centers of ﬁrms ranging from biscuit bakers to drug When Testing Makes Sense makers have always relied on it, as have direct-mail marketFormalized testing can provide a level of understanding about ers tracking response rates to different permutations of their what really works that puts more intuitive...