The Cost of a Failed Experiment

It seems that the “Fail Fast, Fail Often” idea escaped the field of psychology and gained a lot of traction in the business world. Even banks and telecoms with huge technical debt, utterly incapable of fast development are trying to move that way.

“Forbes” is, of course, completely convinced:

Iterating fast failures achieve a desired result faster than perfecting the solution.¹

I am not against rapid development and “testing in the wild” but consider this question: Would you like the manufacturer of your car to develop a new breaking technology this way? I guess not. So the applicability of the “fail fast, fail often” strategy clearly depends on the context. I have been thinking and discussing it with people smarter than me and I think the single most important variable that influences the choice of this approach is the cost of a failed experiment.

Let’s look at the extremes to validate this assertion. If your business is online marketing and your weapon of choice is a landing page, it doesn’t make sense to try to design a perfect one. It is more economical to generate and deploy a large number of them, distribute your internet traffic among them and simply see which one(s) perform the best. The cost of a failed experiment—a random user hitting a bad page—is essentialy zero. You will never see them again, anyway. There is nothing wrong with “spray and pray” approach in this case.

On the other extreme we have Elon Musk who wants to send people to Mars. It must work the first time they try it. SpaceX must do abolutely everything in their power to make sure that the first trip with people on board is a success. All of it: the launch, the trip and the landing. In order to achieve it, there will be an enormous amount of up-front design and prototyping but there will be no testing of unfinished or half-designed product with end users in real circumstances. They will not send two ships full of people with two different landing procedures just to see which one provides better survivability.

I hope you agree that there are situations where the “fail fast, fail often” is not an option or where failure is not an option at all. I bet you now think that they are rare and definitely do not concern your line of business.

Let me introduce some research.

John Gottman has been studying marriages and tracking reasons for divorces for over forty years. In his book “What Predicts Divorce?" he states that “the ratio of positive to negative [interactions] is about 5 for couples whose mariages are stable”. He found out that the absolute numbers of positive and negative interactions between spouses doesn’t matter as long as there are five positive ones for each negative one.

What does it have to do with business? Since your cost of acquiring a new customer is probably between $100 and $300 you care about a long-term relationships with your customers. Just like you care about your marriage or partnership in the long term.

So, suppose you have signed up for the “fail fast, fail often” approach, maybe following in the footsteps of Spotify or Google, and you are constantly tweaking little aspects of your product, carefully studying their impact on your users, constantly taking little steps towards the ever-changing perfection. Let’s say that your team is exceptionally good and only one in three of these small experiments fails and you annoy only a tiny percentage of your users with it. Let’s assume it is 0.5% of your daily unique users.

Here’s the trap you may be falling into.

After a few hundreds of these little experiments (you have been running many of them in parallel, weren’t you?) you managed to annoy the vast majority of your users at some point in time. Not much, since the experiments were small but you left them with a negative emotion. You reduced your “brand-love level” not only in their minds but in also anybody’s who cared to listen to their stories.² How are you going to make-up for it? You probably think that the other two experiments, those that did not fail, should compensate for this minor setback. Unfortunately this is not the case. Due to positive adaptation these little improvements will not be counted as positive counterbalance to your failure. First of all, they were tiny—because by betting on “fail fast, fail often” you hedged against big changes and big failures—so they failed to induce dopamine in your customers’ brains. Secondly, they are something “normal”. Something your customers expect from you since you have been doing it for a while. And you don’t get any “brand-love credit” for staying within customers’ expectations.

The only way to improve the “brand-love level” after any failure is to deliver something beyond your customers’ expecations. And you will need more than one of these. Do you have it?

P.S. Here’s an interesting homework.
Think of a product or a service you have a long relationship with. Ask yourself if you would quit them if you had a cheaper alternative and the switch was easy. If your answer was “no” think of a different brand. Once you found one you would leave without much doubt, ask yourself how did you feel when you started your relationship with it. It was probably a positive feeling, maybe you were even excited. I was when I subscribed for Spotify or when I signed up with my current accountants. Both promised to make my life better and for a while this feeling persisted.
Next question: what brought this initial positive feeling to level so low that you would gladly go? I bet it was something more like “death by a thousand cuts” rather than a spectacular failure. All these little disappointments, these little slip-ups when something didn’t meet your expectations and nothing to compensate for it.

I have been doing such self-observations for a while. As I write it, my accountants are making-up. They made it really easy for me to apply for the COVID-19 relief plan from my government. I was expecting their help but the way they did it exceeded my expectations. At the same time, in line with Gottman’s research, this one big effort improved my judgement of them but not to the level where I wouldn’t switch if an opportunity presented itself. Maybe the “magic ratio” in business is lower than 5-to-1 needed for a stable marriage but it doesn’t seem to be 1-to-1.