Jason Cohen – How data, statistics and numbers will make you do the wrong thing | Business of Software

Checking form...

[x]

Get alerts for new BoS talks


2014 Reg Open

2014 tickets on sale

0 0 0 Days Hrs Mins
2014 Reg Open

Registration open


Jason Cohen – How data, statistics and numbers will make you do the wrong thing

This is a summary of Jason Cohen‘s Business of Software 2012 presentation.

Metrics – You’re Doing it Wrong

The way you’re approaching metrics in your business is wrong. How you’re using data is wrong, because the tools you’re using are wrong.

When asked to guess how many jelly beans are in a jar, individual answers are off by 67%. However, if you average all the answers together, the average is only off by 3%. People suck at guessing, but averaging guesses across people is a nearly perfect predictor of truth…sometimes.

The wisdom of the crowd only works for certain things. When asked to vote for the funniest joke, the wisdom of the crowd didn’t produce it. Crowds are wise when there’s a correct answer. Crowds are useful in objectivity, and destructive in creativity. Yes, crowds are not just neutral in creativity, in fact they are actually worse (i.e. destructive).

Take A/B tests as an example. A/B tests are usually not done right. Picking B over A because it is beating A by a little bit can be very destructive. Sparefoot, an Austin startup, ran an A/A test using Google Website Optimizer, and found that Google Website Optimizer had one A beating itself!

When what you are testing for is rare, the results are overwhelmingly wrong.

Jason has a great (and super cute!) article on his blog on easy significance testing (and hamsters).

The way you determine whether an A/B test shows a statistically significant difference is:

  1. Define N as “the number of trials.” N = A + B = total number of conversions
  2. Define D as “half the difference between the ‘winner’ and the ‘loser’.” D = (A – B) / 2
  3. The test result is statistically significant if D^2 is bigger than N.

Data Points

Seek large outcomes from more traffic (as in 100,000′s of data points, not 10,000′s), especially if you are a small company. Go for the big stuff and shy away from the small stuff. A great example of this is Google’s famous 41 shades of blue test. Google ran this experiment to determine which shade of blue received the most clicks. Running this test with 2 shades of blue, and picking one winner with a 95% confidence level, leads to a 5% chance of a false-positive. Running this test with 41 shades of blue, leads to an 88% chance of a false-positive.

Test theories, not headlines. Don’t spitball headlines. First form a theory about why a change would be better, then test it. If the theory turns out to be invalid, think about what other assumptions could be wrong. Invalidating a theory gives you an opportunity to think deeper, and come up with another theory. Examples of theories:

  • People from the UK like to see a lot of vowels in the word color.
  • On this page people are ready to buy.
  • At this point, the user is ready to talk to customer service.
  • People that land on this page by searching Google with the keyword “security” would like to see a testimonial about security.

Which metrics matter?

Which metrics actually matter?

  • Growth rate?
  • Cancellation rate?
  • Conversion rate?
  • Total revenue?
  • Support costs?

Which variables should I care about? The ones that have the biggest impact on growth, revenue and cash.

Let’s take a hypothetical affiliate program for a SaaS product as an example, and figure out what’s important. Affiliate program parameters:

  • 20 sales/month
  • $100 average affiliate on time payout
  • $20/month average new monthly recurring revenue (MRR)

Using a simple model in a spreadsheet, it looks like we will break even in about four to five months. Now let’s add a 15%/month growth. That 15% growth causes 50% more costs. But that doesn’t count cancellation rates, or affiliate customers being lower quality (i.e. cancel more). The end result: your dead (with a 10%/month cancellation rate). If we then increase the price by $10/month (50% more MRR), we’re back to breaking even at 4 – 5 months.

Affiliate program optimization priorities:

  1. Increase MRR.
  2. Start with small payouts (threshold). Don’t try to optimize the hell out of it, just make sure to stay within the box.
  3. Prevent cancellations by affiliates.
  4. Growth is bad until 1 – 3 are solid.

Conclusion

Strategy:

  • Pick 1 – 2 key metrics to optimize.
  • Pick a few thresholds to watch via simple sensitivity analysis.
  • Fold real data back into the model.

Tactics:

  • The wrong process is worse than nothing.
  • Test theories, not “see what sticks.”
  • Use real significance tests (hamster test), and seek large effects.

[I'd like to thank Bill Horvath, founder of DoX Systems, for sharing his notes with me.]

BUSINESS OF SOFTWARE – FOR PEOPLE BUILDING GREAT SOFTWARE BUSINESSES.

This year will be the 7th Business of Software Conference – 15-17th September 2014, Boston.

A three day conference for founders who want to build sustainable, profitable software businesses. BoS has always been a special conference for our delegates and we want to keep it special. Attendance is restricted to just 400 attendees in 2014. Registration open now.

This year we are also running Business of Software Conference Europe, 25-26th June. For more details, visit the dedicated BoS Europe site. Two days in the birthplace of computers – Cambridge, England.

Join our mailing list to be the first to know when this year’s talks go live and to hear the latest news from the conference.

Eventbrite - Business of Software 2014

Comments are closed



Loading