So here is Patrick McKenzie’s brilliant talk at Business of Software 2011 on engineering your marketing outcomes. Marketing comes naturally to some people. Unfortunately, few of them go on to found software companies. Patrick talks about types of marketing that use engineering skills and won’t be rabidly opposed to your developers. Instead of spending a few weeks cranking out one more feature that won’t be seen by 1% of your users and increase the value of your software by <1%, learn how you can use the same few weeks to create scalable systems that take much of the guesswork out of marketing.
BUSINESS OF SOFTWARE – FOR PEOPLE BUILDING GREAT SOFTWARE BUSINESSES.
This year will be the 7th Business of Software, a three day conference for founders who want to build sustainable, profitable software businesses. BoS has always been a special conference for our delegates and we want to keep it special.
Attendance is restricted to just 400 attendees in 2013 and we have 200 places taken and the next 100 tickets (as of April 20th) will be sold at the second Early Bird Rate.
If you want to see all of the action from Business of Software 2012, the videos of the talks are available in one place now:
Transcript of Patrick McKenzie at Business of Software 2011:
Patrick McKenzie: [Deep voice] Hello Ladies.
[Normal voice] And Gentlemen. [Laughter]
So, if you weren’t here to the talk last year I basically did about an hour-long takeoff on the Old Spice guy I was telling you about. Does everyone …OK …Raise your hands if you were here at the talk last year… [Everybody raising hands]
OK. Put your hands down if you have forgot… No, keep them up! [Laughter]
Alright. Now put your hands down if you have forgotten the topic of that talk. OK, Good. So some people remember even a year later.
So that’s the reason why I like humor in presentations — because it helps people retain what you’re speaking about. Now, right after Business of Software last year I flew back to Japan where I live, and I got off the plane and I gave a phone call to a lady who I met right before coming to Business of Software and she said, “Hey, what did you do on your business trip?” So I gave her the Old Spice speech. I said — and I got to get a new speech this year, and I probably can’t sustain the Old Spice guy for sixty minutes and the joke will be old.
And she says: “That’s OK; I think you’re cooler than the Old Spice guy anyhow.” So, that was a sign to me and we’re now dating. [Laughter and applause]
So, the most common question I get asked is “Why are you in Japan?” And because it’s kind of relevant to the topic of the talk, I’ll tell you. People always ask me “Is it because of Japanese girls?” And that did happen but it was, like, eight years later… The actual reason I’m in Japan is not because of Japanese girls, it’s because of Indian men. Specifically… [Laughter]
Ten years ago I was an engineer with very little self-confidence skills — and now I’m an engineer with very little self-confidence ten years older.
But I was in college and reading the Wall Street Journal. And the Wall Street Journal was saying “The Indians, they’re going to steal all our programming jobs and we’re going outsource everything to India for one quarter of the cost. And if you are getting a programmer right now it’s a mug’s game; you will never be able to compete with all of those folks.” And I’m, like, “Oh God, they’re right!”
So, I thought: “How I am going to get a nice, safe job at a big mega corp like Microsoft like my parents have always wanted for me?” I thought: “I’ll play the Van diagram game: you take one thing that’s pretty hard like programming, take another thing that’s pretty hard like, say, speaking Japanese and you intersect the two.”
And the people who can do both — there must be, like, four of them, right? So, if I go to Japan and become bilingual then I can come back and get a job at Microsoft and no one will ever be able to take that thing here away from me until the day I die.
And that didn’t really work out. But…
So, I’m in Japan and… We have lots of young engineers in the audience? Let me… Perhaps help put your mind at ease regarding the outsourcing issue.
I actually managed a Japanese company’s outsourcing operations to our Indian subsidiary for three years.
And I will distill that into one anecdote which… This is absolutely true.
We made administration systems for universities and at one point we had… we sent the system over to the Indian subsidiary to get manually Q&A’d, you know, push all the buttons, make sure everything works. So, keep in mind in this conversation everything is happening via email with, like, a four-hour delay between each email.
Three days after we sent it over and expected to be, you know, mostly started and we asked: “What’s the status?”
“We haven’t started yet, the system isn’t working.”
“Oh, really? Why isn’t the system working?”
“We can’t log in.”
“Oh, really? Did you follow the page of instructions that would get you from the subversion and the database dump into a working system and then be able to log in?”
“Yeah, we did that but the system isn’t working.”
“Did you get an error message?”
“Yes, we got an error message.”
“What did the error message say?”
“I don’t know, I can’t read Japanese.” [Laughter]
“OK… Can you take a screenshot of the error message with the printscreen and paste it into, you know, MS Paint and send that over?”
And they do. And they show me a picture of the login screen and I write back: “That is not an error message, that is just a weather report, please continue with the instructions and push the buttons that we have marked for you and see if anything comes up.” And they say, “We don’t agree that that is a weather report. [Laughter]
I’m like, “That is interesting because I can read the text and, according to our previous conversation, you can’t.” [Laughter]
“Why don’t you think it’s a weather report?” And they said…they had two compelling pieces of argument on why it couldn’t be a weather report: it was an on the login screen and written in flaming red text.
Why would you ever do that to a weather report? And I said, “Oh, it’s on the login screen and written in flaming red text because, that weather report, if you ignore it, you’ll die!” [Laughter]
And I told this story to Americans all the time, and one of them asked me: “What kind of system are you writing such that a) Weather is routinely lethal and b) It’s so routinely lethal that that’s already special-cased?”
And all I can say is, “Welcome to Japan!”
There are 30 words of Japanese that have made it into the English language and 15 of them are for meteorological phenomena that can kill you. [Laughter]
So, what am I doing these days when I am thankfully quit off that day job and, will never do outsourcing management ever again? But these days my kind of claim to Internet fame, aside from being on “Hacker News” because it’s my job — because I’m gainfully unemployed — is I’m kind of like the Internet’s Bingo guy.
I made Bingo Card Creator, it’s a really simple software that makes bingo cards for elementary school teachers. I won’t tell you too much about that specifically, but we are going to talk about some of the things that I have learned from it. Because I kind of get to use this as my laboratory where I can screw it up as much as I want, and aside from the 200K users, no one, like a boss, will yell at me. And, just so you get some perspective of where I am, Dharmesh fleshed his customer graph the other day so I thought: “Wow, he stole mine!” Like we started the same point we end near the same number, about 5000 customers and the curve looks almost exactly the same! But a key thing to keep in mind: my graph was cumulative and Dharmesh charges between like $1000 and $10,000 a month and I charged 30$ once. [Laughter]
So, key takeaway: if you want to make a billion-dollar business selling software as a service at the 10.000 a month price point is a great idea — and selling 30$ a month once, you have to do a little more.
So, if you remember Jason Cohen’s talk from last year, he said that you get many pieces of competing advice at events like this, and if you are reading the entrepreneurial blogosphere; and it’s important for you to keep in mind where someone is coming from.
And I think that’s important when you’re listening to my presentation. So, I’ve prepared this lovely little four-quadrant graph on the axis of what market you’re addressing and on the spectrum of funded to bootstrapped like me.
And I put up some companies that I know and respect and appear at other places on this graph and, of course, because the only purpose of having a four-quadrant graph is to put yourself up into the right. You can see that I’m more evolved by them. [Laughter]
Well, honestly, what you were you expecting from a guy named McKenzie, right? [Laughter]
But one of the reasons I got into engineering is this really simple and powerful idea that math and science always work.
And I think many of the engineers here really have an emotional attachment to this one. And we don’t really have an emotional attachment to marketing, I mean…
Can I… just have a quick show of hands of who here is mostly an engineer? OK… hands down. Who here is mostly a marketer? OK, yeah. That’s what I thought.
And who here is Paul Kenny? [Laughter]
OK. So for you marketers who haven’t caught on to it yet, the stereotype of marketing in the engineering field is that, marketing is basically witchcraft. There’s some, like, eye of newt and toe of dog, or whatever that goes into the pot — we have no clue how anything works and it’s, like, vaguely evil. I mean, you’re selling stuff to people…
And it’s OK when people buy our stuff, but when people buy other stuff that’s kind of unnatural so we’re just creeped out by it.
But something that I’ve come to appreciate over the years of running my own business is that marketing is not really witchcraft, it’s just about changing people’s behavior. And this is not something that is totally unnoble. We know we can change people’s behavior, we can do it all the time by stuff as simple as changing our tone of voice.
As any lumberjack will tell you, “All good things come from timber.” [Laughter]
OK, Tim told you about me.
And this is, you know… We change people’s behavior in engineering too. That’s why we build products, we want you to change from a manual process to this better process that we have; that’s why Steve Jobs and an excellent product design, and can turn around how an entire world uses their cell phone.
And if we get into this we think that: “Wow, marketing is really sort of an engineering discipline.”
And so I want to talk about one particular type of way of changing people’s behavior. If you think of it, your business is set up to cause a series of fortunate events.
There’s billions and billions of people in the world and you really are trying to convince some small fraction of them to first come to your website. And of that small fraction an even smaller fraction are going to start your free trial, — if that’s your business model. And of that small fraction an even smaller fraction are going to actually succeed in paying you money.
And this graph is called a funnel in traditional marketing and traditionally this is the point where I have like a clip art slide of a, you know, cooking funnel.
But I thought I would use something a little more appropriate because funnel-web spider. Yeah, bad joke, I know.
Anyhow, poor funnels kill companies because when you think of… When you are trying to influence people’s behavior over a multiple stages of steps, if you execute on this poorly, your company will not succeed and, you know, you go home crying and your employees don’t have jobs anymore.
Because think of where you find funnels in your business. You find funnels in email. If you remember Rob Walling’s presentation from last year, email’s one of the most important things you can be doing.
OK, I’m going to collect some percentage of people’s emails but I’m going to email most of them, some percentage of them are going to open, some percentage are going to click on the link, and after clicking on the link some percentage will actually take an action that helps my business. That’s a funnel.
If you don’t do that right your business will not have that level of success from the email marketing.
For starting free trials, you know, some percentage will see your landing page, some percentage will convert into the trial, some percent will have a good experience with the trial, and some percent will go on to actually pay you money.
For your sales cycle, some percent of people will come from an undifferentiated visitor taking whatever action is required by your business to turn into a lead. Some percentage of those leads will talk to salespeople, some percentage will have a successful talk and you’ll get passed up to the decision maker, some percentage will signal actual buying intent and some percentage of those will actually convert into deals.
Even the core use of your product is often a funnel. Many of us make software that helps people do their jobs so there’s some sort of a workflow involved in it. And there’s many stages, you know, A to B to C to D, before a customer actually gets value out of it. And as people use the software, going from A to B, and B to C, and C to D, people stop using it — and people who don’t get all the way to D don’t get the value from your software.
And if they’re not getting in the value than what, really, was the purpose of you writing it?
And check out, of course, put something in your cart, get to the check out page, put in the credit card details, actually click the Confirm button to let us charge your Visa, this is also a funnel.
And if you fail at execution of these things your business just will not work.
And there’s many other reasons your business can’t work. If you can’t produce software that works or if you don’t create actual value for users, yeah, that will kill your business too.
But everyone in this room has that down already. And I don’t know if you all have funnels in our blood yet, so I’d like to change that.
Because what you do with a funnel: first you describe it, and that’s as simple as saying “My steps are, in order, are this, this, this and this.”
Then you measure it, then there’s that nebulous no-one-knows-what-comes-here step, and then you make a lot of money.
So if you’re wondering: “How do I measure a funnel?” I’m not here to sell you any tools, but I use [Cosmetrics], personally, there’s a bunch of things that you can use…
Google Analytics is free, any of these can get you up and running in, you know, minutes of engineering time and you can have information about your funnel in less than a day.
So the missing like “Aha!” step is that after you know what your funnel is and you know where people are falling out of it, what stages are the ones that aren’t converting as well as you think they should be? You can optimize those.
So let me give you an example. Let me give you some general advice on optimizing: shorter is generally better.
If there’s a step in your funnel that, when you think about it, doesn’t add value to your business and doesn’t add value for your customers, you should probably cut that step out.
That seems obvious. But there’s a lot of companies that I’ve looked at, where’s there’s just, you know, an extraneous…
Six years ago I worked with Fog Creek — we’ve talked about this previously — but like n years ago they sold manuals. And for a manual they actually need to, like, ship something to an actual human. So we asked for their shipping address, right?
And right now Fog Creek sells nothing that can actually ship to a human, but the shopping cart has been coded the same way — for the last ten years — so there was a step asking people for their shipping address, right?
And every step the funnel gets longer, even if it’s something simple, like, “Could you please write your address again?” Will cause less people to buy their very expensive, very good enterprise software.
And so, just taking that step out makes them an appreciable amount of money.
Improvements in a funnel compound multiplicatively. What does that mean?
It means that if you increase the number of people getting through step A by about 10%, and you increase the number of people getting through step B by about 10% you don’t get 20% increase to the bottom line, you get about 21 %.
And yeah, the engineers in the room are going: “Wait XXXX there’s an assumption you have to make that those are independent random events,” but I get it. You get an A+ in statistics but you can generally fudge this in real life and it still works.
So the nice part is that you get better results than expected the more and more you work on your funnels.
So when you first write out one of your funnels — and this is a funnel for Bingo Card Creator — and I won’t go through the actual steps — but it turns out there’s a lot of work at making Bingo Cards, who know? You’re going to find there’s some places that just really catch your eyes — so that really shouldn’t be that difficult.
What are my customers failing there? What did I do wrong?
And so I just put red arrows on them, just like tie a string on your head: “OK, it seems like the open word list and creating cards steps were accessibly hard.”
And prior to measuring my funnels I thought: “Well, I think, like, my intuition is 70% of my users are actually — who sign up for the service — are actually successfully creating bingo cards.”
And I found less than half were.
And that was kind of devastating to me. I did all this work creating software and spent all the money on AdWords to get them to my site, and over half of my users were not getting even the smallest scintilla of value out of my software.
I mean it’s not called Bingo Designer Voyeur.com. It’s Bingo Card Creator. If you’re not actually creating Bingo cards — something is wrong!
And so the customization step — I was really losing a lot of people about that — and I thought: “Well, what could possibly be going wrong with this customization step?” and I actually looked at the screen and it started at two options and then, every time a customer had a feature request, I said, “Oh, that sounds like a great idea,” and I added it — and I added it, and I added it, and I added it…
And this was really intimidating people.
And so I thought: “Wait! Are people actually using these features that I’ve added over the years?”
I’ve had nothing but positive feedback about them. But if we check in the database — very few of my customers were actually using the features that I had spent most of my time writing.
And the added features were disimproving the experience for most of my customers. It was just making it less likely that they were going to succeed in their task of teaching children tomorrow.
I was hurting their lives with adding more features to this software.
So I thought: “Maybe there’s something I can do about this that’s kind of easy. Maybe I can just take the features, put them back to where they were four years ago, and for the people who actually love having those twelve knobs to fiddle, just click on this little “click here” link and they show on automatically.”
But then when you rerun people through the funnel… I had a radically better business result. 16% — and we called it a lift, — 16% of people who wouldn’t have gotten through the funnel, actually managed to get through the funnel
Created more value to my customers and directly had a huge increase on my bottom line for only about an hour of work.
And that’s really powerful, right?
You’re thinking: “Yeah. That’s really powerful; but you kind of run a little toy business. Does this actually work with real companies?”
Yes! So I’ve had the opportunity to consult with a few companies and I can’t tell you actual results from all of them but I’m going to tell you general results from some of them. And they just get me drunk later… [Laughter]
I don’t drink! Ha-ha! [Laughter]
So, Fog Creek — can I get a shout-out from Fog Creek in here? Wheee!
sp2: Woo, woo! [from the audience]
Fog Creek are some of my favorite people; and we worked together last year and a little this year on revamping their website and kind of getting them more into being an engineering- and marketing-driven company. A company, as opposed to a company “by the engineers, for the engineers, of the engineers”… [audience mutters; chuckles]
Yeah. We all went through this. OK.
So, they had this page called “the details page” on the old version of the website. And it was a pretty important detail: the details page was the only way Fog Creek could actually get paid.
This is like the entrance to the checkout funnel. And you can just tell by looking at it, kind of difficult to get through there, right?
I actually — so they were paying me money to come use their website and attempt to buy their product. And I have an engineering degree, and I’ve been doing this for the last ten years or so and I couldn’t figure out how. I was at the point where I nearly, like, got on the phone and called tech support to say, “Excuse me, I’m an outside web optimization consultant. How do I actually buy this?”
And it turns out you could actually buy it through the website.
We made that a little more explicit. This conveys in a much more comprehensible, you know, one look fashion, OK? Here’s what we offer, here’s what it costs, here is what you do to buy, you know, click on the nice big button.
And I can’t give you an exact — like a numeric result from Fog Creek or else I’m persona non grata there, but suffice it to say the team that was involved in this project, like, their hands were stinging from the high fives for many, many weeks.
So I want to talk about another technique.
Everyone has a sickness. Some people have malaria, some people have TB, some people have “they don’t buy our stuff.” [Laughter]
And we have a way of addressing this sickness. It’s called “Medicine”!
We give… And we know medicine works because we give two different groups of people two different treatments. We’ll give one the placebo, and one the actual drug, and when we see one group perform markedly better than the other we know, “Aha!” that one difference that we controlled for was actually what caused the difference in behavior. This sort of thinking is kind of rare in the online blogosphere for, say, web design or marketing advice, because people really love their leeches. [Laughter]
It’s like, “Why do we use leeches?”— “I don’t know! Everyone has always used leeches and smart people say, ‘Use leeches,’ so you should use leeches too.”
So, not to pick on “Smashing Magazine” because this is rampant in the UX designosphere, but they came out with a thing a couple of weeks ago that said, with reasoned arguments drawn from a lot of research from qualified people, that if you had less than sixteen points a point fund for your body copy you were costing yourself money.
Like that is a testable statement about the condition of the natural world — we should be able to test it, right?
So, I did!
I …it turned out my website was fourteen point copy. And I made one version of the website with the old fourteen point copy and one version of the website with the sixteen point copy, and showed half of the people who came one week one version and during the same the same week half of the people the other version.
And…(they have no idea this test is going on, and the only thing different about the website are themselves because it’s a randomly controlled trial, is whether they are seeing 14 or 16 point.)
And it turns out — the most common result for A/B testing— there was not a lick of difference between the fourteen point and sixteen point.
So if you have something on your “to do” list — you know “Smashing Magazine” says you should get that done— you can take that one off probably.
But you have a lot of things like this, right? Like, what message are people going to respond to? What position are people going to respond to? What pricing are people going to respond to?
And our best answer for this, as a profession, is…leeches!
You can do better. Try two things. Test them, see which one actually works.
And there’s very easy software for doing this these days. Your programming language of choice probably has an A/B testing library where you can code tests with just one line of code. If it doesn’t, you can write one in a week — I did it.
Sidenote here: if you write one for a stack that doesn’t already have it and you put it on your website, all of the OSS bloggers in the world are going blog about your nice new OSS thing being on your website…You will get SEO juice, the Googles will love you and you’ll make a lot of money. End of sidenote. [Laughter]
OK. Do you think DHH gave a link to my blog because he really love teaching bingo cards? No. But he’s a Rails guy and I wrote the Rails A/B testing library.
So, there you go!
So, write your own. If the people that are actually doing the A/B tests are not programmers, then there’s two great options for doing them visually. One is Visual Website Optimizer, which I really like…— it’s, like… I very, very rarely get envious but this guy when he…— he sent me an email prior to launch and he said, “Hey, Patrick, I know you’re the A/B testing guy. Can you to review this website?” And I was like, “My capsule review, if I had thought of this two weeks ago, I would be a millionaire a year from now.”
Can’t tell you what he’s making but he’s doing well. Anyhow… And then optimizing is another option.
So, there’s many places you can test your website, right?
If you go back to your own company and start reading about all this stuff, start reading about what works for other people, you’re going to find that places early in the funnel, before people have strong emotional reason to be attached to you, are great places for A/B testing because you are on knife’s edge there.
Maybe a little push to the left and they fall out, maybe a little push to the right, they fall in. Landing pages are a great reason for that. They have perhaps never heard of you, they just clicked on an ad, they’ve got, as XXXX says, they’ve got their finger hovered over the back button. And you have three seconds to make an impression.
And you should be testing for what three seconds gives you the right impression.
On your pricing page, like we saw earlier, in your shopping cart…
So one A/B test that I did, someone had an intuition, I was selling stuff via PayPal and I thought: “Maybe I should add Google Checkout or maybe I shouldn’t add Google Checkout.”
And I tried to find querying the blogosphere and: “Leeches!” “No leeches!” ”Leeches!” “No leeches!”
So I said, “All right, I’ll show half of the people PayPal and Google Checkout, — and to the other half just PayPal.”
And it turns out that people who saw both PayPal Google Checkout had a higher conversion rate total, and also a higher conversion rate through PayPal.
Like, just putting Google Checkout as an option, maybe if we can get inside the user’s head, maybe that makes me look more credible…
And so they were more likely to purchase even through PayPal than they would have been without seeing the Google button. So…who knows.
So it’s always dangerous to get into users’ head. Look at their actual behavior rather then what you think they think…Or even what they say.
Well, one more thing you can A/B test is the actual behavior of your software. For example, if you’re thinking, “Hey, we’re investing in this amount of engineering resources into creating a new feature.”
If you create a new feature for your users, your goal is to change your users’ behavior into either making their lives better or making something better for your company by increasing the amount of software you sell to them.
If you create a new feature and give it to half of the users and don’t give it to the other half — and nothing visibly changes — than there was no point in having that feature, right?
So particularly, things on these pages you can test. Headlines — people are really, really sensitive to them. Which is kind of screwy from an engineering point of view because I always skip over them to get to the good stuff, but apparently people read them, and it’s one of…
If you look at the eye tracking studies, it’s one of the few things peoples’ eyes actually hit on a web page. Offers, sort of a value proposition to a user. You’re not saying, “Hey, you give us money, we give you software,” it’s: “Hey, give us money, and we give you this package of stuff. Particular software that you can use in a particular way, which include X amount of consulting services, yadda-yadda-yadda.” You can, like, tweak those knobs up and down and see what people really respond to.
One of the things that HubSpot said last year was that people really responded better to their core offering when they increased the amount of, like, hand holding that was involved in it, independent of the amount of hand holding that was actually consumed, because people just thought: “Ooh, software is scary.”
Calls to action, you know, the little text on your button of “sign up” or “start doing whatever now,” prominent graphical elements and important micro copy.
Thirty-seven signal pages are really good for this, you know, if you look at any of their signup pages right to the side of the actual form, they’ll have a micro copy that’s designed to address your objections against doing that form.
So right next to any email submission they’ll say, “Don’t worry, we won’t spam you. Here’s why.” Right next to a credit card thing: “This is totally secure. We use bank level security,” which, as we all know, means absolutely nothing, but people really respond to it. [Laughter]
So, I’ve been doing this bingo thing for the last five years and last Saturday I taught a group of software engineers A/B test in forty-five minutes.
And because software engineers respond well to live coding, I said, “Hey, I’m going to pull up my website and we’re going to live code and A/B test right now, just throw something out there — because I only have one minute available for this.”
“What’s something we could change?”
And they said, “Hey, why don’t we change this header, from ‘Make Your Own Bingo Cards’ to ‘Create Your Own Bingo Cards Now!’—exclamation point.
I said, “That’s a great test.” And so, I can code in like two seconds, doo-doo-doo-doo, and a week from now, we’ll have great information that says I’ve been doing this for the last five years and you’ve known about the existence of this website for the last two minutes, and therefore I did better. Right? [chuckles]
Well, that won’t make a good story at all.
So, the version that I’ve been working at for five years, I sent 20 thousand people to each of these versions, in full scientific glory, and 40% more of them bought…the only difference was something inserted by somebody who had no marketing background and who had no clue of who my customer even was… [inhales, exhales] [Laughter]
But this means that the Bingo Cards are now diamonds! Seriously. [Whistling]
So 37signals has done extensive A/B testing — you can read some of their stuff online. And believe me, if you haven’t started reading about A/B testing yet, you have no idea how deep the rabbit hole goes.
There’s so many things to read about, but one of their things is, they just played around with the headline on their pricing page and 37signals, they’re an Internet Megabrand — they’ve been around for the better part of the decade now — and just playing around with text on their pricing page has a huge, measurable, business result.
I can’t give you actual numbers from Fog Creek, but Fog Creek has been around for the ten years; they make excellent software for making great developers better and their software probably has 100 man-years invested in it.
And that 100 man-years of effort doesn’t matter in the first three seconds that somebody is at your website, so much as the headline does.
So you really need to put appropriate resources into making sure the headline and design and your value proposition really speak to that customer before they hit the Back button.
So this is the takeaway for this part of the presentation. Systematic A/B testing absolutely prints money. I’ve taken this to many different companies. The ones who actually get into the practice of: “OK, this sounds like freaky Voodoo. I mean, how can changing two words on your homepage actually increase sales by 40%?”
“But ah, we’ll give it a try, and we’ll run an A/B test, and see what happens.”
And if you get into actually doing this on a routine basis and making it part of your normal practice, it’ll collect a 10% win there and a 20% win here, and pretty soon 5 and 10, 20%, add up to real money. And it… If you guys are already starting from… We have a software company that has 10 or 20 employees and millions of dollars of revenue — 5% adds up to real money really, really quickly.
And especially I’d like to talk to the folks in the room who have more mature products; you know, you’ve been doing this for 10 years, your customers seem pretty happy, your feature roadmap is getting kind of sparse because you’ve implemented most of what you set out to implement, and you think your software provides most of the value that you want your software to provide.
Think of how much extra work it is to make that mature product 1% better, you know. Maybe you’ll get the 102nd feature into it, and it only has 101 right now. And maybe, you know, after two man-years of work implementing that feature you’ll be able to say, “Yes!” to customers, and some small percentage of them, maybe 5%, will use it and it will make their lives fractionally better, and that will increase your sales by 1%. And that’s for years of work.
Instead, you can make this A/B testing a routine thing in your play book. And you’ll find, for mere hours or weeks of work, that you’re getting, you know, a 5% improvement to the bottom line, a 5% improvement to customer success with the product, a 5% improvement to number of schoolkids who you helped to teach to read this year — for almost no effort at all, comparatively.
So, I really want to encourage you. If you do one thing from this presentation, go back and start trying it for a while. It’s not much work to get started. Just try it. You, probably, will like it.
So, if this A/B testing is so magical and gets such awesome results from it, why doesn’t everyone actually do this? Well, one reason is that, you know, we have cultural issues about it. We’re engineers — this smells, vaguely, of marketing. Also, there’s a tendency for A/B testing to tell you things that you really don’t want to hear. Designers don’t really want to hear that all the best design practices, well not all the best design practices. Many good, like… best practices are best practices for a reason. It’s knowing which 80% of them are best practices for a reason, and which 20% are best practices because someone was able to state in a confident tone of voice they are best practices.
But, you know, that last 20% can really hurt you. Here’s an example: many designers will tell you — and this is written in books — that your website design should be consistent on every page of the website because that makes it much easier for users to find their way around the website and do what they need to get done.
And what Amazon will tell you — because they’ve tested this extensively — and what everybody who I’ve tested with will tell you is that when people get into the directed mode of, like, trying to purchase your software, it’s better to break your website’s design and make it look not like the rest of the website. Just give them two options: either you go for it and purchasing, or you go back to the homepage, rather than: “OK, you can click out to the blog, you can click out to our documentation, blah, blah, blah.”
That will greatly increase the number of people who have successfully purchased the software, succeeding in their task, at the moment because they’re on your purchasing page to purchase it, right? And if you ask customers… If you, like, collect metrics about what peoples’ use of the software is after that, you know… Proxying for their happiness because your software is making them happier, right? Just that change will increase user happiness and user success with your software. “Well, I’m making you a pile of money, so you should probably do it.”
So, one of the things to make your testing — make your organization an A/B testing organization is, you really need to eliminate all friction about doing A/B testing. Some of my clients have, you know, workflows where it can take up to an hour for things to get from the point where it’s on someone’s computer to where it’s visible on the actual Internet. And if it takes an hour for you to change copy on your homepage, you’re going to do it like once a year whether it needs to be updated or not, right?
So, instead, you want that change to be virtually instantaneous. And you can do this with the A/B testing libraries we talked about earlier. If you’re writing code for A/B tests, you want to write as little code as possible because engineers are lazy. That’s like our job description. And you want to empower your engineers to be lazy by saying, “Hey, writing an A/B test is as easy as — take the code that’s already there, put it in an “if” statement, write what you were considering doing, and then you’re done!”
And when you do that, you’ll find that you’re not an A/B test every blue moon when, you know, a spirit moves you. You’re running A/B tests every day. I was running… The reason I got into this whole low-touch sales processes towards large numbers of potential prospects was because I only had five hours a week to work on this business for most of the time I was running it because I had a day job.
And the… One of the few things I could actually do — because five hours is not enough to write new features — is, on Sunday evening I could push an A/B test live in a few minutes, and then the next Saturday I could look at the results for it and gradually improve my business over the years doing that. And, you know, you can pound a 5% improvement every… in a week. If you get an improvement one week out of every four you try, then 5% a month for a year is like a 70% improvement in the bottom line, which works pretty well.
You also need to have a culture of testing at your company. Many companies default to a culture of: “Let’s ask what the boss thinks.” We once had a six-hour discussion at my company, my old Japanese company, about the difference between — and I’ll translate it into English — the mailing list sign up, which we had 40 sign ups a quarter… that the mailing list sign up was either “Submit email,” or “Submit email address.” And this was, like, the six-hour meeting from heck, you know, just figuring out which of these two things we should use. And finally, we decided it the traditional way. The boss said, “I have heard all of your opinions. And I am the boss, I will make the important consequential decision to say, ‘Submit email.’” If you have a culture of testing in your company, that discussion would have lasted for about five seconds. It’s like: “Yeah, throw them both up at an A/B test. We’ll see if it even matters.” And that will save you a lot of time.
So, the third major technique I’m going to be talking about is about the first run experience. And it’s really important, because of all the people who have ever used your software, you can only say one thing about all of them: they’ve all run it the first time. How many, like… Do people… Raise your hand if you have a good answer for how many people run the free trial version or the actual version of your software the second time, percentage-wise? Like, raise your hand if that’s something you measure.
OK. So, we got one or two hands.
This is another thing you can implement in very little time, and the number is going to shock and dismay you. 40% for my company, when I started measuring this, I got into most of my consulting companies, consulting clients, 30%, 40%, 50%, see the software the second time. You know, often there are, like, hard costs of several tens of thousands of dollars to get into a particular prospect’s hands, and they play with it for three minutes and then stop using it, and that’s the end of that customer relationship. That’s clearly sub-optimal, right? So, you have to totally nail that first impression. And if you structure the first impression… the first use of the software as a funnel, where we think they are going to take activities A and B and C and D, and you measure where customers are falling out of it, they import their data, but they don’t go up to setting up their admin users and they don’t complete the stuff after that, then you can use A/B tests for improvements at that point in the funnel to make it work better.
And there are some examples of sites that do this to… s