Engineering your marketing outcomes | Patrick McKenzie, patio11 | BoS USA 2011

Marketing comes naturally to some people. Unfortunately, few of them go on to found software companies. Patrick talks about types of marketing that use engineering skills and won’t be rabidly opposed to your developers.

Video & Transcript below

Instead of spending a few weeks cranking out one more feature that won’t be seen by 1% of your users and increase the value of your software by <1%, learn how you can use the same few weeks to create scaleable systems that take much of the guesswork out of marketing.

Video

Transcript

Patrick McKenzie: [Deep voice] Hello Ladies. [Normal voice] And Gentlemen. [Laughter]

So, if you weren’t here at the talk last year I basically did about an hour-long takeoff on the Old Spice guy I was telling you about. Does everyone …OK …Raise your hands if you were here at the talk last year… [Everybody raising hands]

OK. Put your hands down if you have forgot… No, keep them up! [Laughter]

Alright. Now put your hands down if you have forgotten the topic of that talk. OK, Good. So some people remember even a year later.

So that’s the reason why I like humor in presentations — because it helps people retain what you’re speaking about. Now, right after Business of Software last year I flew back to Japan where I live, and I got off the plane and I gave a phone call to a lady who I met right before coming to Business of Software and she said, “Hey, what did you do on your business trip?” So I gave her the Old Spice speech. I said — and I got to get a new speech this year, and I probably can’t sustain the Old Spice guy for sixty minutes and the joke will be old.

And she says: “That’s OK; I think you’re cooler than the Old Spice guy anyhow.” So, that was a sign to me and we’re now dating. [Laughter and applause]

So, the most common question I get asked is “Why are you in Japan?” And because it’s kind of relevant to the topic of the talk, I’ll tell you. People always ask me “Is it because of Japanese girls?” And that did happen but it was, like, eight years later… The actual reason I’m in Japan is not because of Japanese girls, it’s because of Indian men. Specifically… [Laughter]. Ten years ago I was an engineer with very little self-confidence skills — and now I’m an engineer with very little self-confidence ten years older.

But I was in college and reading the Wall Street Journal. And the Wall Street Journal was saying “The Indians, they’re going to steal all our programming jobs and we’re going outsource everything to India for one quarter of the cost. And if you are getting a programmer right now it’s a mug’s game; you will never be able to compete with all of those folks.” And I’m, like, “Oh God, they’re right!”

So, I thought: “How I am going to get a nice, safe job at a big mega corp like Microsoft like my parents have always wanted for me?” I thought: “I’ll play the Van diagram game: you take one thing that’s pretty hard like programming, take another thing that’s pretty hard like, say, speaking Japanese and you intersect the two.”

And the people who can do both — there must be, like, four of them, right? So, if I go to Japan and become bilingual then I can come back and get a job at Microsoft and no one will ever be able to take that thing here away from me until the day I die. And that didn’t really work out. But…

So, I’m in Japan and… We have lots of young engineers in the audience? Let me… Perhaps help put your mind at ease regarding the outsourcing issue. I actually managed a Japanese company’s outsourcing operations to our Indian subsidiary for three years. And I will distill that into one anecdote which… This is absolutely true.

We made administration systems for universities and at one point we had… we sent the system over to the Indian subsidiary to get manually Q&A’d, you know, push all the buttons, make sure everything works. So, keep in mind in this conversation everything is happening via email with, like, a four-hour delay between each email.

Three days after we sent it over and expected to be, you know, mostly started and we asked: “What’s the status?”

“We haven’t started yet, the system isn’t working.”

“Oh, really? Why isn’t the system working?”

“We can’t log in.”

“Oh, really? Did you follow the page of instructions that would get you from the subversion and the database dump into a working system and then be able to log in?”

“Yeah, we did that but the system isn’t working.”

“Did you get an error message?”

“Yes, we got an error message.”

“What did the error message say?”

“I don’t know, I can’t read Japanese.” [Laughter]

“OK… Can you take a screenshot of the error message with the printscreen and paste it into, you know, MS Paint and send that over?”

And they do. And they show me a picture of the login screen and I write back: “That is not an error message, that is just a weather report, please continue with the instructions and push the buttons that we have marked for you and see if anything comes up.” And they say, “We don’t agree that that is a weather report. [Laughter]

I’m like, “That is interesting because I can read the text and, according to our previous conversation, you can’t.” [Laughter]

“Why don’t you think it’s a weather report?” And they said…they had two compelling pieces of argument on why it couldn’t be a weather report: it was on the login screen and written in flaming red text.

Why would you ever do that to a weather report? And I said, “Oh, it’s on the login screen and written in flaming red text because, that weather report, if you ignore it, you’ll die!” [Laughter]

And I told this story to Americans all the time, and one of them asked me: “What kind of system are you writing such that a) Weather is routinely lethal and b) It’s so routinely lethal that that’s already special-cased?”

And all I can say is, “Welcome to Japan!”

There are 30 words of Japanese that have made it into the English language and 15 of them are for meteorological phenomena that can kill you. [Laughter]

So, what am I doing these days when I am thankfully quit off that day job and, will never do outsourcing management ever again. But these days my kind of claim to Internet fame, aside from being on “Hacker News” because it’s my job — because I’m gainfully unemployed — is I’m kind of like the Internet’s Bingo guy.

I made Bingo Card Creator, it’s a really simple software that makes bingo cards for elementary school teachers. I won’t tell you too much about that specifically, but we are going to talk about some of the things that I have learned from it. Because I kind of get to use this as my laboratory where I can screw it up as much as I want, and aside from the 200K users, no one, like a boss, will yell at me. And, just so you get some perspective of where I am, Dharmesh fleshed his customer graph the other day so I thought: “Wow, he stole mine!” Like we started the same point we end near the same number, about 5000 customers and the curve looks almost exactly the same! But a key thing to keep in mind: my graph was cumulative and Dharmesh charges between like $1000 and $10,000 a month and I charged 30$ once. [Laughter] So, key takeaway: if you want to make a billion-dollar business selling software as a service at the 10,000 a month price point is a great idea — and selling 30$ a month once, you have to do a little more.

So, if you remember Jason Cohen’s talk from last year, he said that you get many pieces of competing advice at events like this, and if you are reading the entrepreneurial blogosphere; and it’s important for you to keep in mind where someone is coming from. And I think that’s important when you’re listening to my presentation. So, I’ve prepared this lovely little four-quadrant graph on the axis of what market you’re addressing and on the spectrum of funded to bootstrapped like me.

And I put up some companies that I know and respect and appear at other places on this graph and, of course, because the only purpose of having a four-quadrant graph is to put yourself up into the right. You can see that I’m more evolved by them. [Laughter] Well, honestly, what you were you expecting from a guy named McKenzie, right? [Laughter] But one of the reasons I got into engineering is this really simple and powerful idea that math and science always work. And I think many of the engineers here really have an emotional attachment to this one. And we don’t really have an emotional attachment to marketing, I mean…Can I… just have a quick show of hands of who here is mostly an engineer? OK… hands down. Who here is mostly a marketer? OK, yeah. That’s what I thought.

And who here is Paul Kenny? [Laughter]

OK. So for you marketers who haven’t caught on to it yet, the stereotype of marketing in the engineering field is that, marketing is basically witchcraft. There’s some, like, eye of newt and toe of dog, or whatever that goes into the pot — we have no clue how anything works and it’s, like, vaguely evil. I mean, you’re selling stuff to people…And it’s OK when people buy our stuff, but when people buy other stuff that’s kind of unnatural so we’re just creeped out by it.

But something that I’ve come to appreciate over the years of running my own business is that marketing is not really witchcraft, it’s just about changing people’s behavior. And this is not something that is totally unnoble. We know we can change people’s behavior, we can do it all the time by stuff as simple as changing our tone of voice. As any lumberjack will tell you, “All good things come from timber.” [Laughter] OK, Tim told you about me.

And this is, you know… We change people’s behavior in engineering too. That’s why we build products, we want you to change from a manual process to this better process that we have; that’s why Steve Jobs and an excellent product design, and can turn around how an entire world uses their cell phone. And if we get into this we think that: “Wow, marketing is really sort of an engineering discipline.”

And so I want to talk about one particular type of way of changing people’s behavior. If you think of it, your business is set up to cause a series of fortunate events. There’s billions and billions of people in the world and you really are trying to convince some small fraction of them to first come to your website. And of that small fraction an even smaller fraction are going to start your free trial, — if that’s your business model. And of that small fraction an even smaller fraction are going to actually succeed in paying you money.

And this graph is called a funnel in traditional marketing and traditionally this is the point where I have like a clip art slide of a, you know, cooking funnel. But I thought I would use something a little more appropriate because funnel-web spider. Yeah, bad joke, I know. Anyhow, poor funnels kill companies because when you think of… When you are trying to influence people’s behavior over a multiple stages of steps, if you execute on this poorly, your company will not succeed and, you know, you go home crying and your employees don’t have jobs anymore. Because think of where you find funnels in your business. You find funnels in email. If you remember Rob Walling’s presentation from last year, email’s one of the most important things you can be doing.

OK, I’m going to collect some percentage of people’s emails but I’m going to email most of them, some percentage of them are going to open, some percentage are going to click on the link, and after clicking on the link some percentage will actually take an action that helps my business. That’s a funnel. If you don’t do that right your business will not have that level of success from the email marketing. For starting free trials, you know, some percentage will see your landing page, some percentage will convert into the trial, some percent will have a good experience with the trial, and some percent will go on to actually pay you money.

For your sales cycle, some percent of people will come from an undifferentiated visitor taking whatever action is required by your business to turn into a lead. Some percentage of those leads will talk to salespeople, some percentage will have a successful talk and you’ll get passed up to the decision maker, some percentage will signal actual buying intent and some percentage of those will actually convert into deals.

Even the core use of your product is often a funnel. Many of us make software that helps people do their jobs so there’s some sort of a workflow involved in it. And there’s many stages, you know, A to B to C to D, before a customer actually gets value out of it. And as people use the software, going from A to B, and B to C, and C to D, people stop using it — and people who don’t get all the way to D don’t get the value from your software. And if they’re not getting in the value than what, really, was the purpose of you writing it?

And check out, of course, put something in your cart, get to the check out page, put in the credit card details, actually click the Confirm button to let us charge your Visa, this is also a funnel. And if you fail at execution of these things your business just will not work. And there’s many other reasons your business can’t work. If you can’t produce software that works or if you don’t create actual value for users, yeah, that will kill your business too. But everyone in this room has that down already. And I don’t know if you all have funnels in our blood yet, so I’d like to change that.

Because what you do with a funnel: first you describe it, and that’s as simple as saying “My steps are, in order, are this, this, this and this.” Then you measure it, then there’s that nebulous no-one-knows-what-comes-here step, and then you make a lot of money. So if you’re wondering: “How do I measure a funnel?” I’m not here to sell you any tools, but I use [Cosmetrics], personally, there’s a bunch of things that you can use…Google Analytics is free, any of these can get you up and running in, you know, minutes of engineering time and you can have information about your funnel in less than a day.

So the missing like “Aha!” step is that after you know what your funnel is and you know where people are falling out of it, what stages are the ones that aren’t converting as well as you think they should be? You can optimize those.

So let me give you an example. Let me give you some general advice on optimizing: shorter is generally better. If there’s a step in your funnel that, when you think about it, doesn’t add value to your business and doesn’t add value for your customers, you should probably cut that step out. That seems obvious. But there’s a lot of companies that I’ve looked at, where’s there’s just, you know, an extraneous…

Six years ago I worked with Fog Creek — we’ve talked about this previously — but like years ago they sold manuals. And for a manual they actually need to, like, ship something to an actual human. So we asked for their shipping address, right? And right now Fog Creek sells nothing that can actually ship to a human, but the shopping cart has been coded the same way — for the last ten years — so there was a step asking people for their shipping address, right?

And every step the funnel gets longer, even if it’s something simple, like, “Could you please write your address again?” Will cause less people to buy their very expensive, very good enterprise software. And so, just taking that step out makes them an appreciable amount of money. Improvements in a funnel compound multiplicatively. What does that mean? It means that if you increase the number of people getting through step A by about 10%, and you increase the number of people getting through step B by about 10% you don’t get 20% increase to the bottom line, you get about 21 %.

And yeah, the engineers in the room are going: “Wait XXXX there’s an assumption you have to make that those are independent random events,” but I get it. You get an A+ in statistics but you can generally fudge this in real life and it still works. So the nice part is that you get better results than expected the more and more you work on your funnels.

So when you first write out one of your funnels — and this is a funnel for Bingo Card Creator —  and I won’t go through the actual steps — but it turns out there’s a lot of work at making Bingo Cards, who know? You’re going to find there’s some places that just really catch your eyes — so that really shouldn’t be that difficult. What are my customers failing there? What did I do wrong? And so I just put red arrows on them, just like tie a string on your head: “OK, it seems like the open word list and creating cards steps were accessibly hard.”

And prior to measuring my funnels I thought: “Well, I think, like, my intuition is 70% of my users are actually —  who sign up for the service — are actually successfully creating bingo cards.” And I found less than half were. And that was kind of devastating to me. I did all this work creating software and spent all the money on AdWords to get them to my site, and over half of my users were not getting even the smallest scintilla of value out of my software. I mean it’s not called Bingo Designer Voyeur.com. It’s Bingo Card Creator. If you’re not actually creating Bingo cards — something is wrong!

And so the customization step — I was really losing a lot of people about that — and I thought: “Well, what could possibly be going wrong with this customization step?” and I actually looked at the screen and it started at two options and then, every time a customer had a feature request, I said, “Oh, that sounds like a great idea,” and I added it — and I added it, and I added it, and I added it…And this was really intimidating people. And so I thought: “Wait! Are people actually using these features that I’ve added over the years?” I’ve had nothing but positive feedback about them. But if we check in the database — very few of my customers were actually using the features that I had spent most of my time writing. And the added features were disimproving the experience for most of my customers. It was just making it less likely that they were going to succeed in their task of teaching children tomorrow. I was hurting their lives with adding more features to this software.

So I thought: “Maybe there’s something I can do about this that’s kind of easy. Maybe I can just take the features, put them back to where they were four years ago, and for the people who actually love having those twelve knobs to fiddle, just click on this little “click here” link and they show on automatically.” And you can imagine this isn’t a lot of work —a wee bit of Javascript coding, less than an hour to collect these stats, less than an hour to do the implementation…But then when you rerun people through the funnel… I had a radically better business result. 16% — and we called it a lift, — 16% of people who wouldn’t have gotten through the funnel, actually managed to get through the funnel, created more value to my customers and directly had a huge increase on my bottom line for only about an hour of work. And that’s really powerful, right? You’re thinking: “Yeah. That’s really powerful; but you kind of run a little toy business. Does this actually work with real companies?” [Laughter]

Yes! So I’ve had the opportunity to consult with a few companies and I can’t tell you actual results from all of them but I’m going to tell you general results from some of them. And they just get me drunk later… [Laughter] I don’t drink! Ha-ha! [Laughter] So, Fog Creek — can I get a shout-out from Fog Creek in here? Wheee!

Fog Creek are some of my favorite people; and we worked together last year and a little this year on revamping their website and kind of getting them more into being an engineering- and marketing-driven company. A company, as opposed to a company “by the engineers, for the engineers, of the engineers”… [audience mutters; chuckles] Yeah. We all went through this. OK.

So, they had this page called “the details page” on the old version of the website. And it was a pretty important detail: the details page was the only way Fog Creek could actually get paid. This is like the entrance to the checkout funnel. And you can just tell by looking at it, kind of difficult to get through there, right? I actually — so they were paying me money to come use their website and attempt to buy their product. And I have an engineering degree, and I’ve been doing this for the last ten years or so and I couldn’t figure out how. I was at the point where I nearly, like, got on the phone and called tech support to say, “Excuse me, I’m an outside web optimization consultant. How do I actually buy this?” And it turns out you could actually buy it through the website.

We made that a little more explicit. This conveys in a much more comprehensible, you know, one look fashion, OK? Here’s what we offer, here’s what it costs, here is what you do to buy, you know, click on the nice big button. And I can’t give you an exact — like a numeric result from Fog Creek or else I’m persona non grata there, but suffice it to say the team that was involved in this project, like, their hands were stinging from the high fives for many, many weeks.

So I want to talk about another technique.

Everyone has a sickness. Some people have malaria, some people have TB, some people have “they don’t buy our stuff.” [Laughter] And we have a way of addressing this sickness. It’s called “Medicine”! We give… And we know medicine works because we give two different groups of people two different treatments. We’ll give one the placebo, and one the actual drug, and when we see one group perform markedly better than the other we know, “Aha!” that one difference that we controlled for was actually what caused the difference in behavior. This sort of thinking is kind of rare in the online blogosphere for, say, web design or marketing advice, because people really love their leeches. [Laughter] It’s like, “Why do we use leeches?”— “I don’t know! Everyone has always used leeches and smart people say, ‘Use leeches,’ so you should use leeches too.”

So, not to pick on “Smashing Magazine” because this is rampant in the UX designosphere, but they came out with a thing a couple of weeks ago that said, with reasoned arguments drawn from a lot of research from qualified people, that if you had less than sixteen points a point fund for your body copy you were costing yourself money. Like that is a testable statement about the condition of the natural world — we should be able to test it, right? So, I did!

I …it turned out my website was fourteen point copy. And I made one version of the website with the old fourteen point copy and one version of the website with the sixteen point copy, and showed half of the people who came one week one version and during the same the same week half of the people the other version. And…(they have no idea this test is going on, and the only thing different about the website are themselves because it’s a randomly controlled trial, is whether they are seeing 14 or 16 point.)

And it turns out — the most common result for A/B testing— there was not a lick of difference between the fourteen point and sixteen point. So if you have something on your “to do” list — you know “Smashing Magazine” says you should get that done— you can take that one off probably. But you have a lot of things like this, right? Like, what message are people going to respond to? What position are people going to respond to? What pricing are people going to respond to?

And our best answer for this, as a profession, is…leeches! You can do better. Try two things. Test them, see which one actually works. And there’s very easy software for doing this these days. Your programming language of choice probably has an A/B testing library where you can code tests with just one line of code. If it doesn’t, you can write one in a week — I did it.

Sidenote here: if you write one for a stack that doesn’t already have it and you put it on your website, all of the OSS bloggers in the world are going blog about your nice new OSS thing being on your website…You will get SEO juice, the Googles will love you and you’ll make a lot of money. End of sidenote. [Laughter]

OK. Do you think DHH gave a link to my blog because he really love teaching bingo cards? No. But he’s a Rails guy and I wrote the Rails A/B testing library. So, there you go!

So, write your own. If the people that are actually doing the A/B tests are not programmers, then there’s two great options for doing them visually. One is Visual Website Optimizer, which I really like…— it’s, like… I very, very rarely get envious but this guy when he…— he sent me an email prior to launch and he said, “Hey, Patrick, I know you’re the A/B testing guy. Can you to review this website?” And I was like, “My capsule review, if I had thought of this two weeks ago, I would be a millionaire a year from now.” Can’t tell you what he’s making but he’s doing well. Anyhow… And then optimizing is another option. So, there’s many places you can test your website, right?

If you go back to your own company and start reading about all this stuff, start reading about what works for other people, you’re going to find that places early in the funnel, before people have strong emotional reason to be attached to you, are great places for A/B testing because you are on knife’s edge there.

Maybe a little push to the left and they fall out, maybe a little push to the right, they fall in. Landing pages are a great reason for that. They have perhaps never heard of you, they just clicked on an ad, they’ve got, as XXXX says, they’ve got their finger hovered over the back button. And you have three seconds to make an impression. And you should be testing for what three seconds gives you the right impression. On your pricing page, like we saw earlier, in your shopping cart…So one A/B test that I did, someone had an intuition, I was selling stuff via PayPal and I thought: “Maybe I should add Google Checkout or maybe I shouldn’t add Google Checkout.”

And I tried to find querying the blogosphere and: “Leeches!” “No leeches!” ”Leeches!” “No leeches!” So I said, “All right, I’ll show half of the people PayPal and Google Checkout, — and to the other half just PayPal.” And it turns out that people who saw both PayPal Google Checkout had a higher conversion rate total, and also a higher conversion rate through PayPal. Like, just putting Google Checkout as an option, maybe if we can get inside the user’s head, maybe that makes me look more credible…And so they were more likely to purchase even through PayPal than they would have been without seeing the Google button. So…who knows. So it’s always dangerous to get into users’ head. Look at their actual behavior rather then what you think they think…Or even what they say.

Well, one more thing you can A/B test is the actual behavior of your software. For example, if you’re thinking, “Hey, we’re investing in this amount of engineering resources into creating a new feature.” If you create a new feature for your users, your goal is to change your users’ behavior into either making their lives better or making something better for your company by increasing the amount of software you sell to them. If you create a new feature and give it to half of the users and don’t give it to the other half — and nothing visibly changes — than there was no point in having that feature, right?

So particularly, things on these pages you can test. Headlines — people are really, really sensitive to them. Which is kind of screwy from an engineering point of view because I always skip over them to get to the good stuff, but apparently people read them, and it’s one of…If you look at the eye tracking studies, it’s one of the few things peoples’ eyes actually hit on a web page. Offers, sort of a value proposition to a user. You’re not saying, “Hey, you give us money, we give you software,” it’s: “Hey, give us money, and we give you this package of stuff. Particular software that you can use in a particular way, which include X amount of consulting services, yadda-yadda-yadda.” You can, like, tweak those knobs up and down and see what people really respond to.

One of the things that HubSpot said last year was that people really responded better to their core offering when they increased the amount of, like, hand holding that was involved in it, independent of the amount of hand holding that was actually consumed, because people just thought: “Ooh, software is scary.” Calls to action, you know, the little text on your button of “sign up” or “start doing whatever now,” prominent graphical elements and important micro copy. Thirty-seven signal pages are really good for this, you know, if you look at any of their signup pages right to the side of the actual form, they’ll have a micro copy that’s designed to address your objections against doing that form.

So right next to any email submission they’ll say, “Don’t worry, we won’t spam you. Here’s why.” Right next to a credit card thing: “This is totally secure. We use bank level security,” which, as we all know, means absolutely nothing, but people really respond to it. [Laughter]

So, I’ve been doing this bingo thing for the last five years and last Saturday I taught a group of software engineers A/B test in forty-five minutes. And because software engineers respond well to live coding, I said, “Hey, I’m going to pull up my website and we’re going to live code and A/B test right now, just throw something out there — because I only have one minute available for this.” “What’s something we could change?”

And they said, “Hey, why don’t we change this header, from ‘Make Your Own Bingo Cards’ to ‘Create Your Own Bingo Cards Now!’—exclamation point. I said, “That’s a great test.” And so, I can code in like two seconds, doo-doo-doo-doo, and a week from now, we’ll have great information that says I’ve been doing this for the last five years and you’ve known about the existence of this website for the last two minutes, and therefore I did better. Right? [chuckles] Well, that won’t make a good story at all.

So, the version that I’ve been working at for five years, I sent 20 thousand people to each of these versions, in full scientific glory, and 40% more of them bought…the only difference was something inserted by somebody who had no marketing background and who had no clue of who my customer even was… [inhales, exhales] [Laughter] But this means that the Bingo Cards are now diamonds! Seriously. [Whistling] So 37signals has done extensive A/B testing —  you can read some of their stuff online. And believe me, if you haven’t started reading about A/B testing yet, you have no idea how deep the rabbit hole goes.

There’s so many things to read about, but one of their things is, they just played around with the headline on their pricing page and 37signals, they’re an Internet Megabrand — they’ve been around for the better part of the decade now — and just playing around with text on their pricing page has a huge, measurable, business result.

I can’t give you actual numbers from Fog Creek, but Fog Creek has been around for the ten years; they make excellent software for making great developers better and their software probably has 100 man-years invested in it. And that 100 man-years of effort doesn’t matter in the first three seconds that somebody is at your website, so much as the headline does. So you really need to put appropriate resources into making sure the headline and design and your value proposition really speak to that customer before they hit the Back button.

So this is the takeaway for this part of the presentation. Systematic A/B testing absolutely prints money. I’ve taken this to many different companies. The ones who actually get into the practice of: “OK, this sounds like freaky Voodoo. I mean, how can changing two words on your homepage actually increase sales by 40%?” “But ah, we’ll give it a try, and we’ll run an A/B test, and see what happens.”

And if you get into actually doing this on a routine basis and making it part of your normal practice, it’ll collect a 10% win there and a 20% win here, and pretty soon 5 and 10, 20%, add up to real money. And it… If you guys are already starting from… We have a software company that has 10 or 20 employees and millions of dollars of revenue — 5% adds up to real money really, really quickly.

And especially I’d like to talk to the folks in the room who have more mature products; you know, you’ve been doing this for 10 years, your customers seem pretty happy, your feature roadmap is getting kind of sparse because you’ve implemented most of what you set out to implement, and you think your software provides most of the value that you want your software to provide.

Think of how much extra work it is to make that mature product 1% better, you know. Maybe you’ll get the 102^nd feature into it, and it only has 101 right now. And maybe, you know, after two man-years of work implementing that feature you’ll be able to say, “Yes!” to customers, and some small percentage of them, maybe 5%, will use it and it will make their lives fractionally better, and that will increase your sales by 1%. And that’s for years of work.

Instead, you can make this A/B testing a routine thing in your play book. And you’ll find, for mere hours or weeks of work, that you’re getting, you know, a 5% improvement to the bottom line, a 5% improvement to customer success with the product, a 5% improvement to number of schoolkids who you helped to teach to read this year — for almost no effort at all, comparatively.

So, I really want to encourage you. If you do one thing from this presentation, go back and start trying it for a while. It’s not much work to get started. Just try it. You, probably, will like it.

So, if this A/B testing is so magical and gets such awesome results from it, why doesn’t everyone actually do this? Well, one reason is that, you know, we have cultural issues about it. We’re engineers — this smells, vaguely, of marketing. Also, there’s a tendency for A/B testing to tell you things that you really don’t want to hear. Designers don’t really want to hear that all the best design practices, well not all the best design practices. Many good, like… best practices are best practices for a reason. It’s knowing which 80% of them are best practices for a reason, and which 20% are best practices because someone was able to state in a confident tone of voice they are best practices.

But, you know, that last 20% can really hurt you. Here’s an example: many designers will tell you — and this is written in books — that your website design should be consistent on every page of the website because that makes it much easier for users to find their way around the website and do what they need to get done.

And what Amazon will tell you — because they’ve tested this extensively — and what everybody who I’ve tested with will tell you is that when people get into the directed mode of, like, trying to purchase your software, it’s better to break your website’s design and make it look not like the rest of the website. Just give them two options: either you go for it and purchasing, or you go back to the homepage, rather than: “OK, you can click out to the blog, you can click out to our documentation, blah, blah, blah.”

That will greatly increase the number of people who have successfully purchased the software, succeeding in their task, at the moment because they’re on your purchasing page to purchase it, right? And if you ask customers… If you, like, collect metrics about what peoples’ use of the software is after that, you know… Proxying for their happiness because your software is making them happier, right? Just that change will increase user happiness and user success with your software. “Well, I’m making you a pile of money, so you should probably do it.”

So, one of the things to make your testing — make your organization an A/B testing organization is, you really need to eliminate all friction about doing A/B testing. Some of my clients have, you know, workflows where it can take up to an hour for things to get from the point where it’s on someone’s computer to where it’s visible on the actual Internet. And if it takes an hour for you to change copy on your homepage, you’re going to do it like once a year whether it needs to be updated or not, right?

So, instead, you want that change to be virtually instantaneous. And you can do this with the A/B testing libraries we talked about earlier. If you’re writing code for A/B tests, you want to write as little code as possible because engineers are lazy. That’s like our job description. And you want to empower your engineers to be lazy by saying, “Hey, writing an A/B test is as easy as — take the code that’s already there, put it in an “if” statement, write what you were considering doing, and then you’re done!”

And when you do that, you’ll find that you’re not an A/B test every blue moon when, you know, a spirit moves you. You’re running A/B tests every day. I was running… The reason I got into this whole low-touch sales processes towards large numbers of potential prospects was because I only had five hours a week to work on this business for most of the time I was running it because I had a day job.

And the… One of the few things I could actually do — because five hours is not enough to write new features — is, on Sunday evening I could push an A/B test live in a few minutes, and then the next Saturday I could look at the results for it and gradually improve my business over the years doing that. And, you know, you can pound a 5% improvement every… in a week. If you get an improvement one week out of every four you try, then 5% a month for a year is like a 70% improvement in the bottom line, which works pretty well.

You also need to have a culture of testing at your company. Many companies default to a culture of: “Let’s ask what the boss thinks.” We once had a six-hour discussion at my company, my old Japanese company, about the difference between — and I’ll translate it into English — the mailing list sign up, which we had 40 sign ups a quarter… that the mailing list sign up was either “Submit email,” or “Submit email address.” And this was, like, the six-hour meeting from heck, you know, just figuring out which of these two things we should use. And finally, we decided it the traditional way. The boss said, “I have heard all of your opinions. And I am the boss, I will make the important consequential decision to say, ‘Submit email.’” If you have a culture of testing in your company, that discussion would have lasted for about five seconds. It’s like: “Yeah, throw them both up at an A/B test. We’ll see if it even matters.” And that will save you a lot of time.

So, the third major technique I’m going to be talking about is about the first run experience. And it’s really important, because of all the people who have ever used your software, you can only say one thing about all of them: they’ve all run it the first time. How many, like… Do people… Raise your hand if you have a good answer for how many people run the free trial version or the actual version of your software the second time, percentage-wise? Like, raise your hand if that’s something you measure. OK. So, we got one or two hands.

This is another thing you can implement in very little time, and the number is going to shock and dismay you. 40% for my company, when I started measuring this, I got into most of my consulting companies, consulting clients, 30%, 40%, 50%, see the software the second time. You know, often there are, like, hard costs of several tens of thousands of dollars to get into a particular prospect’s hands, and they play with it for three minutes and then stop using it, and that’s the end of that customer relationship. That’s clearly sub-optimal, right? So, you have to totally nail that first impression. And if you structure the first impression… the first use of the software as a funnel, where we think they are going to take activities A and B and C and D, and you measure where customers are falling out of it, they import their data, but they don’t go up to setting up their admin users and they don’t complete the stuff after that, then you can use A/B tests for improvements at that point in the funnel to make it work better.

And there are some examples of sites that do this to sites that do this to… some tactics that might work for you. Ah Peldi.. So, I really love Balsamiq Mockups. And this is not actually the Balsamiq Mockup screen, because it starts completely empty. And no one has any clue when they start with software that’s completely empty. What do I do here? And no matter what your software does, if you start people at a complete, like, Garden of Eden state, most users are going to be: “Oh dear, what do I do first? I don’t know. How do I get value out of this? It’s not obvious; there’s just a piece of white paper ahead of me.” If you’re a designer, white paper is fun, but actual users have lives that they want to get back to and white paper is just: you have to do a lot of work prior to getting back to your life, and most of them are just going to crinkle it up and throw it away.

So, what Balsamiq Mockups actually look like when you start it is this. It has fake data imported into it which both tells you how you can get start with the software, you know, “Click here,” “Click here,” “Click here,” “Drag this,” “It’s risk-free,” “There’s a control Z available at any time,” and it starts to expose you to some of the richness of the software. You know, this thing that you’re seeing and you’re editing pane was clearly created just with the software. You can make something just, you know, just as vibrant and rich with just a few clicks of your mouse.

And so… You know, I don’t have access to [Polly’s] stats, but I’ll bet if you gave people those two interfaces and saw which one got more uptake among customers, the one that showed them what they could do with the software would do much better. So, one thing that we don’t do enough is customizing users’ trial experience.

Raise your hands if every user who starts your free trial gets — they start at the same point.

Almost everybody, right? Raise your hands if prior to a user starting a free trial, there is some way you can know something about that user. I think that’s true for almost everybody, because the user came from somewhere. They went to their local Googles and they described in excruciating detail what their paying point is. Many people this week are describing to their local Googles that their paying point is that they don’t have Halloween bingo cards. And so, they come to my website, and I know, because Google would tell me, they searched for Halloween bingo cards.

So, if they sign up for a trial, should I give them a blank screen that doesn’t give them any directions? No. I should look: “Hey, you want to make Halloween bingo cards? As a matter of fact, the Bingo Card Creator thing will actually do that. So, I’ll give you easy options for starting that.” And those easy options for starting it increase the conversion to actually using the software among people who have… you know, who I can determine their intent by upwards of 20%. It’s absolutely huge.

You can do that: know what search terms people are using, what advertising criteria do they respond to, what industry did they say they were in, what use case were they looking at on our website? You know, we have all this data sitting in our — sitting on the server somewhere. Use that data to make their lives better, because that’s why you’re here.

Anyone ever use this multibillion dollar software application? It’s World of Warcraft. If you get dropped into World of Warcraft and… I’ve wasted a couple of years of my life on this. I used to be a (?) Guild leader — don’t judge me. [Laughter]

But if you get dropped into it, the first thing they do is put a popup that says, “You right-click on stuff to interact.” And then you close the popup and there’s this guy standing right in front of you with a big exclamation point on his head. And since you’ve only learned, like, one thing you can do in this world, you right-click on him and he says, “Hello. I am the Quest Giver. I am going to teach you about how you play this game to give us money.” [Laughter]

And he says, “Right behind you there is a wolf, you have one button, it’s a Z, kill the wolf with Z, Z, Z, Z, Z, Z!” And you go Z, Z, Z, Z the wolf and it’s like, “Bing! You succeeded in killing the wolf, great! Go back to the Quest Giver.” You go back to the Quest Giver and he says, “Great job on killing the wolf with Z. Now there’s ten wolves, and I’ll give you two buttons: Z and X. Kill the ten wolves and save our planet!”

And you go kill the ten wolves and you come back. And it’s already… it’s teaching you how to interact with the software at the same time while you are feeling, “Wow, I’m really getting success with this, I’m saving the world! I’m about to spend the next 720 hours of my life playing this game.” [Laughter]

You can do this in your software too, right? A guided tour in the middle of the software that shows them, you know, shows them the really fun bits and start — makes them feel like: “Wow, I’m already getting value this quickly!” Rip off this idea. Play World of Warcraft if you need inspiration. But stop after the first five minutes. [Chuckling]

So, a lot of us, you know, we spend so many years building the software, and we know that there’s huge amounts of value hidden in the software but we hide it, you know. They might be a data-import step but you have to really start using the software for months until you start to really feel the productivity gains in your own life.

Many of us have this problem, right? It is? Yeah. So, you typically will not be able to convince your users: “OK, drop whatever you are doing. I want you to change the way you do business for the next few months. Use our software and come back to me and in the next few months, you’ll, like, sort of feel life is better. And you’ll know our software did it.”

That’s a very hard sell. What instead you should do is spoil the good bits, figure out the, you know… You think your software is better. Why? It is still that to one, you know, the cappuccino of goodness and then give that to them right away. Like, I have this site called the “Appointment Reminder,” and it makes appointment reminder phone calls to, say, a doctor’s office to his patients to tell them, “Hey, please come to your appointment at 2 o’clock.”

My value proposition to the user is two-fold: A, I’ll make these reminder calls for you so that you don’t have to do yourself and you don’t have to hire a person to do it; and B, because people get the reminder call, your no-show rates for appointments is going to go down, and that’s going to make you money.

Then those two things like save time, make more money are the generic… You can sell anything to enterprises if you can hit those two bullet points. So… but actually using the software it’s kind of a hard sell because I have to tell people: “OK, first you need to stop whatever you’re doing for scheduling appointments right now, start scheduling them with this software. Several weeks from now when your customers come to their appointments you are going to see that fractionally more of them actually come, and that’s going make you huge amounts of money because fractional — like, fractional, decimal point of anything times cost of healthcare equals, like, a mountain of money.” [Laughter]

So, that’s a hard sell, right? So, instead of doing that I have a quick little demo that I show people. And it’s like as soon as you start using the software I say, “Hey, what’s your phone number?” and people give in. And I give them a call on the phone which demonstrates that I can call phones and everyone… You are using a computer and you call a telephone. People feel that’s a very magical interaction. “What, the computer called me? Hello?” [Laughter]

And a comely lady who goes to some college who I paid five dollars to and recorded this script gives them a sales pitch for the software, basically… in their ear and they listen to the sales pitch because whoa, a computer is talking to them on their telephone. And she says, “Yeah, this is, you know… We’ll schedule appointments for your customers it’s very, very easy, just as easy as putting your phone number was there.” And then we tell them that their appointment is and it says five minutes from now.

So, your fake appointment is five minutes from now. Are you coming to your fake appointment? Press 1. If you can’t come to your fake appointment, press 5. And then they hit 1 and immediately on their screen, it pops up and says, “You confirmed your fake appointment,” and if they cancel, “You canceled your fake appointment,” but that’s not a problem because we could email you right now and say, “Mrs. Smith is not coming to her 2 o’clock, so you can book someone for this and save the revenue for the 2 o’clock slot.” And now, instead of perceiving value six weeks after using the software, they perceive value roughly like one minute and 48 seconds after starting using the software.

And people who go through this process are much, much more likely to actually make the time and investment to start using it. And once they start using it and they see, because, you know, I send them stats every week: “Your prior no-show rate was 28.5%. We’ve got it down to 5%. The actual amount of money we made you is blah.”

People really respond possibly to that. A word of caution: we got into this business to make good software and help people’s lives, right? This whole… the notion of using math, using science to improve our marketing such that we can… such that we can improve our businesses is a powerful tool, but it’s a tool, not a cause.

The kind of pathological end game for, you know, making minor incremental improvements to your business is that you turn into Zynga, where you come up with a business model which is intentionally addictive to people and basically destroy your — their lives to line your own packet.

And none of the people in this room are going to have that problem, but just remember you have the great product or service first, and then you incrementally improve it towards even higher steps of greatness.

I often think that in this software industry we have wonderful, wonderful gifts with the power to change the world and we don’t make great use of them. I mean, Harry Potter would have been a pretty bad book if, you know, Hermione came up with a new spell and was it “expecto,” poking. [Laughter]

And, you know, it would have been a pretty boring book too because the villain would have been: “All right, I’ve got this great, evil plan. I’m going to make everyone cultivate fake mandrake root, and them I’m going to cast a curse of withering on the fake Mandrake root so they have to come back and cast a counter-spell every four hours. If they don’t cast the counter-spell, they have to plant more fake mandrake root.”

Terrible book, right? I’m sorry, I don’t mean to hate on Zynga. It’s just… I don’t hate Zynga, I just love all things that are good and true in the world. [Laughter, clapping] And so, you know, use your enormous powers for good, not evil. So, there’s some more ideas for how you can use your engineers to improve your marketing outcomes on the bingo card, but don’t try to read it because the text is way, way, way too small. You can find it on my blog later. I’m always happy to talk about this or any other subject with pretty much anybody. My contact information’ s up there, I’m all over the internet so if you just post a story on Hacker News, I’ll probably see it. People have done that to get in contact with me before. It’s like, “Ask Paddy blah blah blah.” Email… Even my teachers can understand it — presumably know how to use it but…[Laughter] It will work. Anyhow, thanks very much for being an attentive audience. We have about twelve minutes, and I would be happy to take your questions. [Applause]

Audience: When you’re doing A/B testing on offers, how do you avoid pissing people off by giving them different prices?

Patrick McKenzie: So, this is something that your A/B testing library has already figured out for you. It’s usually going to track people with cookies and say that a particular browser will only ever see one of the offers or the other. But people have multiple devices these day, and sometimes that doesn’t work. So rather than testing prices, you might test, like, subtly different products in that, you know, they’re not seeing the difference between vanilla ice cream at $1 and vanilla ice cream at $2; they’re seeing vanilla ice cream at $1, vanilla ice cream and this little, like, magic sprinkle at $1.25.

And when they see that, because they only get to see one of them, they’ll see: “Oh yeah, it’s $1.25 for the vanilla ice cream offer.”

But if someone says, “Hey, wait. These two are different.” “Oh, yeah, they’re different. We have different product mixes, we test them out.” People very rarely get mad about this. In 5 years I have 200.000 users for the software. Only one person has ever caught on their being an A/B test. And that’s partially because, you know, you implemented it successfully, and partially because people just don’t care. You and your software all the time. It’s one of the most important things to your professional life.

Your users, they have lives that, you know, 23 hours and 57 minutes of the day, are some place much better than your software. They do not notice the differences in it. So, the one person who did notice the difference said, you know, “I paid money for the software and I didn’t get the color feature. And I went over my sister’s house, and I logged in to her free version and she had color. What’s up with that?” And I gave her her money back.

So, total cost of A/B testing for me over five years, 25 bucks. [Laughter] Next to diamonds. Any other questions? Yes?

Audience: Something I’ve struggled with is the first-time use experience, and the… Kind of want you to over-label the software. Like, on a field or whatever, you tend to put extra labels and extra help. They’re really only useful to the user that first time. You know, after that first time, they know what it is, and you don’t need it anymore. Probably get into overlabel for the first time and then you demo it to someone and it looks really complicated because you got so many labels. And kind of… How do you make all that work?

Patrick McKenzie: So, one of the things I like doing, there’s no commandment to like, the engineering books of ethics that says your software has to perform the same every time it’s run. So, one thing you can do, you know if someone is using it for the first time, just put a notification on the top of the screen, or maybe a modal window that says, “Hey, we noticed you’re using this for the first time. We’ve prepared a tour for you. Would you like to take the tour? Yes or no? You can always retake the tour by going to “settings” and clicking the “take the tour” button.” And if they’re in tour mode, you give them the, you know, either the simplified version of the experience, or the “We put labels on everything so that you know what it is” version of the experience. And when they’re not in tour mode, you’ve give what you think as the usual use of the software.

And A, you’ll find that people that do the tour convert much better in most cases, if you do the tour right; and B, that won’t compromise the use of the software when you’re an expert eight years down the line — you already know what all the features do. Does that help? Thanks much.

Audience: You’ve got your feature when somebody’s signing up for a trial. And… Like, for example, you detected that they were looking for Halloween cards. Is there not a chance that the user will feel invaded, like you know what they were looking for? Because sometimes, I think, people have a sense of safety when they’re browsing the Web, they just… nobody knows — they haven’t filled in any forms yet, they’re still safe. But then your website just is interacting with them. Which is cool. For some people it’s probably — it’s quite clever, there’s another percentage of people who react negatively to that kind of attention.

Patrick McKenzie: OK. So the question is, if you’re too clever about the user’s needs, won’t they think you’re spying on them, and won’t they react negatively? And I have multi-part answers to this question. One depends on your users. Engineers are weird, we care about things like that. Real people are less weird in that they don’t care how software works. This is, you know, the notion that someone could see that I was searching for Halloween bingo cards and that could, like, be broadcast to the entire world. Totally not at the top of their list of concerns this day. They want to get back to their kids faster. You provide them with that, they will be totally happy with it. So, know your market on that. If you’re worried about it, test in isolation. Give 10% of your people the more, you know, Facebook-y version of the world, and 90% of the people the traditional “we don’t pretend to know anything about you,” and see which one works better. See if you get any complaints.

I have, you know, over 200.000 people have used this software, several tens of thousands have gone through this interaction with the software and anticipating their needs and I’ve never once had a complaint that said, “Whoa, whoa, whoa, when I went to your website to make the Halloween bingo cards, I don’t want to tell you I was looking for Halloween bingo cards.”

Funny little sidenote on that. So, I did get one customer request support that said, “I couldn’t make the bingo cards. Can you please help me with them?” And I logged in to her account, which was not something she expected me to be able to do, and I’m like, “Yes, was the bingo cards you were having trouble with the… the… the… ‘sexual health for seniors’ bingo cards?” And she says, “Ah, yes.” [Laughter] And I said, “Ah. So, one of the fields… Have you ever had multiple pa… That one was a little too long for the software. If you truncate that one you can get the cards to print out the way you expected.” And she said, “Oh, thanks. And thanks for being so understanding about this.” And I’m like: “[inhales sharply]” [Laughter] “No problem, Ma’am.” But yeah, one awkward experience in five years — not too bad.

Audience: One idea about the Halloween. Maybe you could test, say, “Great for Halloween, Christmas…” No wait… Yeah, I know, but it’s like more generic that… Maybe it would be worth testing to say, “Oh, it works for all sort of holidays, like Halloween,” or, you know, whatever they typed.

Patrick McKenzie: So, thank you for the idea. I’ve tested various variations on that theme. Not to bore you with, like, anecdata from my business. But one the things that I did was saying — the most popular uses of our software are X Y and Z, and customers didn’t really respond to that.

When you put it in a discoverable thing on the left-hand side of the screen and say, “Hey, here are pre-made stuff,” and you populate the pre-made stuff with the ten most popular this, or you can search — people really love that — but when you kind of… you throw ten things to them like, you know, “People really love this for wedding showers, baby showers, Halloween, Thanksgiving, Christmas.” “It’s the middle of October, and I’m not getting married. I just want the Halloween stuff. Give me the Halloween stuff.”

So, if I have good data that says someone is really interested in Halloween this week, then they’re probably interested in Halloween this week. I have a funny story there, but ask me for it later.

Audience: How much instrumentation is appropriate to put within an application to track features usage?
Again, getting back to the creepy aspects. You want to make your users successful; if you see what they’re doing, you can perhaps make them more successful. But I don’t know if you can comment on instrumentation within… Yeah.

Patrick McKenzie: Sure. So, keep in mind — and this is why we had the lovely four-quadrant graph — I sell bingo cards to elementary school teachers. I’m not in a regulated industry, so I can get away with, basically, logging everything you ever do. And if you are, you know, if you have XYZ compliance to worry about, you probably can’t get away with doing that.

But, in general, I log everything and then I expose the data to myself and only myself, where I think it would be appropriate, and I make some data publicly available on my website, because I’ve published stats for years. And that stuff goes through a fairly rigorous… Like, I watch it for anything that could possibly be personally identifying, because I worry about that sometimes. But, you know, in general if it’s just going to be sitting on a spinning platter somewhere, it’s better to have more data than less.

And what you do, like… Customers aren’t too annoyed with data being collected because people… Like, engineers care about that — the real world, not so much. What they’re really annoyed is when it gets exposed in ways they didn’t anticipate. So, you know, make sure that you’re only seeing stuff that you should be seeing and that when you, you know, do sensitive things like logging into someone’s account, you have their permission or you massage that messaging in a way that makes sense to them, like: “Oh, to help you fix that problem, I just took a look at what the computer had recorded about you making those cards, and I saw blah.” Or, you know: “We took a look at the…”

A great tip that I heard last week at Brooklyn Beta: you have exception notifications that get sent to you, right? And you can mail the people who are in charge of the account for that and say, “Hey, the automated system picked up that there was a problem with your account. And because I’m the head of engineering — or the head of the project — I took a look at it and I fixed the problem. And I just wanted to tell you that I fixed the problem.” Your customers will love that. And if you massage that messaging at the front like: “Well, yeah, I, you know… I see where the user experience falls, but it’s not me, it’s the computer.” Make the computer the bad guy? People don’t usually have a problem.

Sure, Peldi?

Peldi: I’d be careful with desktop software. There’s a very different expectation there and it should be an opt-in if you want to send data to the server. It’s the same thing, XXXX but I think XXXX

Patrick McKenzie: Peldi is absolutely right here. Peoples’ user experience expectations for desktop software are a little different. I’d be really careful with desktop software too. If you’re writing it, you should stop. [Laughter, clapping] Sorry. So, there’s a blog post… That’s a pretty controversial thing to say. I have 2000 words on the blog post. Search for “Why I don’t make desktop software anymore,” and it’ll be in like the top five.

But yeah, your life and your customers’ lives will get much better when you move into the SaaS model. The difference between HubSpot and me, they charge money every month. It’s great stuff. And not only will — your life’s better and they’ll be better, but the expectation for everyone using the Internet is that Google Analytics is on like 40% of the pages on the Internet, and that’s OK.

And that everyone has cookies on all the time, and that’s OK. And, yeah, there’s like a weird little microsegment of the population called engineers who actually know what a cookie is and don’t want them on their hard drives. And that’s OK too. They don’t have to pay us money because they never will pay us money anyhow.

Alright. I think we’re just about done. Feel free to come up to me with the rest of the conference or send me an email. Thanks very much for listening.