Patrick McKenzie & Bridget Harris: Lessons in Scaling a Non-Profit Data Brokerage Startup at Breakneck Speed

Patrick (@Patio11 on the internet), discusses how a tweet on 14 January 2021 led to a talented but disparate group of technologists getting together on Discord to start hacking on a ‘weekend project’.

Within 12 hours, data was being shared with the public and, on the same day, vaccine seekers had found locations and booked appointments for their Covid-19 vaccinations.

The weekend project ended up becoming the de facto national vaccine availability database for the U.S. Ultimately, by the time the website was shut down at the beginning of August, it had made a significant impact on the people it served.

Patrick talks with Bridget Harris about how the project started, was organized, iterated, and evolved at a pace that would make a Tesla blush.

From phoning pharmacies to working with consumers, private sector, and government agencies alike to build the best public data set of vaccine availability in the US in less than 3 months – the pace and evolution was startling.

Transcript

Bridget Harris 
I’m also very excited about this session, because up until a couple of weeks ago, I didn’t know Patrick had done this on behalf of many 1000s and millions of people over this year. I think that probably I’m safe and saying that all of us for the last couple of years have felt pain and anguish about wanting to do something to help the efforts of combating this terrible disease.

Patrick I think it’s fair to say, you are a great friend to Business of Software, there’s many people on this call, who know you and are personally delighted to see you and want to hear about your story and the latest updates from Patrick. As Mark said, he is a long standing contributor, not just to Business of Software, but to the Internet web world in general.

I personally, as a software founder used and read all his blogs and his advice very much at the beginning. And in terms of learning, a learning community and a sharing community, Patrick is really one at the forefront of that culture. He has now gone on to do that classic thing, which is to scale and amplify his advice through the power of Stripe and Stripe’s programmes that they run to encourage more businesses to start up and Patrick’s right there as an advocate. So that is Patrick. I hope I haven’t missed anything out.

Patrick McKenzie 
I’ll give the obligatory warning that while I do work at Stripe these days, and the CEO of Vaccinate the States, which we’ll talk about in a few moments. All comments today are in my personal capacity. But it’s great to see everybody again, lots of faces, I recognise in the in the gallery view, I’ve been coming to Business of Software since 2010. So wonderful to have the opportunity to speak with you today on something that is, I think, squarely in the middle of lot  of our interests. 

On one hand, it’s a rollicking good time startup story. It is applying some of the techniques that we’ve talked about over the years to very real problems in the world. The third thing that I love about this community is how sort of optimistic we are about the potential of software and will make a positive impact on the world we live in and knock on wood, we have a fun example of that. So glad to get into the talk here.

Bridget Harris 
I’m very lucky that I get to ask all the questions for the next 15 minutes or so. But I think afterwards, we have a Q&A scheduled alongside the other ones so that if you’ve got some questions you want to ask specifically there’s a special time set aside. But also, if I can cover questions in the chat, I’ll try to as well.  So let me start I’m going to share my screen.

If anyone in California wants to do a civtech project which matters:

1) Find which healthcare providers have inventory
2) Publish #1 constantly

There is no step 3, and a lot of room for different approaches on step 1. https://t.co/VZbPNAe7X6

— Patrick McKenzie (@patio11) January 14, 2021

So I wanted to start with with this because this was actually linked this is linked through to the tweet that might refer to earlier that Patrick tweeted to set things off. And I think you’re right in saying, Patrick, that this is the sort of the story that somebody that this is a very vulnerable population of people trying to get vaccinations at the very beginning of the year, when vaccinations were very early on, and you wanted to sort of pick it up from at this point what what what was happening, and particularly California at this time.

Patrick McKenzie 
Sure. So for those of you who don’t know, me, I’m broadcasting live here from Tokyo, I’ve lived in Japan, for my entire adult life. And like many people, I was following the news with regards to I think one of the singular experiences is this pandemic, as if you’ve had a life that has sort of crossed multiple time zones, then you’ve experienced the pandemic in multiple places at once. So I have my family I was worried about in Chicago, and many professional peers that were living in Silicon Valley and was trying to keep up with them. And of course, my wife and kids and I live here in Tokyo.

And a thing that was coming out of the news coverage in California was that, good news, America had finally started rolling out its vaccination campaign. Bad news, the information environment with respect to receiving the vaccination in California was catastrophic. There were multiple reports in the media of people, largely people who were managing care for others. So perhaps the child of elderly parents, who were sequentially calling 20 or 30, or more health care institutions to try to find one place that had current supply of the vaccine so that they could get their parent in line for a dose which under the state of California is rationing policy, their parent was theoretically eligible for and I was enormously frustrated as a technologist reading this.

Bridget Harris 
At the time even though California had millions of doses, only 37% of the doses it had sent to it were getting out there. So there’s essentially a massive latency in the vaccines that were available versus take up.

Patrick McKenzie 
Yep. And it seems like such a thing where, on the one hand, we, California is blessed by the world’s leading software industry. And on the other hand, it appeared based on the state of play in early January, that we hadn’t thought to put together a website to perhaps make it available. All of the patients needed to call all of the hospitals to get individual updates, which will all sound the same for all patients calling any particular hospital.  So, because when one is frustrated, one immediately goes to Twitter and dumps things out to the internet. I went to Twitter and said, this thing that, you know, some geeks should just step into the void here. And when I said some geek, I I meant basically, anybody but me. But all you need to do is to find the information that everyone is seeking right now publish, it’s constantly be and, and I thought, immediately after tweeting this, well, it’s kind of lame just to tweet this and then walk away having done a classic consultancy move.

Bridget Harris 
So then what happens what happens after this tweet?

Patrick McKenzie 
I tweeted a few more things in the in the thread attempting to inspire people to take up the torch and maybe offer to provide some financial assistance if anyone did. And there started to be a little bit of buzz on it. And a gentleman named Carl Yang who I did not know this at the time, but I was up in Tokyo, Carl Yang was down in Indonesia. He said hey, actually, this is a really good idea. And the minimum step to get started is just start working my network and like pull in people who would be useful here. So Carl, who is a essentially a PM, has worked in growth in the tech industry, etc. Has a lot of friends also living in California, he threw up a Discord server, and  started pinging people and bringing them in engineers, etc, that he knew from the community.  Discord for those of you who are not familiar with this, it’s like Slack, except for video gamers. Which we got immense joy of discord remained our primary organisational technology for the entire run of Vaccinate the States. It was a great fun trying to explain that to folks in like the healthcare industry or government who were, you know, just getting onto the slack wave and not yet as seen what we use to kill dragons. More metaphorical dragons. Or actually I suppose third less metaphorical but more virtual dragons.

Bridget Harris 
There were other people out there as well. So you had this person Cary Craver, she was doing something for Texas. So essentially, a whole lot of people jumped on that thread, saying, ‘Yeah’, ‘we can do this’, ‘We’ve got the tools’.

What was the most efficient way to get the information into the system?

Patrick McKenzie 
A few people in a few places in the United States are working on it on there, for their local context. Carl roped together about 10 for organisers to start hacking for California. It was about, let’s say, the time zones will throw everything off here. So the official timezone for the rest of this presentation is California time. It was late night, say 8pm or so California time when everyone piled into the discord, and started hacking with the desire to get something out early on Thursday morning.

I jumped into the discord and thought, oh, I hadn’t haven’t been alerted to this on Twitter, and thought, oh, you know, it’s great that people are doing this wonderful thing. Clearly, I will not be involved in any meaningful way. But I can at least like tweet out when they launch something. And so people were already like, rapid iteration on code was happening, and people were discussing the strategy for getting information into the system. I was pretty sure at the point where I can post the tweet that, like the natural tech way to do this is like build a website and either scrape the information or have some sort of crowdsourcing system to allow users to report information of where they found the vaccine. And I thought that is probably not the way to do it. The most efficient way to do it is to actually run an internet scale call centre. And so while we were having those design discussions, I was advocating for a call centre and they said, Why do you care about call centres so much? I said, because I paid for college working, working in one, and trust me, this will be more effective.

We worked through the night tacked up on things and had basically the MVP ready with no data in it when pharmacies open for business the next morning in California at about 9:30am. At about 9:31am I made what I think was the second call for organisation. For those of you familiar with the Lean model, you sort of like attack that. The next thing that you don’t know that could throw off the entire project, and at 9:31am, the thing that we didn’t know was, is it possible from outside of the formal healthcare system with no authority, etc? Can you get information from it? And so I called on my cell phone from Tokyo into the one pharmacy whose phone number I knew in California, which happens to be a Walgreens in San Francisco. I called them and said, “Can I speak to the pharmacist?” I got transferred to the pharmacist! I said, “so if I had a 65 year old who needs the COVID vaccine, could they get that from you? And if so, how?” And they said, “Oh, bad news. I don’t have supplies yet of the COVID vaccine, we will publish on our website, once that is ready, or you can keep calling. And I think the best place that you could potentially find in San Francisco right now is blah.” I said, “thank you very much. That’s super helpful”, went back to discord and said, “Good news, guys. You don’t need any credentials at all to pull supply information out of the United States healthcare infrastructure, the pharmacist will just tell anybody over the phone.” So like, we have a chance.

Humans and Tech

Bridget Harris 
So this is one of the interesting things, I think we’re going to be talking about Patrick, which given the fact that our whole audiences are software geeks, part of the secret sauce of what you did was human powered. It was human intelligence. It was the simplest, most agile thing to do, which is just pick up the phone and and call. But one of the things that really struck me is don’t forget, we’re actually still two days before Biden gets sworn in. There’s still quite a lot of uncertainty about how the vaccination programme is going to roll out across across the state. As you said, you tweeted on the 14th/15th, you launch something. And you are up by the 19th. I saw a doctor on Twitter the other weekend, she had a 17 tweet storm,  annoyed at how hard it was to find a vaccination availability for her parents. So there she is, she’s a doctor, she’s in between, you know, running her day job, obviously right in the middle of the crisis. And she she can’t quite believe that she can’t get an appointment for her parents. And then there’s Vaccinate the States. Your you know, your first early couple of you don’t have been doing this for five days. And It’s already ricocheting around the internet, or at least around people in California looking for it. Have you ever started a startup and had that much impact so quickly?

Patrick McKenzie 
The timescale for a lot of this blows my mind. And I was thinking when I was reviewing sort of primary source documents to refresh my mind on the timeline. It feels like sort of a dream. Did we actually do that? The timestamps seem to say, yes. We went live on that first day, on the Thursday am we had our first confirmed report of someone getting vaccinations scheduled for their parents that afternoon at about 3pm, California time. Then we had the report of that user that their parents were successfully vaccinated on day five. So that was the first like end of the loop where an actual health care outcome was improved. And they said they cried at the vaccination.

Bridget Harris 
Actually, this is a thing that we’ve all now as adults going through this will remember every day matters. And so you have a statistic that you’ve learned about what how much it matters for what’s one day mean. Don’t explain that so we can understand the impact.

Patrick McKenzie 
So this is I think, the most important number that you can hold in your head for at least the stage of the pandemic that we were at in January slash February of 2021. In the United States. To an approximation based on the current status of the pandemic, who is getting the vaccines and the relative risk of severe health consequences if they got COVID, pulling vaccines 10,000 dose days into the future saves one life and expectation. So for example, if you have 500 people, and you pull the vaccine forward by three weeks, that’s the over 10,000 so you have saved more than one life of expectation.

Bridget Harris 
So it’s a race against that initial growth?

Patrick McKenzie 
There are two ways to get COVID antibody is one way is by the vaccine, the other way is by a potentially lethal disease. Everyone in society was in a race against that disease. And this became sort of our mantra and internally everyday matters. Every dose matters. What can we do to increase the urgency increase the speed and and solve the bottlenecks that we are seeing in the national rollout of this?

Bridget Harris 
You mentioned a few people, obviously, because I know you led the project, but there’s a lot of people that kind of came together. And forgive me if, if you watch this, and you’re not, I haven’t screenshotted. Forgive me. I’ve done this not Patrick. So I’m not deliberately excluding anybody. But these are obviously some of the people who threw their name into the hat at the beginning. Did you know them all? Or did you just did they just all find it? Did you all find each other?

Patrick McKenzie 
I literally knew nobody. In this project until we started working together. It was a quote unquote, randos on Twitter. I think Carl knew a lot of the the core organisers of which we had about 10. On day one, we eventually had about 500 volunteers at various levels of involvement in the project, a team of about 10 people who were working as employees and other 10 folks that were working as contractors doing phone calls for us. And then depending on the the exact time in the project between five and 25 folks working in a contracted call centre.

Bridget Harris 
So what was amazing, as well as the speed of obviously having to move faster, literally get jobs, jobs booked and parents into centres being vaccinated is how quickly you did the organisational capacity. So you took all of these people you didn’t know them, everybody’s on LinkedIn and Twitter and finding each other. And you turned it into a staffed operation, which is pretty impressive.

Patrick McKenzie 
It felt enormously fortunate as the CEO of this thing to have a team of massively competent individuals just like show up out of the ether and say, Okay, I’m here, what can I help on? I still almost don’t believe it was possible. But we had the the right set of people, and we started working on networks in California and elsewhere, it was truly a global effort in a lot of ways and saying, Hey, can you, you know, put your life on hold for a couple of months? Or in the alternative? Can you even just give us an hour and help us out on this one specific thing that we need right now? And folks were extraordinary, extraordinarily generous with their time and talents.

Bridget Harris 
At this stage, you decided that you had to relocate west coast because you couldn’t do this all from from here?

Patrick McKenzie 
Yeah. So there were a variety of reasons. I was speaking to one executive about this. And they said, You know, I think I’m a little worried for you, you seem to have been working basically the entire clock, because Tokyo and California have sort of punishing, work hour overlap. And I said, Yes, but I have a solution to that I’m hopping on a plane in in two days. And we’ll be running this from San Francisco for the duration. And they said, Wait, what? And so is explaining the logic like we had, we had a breakthrough recently, it’s clear, this is going somewhere. So I need to be there. And I will get there. And they said, so. So where are you now? And I said, Oh, I’m an Airbnb on the way to, on the way to get my test results to get to the to get to the airport. And then after I get to San Francisco, I will probably have to, you know, buy clothes because I have none on me right now. I literally got to America with just my laptop,

Bridget Harris 
Tell me a bit more about the volunteers. Because again, we haven’t been you said that there was 400/500 volunteers, but you obviously tapped into a huge amount of willingness of people to get involved. And how on earth did you coordinate 500 people? What were what were they will do?

Patrick McKenzie 
Oh goodness. So we the the core of the operation for most of our life was twofold.  One, we were thoroughly a software organisation and a lot of geeks organised and discord in the early days. Many of our volunteers were sort of recruited through either our personal networks or me posting on Twitter. And so we have a lot of engineers actually, you know, just being in phones rather than writing. Writing the the code, as we progressed, became more of a sort of engineering on one side and operations. On the other we’re operations main job for the vast majority of our lifecycle was, like, make sure we have good information that we can expose to people that want to get health care. So they were making phone calls, they were doing data entry, they were, we had this thing called Web bankers in in comparison to phone bankers were they were sort of like screen scraping with their eyeballs. Every Facebook page from a county health department in California. This fire department has tweeted out that they have 10 doses, they’re expiring by the end of the day, yada yada, feed that all into a system and figured out how and then the engineers figured out okay, how do we get structured data out of that and then put it on websites such that people can find the can find the information.

Bridget Harris 
We’re going to dive into more of that, you know, in a bit but just to, again, to emphasise the sheer speed with which you, you and your team worked day one, you, you launched the website, day 104, you’d managed to find 5000 different locations that were that were informing you about vaccination availability. I’m sorry if people can’t see this on the screen exactly. But it’s just talking about exactly how fast you moved. When you have to, you know, in a public health crisis, when you have to move, in some ways, in some ways with where things are easier when everybody’s on the same page, because everybody’s got the same goal.

The North Star Metric

Patrick McKenzie 
It was a very mission driven organisation, and our Northstar metric, which we repeated incessantly was shots in arms. I’d had a bit of personal reading about the progress of the pandemic, at many institutions, a lot, were still institutionally taking weekends off, taking holidays off, etc. And I thought, it’s important for people to have some bit of downtime. But like, the virus never takes a holiday. And so if you’re doing pandemic response, your organisation almost definitionally has to never sleep. And so, we got a heck of a lot done in these first 100 days. By total coincidence, that was largely dictated by external constraints, rather than our engineering schedule we ended up on day 100, going from Vaccinate CA to Vaccinate the States and broadening our efforts to the entirety of the United States of America.

Bridget Harris 
I wanted to talk more about that sort of that call centre operation that you put into, into place. But I think at this stage, we will have to just give him and his team round of applause for doing this within, you know, four months.

Patrick McKenzie 
The important thing is not ultimately the it’s a number of pins that we were able to put on the map, although we were very happy to be able to put up some of the 1000 pins on the map. The really important thing is that it ultimately impacted healthcare outcomes for people who are seeking vaccine. This gets into a little woolly to talk about the impact, because I think that we discovered pretty early in the lifecycle was that since we had an incredibly aggressive timeline, we were not going to be able to both become sort of the somewhat canonical source of vaccine location information, and also build a customer brands such that hundreds of millions of Americans, Americans were able to use us. By the midpoint in the project, we were largely working with partners that already had the reach to Americans to publish the information to them. So some of those publishers are things like County Health Department’s, some of them are some of the largest publishers in the world, like, say, Google. And most of our impact happened through those publishers. So now I get to work through some kind of woolly mammoth. Google for understandable business reasons will not tell us how many people use the Google search engine on any given Monday. Sad but true. Or what their query volumes were for particular queries. But we have reason to suspect that many millions of people were able to successfully get Vaccine Information from us. And based on sort of our anecdotal understanding from talking to users, and our metrics, while we didn’t have patient identifiable information anywhere, we think probably about 20% of the people who were exposed to our information would have successfully been able to connect to the vaccine as a result of that. And so when, when we do the math, we think, Okay, we’ve probably got millions of vaccines delivered at the margin, exactly how many millions is, is a little up in the air, and then you run it through that, okay, make some assumptions about how much that accelerated it versus the counterfactual where we didn’t exist, probably saved 1000s of lives and expectations. But again, difficult to know, since at the end of the day, we’re not Google etc.

Bridget Harris 
The other thing to remember, of course, is that the administration throughout these months, were making constant announcements. I mean, they were bringing forward their milestones of what they wanted to achieve in terms of every adult American was going to have access to a vaccine by the end of July. Then it was by the end of May. Then it was by the end of April. So presumably you had a lot of momentum behind you as well, because all these vaccination centres are being told they need to vaccinate people, and it’s connecting the people who need the Jags with the people who’ve got the vaccine?

Startup working with government

Patrick McKenzie 
There were some there, pluses and and minuses and working in a, the kind of project where things that you are directly working with could be mentioned by the President and to the Nation Address anytime. If your familiar with the stone soup thing where somebody brings the carrot, somebody brings the rice and somebody brings the pees and together a wonderful soup emerges. We had our little part of a very large national response effort. And parts of that were, in some ways complicated by the political realities of the situation. The fact that, for those of you who aren’t aware of the pandemic response in the United States, was done by over 3000 governments in parallel, the federal government, the state governments in each county government. Responsibility and accountability was more diffuse than it was in some nations. And so, this just brought some challenges. And it’s also great to hear as a product owner, your roadmap entirely changed because the President has made a speech. But we we tried to move as fast as we could and roll with the punches.

Bridget Harris 
So let’s start with that then Patrick. As you just said, you’re you’re used to private enterprise, sort of software outside of the usual zone of federal state, public sector bureaucracy. And as far as I’m aware, without sounding without teasing you too much, your previous leadership experience had been as a WoW, World of Warcraft guild leader. So this is your kind of your current, you know, Team Lead experience of managing people, which is quite geeky, obviously, quite based around gaming. And you said yourself that you’re using discord as a way of coordinating it. So you’re bringing to this what’s essentially a very, you know, sort of emergency style public health and it’s a public health emergency, there’s no room for disruption or messing about is it you just got to get the job done? What were the advantages that you felt you you and your team brought to your ability to move because you were using coming from a different organisational culture?

Patrick McKenzie 
So one advantage is that we were steeped in the software slash startup culture of moving very quickly. And although this seems vacuous, isn’t when you work in the government, we can successfully write software, the particularly in the wake of the healthcare.gov rollout a couple of years ago, in the United States, there was a bit of a once burned, twice shy feeling within certain powers of corridors of power. The federal government should just stay out of the software game, the risk of like putting website out there and having it blow up on day one is just too high. And so the government made some choices which were correct along some axis of axis of not exposing them to a political risk, but probably incorrect among some axis like, get something in the hands of patients and people who are assisting patients so they can get accurate, accurate information as quickly as possible.  We didn’t have those sort of institutional constraints, we also didn’t have institutional constraints, like we need to run a competitive bid process and, you know, follow 300 pages of regulation, to be able to ship a change to the website. The for those of you who’ve worked in government contracting before, think there are probably relatively few people on this call who have a positive opinion of government bid processes. We did not get FedRAMP certification, etc, etc. To counteract an unfortunately very common narrative in tech guys, there’s some things that government and the rest of civil society very, very much does bring to the table. The vaccines for one, whoo! If we had not as a society successfully created a working and safe Coronavirus vaccine within two days of sequencing the genome of the virus. I don’t think we appreciate it enough how much it’s like a mind blowing, human history scale scientific achievement that is and and getting the vaccine procured and produced in quantity, etc. But then connecting people to it is difficult.

Bridget Harris 
Let’s get into the nitty gritty of that because this is this is the real stop and think about this. I’ve looked online and there’s loads of really quite painful efforts to try to get availability of vaccine uploaded by CSV and oh my god logins and email pingbacks etc. And what you did was you set up an operation where you just literally phoned round. And you were just doing a human powered ring round and web scraping. So tell us a bit about the tech on on all of that. How did you do it all?

Patrick McKenzie 
Yep. So we tried to avoid the, you know, let’s spend the first six weeks of this effort building out a wonderful technology and we’ll get Kubernetes spun up and etc, etc.  We were laser focused on getting value onto the internet as quickly as possible. So the early life this, we ran everything out of airtable, which props to airtable. I don’t know if Vaccinate CA would have gone the distance without it. We had static sites built in Jekyll that on day one, just published to Google Sites We eventually did something differently. We were using Google Maps as our core mapping engine, which pretty core to the use case for the situs, we want to show you a map with pins near you and the sort of metadata about those pins. So you can find, given your personal situation, which location near you that has the vaccine is one that will actually give it to you, and how do you get access to it there.  We eventually did hit scaling limitations on airtable. Go figure. We eventually had a more standard Postgres. Let’s see Django as the web as the application Postgres on the back end for running the killer app where most of our value is provided. Still, to do a static site to the date, the project ended, it was just a static site with a flat HTML and flat JSON results getting served up with JavaScript magic. But we switched to mapbox as our mapping provider, we ended up making a heck of a lot of mapping calls, as you might imagine that they were, they were quite generous with regards to giving this nonprofit organisation nobody had heard of a lot of their time and talents to get that to work.

Bridget Harris 
And what what barriers, did you find that in a way that you kind of problems that you’ve you solved along the way that you got better at and what were the unexpected things that that that you didn’t, you didn’t because of the way you’re choosing to do it, you uncovered problems?

Patrick McKenzie 
Oh, goodness. So many learnings. And I could, I could talk for hours on all of the the fun micro infrastructure of the United States healthcare, IT departments.  One thing that we discovered very early, literally on day one was that we, we had to instruct our callers to ignore lies. And the amount of lies in the information ecosystem was much higher than you could then you would naively imagine, or at least that I naively imagined. For example, corporate, for a lot of large pharmacy chains, had a thing where if you call the pharmacy, a recorded voice would tell you, if you’re calling about the COVID vaccine, the pharmacist doesn’t have any information to get to give you up can check it up on our website. The websites had no useful information on them. We would tell our callers, okay, so recorded voice is telling you that the recorded voice is lying to you. Wait until you hear the prompt for speaking to the pharmacist and then speak to the pharmacist. And I think that we hammered into people over and over again, was if a website says that they don’t have the vaccine, and the person that can legally put the shot into your hand says that they do. The pharmacist is right. The ground truth is the only truth that matters. Let’s get the ground truth to our users.  Other other fun things that we learned, we had a sharply limited number of calls to make per day. A reason that we cottoned on to this call engine is the thing that will lead us to success and that many other initiatives did not was that we were willing to do that thing that didn’t scale and pick up all the all the phones, because we knew, like how many hospitals can there possibly be in San Francisco? It’s probably like 20. How many pharmacies can there be in the state of California? And on day one before we actually like looked at that number and like 5000 plus or minus? Turns out it’s about 6000 plus or minus good enough for government work? But we had only 500 calls that we can make a day against 6000 pharmacies.  And so how do we prioritise those calls in the early days of the effort?  The thing that we did was we had a call Captain role, which was partly in charge of like doing rah rah for the callers and saying, Okay, we found our first vaccine of the day, that’s awesome. Let’s keep getting those phone phones people worked in high pressure sales environment. We were a high pressure sales environment except trying to give something away for free So the the other thing that call captain was in charge of doing was reading the results as they came back in and retargeting our calls based on it. And so one day we realised this is when we had about a 4% success rate and get in getting the vaccine. So if we called 500 places that that day, we would add an expectation have only only find 20 places that had administrable doses of vaccine. Within a 10 minute span, we got three safe ways that said, Yes, we had the vaccine. And we said, well, we’re not medical experts here, and we’re not logistical experts here. But that seems statistically unlikely, seems to this geek that what is likely happening is that Safeway has a unified infrastructure backing it for getting this vaccine to them. And that that infrastructure has likely gotten the vaccine, it is sending it out to safe ways. So Safeways are much more likely as of today to have vaccines than any other pharmacy in California, let’s call every Safeway in the state of California. And so we sort of dynamically retargeted our calling efforts to do a survey of safe ways which we got done in the next two hours and found dozens of locations that had vaccine.

Empowering volunteers

Bridget Harris 
When you said retarget, can I just ask you like literally how are you doing that? Mary Lauran asked, What was your limit of 500 calls a day? How are your people who are making the phone calls? What were they seeing literally what are they looking at in front of their screen? How are they inputting the information? Literally what they were looking at?

Patrick McKenzie 
It was an air table app in those days, which would would sort of walk people through the call script. And so you say, you know, click here, and we would say, Okay, the next place in the queue is the Walgreens at, you know, Walgreens number 563. Here’s their phone number, here’s their address. And then it was like make this call or skip to next.  Because sometimes you can tell from the name that it wasn’t going to be helpful, you would make the call and a screen would would pop up with the script in line and it would say, Okay, you’re calling Walgreens, you hit four immediately to speak to the pharmacist. And then then you just start reading from the script. And the pharmacist would give you, you know, humans talking to humans information. And the script would attempt to turn that into structured data. So like click this button, if they say yes, they haven’t clicked this button, if appointments are required, click this button if appointments are required through their website, type in the URL here, etc. And so that interfaces with caller spent most almost all of their time working with.  But an interesting thing about this was that in the course of building this machine that was making hundreds of calls a day and having humans actually reading the structured information as they came back and having our, this is critical. Our callers who in many organisations, people who operate phones are not institutionally trusted with authority to talk to the people who are like building the engineering systems. We had everybody in the same discord from day one. Our callers would would feel some initiative with respect to like feedback into us things they were learning, or deviating from the script wildly. For example, one caller was calling a particular pharmacy in a particular county in California. And they were going through the script. Oh, you do have the vaccine? That’s great. How do I get it? Oh, I need an appointment. Where do I make that? And the pharmacist said, Oh, you make it up the county’s website at WWW dot blah. And the caller said, Sir, I’ve been on the county website several times today. And your pharmacy is not listed on the website. And the pharmacist started swearing a blue streak. And after the pharmacist is connected, he called up the county health department to figure out why isn’t my pharmacy listed on the website, because if they aren’t, if we’re not on that website, we literally cannot get people coming in asking for the vaccine. And it turned out that the county had hired a consulting company to build them a COVID portal. One thing that they need to do was go to all the pharmacy chains and get a CSV file from them describing all of their locations and phone numbers and daily dosage bandwidth to upload to the to the portal, one of those CSV files got dropped on the floor because as you can imagine, these are like getting emailed back and forth, yada yada. That CSV file had 11 locations for pharmacy, each of which had between 50 and 100 doses of the vaccine sitting in the freezer, and nobody had noticed. And so, when we were when we were the pharmacists were able to tell the county Hey, you know, perhaps we should be getting appointments. They had 600 appointments come online within their system within minutes. And all of those were immediately snapped up and 600 people got dosed within next presumably 48 ours. And the that was one of the key turning points on Oh dear, we should be intensifying separate because we, we seem to have created a software delivery COVID vaccine and have manufactured the first 600 doses of it. Clearly we should be scaling lists to find these other infelicities within the system and report them to people who are capable and empowered to change them. So we weren’t just helping people to get access to information that existed somewhere, but rather like changing the state of play for them.

Bridget Harris 
Kind of just going from the original, you know, quite obviously very, very urgent and instant startup literally start up from nowhere. In the first couple of weeks you then professionalised quite a bit. So you did have some people who were taken on and sort of staffed, you raised over 1.1 million is that right? So I don’t know if you can reveal who was funding you? Can you raise you did you personally organise a fundraising effort to help pay for that stuff, and at what point did you professionalise into professional call centre operators. FUNDING

Patrick McKenzie 
So as you might imagine, for a project that literally started with a bunch of geeks tweeting at each other, we thought this was going to be like a quick hack on things that we’ve worked over on the weekend and did end up taking over the legs for a couple months. On day eight, I went to the team, to this point that sort of like prevailing norms of the project were like an open source project, we’re all committed here, there is no hierarchy, yada, yada, went to the team on date and said, there is some chance. And it’s very uncertain, but this might be the most important thing that any of us work on in the next couple of years. If that is true, we we need to be radically better staffed than we are right now we need to be able to make phone calls consistently. Rather than sort of being dependent on our volunteers coming in every day to do it. We need to have some sort of, like organisation that civil society can interact with, because, and this sounds like crazy talk to everybody on date said there is, there is some level of results that we can generate that will get us into the White House if we need to get into the White House to get to impact the pandemic as much as we possibly can. But we probably can’t do that without an org and the leadership structure and the CEO, etc. That’s going to require money, I suggest you make me the CEO, I will go on a rapid fundraising effort and raise the valleys quick, a seed round, and tell investors basically that I’m going to squander their money as quickly as possible while driving my total addressable market to zero. So, some startups fail due to bad luck. And some startups fail due to inaction. I was I was telling people, I intend to fail very quickly, at least for the typical metrics of startup success. So we, we raised money from a number of places in tech, Stripe generously gave me some time off to work on this and supported us in other ways. I probably can’t run through the major donors at the moment, but wonderful thing of 501 C, three, C 3501. C, three charities, as you’ll be able to read it on the public documents next year anyhow. And we ran the project through from January through orderly shutdown in early August, using that money.

The end game

Bridget Harris 
How did how did you merge into the the nationwide effort?

Patrick McKenzie 
We worked closely with the National blessed effort, which was run out of the Boston Children’s Hospital, eventually called vaccines.gov. The pluses and minuses of doing things with the government to pluses and minuses of the government’s decision making processes. At some point in the pandemic they said, we don’t think we will be able to staff the digital side of this. We need to find a partner with appropriate gravitas who has people working on the ground already, and Boston Children’s Hospital had been running a vaccination promotion site for a period of 10 years, including times back when my major professional focus was selling bingo cards to elementary school teachers. So props to Boston Children’s Hospital for having staff organisation. But they were there.  They got told by the powers that be Congratulations, you are the American national effort for giving this information to the country. It was a little rough relative to their resourcing available to do that. We were better resourced along some dimensions. And so we’re able to, to help them with things like, for example, making calls. They had a federal mandate for the CIF. I remember the name of it, federal retail pharmacy programme. So the Walgreens and CSPs that were receiving doses from the federal government were required to report the information into a data lake. Learned that were this year, goodness! They would report that information to a data lake, which would eventually bubble up to vaccines.gov. But so data was getting put in the data lake, but nobody had any clue whether it was accurate or not. And due to the way that the organisational boundaries broke down, nobody was resourced to actually check. So we were talking with them, they said, could you just call these people who are, you know, in our data lake and like, check that these submissions that we’re getting are accurate. And so we were able to help them spot check their information, get the information that was on vaccines.gov, up to maybe a bit of a better standard than it was when we started. And also sort of like, pull subsets of their information in so that we could target our calls more efficiently.  So we had, on the one hand, this sort of federal formal, legally mandated excetera effort that was getting some subset of information pushed in from the private sector that was actually administering vaccines, private and public sector, well, mostly private in that programme. And then we were conflating that with our phone calls with the web bank or operation that was getting information from counties, Facebook pages, with reports from the media, etc, putting into one place, and then pushing it out to publishers like Google, like the county health departments. Interesting thing about government contracting, the Alameida County, which is a county in California, got in touch with us and said, Hey, you seem to be bright technologists who are working on this thing. We’re putting together an RFP for a consultancy, where we want to add an interactive map to our website that will show people where to get the vaccine close to them. And we said, interactive math on where to get the vaccine, we could do that for you. And they said, Wait, what does that even mean? Let’s have a call to discuss this at 11am. Tomorrow, we came to them at 11am with here is a URL of some JavaScript just embedded in your web page, boom, you’re done. And they’re like, Oh, that was the best RFP process ever. So we eventually got that deployed into county surfing 10s of millions of people in the state of California, for nothing. And knowing what to know about contracting, the cost of getting that feature done for one county website would have exceeded the entire cost to to run our operation for the duration of pandemic, even with no data at all, just the just the software writing itself.

Bridget Harris 
I mean, speaking, as somebody literally does run an appointment software business, we have a we have an internal expression to appointment management that people want when they come to us looking for solutions, which is Cat herding. And it’s only so far that software is going to fix a very, you know, easy, easy to fix problem. beyond it. If somebody’s going, No, I want them to be over there and with this group, and they’re not on Tuesdays, but you know, available in the middle of the night, whatever. And then we don’t know which room we’re gonna book at the same time, you know, that is Cat herding. And I love the fact that you just cut to the chase in terms of call pharmacies, because that’s, they’re the ones who are going to know.

Patrick McKenzie 
A blessing and a curse. We punted on the actual like scheduling problem, because we didn’t think we would do it in the time that was required. One of the things that hopefully the United States will learn from this effort is that having several 1000 independent systems for scheduling appointments for this national emergency did not optimise for  actually getting an appointment. But we were just pointing people at the front end where they could where they could schedule and then so they knew it was right. Because you’re saving people is much more likely to be worth it than if they were just Yeah, grabbing their closest pharmacy or health care provider.

Bridget Harris 
Yeah, what you’ve got is 100 people every hour making the same 100 calls to the same pharmacies trying to find it whereas you’re making one cool finding out and then they then they know that they can, they can call and make the appointment. Soyou come, Patrick, you’ve got you’ve had sort of years and years and years of working of being immersed inside the entrepreneurial startup private sector world, obviously making money losing money. You know, the whole the whole world that we that we inhabit? What does it feel like for a short period of time to actually be in charge of a public sector project? What were your takeaways?  What do you think that sector learned from you and your discord guild? And what did what did you learn from them?

What startups and governments can learn from each other

Patrick McKenzie 
I would like to think that. Hmmm. I think that both sides have a lot to teach each other. I think whether the lessons were learned, it’s certainly work in progress for everyone. When we were doing our due diligence calls in day one, I got told very firmly by someone in public health and in California, don’t! I’ve seen this movie before, and it’s about tech bros who know nothing about health, you’re going to run a website and it will be only in English. You will forget to localise it into languages that are spoken by less Californians. And I said, with respect, I appreciate as someone who lives in Japan, that not everyone in the world speaks English. But that’s a reason to localise the website as quickly as possible. That’s not a reason to not do anything.  That bias towards action and getting incremental quantities and incremental quanta of value up on the internet is something that I think is powerful in the startup mindset. I think people in tech underestimate the amount of resources available to the United States federal government, when the United States federal government decides it really, really wants something. And the United States government well ahead many challenges over the course of the last two years, really, really wanted to get several 100 million doses of the Coronavirus vaccine manufactured and distributed across the country. And that did happen. So props.  There are many counterfactuals of how this code could have gone better how it could have gotten worse, but we are in a much better position than many people, myself included, would have expected us to be in the early days of the pandemic. Again, can’t say enough good things about the the scientists, the pharmaceutical infrastructure, etc, etc. That got this virtually miraculous drug done in literally days. Unfortunately, languished for a very long time in regulatory approvals, etc, etc. That is another topic entirely. But perhaps you want to get that from someone who actually does have a doctorate in something other than slaying dragons.

Bridget Harris 
So where are they all now? All your team? Where’s everybody going back to? Where does the project you explain now it’s all gone to vaccinate.gov? That’s, you know, essentially there’s a some of the heat of the of the fire has gone out a little bit because obviously, thankfully, a lot of people now are vaccinated or apparently have their jobs. Where’s everybody gone now? And are you back to your day job?

Patrick McKenzie 
I have gone back to my day job at Stripe. Many of our team members went on to their next adventure. I’ve been writing a lot of recommendation letters. We’re all keeping in touch in case, God forbid, we we ever need to do this thing again, hoping not to but you know. One learning from it was that the value of having people who are heavily networked in both technology places for getting things done very, very quickly by working around the formal processes for getting things done was astoundingly high. Keeping that network alive is something cross fingers I hope to achieve over the next couple of years.  The pandemic isn’t over, not in the United States and that broadly for humanity. It’s it remains critically important that we continue attacking it with focus and urgency every day. The current focuses on the demand side of things, which was something we were we were largely not working in were our we didn’t have the right skill sets and platform to contribute as materially to that as we did on the supply information, gap class solutions.  Oh, I see a comment in the chat. And it’s a great one. On Twitter. You mentioned your org received valuable contributions from surprisingly young folks. Can you tell the group more about that? Yes. The most important person to the Vaccine Information Infrastructure in Illinois was literally 13 years old. We found that out several weeks into the project, after they had been scraping from a variety of sites across Illinois and feeding that information to us, so that we could more quickly call the right places in Illinois to verify the information. So we had a quick chat with their parents, and then continued working with them, obviously, because that is the right thing to do, given that they’re saving lots of lives to the pandemic. So, at the end, when I recounted the story, somebody reminded me, you know, we had 2 x 13 year olds, right? And I’m like, oh, goodness, I didn’t even realise that. There were a lot of people from a lot of different walks of life, in technology outside of technology, across the United States and across the world, who contributed to this and I’m enormously thankful to all of them for selflessly helping as many people as possible to get vaccinated as quickly as possible.

Bridget Harris 
Well, again, we commend you, Patrick. And I feel like I am sure everybody in BoS feels like one of our own was out there doing something really, really significant and impactful for a during a period of time when a lot of us has been feeling pretty, pretty useless about what what we can do to help. As you said at the beginning, the effect of bringing back those that that horrific ratio of who how many people die, based on the absence of being vaccinated, you brought forward that number so that 1000s of people got vaccinated before they got COVID. And saved their lives as a result. And that’s, that’s something for you and the rest of the team involved to be extremely and proud of for the rest of your lives.

Patrick McKenzie 
If I can give one little addendum to that, the this project wouldn’t have been possible, not just without the people who were sort of in the room doing it, but without the technology that the technology industry has built the last couple of years. I was telling the folks that Zapier and Zapier was one of the groups that that generously supported us. I said, you might not appreciate this. But in addition to the financial support that Zapier gave, the actual Zapier product solved really specific pain point for us in the first couple of weeks of the rollout of this. I can tell folks that in the q&a if you want. I said, you might not have felt when you were going to go into work in January and February that you personally were doing something to, to combat the spread of the pandemic. If you were a system reliability engineer at Zapier, you were because you were enabling us to be maximally effective with regards to our small part of the puzzle. For folks who feel like your only contribution has been been showing up to work for the last 18 months and doing the thing that you do every day, showing up to work and doing the thing that you do does help people in the tech industry. Maximising for the impact of that seems to be a good thing to devote the rest of our careers to.

Bridget Harris 
A fine way to end I can keep asking Patrick questions all day on this but let’s open this up.

Audience Questions

Audience Member
Yeah. So I’m really curious. It was really interesting to hear, Patrick, what you your thoughts on what tech can learn from government and what government can learn from Tech?

I think often there’s a lot of focus on what government can learn from tech and not so much the other way around. It worked in in nonprofits, and I used to live in DC and was like very proximate to government stuff, had a lot of respect for government also understood how a lot of things could be better. And before I got into tech, I was kind of thinking about what was next and talked to my, my older brother who’s like, been in tech for his whole career and I told him, I was considering different government positions. And he was like, I don’t want to work in government. I was like, why not? And he was like, you know, like, Bridge to nowhere, which is like such a trope that has been mobilised by like certain folks who have certain interests about a story of what government is. And I’m curious from so I have often gotten the sense that just like Silicon Valley has so much more to understand and could have a much better impact in the world really by like understanding government and, and knowing how to collaborate better with government, but without really having a specific sense of how that is. So I’m curious as from your perspective, as someone who has really worked at this intersection in a very tangible way on this one specific topic, like, Could you say more about what you think tech could benefit from understanding and what benefits that could potentially have if that bridge, where if that gap were more bridged?

Patrick McKenzie 
Oh, many, many things that one could go into here. One that was particularly salient in my experience, as you mentioned, there’s a sort of perceived antagonism between tech and government and perhaps a perceived antagonistic triangle between between tech media and government. And we were very careful with managing each leg of that stool while we were doing Vaccinate CA and Vaccinate the States.  The was a tragic misalignment, particularly early in this year between the perceived political realities both within government and within the halls of power in tech and tech government collaborations in specific. I don’t want to bring up any potentially controversial things. We we were a strictly apolitical organisation both for the usual 501 C reasons and , it’s impossible to get anything done in the United States if you were antagonising either or both tribes. The anyhow.  In the context that America found itself in in January, the government was maximally not wanting to show the tech industry in a wonderful light, particularly for that would show the government in a poor light. And the tech industry was maximally attempting to avoid a newspaper headline along the lines of tech industry upstages the government on America’s number one national priority.  From my perspective, that prevented what could have been very productive phone calls between high level decision makers in both camps, saying, Okay, it’s very clear that we have a need for this particular thing delivered very, very quickly. You are more advantaged than us with respect to bringing that to market, do it as quickly as possible, or you are more advantaged than us with regards to one of the reasons. Tech was reticent to act on this internally was perceived lack of legitimacy in a time where the urgent need for legitimacy in the democracy was highlighted, very highlighted very salient for a lot of people. And so without the legitimising function of government and representative democracy, Tech was not willing to do the things that were within its power, which would have helped Americans get access to vaccine. And so let’s say a mutual appreciation for the institutional constraints that one was working, working over, working under rather and a, frankly, I think mutual bridge building effort, and we can’t afford to be at loggerheads, particularly when time is of the essence that everyday matters would be very useful. I think that I’ll say some good results of that. The main observation about 300 pages of regulations, but a lot of the the sort of intent behind the regulations is in the right place with regards to one thing that many stakeholders including in the government were very worried about in California, the rest of the United States was vaccine equity, for example. And there was concern that tech centric attempts to alleviate the problem would have negative impacts for vaccine equity, have long and complicated thoughts on vaccine equity. But for example, the the was a useful thing for our organisation that a lot of folks had already put some thought into. Okay, here are the top 10 as spoken languages in California by sort of lack of access to, to English healthcare information in their community, and like a, you know, staff rank ordered list, which was helpful for us, because we then knew which translations to go after quickly, was also really helpful to have the feedback loop between us and the government, which developed over time, we had a sort of Batphone into the California Governor’s office where when things came up that were at the intersection of what we cared about and what they cared about, we could call them and say, just as concerned citizens, because we are acutely aware that we also have a legitimacy issue. We would like to report that we saw this, perhaps someone should chase that down. One example of that was that was the public policy of the state of California that a vaccine not be allocated to people in reliance on their immigration status. That public policy was not uniformly followed by people who were in charge of administering the vaccine. And so when we became aware of that, we would feed back into the government office if they could have someone in the regulatory apparatus, give the health care provider at issue, a quick call to remind them of their legal responsibilities in California.

Bridget Harris 
If I understand Mary Lauran’s question it is around the capacity of the public sector to innovate at all. And if it does, it tends to very, very slowly. So it’s the slowest latest adopter to anything that comes on board. And that’s because it’s institutionally risk averse. And everybody works for public sector, governments, you know, organisations are, are taught to be risk averse. So, it seems to me that you and your team, it wasn’t just so much that the software solution that you threw up, it was actually just the problem solving thing that you came, you came, you came to the to this topic, just with a desire, I think, and one of the tweets that I showed, is just a bias for action. You know, it’s just a, we’re going to try to do this, we’re going to problem solve, we’re going to get it done. And that’s a pragmatism that most public sector organisations are often absolutely not allowed to do to make what you’d call common sense decisions just to go in there and start phoning pharmacies, there’ll be a host of people whose job it would be to, to stop other people from doing that for all our risk assessment, reasons that would justify their paycheck. And so there is something there, which is there’s presumably there has to have been some kind of disruptive force on those people on the people who are working out out of the governor’s office office, who could see the problem solving and the energy that you guys brought to this topic. That wasn’t it wasn’t just about being some software genius, like you said, most of the time, you’re scraping Dreamweaver sites, you’re not doing anything, you know, super AI innovative, you’re just fighting your way around a whole load of crappy websites to get to the information.

Patrick McKenzie 
I think that risk aversion is a true thing and is one piece and a complicated puzzle. And they don’t have all the answers to this complicated puzzle at the end of the day, and I’m quote, unquote, just the software guy. I think society should take a big look at institutions and institutional design in light of what we’ve learned just last year. One of the things that continues to surprise me about the the vaccination effort, right at societal level, is that authority and responsibility was extremely diffuse. Was and is extremely diffuse. The joke with people that I was sort of the vaccine information czar for the United States for a period of six months. And that joke is closer to the truth than I think the United States should want it to be. There is an org chart. Because of the cycle times in government, that org chart gets updated very infrequently. And the org chart, quite literally did not have somebody in charge of this. And every organisation thought it was not within the organisational remit to fix the lack of someone who is actually like waking up in the morning and saying, I have nothing to do today but this. Nothing I do this quarter matters other than fixing the Vaccine Information problem. In a lot of places, there was a sort of like, no, that the jobs being done were important, you know, like individual county health departments, were physically putting jobs in the arms of a lot of people, etc, etc. And they, they would say, Oh, the state should give me this state’s pointed at the feds, the Feds said we can’t do anything without XY and Z. And that, from like, institutional design perspective, we clearly should have had should have wanted for there to be people who have the appropriate level of power to enact change the situation and a desire to use it. I don’t know how we square that circle as a democracy that has the form of government that we have. But I think there’s there should probably be a conversation on okay, how, how do we want to change our decision making apparatus, apparatuses, decision making bodies with respect to this? How do we want to be better prepared for the next disaster and indeed, for the day to day operation to government, and government and society more more broadly?

Bridget Harris 
I couldn’t agree more. I’m sorry to keep on going on about this, but I want to go back to the call centre question.

Patrick McKenzie 
I’m gonna go back I love geeking out about call centres!

How hard or easy was it to transition from top-heavy volunteer organisation to more traditional volunteer base?

Bridget Harris 
It’s actually to do with the political organisation of volunteers, which is, again, it’s a sort of something that people pass over as if it’s super easy. It’s not easy to manage volunteers. It’s not easy to manage volunteers and tech, and actually something that I didn’t realise you and I have in common. A long time ago, ran a campaign where I did exactly the same thing. It was a political campaign, but I was getting volunteers to scrape political websites and communities. We scraped loads of websites to get contact details, stick into a campaign website or campaign database and run a campaign. What you’re doing is campaigning, but for obviously, a much more laudable reason than than the one I was for. So what did you learn from dealing with all of those volunteers? How how hard or easy or joyful or painful was it?

Patrick McKenzie 
So a lot of things, interestingly, I heard from, from someone who have previously worked in the Obama White House that one of the reasons that the various government governments in the United States had cottoned on to our solution was that they are for institutional constrained reasons. They can’t work for volunteers directly, they can work with NGOs, but for labour relations, etc, reasons are forbidden by regulation for having lots of unpaid staff, which we ran at certain points that largely on the goodwill for unpaid staff. With respect to running them, the we had a sort of top heavy volunteer organisation early in that we were largely having, you know, engineers and Silicon Valley executives driving the calls to pharmacies. And we were sustain that for about a week or so before our volunteer base largely decided. I’ve made 200 calls, I’m out. And then transitioned to a more traditional volunteer base for civic oriented, civic oriented projects in the United States, which was interested, interesting, in a lot of ways, we had a volunteer coordinator, Debbie, who had been doing this as a volunteer herself, but for for many, many years. And she said, You know, when we are doing grassroots political action in the United States, or, or rapid response, here are the things that we use to, to keep a lot of volunteers coming back every day, here’s the engagement strategies that work. By the way, we’re going to have to start explaining to people we didn’t have like orientation meetings for how to use discord for the engineers just started on day one, because we didn’t think anyone would need that. But we eventually had to switch to Slack, which more of our volunteers were comfortable with, and really, really worked on the UX stuff like the individual call flows, etc, to, to keep the volunteer base engaged and keep them effective. Also had and this ties back into the last point, as the government started to recognise that we were a high quality partner, and good to work with etc, they would do things like California had a statewide volunteer initiative. And they, you know, have a lot of people who are getting in touch with the state on how can I help, they had to state my volunteer initiative to sort of connect residents of California with projects that could, you know, shovel ready, work, you know, use more folks that were ready to do things and so they, there was a virtuous circle where they said, Oh, you know, if, if you need it, vaccinate ca has been doing at or if you were looking for an opportunity, vaccinated ca has been doing excellent work for us, and they can plug directly into their infrastructure and start making calls immediately. That immediacy of value created was something that we spent a lot of effort on, both sort of, selfishly from our perspective to, to get value from people as soon as they started working for the org, but also to, to give people like the immediate sense of a win and, and help them, you know, want to continue with the effort. So we did things like when we were as we developed sort of like a predictive model for whether the next pharmacy we called would be, would be more or less likely to have the vaccine and we would attempt to put at tempt to get people wins earlier in their career with volunteering with us, such that it wasn’t the hearing, no hearing Hearing no experience, but rather like you make your first call and boom, it’s your you’re calling someone who actually has the vaccine, you feel like you’ve contributed. And so that makes you make the second call the third call and etc.

Bridget Harris 
Well, I mean, you know, you’re making a really good point, which is that you can’t keep volunteers on on track forever. And you’re actually in some ways, you must have brought in some of your other startup skills of marketing and messaging and appealing to emotions and keeping people keeping people going.

Patrick McKenzie 
Vesna asked an interesting question.

Audience Member
What I wrote was just wondering if you had any dealings with vaccine hunters in Canada, their volunteer base, and very similar to what you’re talking about. I followed them, like I said, obsessively. I don’t think I did any work until everybody was vaccinated my family. It was just amazing and obviously, what you’ve done is amazing on the other side of the border there, too. So wondering if there was any collaboration?

Patrick McKenzie 
For folks who might not have seen this, there was a pattern that we saw, both with vaccine hunters, the org in Canada and via various sort of organic initiatives. In many places, where somewhat informal groups would spring up on Facebook was the one that we saw a lot, where folks would sort of trade tips for getting vaccinated and what a lot of people sort of became informal health care facilitators in that they would spend a lot of their time on this Facebook group, then people would say, Okay, I’m in, here’s my locality. Here’s my risk factors, does anyone have a vaccine for me, and the folks who had better access to the healthcare system or better ability to convince the healthcare system to give them access to health care, which is unfortunately, a major issue in the United States would help these folks in like dudes late work to actually get them an appointment, and in some cases, actually get them to the appointment by literally calling Lyft or Uber in some cases. So we were extremely familiar with the model. I don’t know if we had a contract specifically with the vaccine Hunter organisation in Canada. We had a group within our Discord have vaccinate other places, and we had an open access policy where anybody from anywhere in the world could come in, ask our questions, attempt to work with our volunteers, if that was something that was mutually interesting. And what we ended up doing was sort of like providing a tools for a lot of the folks that were doing this to be more efficient about doing it. Like, for at least some users, the mere availability of an appointment at a particular website on the internet doesn’t solve the problem for them, they might have linguistic issues, although we’d attempted to address that with translations they might have. You know, many people have a variety of risk factors with regards to health care. And so sometimes those risk factors are correlated with difficulty in executive functioning, or in accessing technological resources or, etc. And so we would give to people who were helping those folks the ability to like very quickly find the relevant appointments for them. And we had a, we had a lawyer in California, say that she successfully navigated 70 people to appointments using our website. And then after she was done with your list of friends, family and neighbours, she started making calls for us to help other people get their vaccine, that that sense of selflessness of something I saw over and over again in the effort and was really personally motivating to me there was a gentleman from Israel, Israel, for those of you who are following on international news, had a very well executed vaccine campaign very early. And he read about the our effort on the way to get his own shot. And immediately after getting discharged from the the clinic, hopped on his phone and started calling pharmacies in California so that people could get their shots in California. It’s fine, I believe, having no pre existing relationship with the state of California. So I was moved by that and a lot of other little micro anecdotes of grace during this experience.

Audience Member
That’s awesome. Thank you. I love all of these efforts. Yeah, thank you.

Bridget Harris 
It is very moving. I think when I went in to get my second jab, which was earlier in the year, and there’s all these volunteers and centres like the photograph, most are up at the beginning. And everybody’s, you know, every, you know, everybody’s volunteering to get this to get this job done. And and, you know, I nearly burst into tears, I thought that probably wouldn’t be appropriate. It’s not about me. But we, you know, it’s very, very rarely do you feel part of a global effort. And obviously, there’s still this work is still ongoing around many places in the world that haven’t had access. How Does anybody else have any other questions? Go ahead.

Audience Member 
I’m in two worlds at the moment. And I actually work as a cybersecurity automation engineer in the European Commission. And I’m also working in an AI startup in Silicon Valley, as a contractor. And so I see the two speed worlds, the very slow bureaucratic organisation, and the very quick moving Silicon Valley culture. And I was interested about your points about like, the foundations of technology, and how you used a lot of the building blocks that helped you accelerate delivering these solutions. And because I struggle in the European Commission, because I have great ideas of getting all the students in the European universities to contribute to the cybersecurity battle and getting them engaged. And like, I’ve got ideas, but I just can’t get it into into action. Let’s see. So I was just kind of curious about how initially, you had the vision and how you managed to work out the path of building bringing the components together and seeing where you were in the start of the pandemic and the chaos that was going on, and how you understood that you could get from where you were to where you needed to be to get something that really made a difference.

Patrick McKenzie 
So I don’t have the fully generalizable answer to this question. If I did, I’d be a lot more successful than I am. There was a little bit of learning things on the fly and, and running by feel and a little bit of actual plan. To the extent we had an actual plan. And this is one of my, my own personal contributions to the organisation. The I told folks very, very early, that we should present ourselves as a citizens effort rather than say, a natively tech effort and present ourselves as professionals coming to contribute our professional skill sets against the, against the pandemic, and then have a very, very considered a media strategy. And this resulted in a good deal of good press for the citizens industry of early in like week, one week two of our organisational existence we had, because there was this massive pent up demand in California and the news media has sort of institutional need for following up on the story of like, you know, people are extremely frustrated as there is no good information and yada yada yada. We were sort of like a counterpoint to that narrative on, you know, narrative building like, problem, proposed solution. Whoo. And so we we fit a need for them to tell a feel good story about it. And we, I think we got more, and I told people just internally, we are getting better press at the moment because of that, considered engagement with with the media, then we deserve based on how many people we have helped as of today. And you shouldn’t feel good about the press for us being our name and lights, we should feel that like instrumentally. What does this accomplish?  For us, the thing it accomplishes for us is getting us closer to the circle of trust with regards to institutions that have credibility and mandates with regards to these topics. And we should stepping stone from where we are right now. We’ve been anointed by this good press as an organisation that is for the public’s benefit and doing some things right. We should continue attempting to sort of climb that credibility ladder. Let’s start getting into local governments. Let’s start working with anybody who will talk to us at all in the state of California and parlay that into eventually, we built trust with them. We got introductions from them to the federal government initiative, and then climate letter Knowing now what, uh, what I know, I think we could have been more aggressive in we were very while we said our our only goal shots in arms were willing to sacrifice on any other axis to to get more shots in our faster we were afraid of stepping on toes in the first couple of months of our effort. And if I got to do over, I would have stepped on a few more toes and maybe been had a few more uncomfortable conversations with Save the national effort to to accelerate our ability to deliver impact there, but that sort of like considered mapping out who the stakeholders are, could we know who knows them? Who, who, like, Who can we influence that will like ladder up to being able to influence someone’s decision to to direct this decision ended up being very useful for us at their microchip for for folks that are working with media organisations and influencers. There are often people who are as sort of their like barrier for working with people outside of the system is lower than the barrier of most participants of the system. But they are read by lots of participants in the system. And so those are good first folks to have conversations with because if you establish credibility with them, they can help you bootstrap credibility to sort of like credentials or gatekeeper institutions without you being directly credible to credentialed or gatekeeper institutions. And so an example of that was Tyler Cohen is very well read in policy circles, I have, for various reasons, something of a indirect relationship with him. And we were able to get our stuff into policy circles through Tyler Cohen, versus saying with them directly, hey, I’m the bingo guy, I have this great idea for solving COVID which might not have gone over all that well.

Bridget Harris 
So can I get the inverse Austin’s question? What would you find easy to do? If this was just one of the startups that you help advice and support? Or indeed one of your own? That you’d have found easy to do that actually you found quite a challenge? Was it the politics with a small p? Was it the interplay between policy people, government people and what you were doing, or actually, because a lot of what you’re saying, you know, you could swap all of this out for a marketing and product, you know, product market fit, and all this other all this language that we’re very familiar with? You could just apply that to your experience, was it parallel? Or was it were there things that you could you couldn’t do in Vaccinate CA that you would have found effortless in a startup?

How to get state and local initiatives on the map without the feds

Patrick McKenzie 
There’s a famous quote in American government. The quote is,

“You go to war with the army you have not the army you wish you had”.

True of government organisations as well. There was a sort of pre existing chain of players that had been anointed by the federal government as the blessed source of the data lake etc. We had to work with them. Our publishing partner said in no uncertain terms, for internal political constraints, we must, must, must, must work with the blessed initiative. And so we want to work with you we would, we would feel better working with you. But we feel we can’t give him the current environment. So we can work with you to the extent that you can work with the blessed initiative. 

There was this one time where blessed initiative has a vendor on the back end, which is maintaining the data lake. Data lake! This is a wild term I learned this year. It’s a spreadsheet. 

So conceptually speaking, the spreadsheet had a list of all the pharmacies in the United States with their phone numbers and addresses, etc, etc. And because the federal government vaccines.gov was able to put a pin on the map, that spreadsheet must also have contained longitude and latitude information. One of our partners urgently needed longitude and latitude information. And so we went to the to the federal initiative and said, “Could we please get lat lon added to the export that you do daily, which ultimately forms the basis of most of the information ecosystem around this? It would unblock work at some of the largest organisations in the world?”

And they said, Hmm, interesting idea. Let’s talk to the data lake people because we have no technologists internally on this data lake people said, huh, please file a change request. And we will get that scoped out. We said it’s literally adding two data bits you have to a CSV file, do you really need to go through the whole process? They did Because of NDAs, I will not tell you exactly how long it took to get those two fields added to the CSV. But let me say that is plural weeks, where it would have been, like, I literally would not have emailed the engineering team to do it, but rather just done it myself if I had had access to that codebase. So that that’s one of the like, not so great examples of how this worked out.

There are some like government is the job of coordinating people who have very different needs and very different preferences and into some amount of activity for the common good. There were some cases where you wish that the coordination or people have different preferences than the one they had.

Here’s an example. We advocated that all vaccines that we were aware of be put on the map, as it were. And the government said, Well, that’s interesting but organizationally, we are the federal government, and we are only capable of either putting federal programmes on the map or perhaps we could think of a way to do state programmes, if we had say, unanimous buy in from American governors that they wanted their states exposed through the federal system, and said, that seems like you need to get 51 signatures to do this thing, including signatures literally by governors, instead of just doing it, why don’t we just do it? And they said, that’s politically non starter for us and we several levels down from National Command Authority, do not think that we can spend our limited amount of time talking to the higher ups on this particular issue.

We said, Okay, our workaround for this was, can we augment the authority that you the federal government have with information that we are independently developing with regards to state and local initiatives, and then feed that directly to our publishing partners, while telling our publishing partners truthfully, information is coming from the feds and the states, etc (Without the feds and the states having agreed to do that)? They said, Well, you as private citizens can say any words you want on phone calls with other practices. And so we’re like, oh, great, then we will be saying some words. And so that happened, but it feels strange to me that it needed to happen.

Bridget Harris 
Patrick, you’re not applying for the job of CTO of the United States anytime soon! What are we going to do about this? How long do we have to wait before tech catches up? When does government catches up with tech, because you do have a mismatch, you’ve got all of basically the problem solvers, the innovators, the bias to action people, they’re all in the tech world. Sometimes on completely ludicrous projects. Very interesting examples. How are we going to square that from your insight, experience? Now? What would you what advice would you be giving the public policy world now to think about how we enable tech better?

Patrick McKenzie 
There are some interesting initiatives such as the United States Digital Service, which essentially is a second career for people who are technologists to give back by doing a tour of duty within various governmental organisations. There is that they have many challenges, the scope of whether that’s the the appropriate engagement model is an open question. And ultimately, I think that the big thing that needs to happen is more software people working within the government literally awful levels, and getting closer to the levers of power. Just like that’s for for everything for geeks, like hey guys, let’s get closer to the levers of power.

Bridget Harris 
We’ll have a discord for them, you know, discord.gov

Patrick McKenzie 
Interestingly that that is closer to reality than you might think. The number of signal and WhatsApp groups etc. in Washington DC, by AI folks coordinated across government organisations might be terrifying to the organisations that are coordinating.

Bridget Harris 
One of my greatest fears coming from the gaming world is democracy is coming to online communities who are starting to organise in a very naturally political kind of way. Because they realise that the owners have the tools, you know, the means of production, as it were, which are the people that are in the technology companies that run the games, they have a lot of power over them. So the two ends are coming together quicker.

Patrick McKenzie 
My funny gaming plus pandemic anecdote is that there actually was an unplanned pandemic in World of Warcraft, back in, I want to say 2012 or so, the Corrupted Blood incident, if anyone wants to Google this. The Corrupted Blood incidents, was very interesting to the Centres for Disease Control the American anti epidemic, anti pandemic agency, among other reasons, because at least some of the players who got Corrupted Blood, thought it was hilarious, and, and would do things to increase the probability of other players catching the Corrupted Blood to keep the pandemic going. So this was the negative for the people who, from your perspective of like playing the game, you want to play the game and uninterrupted fashion, but some people thought it was like really, really funny. So they would spread the disease intentionally.

And, and the CDC said, That’s really interesting. We had never modelled people intentionally trying to make a pandemic worse. But like, Blizzard, can you give us your data on this so we can study it. Unfortunately, Blizzard didn’t have the right systems to, give them this sort of thing. But there were some uncomfortably on the nose. The parallels with between that incident and the some of the things we saw in sort of user behaviour in the last year and a half. And you’re just like, I’m glad the CDC made that call to Blizzard and tried to say like, Hey, we’re the CDC. You’re a video game company. But it seems like you have something we could possibly learn from can we get you to get your engineers and data scientists and product people in a room and tell us what you know about pandemics that we don’t?

Because we want to know all the stuff about pandemics like that showed some vision and verve and willingness to adapt from the CDC in 2012. I would hope that all government organisations have that vision and fervour and willingness to adapt all the time. We’ve seen that that isn’t always the case. And I don’t know how to fix that. Not always been the case.

Bridget Harris
I was just gonna say that, that speaking from personal experience, and living in DC, and knowing a lot of people who work in government, there are a lot of folks who are super smart and motivated and want things to work better. And there’s a tonne, that gets in the way and some of that is that they’re members of Congress who get a lot of benefit out of talking, saying bad things about how the federal government works. And so avoiding scrutiny. You know, not messing up is a there’s there’s a big, you know, motivation to not do something that would get you on the radar of folks who have that interest.

How do you think we can make governments learn to use and utilise technology more effectively?

Audience Member
Yeah. Hi. Oh, I don’t know who it was who was talking earlier about working for the European Commission. I’m in Brussels, as well, although I don’t work for the commission. I’m kind of I’m sitting here partly with a heavy heart, because I kind of know. Well, there was a question I wanted to ask earlier, which is, do you think government has learned anything from you? And my suspicion is probably not a lot. But maybe it could be?

How do you think we can make governments learn to use and utilise the tech industry and the developer community more effectively in the future for for stuff that might happen again, because you have proven that stuff can be fixed super quickly, when you have engaged and brought that community together? How can we get government to wake up to this? Because my suspicion is, they still won’t get it? They they’ll, they’ll run out all the same processes that they would normally do when the next thing happens? And I just, I don’t know, I hope things might change. But how do you think they could they would learn to listen.

Patrick McKenzie 
So it is partly an issue of getting an institute like individuals in positions of power to to listen and to understand that these capabilities are available and go about tapping them. It’s also partly an issue of like institutional assignment in that. There were very few people who like woke up in the morning and said, What can I do today to make the responsive for less successful, but but the way the jigsaw pieces fit together was not optimal. From the perspective of making the responsive for more successful.

There is a real issue in getting people to understand that these capabilities are available and tap them. And one of the the things that we can do to alleviate that is the same thing, software industry always does some marketing and sales problem. Figuring out what the buyers score got, and say, we can be absolutely indispensable for you assuming nothing that you’re scored on if you continue listening to us, and then you will get listened to more often, if you are successful on delivering on that marketing and sales promise. One of the things that is starting to become apparent, at least within the United States political context is that if you have a couple of smart tech people attached to a political campaign, that political campaigns odds of winning, go up rather remarkably. And I could give some examples here, but don’t want to get into the unfortunate realities of talking about individual political campaigns.

So an interesting thing is that after elected win office in that fashion, and then run into, you know, problems actually doing the job, they are often, you know, that they have a couple of technologies that they personally worked with, as they will ask questions like, Hey, do you know anybody who could just like, help us, you know, fix this problem and run round this, etc? Or that thing we did in the campaign? Could we? Like, what would it take to to get Y agency to do that? And then they start asking good questions of the right people, and in some cases, the boring work of government but like write the legislation that allows or instructs the right people to do the right work to to get the ultimate outcome for the for the focus on the nation.

Bridget Harris 
But just to finish off on a positive note about tech technology and vaccinations, I was filling out a consent form the other day for for my kid at schools, and the school is basically now vaccinating all the children in Britain. So as a parent, I was sent the vaccination form I know and they’re just going to block jab, everybody, jab them all as they come into the school. Anyway, it was it was absolutely lovely. I get sent a form, and it’s sort of 20 questions, what’s your name, whatever else so I basically fiddle through, go to the, to the Submit. And just as I do my last question, another question appears. Okay, so fill that out. And go to submit I fill out the last question. Another question appears. And actually I was presented with something like a 12 question form, and it ended up being about 25 questions. And somewhere somebody in the Nudge Unit or the government, whatever has said, You will not get or You’ll get a high drop off rate of parents who can’t be bothered or disinclined or haven’t got the time to fill out a 30 questionnaire thing, but once you get them on the hook, and they’re one question away from submit, my God, they’re just gonna keep doing it until they get to the submit button. And I just thought, well, that is progress. That’s progress. And then they’ll probably have an extremely high rate now of consent forms from parents. Because of that one little, you know, one trick that gets parents to do it.

Patrick McKenzie 
Yeah, the conversion optimization for righteousness is something that we saw this year as well. One of the things that we did was Washington University was running a scale medical trial on flu vaccine, which is a promising, it’s an antidepressant drug, but apparently has some potent anti COVID properties. And they had a grad student do their admission form. It looked like a grad student had thrown something together. And we said, well, if you’re optimising for conversion to get people into this to get the results fast and quickly improve the state of medical care, we would do some things differently. And so we hired a consultant to like redesign the form for them and doubled the rate of people that successfully got through that and so halved the amount of time it took that the trial to produce the results and those results are now they would probably stay at once a to three weeks ago. So that’s been interesting thing to look at.

---

---

Next Events