Pilot Like You Mean It

Episode Transcript

Over the summer, MIT released a report that supposedly concluded that 95% of all the AI pilots out there are failing. And for a couple of weeks after that, it was all you could hear about — all anybody was talking about. And it's still a stat that's getting cited pretty much constantly. I'm going to call it now that this is going to become one of those things that just persists forever, like the "70% of all transformations fail" one that is still getting thrown around. I'm pretty sure that one originally came from the Harvard Business Review, but most people don't even bother to cite the source anymore — we just say it like it's true and we all accept it now. But questionable sourcing aside, the frenzy over this MIT report has really got me thinking about pilots in general and why we're so obsessed over their success or failure. Why do we care so much about how other people's projects are turning out? And if a pilot's supposed failure can teach us crucial lessons that allow us to iterate successfully toward better outcomes, isn't that actually a win? So I think it's time to reframe our understanding of what it means to pilot something — whether that is a framework like Agile, a new technology like AI, or maybe a piece of MarTech. So let's clear this episode for takeoff.

Welcome to the Agile Marketing Edge, the first podcast dedicated to turning agile theory into real world marketing breakthroughs. I'm Andrea Fryrear, CEO of AgileSherpas and your guide on this climb to smarter, faster, outcome-driven marketing. Every week we unpack the what, who, and how behind agile marketing — from building high velocity workflows and slashing waste, to measuring what really matters and scaling success across teams. You'll hear quick hit strategies you can deploy today, plus candid stories from marketers who have traded chaos for clarity and never looked back. Hit follow wherever you listen and let's carve the next switchback together.

This whole pilot uproar after the MIT report reminds me of a Sex and the City quote when Carrie says that it's amazing how upset women can get over the marital status of strangers. We can rephrase that slightly to say: it's amazing how fixated white collar workers can get over the pilot status of strangers. And I don't want to imply that I'm completely above all of this, because when the MIT study came out, I was referencing it — and I will continue to reference it, because the numbers are just so flashy and easy to understand. They make the point about the difficulty of getting AI right really well. So let me quote directly from the study so you can see how clear it is: "Despite 30 to 40 billion — that's a B — in enterprise investment into GenAI, 95% of organizations are getting zero return. Just 5% of integrated AI pilots are extracting millions in value, while the vast majority remain stuck with no measurable P&L impact." It's punchy. It's clear. It's to the point.

And it's the same story if you go back just a couple of years and look at non-AI transformation efforts. Bain and Company did two surveys of 300 really large companies, ten years apart — in 2013 and in 2023. Both times, they found that only 12% of transformation initiatives either met or exceeded expectations. In 2013, 38% failed completely to deliver on their objectives and 50% settled for mediocrity — it was okay, but not great. In 2023, 13% failed to deliver and 75% settled for mediocrity. Again, punchy and clear data. But in both cases, it's kind of depressing. Because if you're somebody who wants to drive change and transformation in your organization, these are the kind of stats that make you want to throw your hands up, throw in the towel, or maybe just throw up. But if you're somebody who doesn't like change, these are the kind of stats you're going to pull out in a meeting — perfect ammunition to dig in your heels and argue for the safety of the status quo.

But there are two really important things lurking behind the scariness of the data in both of these reports. And of course, these aren't the only scary reports — there are lots of them out there telling you about how transformation efforts fail or how difficult it is to adopt AI. But in all of them, there are things lurking behind the scenes that we need to better understand. The first is that everything — every single thing — that impacts the success or failure of a pilot is controllable. It is within an organization's control. Everything that impacts a pilot's success or failure is within your control. The second thing is that even a so-called failed pilot is an amazing teacher. And these two things are really, really important because when we put them together, they should tell us that there is literally no good reason not to try a pilot. The status of other people's pilots notwithstanding, we should not be afraid to try one. Because if you do it right — and note the "if" there — it's either going to work and be amazing and lead toward larger transformational outcomes, which is what we want, or you're going to learn a whole bunch of stuff and your next pilot, your next step down the transformational path, will be even better.

So let's unpack both of those things further, because we want all the ammunition possible to argue for the inevitability of pilots. We don't want to give power to the naysayers in the room who would point to these kinds of reports and say they're arguments against change and evolution.

First up, let's talk about the success and failure factors — the things that are, as I said, within our control. It turns out that the 12% of companies from the Bain report that met or exceeded expectations during their transformation efforts all had things in common. And Michael Mankins and Patrick Litre of Bain put those common factors into six categories. I'm going to adjust those categories slightly here, because we're not necessarily talking about organizational-wide transformations on this podcast — we're talking about transformations inside a marketing team or marketing organization, which is a bit of a subset. Some of the six apply one-to-one with a very clear connection, and some need a little translation. But a lot of it is very much the same.

The first success factor is treating transformation as a continuous process. And this is one I don't need to modify at all — it is absolutely true. The marketing organizations that I have seen transform well don't approach it as a checkbox or a one-and-done project we're going to do this quarter and then walk away from. This is just what work is now. It is always going to be a continuous state of transformation, and approaching it that way is the best option for setting ourselves up for success. And as we go through the other five, you'll start to see how this mindset informs all the other components and sets them up for success as well.

The second success factor is building transformation into the company's operating rhythm. I didn't modify this one either, because I have seen a huge difference between the clients we've worked with who have regular communications and regular integration of transformation efforts into their existing rhythms, and the ones who try to keep it separate — a different thing, off to the side, because someday it will be over and we won't need to integrate it into the regular work. You can see how tightly this is connected to the first factor. If we think transformation is temporary or just something to check off our list, we're not going to bother building it into our regular operating rhythm. But if we think transformation is continuous, then we will work hard to integrate it into the normal work we're always doing. One of the groups we worked with did this so well — they had a regular communications machine that talked about how the transformation effort was going. Every week there were emails that went out. They had dedicated space at monthly town halls where the transformation team would report out on KPIs, which teams were going to be affected next, and representatives from teams that had already gone through the transformation would share their experience. It was a thing of beauty. It made everybody excited to be the next group to go through the process, and it made the whole thing feel real, tangible, and permanent. Building a transformation into the normal rhythms of the company really, really helps.

Third: explicitly managing energy and resources. Not much to adjust here either, but this is one I see marketing teams in particular getting wrong a lot. When transformation is happening at an organizational level — which is what the Bain study was more about — it's coming from a CEO directive, and so at least there's a lot of weight behind the budgeting process. Whereas a marketing leader may have a bit more difficulty pulling together a hefty budget for a transformation project. Even if transformation is going to be ongoing, it can be difficult to pull resources from revenue-generating activities to fund transformation activities. However, if you are serious about this — and you should be — then you need it to be funded with not just money, but also energy and human resources. This cannot be one poor person's side-of-desk project. You need it to have KPIs. Ideally, you have a transformation office or a transformation team. It needs to be clear that this is a priority with high-level executive sponsorship and real energy behind it. This also makes sure there's not a single point of failure — if one person leaves, the whole thing shouldn't collapse. This should not be a single-legged stool. There should be multiple legs so that if somebody does leave, the effort can continue. If this is a continuous process that has been integrated into the rhythms and routines of the department, you don't want it to get ripped out because of one departure. Protect it by having a team, by having it more integrated and insulated from disruptions that could be caused by any single person leaving.

The fourth one is one I struggled with at first, until I thought about the AI report from MIT. This success factor is using aspirations — and not just targets — to set expectations about a transformation initiative. I don't think we give this one enough credit. When we think about what we're trying to accomplish with a transformation effort, a lot of times with an agile transformation we're thinking about speed to market, productivity improvements, and waste reduction. We're looking for improvements to the bottom line, because that's what people want to see — the business benefits of changing their ways of working. And that's valid. But having some aspirational component to the transformation, not just targets, was part of what made the successful organizations successful. If we think back to the AI report, they were fixated on P&L impact. But maybe that's not all that success looks like when it comes to adopting AI. Maybe we're missing some of the intangible benefits of modernizing teams, modernizing ways of working, of people feeling like they're innovative and on the cutting edge. That matters too. Things to think about. And it's worth noting that you don't have to do all six of these things — the Bain study was not saying all six were necessary to land in the 12% who found success. The more you do, the more likely you are to find success, but all six are not required. Three or four of them will get you the highest likelihood.

Fifth: driving change from the middle out. I could not be more in agreement with this one, because what we have seen so many times in transforming marketing organizations is something called the frozen middle — and this is where transformations can stall so, so often. You have individual contributors who are excited about changing ways of working, especially with agile, which is all about team empowerment and autonomy. For a lot of people, it's about being more productive, owning their day, and being able to push back on interruptions and unplanned work. That's such a good feeling — like a load coming off their shoulders. Agile is fun. They feel connected to their colleagues and see alignment between their work and the organization. And then executive leaders love it because they get visibility into the work being done by the teams. They can talk more clearly about activities, help remove non-value-added work and waste from the system, and they often get to stand up during planning or strategy or board meetings and talk about how great the transformation is going. But then you have middle managers who are bearing the brunt of the situation — they're the ones people come to when they're not happy about how the transformation is going, when they're scared, when they're worried for their jobs, when the tool isn't working, or when the executive is getting difficult questions and passing them down. And the managers feel like their job is being transformed away. So bringing them into the effort early and having them drive it is the best way to make sure the transformation doesn't freeze or stall out there. They have a lot of power in making sure things don't change if they don't want them to. Driving change from the middle out can be super powerful for any kind of change, but especially for a ways-of-working change like Agile.

The sixth and final success factor is budgeting for continuous transformation. This is the one I adjusted the most, because in the Bain study they were talking about organization-wide transformations funded with tens of millions of dollars — not something marketing leaders are generally going to be doing. However, we do still need to be serious about the money we raise and budget for a transformation effort. You cannot announce some kind of transformation — whether it's agile, AI, or whatever comes next — call it mission-critical and strategically important, rally everyone around it, and then turn to the team leading it and tell them they have a $10,000 budget. You can't. It's not going to work. Especially if we go back to the first success factor: transformation is a continuous effort now. It's not going to be over after Q3. Transformation is never not happening — the question is just what the next phase looks like. So make it a permanent line item in your budget. It's not going away. That way you'll be able to continue to educate, continue to evolve, and continue to be ready for whatever comes next, instead of being caught flat-footed and scrambling every time the next version of AI arrives and disrupts you.

So those were the six success factors. You don't need all of them — shoot for three or four and you'll have a good shot at your pilot and your overall transformation working. And coming back to the MIT report, it's also worth noting that the things they pointed out as contributing to failed pilots — brittle workflows, a lack of contextual learning, and misalignment with day-to-day operations — are all within our control. To quote directly from them: "The core barrier to scaling is not infrastructure, regulation, or talent. It is learning." So again, all things that we control. If you are in the struggling pilot category, it may be time to turn on some Taylor Swift and look in the mirror, because you might be the problem.

All right. Speaking of learning, let's move on to the second part of the pilot conversation: when a pilot is not successful, what can it teach us?

This really reminds me of when we work with clients who need to adjust their team composition as part of an agile transformation. This happens a lot, because marketing teams are often set up in functional silos — all the designers together, all the email people together, all the marketing operations people together, and so on. And it can be difficult for those folks to try to work in an agile way because they're not set up cross-functionally. What happens over and over again is that we get stalled as clients try to figure out what cross-functionality should look like for them — because we only have two marketing ops people but we need five cross-functional teams. And so they get stuck, and we spin and spin and spin, diagram and move and talk and discuss and debate — should we hire, or can they be part-time on this team or that, and what should we do? And every time, every time, we just want to try. We just want to pilot. We just want to launch a team of some kind. Because while we know some best practices and can give input on what's worked before, everybody is unique. Teams are made of humans — and a little bit of AI now, but mostly humans — and we simply cannot predict how it's going to go. There is no magic crystal ball. We just have to try. But a lot of people get stalled out here.

And this is exactly why thinking about piloting not as the first step on the path to perfection, but as the first step on the path to learning, can help us not be so afraid to start. Because this is what agile and agility is, as well — it's not about seeing how fast we can get to perfect. Agility is speed to learning. And pilots of any kind — agile pilots, AI pilots, whatever kind — are nothing if not the incarnation of agile ways of working. It's like an MVP, a minimum viable product. A minimum viable pilot. We can even just keep using the P. It's a quick way to test an idea, to see if we can deliver value in a new way. And the answer might be no — we cannot deliver value in the way we thought. And that's great, because we learned in a controlled test environment, not by blowing up an entire department or organization.

And I think this is part of the issue with the AI pilots in the MIT study. We've got multi-million-dollar investments — so maybe these are not pilots at all. Maybe they were never approached with the MVP agile mindset. There was no intent to test and learn and iterate and roll things out incrementally. Instead, people started writing checks with seven zeros on them. And of course, when things started not going well, everyone freaked out — because this is not low-risk failure, which is what real pilots are all about. So maybe in the case of the MIT study, these aren't failed AI pilots so much as a case of people not actually setting up a pilot in the first place.

So, if you'll forgive me a terrible pun, it's about time to bring this episode about pilots in for a landing. At this point, having talked about the success factors for a pilot — all of which we control — and the opportunity to learn from a pilot even when it so-called fails, I hope you are coming around to the idea that no matter what happens, a pilot done properly is always worth doing. Because in that situation, there are only two possible outcomes: we win, or we learn.

But if you're struggling with what a good pilot might look like, the Agile Sherpas team is here to help. We provide expert consulting guidance to marketing teams of all shapes and sizes, and you can learn more at AgileSherpas.com. Until next time, I'm your host Andrea Fryrear. Remember — the struggle is real, but so is Agile marketing.