E118: Allison Pollard – Run the Experiment, Evaluate the Hypothesis

Kayanna  00:07

We all have something to share. KnolShare with Dr. Dave.

 

Dr. Dave  00:12

So welcome back to another enlightening episode of the KnolShare with Dr. Dave podcast. I am your host, Dr. Dave Cornelius. Today, we dive into the heart of innovation running experiments and evaluating hypothesis. So let's start our journey. When you think about generative leaders, you know, they're the pioneers with the future, recognizing that staying stagnant isn't an option they need to experiment. And innovation is crucial. So you think about companies like Apple and Google, and Amazon and SpaceX, you know, they wouldn't be the Giants without any experimentation mindset. So the benefits, innovation, adaptation and keeping ahead of the curve. So most of us have heard of the scientific method in school. So it starts with a hypothesis. So in leadership that translate to having a clearly defined objective. Want to launch a new product feature, start with a hypothesis. We're not talking about making wild guesses. We're predicting outcomes and setting ourselves up for intentional learning. So today, we have Allison Pollard. Yay, who's joining us. Hello, Allison.

 

Allison  01:25

Thanks for having me. I'm so I'm really excited about this conversation. Meet

 

Dr. Dave  01:30

you. And I was like thinking in my mind, you know, Allison came from improving ink. And then she went to helping improve. And I think her third stage is improving improvers. What do you think?

 

Allison  01:47

I don't know, maybe, maybe retirement is just like, improved, done. Everything's better everything is is in a good state. And like, I can like set it all aside at that point. I

 

Dr. Dave  02:01

hope so I really do. So what did you just give people a little update from the last time we spoke? You know, what are you doing?

 

Allison  02:13

Yeah, like you said, I'm, I'm with helping improve. I'm one of the CO owners, we have a new class that we've been working on, called Managing amazing people. That's been where the bulk of my time has been going in, you know, figuring out how do we help people managers have better one on one interactions in the workplace. It's so critical that, you know, managers are often in the middle. They're like the connective tissue across organizations, and especially in Agile training, they often get left behind. Or they're told like, Well, now that we have a team, you don't need to worry about as much just trust the team, they'll, they'll be fine. They don't really need you a whole whole lot. You know, you'll remove an impediment every now and then. And it often has led to managers that are very hands off with the teams and more importantly, with the individuals. And that has not been serving us very well across the community. And so Paul, Tavis and I decided, you know, based on what we'd seen in organizations, and especially what we've been doing in executive coaching with some of our clients, there are some core skills that managers need in order to be effective. And so how do they look at things as building a network of mutual support? And how can they be really clear, connected and curious with their people, that they're able to help them to grow and help them to succeed in the workplace. So we've been excited to build out those materials and bring it to our clients.

 

Dr. Dave  03:52

You know, I'm so excited to hear about, you know, all the good work that you're doing. And as you were thinking about as I was thinking about managers that I get left behind, you know, I guess you're working toward making sure people you're not telling him go sit in the corner go that's what we did with Agile I go sit in the corner, we don't need you right now.

 

Allison  04:13

Probably did that at some point. Earlier in my career. I'm like, I've got this with the team. Like you can go chill out over there. Like, I'll tag you in if I ever need you, like not a big deal. But, but again, like that doesn't, that doesn't work all that well, even if it works temporarily, where the team is almost like self managing that they are producing that that they're getting along well enough. There are some real things that managers can do with their positional authority that the team cannot do that can be super useful and super needed. I think especially as organizations are needing to move faster, and just keep up with the number of things happening with their customer. is out in the world on the market managers are going to be essential to keep everyone connected and moving in the same direction. Well,

 

Dr. Dave  05:08

let's just jump right in as you talk about moving faster. You know, from your perspective, you know, why is adapting and experimentation mindset, really pivotal for generative leadership? You know, especially in today's fast evolving landscape? Yeah,

 

Allison  05:24

it's a it's a great question. You know, I, I think, early in my career, you know, I came from a project management background and owned a lot of the process that I started off with, like, well, let me let me go be really smart. Let me read a bunch of books. Let me go learn from others. And I'll come up with what a great new process improvement looks like. And I'll try to like think through all the challenges with it. And so now he was like an expert, here's what I think is the best thing to do and make those recommendations or just go ahead and make those changes. There came a point though, where there were situations that I didn't know what the best thing was to do. Or more importantly, I needed so many other people to be on board with that improvement, or without recommendation, that it actually got easier. When I could lean into that, can we try this sort of approach? And as I got more into the Can we try this? It actually got into a, here's what I'm noticing? Can we try something anything? I don't even know what, and inviting people to take more ownership of the experiment of recognizing those opportunities of defining what is the possible solution? Or like, what is the improvement? You know, what are the next features that we might want to introduce to our customers? And what does that look like? So, recognizing over time, like, oh, I can't be the expert on everything, I can't learn fast enough, I can't seek through all the risks, you know, hard enough on my own, I do need other people to help be part of that process. And again, especially from like an adoption perspective, inviting people to be contributing to it to really be participants in the experiment, from like beginning to end, it gives them a greater sense of buy in and a greater ownership, that just leads to better outcomes.

 

Dr. Dave  07:28

Without a doubt, you know, so when you think about formulating a hypothesis, right, you know, how do you ensure that they're both ambitious enough to drive innovation? And precise enough to be able to that you could test them?

 

Allison  07:46

Yeah, yeah. Well, and in because often, like, when we think about innovation, it becomes this really vague abstract, like, let's make things better, or let's make them awesome, cool. I, I love the sound of awesome, I love better, you know, I'm improving things, like, let's help improve some stuff. But what does that look like, you know, and like, it creates this, for me like energy, you know, like, oh, this sounds exciting. This is this is kind of cool. But I don't know where we're headed. And a lot of times with the clients, I have worked with, you know, larger enterprises, they've been much clearer on like, here's the feature that we want delivered, we put together the business case, it might be something that our competitors already have. And we feel like we're behind, on delivering it. They're very precise, in you know, here's the specifications. We know some of the measurements, we know some of the things that we want to test with it. But it's very, very large. And so it's like, how do we go from, we want something that's awesome. It looks exactly like this to, you know, that ambition, something that's, you know, more aspirational, and that's small enough that we can test it. So for me, it's like, where are we today? And like, how do we kind of nudge things in a in a smaller direction? You know, like, what's the first thing that we would need to know from our customers? How would we, how would we check you know, if they respond to this new feature, if this is something that they want? Like, how can we prove out that there's some value in it? So it's, it's like teasing out some of the assumptions and like finding the hypothesis behind these large features that they had already imagined. And we're on the verge of getting many, many teams to work on delivering.

 

Dr. Dave  09:50

Yeah, that's so true. So let's think about something in a real world. You know, from your experience, where the design of an experiment is significant. difference in the outcome or insights gained? Yeah, so,

 

Allison  10:03

so often with innovation, I find that there's like two different possibilities that people have in mind, there's the, we're gonna create something brand new, right? It's, it's a disruption, like we're going to replace, or we're going to create something from scratch, that is going to be so amazing. It's going to attract new customers, or it's going to amplify our business in some way. So there's this like disruptive innovation. And then there's the other version of we have an existing product. And now we're innovating on what it looks like or how it functions. So maybe operationally, we're saving some costs. Or it's like going into a new market somehow. And when we talk about the design of an experiment, I think back to a team that I had worked with that was responsible for an application used by frontline employees. And so if you can imagine there are 1000s of frontline employees that use this application as they're helping customers, and they're all over the world. So there's a high training cost to this application to help those frontline employees be effective in their jobs initially. So this team that works on the application is told we want you to innovate. And they were getting some language of we want you to innovate and improve the operational cost, like how might we be able to onboard new employees in a faster way, so lighter training that's required for this application, but then there would also hear this language, like, do something that replaces the existing product, like re imagine what the experience is for the frontline employee, as they're working with customers and how they interact with the software? Well, this team then struggled as they're trying to come up with their experiments of, are we coming up with enhancements to the existing product, because that looks one way. And that we would need to, you know, measure it based on the existing application and the use across all of the existing for some, like significant portion of the the current frontline employees that are using it. Or if we're creating a brand new system and reimagining what this thing looks like, we might do much smaller prototypes that are not living in production for an extended period of time, it might really be, we're going to use this prototype for about an hour, we're going to interview a couple of the employees, see how they respond to it, and like take that information and feed it into like our next experiment. So the design, it was like, which path do you want to go, it's very difficult to go both ways, and your outcomes are gonna be very different. So for this particular organization, they decided we want to go more disruptive, we want to reimagine experience, because we do think that will help us in the longer term. But that meant that they actually went with a smaller team that could move in a faster manner, run smaller tests, and be more nimble. And that the evidence they were collecting was, again, much smaller datasets, you know, of, you're interviewing about 20 employees, you're, you're testing things out for an hour or two, you're finding what is useful, you know, what do they respond to, and you're not trying to replace the entire system all at once. So it really was more of like a new product development kind of approach.

 

Dr. Dave  14:00

No, that's excellent. As you you brought data into the conversation, you know, there's tons of data out there, you know, from the different types of experiments, even experiments that's not ours, that we can learn from. So how do you prioritize which metrics to focus on? That's a good question, right? But don't go anywhere yet. And how do you avoid potential Miss pitfalls of Miss rep Miss interpreting the data? Cuz that could happen, right?

 

Allison  14:28

It can one and again, because like, I think to my like, project management background, you know, that we knew, right? A project was expected to create XYZ results. But you know, you just kind of build it and build and build and build and you deliver it well. Often what would happen at the end, we would comb through all the data available, it might not have met the original business objective, right? It might not have generated say the revenue that We had anticipated in the business case, but it did some other good things. And so we're then like pulling all the other good things data, and like broadcasting that, like, look at all the fancy, wonderful things that happen, you know, like, we decreased the number of calls to our Operational Support Center, we, you know, have made this thing better it performs in a faster manner and, and just like showcasing anything that can look like success, even though we had met our original goal. And for me, if I'm gonna run an experiment, I really do you want to be clear, before I start running it, what is the metric that matters the most? What is the thing that I want to be measuring? How am I going to evaluate success, and actually be super clear of, you know, setting that metric of I want 70% of the users to be able to complete their transaction and not be calling our customer support, if that was the thing that was most important. And hold to that, right, once your experiments underway. And it's certainly at the end, you know, if it comes back at like, 65%, there's that inclination to close, you know, like, can I pat myself on the back and say, like, well, we got better than we were? Or do we take that really scientific view and say, 65% is lower than 70%? So let's talk about what the next experiment is, if it's still important to get to 70%? What is the thing that we want to try next or try differently? To help us achieve that metric? Yeah,

 

Dr. Dave  16:38

well, it's interesting, right? Because in science, we just don't run one experiment, we run it multiple times, to try to get to whatever that target goal is. So it's part of our iterative learning experience. Right? So tell us more about a time when an experiment didn't yield the expected results. How did you pivot? And what iterative learning ensued? Yeah,

 

Allison  17:06

yeah. Um, so I think back to when I was starting the new mentorship program with women and agile. So it's been a couple years now. But I wanted to take that experimental approach to building this program. And so there had already been a lot of requests in the women and Agile community, we would love mentorship, you know, it'd be great to have a mentor. And it was like, great, so we're gonna do this thing. And in my head, I thought, there's so many amazing adlis, who are so generous with their time, and they have so much experience and knowledge to share, it would probably be awesome to do some, like masterminds like group mentorship offerings like that, that would get the maximum effects for mentees that are probably like newer in the Agile space, and they really just want some clear answers, you know, on, what do I do? And how does it look at, you know, this, this situation that I'm facing? And then, again, the mentors like they would they would feel value of, of their wisdom, their experience, you know, being able to help others? Well, thankfully, we were sent out a survey to the members to find out like, What do you mean by by mentorship? And what is it you're interested in? And I was completely wrong. So this was the first big aha, have, you know, people actually wanted long term one on one relationships. So my mastermind group idea was out. And they were looking for more advanced help, I would say that they were well into agility and looking at how do I take it to the next level? Like, how do I continue to advance in my career, become a stronger leader, and do some things that are beyond the fundamentals? So that kind of led me to my next okay. How do I like what do I need to know now? So I've determined it's me long term, one on one relationships that the mentees are interested in. And it's a more again, like sticky, challenging kind of stuff that they're they're wanting to work through. As I thought about who I knew, that could make ideal mentors for that. I then interviewed them. And I said, What is it? Like, what would you need? Or what would you want? In order to say yes to mentoring a complete stranger? For six months, you know, like, what would you hope for or what would what would make that a great experience for you? And again, the answer is surprise me because I thought it was going to be I want men to have very clear goals. I want them to be like really motivated, like, I want to know that they have the Time to do things right, like run experiments on their own. And so if someone had a really clear, compelling goal, I'd be like, yeah, Sign me up. That's the kind of person I want to work with. And instead, the interviews, actually told me, it would be really useful to meet my peers across the community, that as a mentor, as someone that is more experienced, that's in a leadership role, I may or may not get to meet people in similar positions across other companies, that would be really valuable to me. And the other thing that I heard was, I would love to know how to be a more effective mentor. Because that's not something that gets taught all that often. You know, we might, we might pick up a couple of tips and tricks here and there in different leadership, or especially like coaching type classes, there's a lot of techniques that get pulled in. But being able to learn alongside other mentors and other people, you know, how can I? How can I be most impactful? And how can I be like a better positive influence on the mentees, that, again, sort of changed like the program direction, so instead of only focusing on one on one relationships, and like crossing our fingers and hoping for a good match, we actually decided to focus on creating a valuable experience for the mentors, especially recognizing that these are women who are in leadership roles that may not be represented all that frequently in the Agile community, especially that we had a real opportunity to elevate them. And I think it is like, how do I pour into their bucket as they then pour into others, knowing that they will have a mentee through the women and agile mentorship program, but likely because of their position and just who they are. They're mentoring other people already in their workplace or in their local communities. So it's a real opportunity to create a stronger ripple effect. That I'm like, so glad that we took this experimental approach early on, because all of my initial ideas, and and again, it wasn't just that I thought, Oh, hey, this sounds cool. Wouldn't it be fine? If we did a mastermind or, you know, if we then just like, focus on on pairing people, I was looking at what else was out there in the industry? Like, what are other people doing in the community, and I saw what was popular, and I thought, well, if it's popular, it must be there for a reason it must be working really well. But tuning in and listening to our constituents, to our members to find out, what did they really desire and like, what would be most useful for them took the program in a different direction.

 

Dr. Dave  22:56

You know, I really love that you use women in agile as, as an example. And that's a really amazing program. And, you know, one of the things that I was really appreciative of is that it not only just peered women to women, but you also started to introduce men as part of the mentoring program in literal, which I thought was really great, because, you know, we need each other, right. It's not like we live in a little bubble of women over here and men over there, we all need to work together and

 

Allison  23:32

the launching new voices program, like they're further ahead, and that they have found some great male allies to work with their protegees and, and help get new speakers out there at conferences.

 

Dr. Dave  23:44

Yeah, I was just so excited to be a part of, you know, one. But one group who have done that, you know, so so that was fun. But, you know, as we start thinking about all of these different experiments, we have to balance risk, right. And for most people think most people think the risk are things that has happened, but risk are things that may happen, they haven't happened yet. We just have to keep our eyes on them. Right? So how do you manage it, manage and mitigate risks associated associated with experimentation, just ensuring the safety of the organization, you know, while encouraging bold, innovative steps.

 

Allison  24:26

It's a it's a tough one. And I think this is where again, my like, project management background has been super helpful. You know, we had to create the risk charter, you know, and like, keep track of, you know, the high priority risks, the lower priority risks, you know, like, what were the issues that were starting to show up? What are our mitigation plans, that when I'm working with a client, and we start talking about some innovative either process changes or innovation with their products themselves? I like to be one of the boys that helps poke holes in things. What about this? What about that? You know, like, let's, let's kind of check on some different angles, that could be gotchas that, you know, could be things that make this fail pretty quickly. So I feel like there's a little bit of like, pessimism that that is needed. And that is useful in that. But at the at the same time, you do need to create this space of, it's okay for us to fail forward. And what is that going to look like? Because I think a lot of times when we are talking about innovative changes, there's not necessarily like rollback, like, and we go back to previous states, you know, like, we have learned something new things have changed. And the world around us continues to change, you know, day by day, minute by minute. And so I tend to think in terms of like, what we've learned from like DevOps and like continuous delivery, you know, so how are we going to be able to like, fail forward, you know, like, what is it going to look like to be able to recover quickly? What are the things that we're going to want to notice? So it's, you know, we talked a minute ago about how are we going to have a hypothesis be testable? And, you know, what are the metrics? Those are going to be key things that we look at, after we have, you know, introduced this new innovation? What is it looking like? What do the metrics tell us, you know, what other things are starting to show up that we had not anticipated? And how can we like come together, and this is part of the the safety of like, we're all going to be able to come together and talk about and explore, like, what is it that we're seeing? What do we make of that? How do we want it to be? And like, how do we continue to, you know, roll with this, you know, be able to advance it, it could go in a different direction than we had originally expected. And that can be really helpful, like, now we do want to pivot, we want to bring it back over this way, or we're going to take it look straight ahead. But keeping the communication open, and helping everyone to ideally, look at the same information have a lot of transparency, is what I have found helpful as we are working through some of those risks and dealing with issues as they come up.

 

Dr. Dave  27:18

Yeah, you know, it's, you know, as a generative leader, you know, we're talking about cultivating a culture of experimentation, in our view. So from your perspective, you know, how can the generative leader instill a company wide culture of experimentation?

 

Allison  27:35

Yeah, I've worked with some amazing leaders. On this. I, I think, at the point where, once they start using the language more regularly, you know, of like, what are we? What are we trying to build, you know, or like, what's our hypothesis here? What are we going to measure? You know, like, what are we learning, like, that creates part of it, especially as they themselves are using an experiment approach to addressing business problems. Like that's where it becomes a bit stickier of the culture like this is now the new way that we do things. I had one client in particular that had latched on to sort of that like improvement, kata approach, right? That is very much that, like, we run the experiment, we see what happened. Now we like run the next one, but it kept going through that iterative cycle. Part of it was, how do we create a culture of feedback, that was something that they desired? So what were the experiments that they were running to, you know, create more feedback for themselves as leaders to be able to give feedback to others, how to get others to give others feedback, you know, it's like all these different directions that they would be able to tell you like, well, here's the thing that I'm experimenting on, like this month, you know, or this week, this is the thing that I'm trying this is what what I'm expecting or what I'm hoping for, you know, behavior wise, and how I might measure it. And they can tell you like, here's how long I'm doing that. I also see leaders that have brought in hackathons as like a practice in their organization. It's often like a special events. But it gives a way for the organization to all participate in something that's experimental. And most importantly, it gives the leaders things to be able to like point back to so if I now want you to act more experimental in your regular product delivery, I might kind of refer to some of the things that happened during the hackathon, or how a hackathon took place to evoke that new behavior, you know, like, import it from the hackathon into our every day. So what is the smaller thing that we can try? You know, how can we know it? In two days or less, you know what customers are responding to. So I find that generative leaders are finding ways to model the experiment themselves, but then also like help share that experience with others. That that they get a sense of like, what does it mean to run an experiment or participate in an experiment? And then how do we find more and more ways of applying this in so many different dimensions in our work?

 

Dr. Dave  30:30

So, on top of that, so are there strategies or approaches you found particularly effective in fostering this mindset of being a generative leaders?

 

Allison  30:43

Oh, my goodness. Um, so for me, I feel like a lot of what

 

Allison  30:53

what I kind of go back to with experiments is like lean startup, thinking, you know, how do we build? How do we measure? How do we learn? And it's not that I need everyone to go read the Lean Startup book. And, you know, I don't I don't need everyone to be experts in it, but just understanding that cycle, and, and I have also found it handy to have in my toolbox, you know, like, here's a lot of different experiments that can be run, that, that being able to propose to companies, you know, like, where might we want to use a landing page where my waiting list, give us a lot of evidence, you know, where can we go lighter and do some, like customer interviews? Or some, like customer shadowing? Like, what is it that they need to learn? And like, how, how accurate or like, how strong? Do we need data to be able to move forward in that general direction. So the, I would say the source that I found super helpful for that was testing business ideas, by David bland. That entire book, like it's such a handy reference to keep at your desk, because it very much is, here's a whole lot of different experiments that can be run. Here's, you know, like, what, like, what factor like what sort of like criteria? Are they testing? You know, is this the value of your product idea? Or is it the feasibility of your product idea, for example, and then also, like, the strength of the evidence. So I found like, opening up my eyes, and helping others to also see the wide variety of of techniques or again, like different kinds of tests that we can do as experiments, gets people thinking more creatively about, you know, how can we move forward in trying to make some innovative changes?

 

Dr. Dave  32:56

Wow. Wow, I just have to give you the hard questions. You know that right?

 

Allison  33:05

Of course, of course, of course.

 

33:08

So, I want to give you, you know, what do you want to leave the audience with, you know, you've said a lot of great stuff today. And I've learned a lot to go back and take after listening to this. And so what do you want to leave the audience with today?

 

Allison  33:24

Um, I think the, I think the core thing for me, again, of like, as, as a generative leader, right, as individuals, like, how are we showing up? And I think back to, like I said in the first question, so often, I thought I needed to be the expert, I needed to be the smartest person in the room, I needed to, like, figure out the design or the implementation, or, you know, like, how everything was gonna, like, look, initially and then propose it to people. And just like, logically, they would say, yes, it's so smart. And, and while some of that might have worked, really, and truly, like, that process becomes so slow and so cumbersome, and it does not engage people the way that we need today, as things are moving so fast, that we need to lean into how do we invite people to experiment? How do we run experiments ourselves, so that we are able to continue to move forward and be able to move quickly, and hopefully together? So I very much think of, you know, how to each of us look at what's the thing that I need to know, like, what do I need to learn next? What's the way that I go learn that? That's quick, that's cheap. That's that's safe, and use that to guide us to our next step.

 

Dr. Dave  34:54

So awesome. So in closing, I want to say thank you for joining me on this episode. sort of now I'll share with Dr. A podcast. You know, as we wrap up, I want you to remember this to innovate and thrive. We must be willing to experiment, evaluate and evolve. That's the mount the mantra of the Future Focus generative leader. And I wouldn't want to say just thank you for tuning in to the knowledge show with Dr. Dave podcast until next time, keep experimenting learning and leading from the center. I want to say thank you Allison always for being such an amazing partner. You know, and showing up for podcast and writing soccer my books and

 

Allison  35:36

I love getting to talk with you every time.

 

Dr. Dave  35:41

Yes, so much fun

 

Kayanna  35:46

Let's talk about it.  Talk, talk, talk! Let's go deep. We all have something to share. Knolshare with Dr. Dave.