Complete training evaluation with Richard Griffin

Richard Griffin is a training-evaluation expert and author of the book “Complete Training Evaluation: The Comprehensive Guide to Measuring Return on Investment.”

In this 30-minute interview, Richard discusses several new approaches to learning evaluation, and explains how to collect and use data to assess training effectively.

He also highlights how to overcome the challenges of calculating ROI on social learning and e-learning, and he tells us why it’s important to get all stakeholders – not just trainers and learners – involved in evaluating training. Plus, much more.

Once you’ve listened to the interview, tell us what you thought in the comments, below, and share your own experiences and insights on training evaluation.

Listen today: Click here to download the podcast
 

Interview transcript

Rachel Salaman: Welcome to L&D Insights from Mind Tools. I’m Rachel Salaman. Billions of dollars are spent on workforce training every year, but how much of that do we ever get back? It’s always tricky measuring return on investment with something as intangible as learning, especially when you take into account the bigger picture. After all, training one person can impact the productivity of a whole team or unit, but how can that be proven?

I went to talk to Richard Griffin, Director of the Institute of Vocational Learning and Workforce Research at Buckinghamshire New University in the U.K. An economist by training, Richard has 30 years’ experience of workforce redesign, skills development, public policy analysis, and employment relations. He’s also an expert in training evaluation and he’s brought a lot of his insight and practical tips together in a new book called “Complete Training Evaluation: The Comprehensive Guide to Measuring Return on Investment.”

I started by asking him whether it was actually possible to measure the return on investment in training.

Richard Griffin: The short answer is yes, absolutely. I think one of the motivations of the book, was this kind of paradox that we invest a huge amount of money in training and a huge amount of resources – something like £45 billion a year in the U.K. But very few organizations, less than 3 percent I think, evaluate the return on that investment. And I don’t really understand that, because it’s not as difficult as people believe it can be. So, if we know why we’re doing the training, we know why and how we can evaluate it, so absolutely we can do it and we absolutely have to do it.

Rachel Salaman: What’s your definition of training evaluation? That might seem obvious to some people, but you point out in the book that it means different things to different people.

Richard Griffin: It does. I think it raises a straightforward answer, that it’s about the collection and presentation of information on the effect of a training program. But I think we’ve really seen that in a very narrow sense, and what I’m trying to get over in the book is that we need to have a wider, more comprehensive view about what evaluation can do.

So, for example, it’s not just about the training event. In fact, if anything, the training event – whether that’s e-learning, self-directed learning, classroom teaching – is a small part of what the effect of training is. And what “Complete Evaluation” does is say we need to take a much wider perspective – we need to think about the effect of training before it begins and long after it’s finished. We might talk about this a bit later, but a lot of training isn’t transferred into improved performance. We also need to understand training in the context of the workplace and what happens within the workplace, and move evaluation out of just being about the training event and integrate it much more in the view of the whole workplace.

Rachel Salaman: You’ve touched on this a bit, but why does evaluation matter?

Richard Griffin: It matters because we invest a huge amount of money in training and development. We also have a massive belief in it. In the last decade, across pretty much every economy, there’s been a comprehensive view that the key to improved productivity, investment and skills is investment in knowledge, skills and attitudes. But we don’t have the evidence that this is the absolute truth. We have a massive belief in it, but, if you pin people down, if we think, well, if I had to go and justify to my chief executive or to whoever whether the training has made a difference, what the evidence shows us is that most people can’t do that because we don’t think about the impact. And there are consequences to that. One is that is that training and development departments aren’t seen as key levers of productivity improvement; what they’re seen as is quite low status. In other words, it’s a good thing to do, but it’s not seen as a critical lever in many organizations.

So, a big reason why we need to evaluate training is to get the evidence that training is a good thing. But it’s not always good – we don’t always get it right and, if we don’t evaluate it, we don’t know whether we’re getting it right or not. This means that we could be putting money into programs that aren’t delivering the goods. But I’ve also talked about it being part of a bigger picture of training, not just about the training and delivery itself. We do need to know whether we’re getting training delivery right, so it can help us. Even if we don’t need to explain the impact of evaluation beyond the training function, we can do it through evaluation to make sure that we’re getting that delivery correct.

Actually, even knowing whether the coffee is good and whether the room is good is important if you’re a trainer. So there’s a huge amount that evaluation can do and must do because we do have this paradox – we’ve got this massive investment, this massive belief and very little evidence, and training has held up well during the recent recession, better than probably a lot of people thought it might do. But I worry that if we don’t get that evidence, we may see some of that belief in training pulling back and companies saying that it isn’t worth the time, money and investment.

Rachel Salaman: In the book you make the distinction between approaches and methods, and you credit Donald Kirkpatrick with laying down the foundations of most training evaluation approaches that have followed his classic approach, which was devised in 1959. Could you describe his model and also bring us up to date with some of the approaches that have been used since?

Richard Griffin: Yes, it’s incredible, I think the genius of Donald Kirkpatrick, who we unfortunately lost very recently, was to simplify a very complicated process. He broke training down into reactions. What do people think about the training? Have they learned anything? Has anything changed as a result of the learning? What’s the impact of that training? And he did that in 1959, and still, 50 years on, the vast majority of approaches to evaluation that are carried out use the Kirkpatrick approach.

I talk about approaches because what I think there’s a lack of is “how to” advice, so what Kirkpatrick did is say: “Well, this is what training is about. If we want to evaluate it, these are the things we need to think about.” He was very clear that there are four levels, and what we do is focus primarily on the reaction level, not the further levels. As you move from level one to level four, the amount of training evaluation declines, and I think the Kirkpatrick Partnership has recognized that, because the model has not stood still, and they’ve relatively recently refreshed it and now call it “New World Kirkpatrick.” They’ve almost turned it on its head and said that we need to think about the result bit of it first – what is it that we’re trying to achieve through the training? And then think backwards about how to gather the evidence for stakeholders that we’ve actually achieved that. So, reactions still matter and we’ll talk about this later, but the key bit is, “What difference has it made?”

So, the Kirkpatrick absolutely dominates the training evaluation field and I’ve got a lot of time for it. Academics have been quite snooty about it and I don’t share that. I think we wouldn’t be where we are now if it wasn’t for Donald Kirkpatrick and the work he did. But what he didn’t do is go into a lot of detail about how you gather data, and the world of work has moved on. It’s very much a model geared towards formal planned training, and a lot of learning in the workplace actually isn’t planned – it’s informal, but you can’t understand training and evaluation if you don’t understand Kirkpatrick. I did look at, for the book, other approaches to training and evaluation, and it’s another paradox because there’s a mass of them. If you put “training and evaluation” in the search engine you’ll millions of responses. I do touch on some of them in the book – some of the ones I think offer interesting insights. A lot of them are actually variations of Kirkpatrick and it would be interesting to see when people look at New World Kirkpatrick what the impact of that is because I think that while it looks good, we haven’t had any research on it yet.

Rachel Salaman: So how has workplace training changed? What are people learning and how?

Richard Griffin: I think firstly training deliverers – L&D professionals – have been brilliant in innovating. So the way we learn now compared to even 20 or 30 years ago is an absolute transformation, especially when we’re thinking about things like simulation, gaming, e-learning, and the use of social media. Even where we learn. For instance, we can learn on the train into work. So, L&D have been absolutely great in being innovative, in addressing changes, and in embracing technology in the way we learn.

The world of work has also changed, and I talk about when my father started work in the 1950s, jobs were very standardized – it was very clear what your role boundary was. For instance, you might only have one or two jobs in your life. My dad had two jobs through the whole of his career, and you had what are called very closed skills and knowledge required to deliver that job. So it was very fixed and training in that circumstance was quite straightforward, or appeared quite straightforward. So if my dad went off on a training program, he’d go into a classroom and he’d learn something. He’d fill in a questionnaire at the end and he’d go back and carry out the job as he did before, but with that new knowledge.

If I think about my own career, and I think about the world of work now, it’s an absolute transformation. There’s much more emphasis – quite rightly – on softer skills. Rather than having closed skills, it’s much more open – jobs are much more flexible. There’s a recognition also, going back to my father’s day, that learning at work was only ever thought about in terms of formal learning. So like those training days you’d go away to, now we recognize that the majority of learning in the workplace is informal. So you’re far more likely to learn something over a coffee in a coffee break or over the water cooler than you are in a training program. What we recognize now in work is that we need to facilitate that – we need a much more expansive learning culture in the organization. However, training evaluation is focused very much on that old way of looking at work, so it focuses on reactions at the end of a training program. It doesn’t bring itself into the wider workplace. It looks at formal learning, not informal learning. But it can actually look at all of that. It must do, because it all links, and one of the messages I try to get over in the book is that workplace learning now is a partnership – it’s a partnership between the learner, the training, and the learning and the workplace. We need to see it as that collective whole to understand it and evaluate it correctly.

Rachel Salaman: We can’t talk about evaluation without talking about data, which is a term some professionals might find a bit daunting if they don’t have a background in social science. What do you mean by data?

Richard Griffin: It’s simply information that can tell a story about the impact of the training. This information can be presented in a number of ways. It can be presented in the form of numbers, which is probably the way we’re most familiar with; it can be presented in the way of words, which I think is incredibly powerful, more powerful than we probably imagine; it can be in the form of images, pictures, which actually are quite surprisingly powerful sometimes; and, it can also be presented by videos – moving images – as well.

So, we need to think about data much more widely than we have done traditionally in training evaluation. One of the things I wanted to do was bust the myth that this research stuff is difficult. You can do a PhD in research methods if you want, and you can buy really thick books about how to do focus groups and what have you. However, what I wanted to get over in the book is that you don’t need to do that – you can get good, credible information without having to do all of that. It’s not as hard as it sometimes may appear, so don’t be put off. But also think more widely – data is not just about numbers; there’s other ways we can gather information about impact.

Rachel Salaman: So what are some of the ways that data can be collected?

Richard Griffin: I think the most common one is a survey. I think what we need to do about surveys is think about when we issue them to people and what we say in them. But they are a very effective way to gather information, and you can do them online as well as in paper form of course. But also, we can use words through interviews and focus groups, and we can use images.

For instance, when I was looking at someone else’s training program for the book, I walked into this room which was absolutely covered with images and pictures. The trainees had been asked to go and look at the images and pick one that described how they felt about the training they’d just had. It was talent management training actually, and I was a little bit skeptical when I saw this and I wondered how good this was going to be. The people were spending about 15 minutes looking through the images. They each took one back and then described why that image said something about the training they’d just completed. It was all put on a board together, and it was a really great way to focus people’s thinking. It really got me thinking about how we could use pictures to tell a story about what the effect of the training is.

But there are observations as well. There’s also feedback, diaries and logs. There’s a lot of ways we can gather information. I think sometimes perhaps we rely on one method more than we may need to, so I wanted people to think a bit more widely about how we can gather the information we need.

Rachel Salaman: How does an HR or a learning development professional choose between the different types of data, how do they know what would be best for their purposes?

Richard Griffin: I think it almost goes back to the first question. The critical thing is understanding what the purpose of the training is and being really clear what the expected outcomes are. Some of that may be very straightforward. So, if it is telesales, then it may be very obvious that the best way to see whether a difference has been made is whether sales have increased after the training.

If it’s leadership though, or diversity training or team working, then that’s a slightly more difficult concept in terms of numbers and it may be better to get information based on people’s description of what’s changed – not just the trainees, but also the people who work with them. But there’s a degree of pragmatism here as well. People aren’t doing PhDs, so what resources do they have, and critically who is this for, who are the stakeholders, what do they expect, and how much time do we have? It may actually need just a series of telephone interviews with a few trainees to get the information that you need. So, I’m not prescriptive – it’s not one size fits all. There is a degree of pragmatism in this, but I think if you recognize that there’s a range of approaches that you can use – a range of methods – then some of those may lend themselves to different training programs more readily. That gives you a toolbox that you can dip into depending on the program.

Rachel Salaman: In terms of analyzing the data, there might be a perception out there that things like words and pictures are a little more complicated to analyze than numbers which you can put through a software program. What are your thoughts on that?

Richard Griffin: You can get software programs that do analyze text. What it would do is go through and find common themes and it will pick those out. I do occasionally use those, but what I tend to do is listen – you record an interview, and you think very carefully about the questions you want to ask, of course. It’s critical to ask what difference has the training made, and that’s maybe exactly the question – an open question you ask. Or, you might be very specific about a particular outcome you’re expecting from the training. But what I find is that if you just listen, and if you do a one-to-one interview on the training program, it may not take longer than 10 or 15 minutes. Or, if you do a focus group, where you bring a number of people together and ask them as a group a question, it shouldn’t take longer than an hour. So rather than typing out the whole script, what I find is, if you just listen to it you’ll hear what’s being said. You’ll pick the things out, and you’ll spot a good quote which you can transcribe. So I find that a much richer way of actually engaging with the views on the training program. Then when you present that, you can identify the outcomes that are being delivered and you can provide examples. So that’s why I think words are quite powerful in a way that numbers aren’t always, because people can describe what has happened.

Also, and this is really important, we don’t always get training right. Not necessarily because the training hasn’t been good, but because other things get in the way. So we know for example that if people learn something in a training program, but then there’s a long gap from when they learn it to when they can apply it in the workplace, then they’re likely to forget it. So it’s not because the training is not good – it’s because they’ve not had the chance to practice. We know if their managers and peers aren’t supportive, that can also have an impact on the effectiveness of training. So by talking to people about the effect of the learning we can actually pick up some of that stuff as well. So it was really good, but I was too busy to go back in the job and do anything with it. So that gives us additional information. You may not have thought about using words, and you may not have thought about using pictures. But just try it and see – it’s not hard, and the book goes into advice about how you can do it and structure and plan it. But it doesn’t take as long as you might think.

Rachel Salaman: I suppose it depends a little bit on how you are going to ultimately present the results or the impact of your training because if your stakeholders want to see pie charts and graphs that’s easier to create with numbers than words.

Richard Griffin: It is, and what I suggest is you combine methods. So at a very basic level I would always ask people to rate on a scale of, say, one to 10, how effective they thought the training was in terms of expected outcomes. So you’d always have a numerical way of describing effectiveness. But, training is this complicated, complex relationship between lots of different things and being able to delve into that which words allow you to do and conversations allow you to do actually does bring a lot more out. And what I found, even working with quite large companies, is presenting a series of quotes and descriptions about the difference that a trainer has made actually does have an impact in a way that graphs or whatever don’t always do. I’m not advocating one route or the other. It will depend, but you may be surprised how powerful that can be.

Rachel Salaman: Tell us about happy sheets and what we might consider using instead of them.

Richard Griffin: Essentially, they’re just asking people what they thought of any event – in this case, how satisfied you were with the training. They are by far the most common method that L&D professionals use to evaluate training, and I think everyone should ask themselves what we’re using them for, why we’re using them, and what they tell us. This is because what we know is actually being satisfied with a training event is no predictor of whether that training is effective. Actually, it may be that effective training creates quite opposite emotions. If we learn something new, that can often be a bit of a struggle. Sometimes, we’re challenging our own beliefs, so asking people whether they’re satisfied with training in terms of impact has no predictor factor whatsoever. It doesn’t tell us anything at all about what happens when people leave that learning event. It may tell you something about the classroom etc and that may be useful, and if people are satisfied and happy when they leave then they can be good ambassadors. Though we need to be careful – there’s always a euphoria effect at the end of training: “I’ve spent my three days doing this, I’ve finished now so I feel quite good about that, so we need to be careful about that.” That does not mean that we can’t find useful things out during the training, and what I discuss is the concept of “utility reaction.” That’s not my idea, what I’ve tried to do in the book is base it on research so it is evidence based. Rather than asking people if they are satisfied with the training, what you need to ask people is, is how useful they perceive it, to what extent can they relate it to their workplace, and to what extent does it meet their expectations and needs.

If we have good utility reactions, then that’s a predictor of whether the training is going to be transferred, and it sounds a bit obvious actually: ask people whether they think the training about their job will make a difference to their job and if they say “yes,” that’s because they can see how they can apply that learning and what difference it will make. Then they’re likely to be motivated to go and do that. That sounds really obvious, but it’s very simple and we just don’t seem to do that very often.

Rachel Salaman: What are the best ways to evaluate the learning outcomes of training and how long a view should we be taking with these?

Richard Griffin: We’ll take the second part of the question first. This notion of learning transfer and decay is really important – there’s a very consistent figure that appears in research where people have tried to investigate the extent to which learning has been retained over time, and it shows that on average only 10 percent to 15 percent of what is learned in a learning event is actually transferred and retained. That means something like 85 percent of learning is actually being wasted.

So it seems to me really critical that we do need to take a longer term view, and I would suggest that we don’t really know exactly the best time to do this. But there’s a kind of sense of somewhere between three to six months after training where it’s worth going back and asking the trainees, the learners, what difference they think the training is still making and asking them to have retrospective views on how effective they thought the training program was. We need to do that first so we can see whether that training has made a difference and retained a difference over time. But if it hasn’t, what actually is getting in the way of it? Was it that the training wasn’t the right answer to the performance problem? Was it that those other things got in the way and people weren’t able to transfer the training? Was there something before the learning required?  I talk a bit about learning readiness, making an assumption always that people are ready to go into training, and that it will work and it will be fine. We just need to do it right, but actually understanding whether people are learning ready, which is something that evaluation hasn’t traditionally been seen as something that it can do, is something it can do – it can ask that question, “Is this the right cohort for the training we’ve got planned?”

So, by evaluating training transfer we can not only find out whether it has worked, but if it hasn’t. This is as important, and we shouldn’t be scared of failure if we learn what the problems are. I think this is the big new frontier for training evaluation, and we can look at what’s preventing people transfer. Now, it may be because it was the wrong answer to the question and we sometimes think that training is a default to any performance issue. But it might not be – there might be other things that would help, or maybe we have a really poor learning culture in the organization, which counters the effect of the learning. So we need to understand that. And there is an unintended benefit, if you ask people six months, a year on, whether or not – it has an effect of actually making them think again about the training. So there’s a bit of a performance boost just by doing that evaluation. People use what are called relapse strategies, so I’ve forgotten something, what do I do about that? Well tough, I’ve forgotten it, or actually do I go back and think about what I learned. So are people doing that? And managers are really critical here – the extent to which they support staff to learn and apply that learning, whether they’re talking to their staff about the learning they’ve been on, whether they’re setting goals related to the learning, whether they’re getting the opportunities to apply the learning. Evaluation can look at all of those things, and we don’t do that and it’s not that difficult to do.

Rachel Salaman: In the book you talk about the importance of getting stakeholders involved in training evaluations. What do you mean by stakeholders here and how should they be involved?

Richard Griffin: A stakeholder is anybody who has an interest, who is affected by the training program. Now it’s obviously the trainees and the learners. It’s also the trainers, but it’s other people as well – it’s the HR directors, it’s the managers, it could be the other staff, it could be the finance director.

Why they matter is that ultimately they are ones who will make judgments about the training program and they might have different views. It may be that as L&D professionals, we are really clear about what we think the training should deliver. What I’ve found once I’ve identified the stakeholders and started talking to them about what they expect from the training, is that their expectations can be quite different, and it’s really interesting that they have variant views on what this learning should achieve. Identifying those at the start gives you the framework for what you then need to look at in terms of the training’s effectiveness.

Now some of that may actually be just misconceptions, and that’s a great opportunity for evaluation to clear misconceptions up. But it may be legitimate that there’s a variety of views about what the emphasis of the training’s outcomes should be, so by identifying stakeholders and asking them what they believe the outcomes should be you discover if there are variants. Then you can share that among the group of stakeholders so you try to build a consensus about the most important ones. The stakeholders need to be kept informed throughout the evaluation, and they can help with it, because sometimes if you need to set up interviews or whatever, having someone more senior in the organization backing you because they understand the importance of the evaluation, and then being involved in it, and then continuing to be involved in it, can be really useful because they can help get people to clear their diaries and meet you or fill the surveys in. If you can get a person at the top of the organization, all the better.

Rachel Salaman: What particular challenges to training evaluation do things like e-learning, self-directed learning and social learning pose?

Richard Griffin: I think e-learning is a gift because one of the challenges sometimes is getting good response rates, so with e-learning they’re already there, because they’re on the screen and they’re doing the learning. So response rates are pretty much 100 percent. That makes the actual mechanics of delivering an evaluation much easier for evaluators. Social learning, informal learning, is probably the most prominent way that people learn in the workplace. The majority of what we learn, for good or bad, is probably through informal discussions, observations, actions and it’s an area that training evaluation hasn’t really touched but needs to and can do. I describe how to in the book, and it’s really powerful. I discuss an example with the Bank of America where it was a call center and the way it worked was that people had very little interaction with each other. But just by simply understanding that it was not possible for people to share insights and observations about work, and change that through just giving people informal breaks where they could come together led to a $15 million increase in profitability. That gives kind of an indication of the power of informal learning. In terms of self-directed learning, we can use reflective logs, we can use diaries, and we can use peer evaluation so we can ask colleagues what they think the difference has been. But that’s quite challenging if people feel that “I’m not doing so well, I’m struggling a bit, I’m not sure I want to reveal that.” That’s again why we need to be much more supportive of learning in the workplace.

Rachel Salaman: So what are your key takeaway tips for HR and L&D professionals looking to increase the usefulness of their training evaluation?

Richard Griffin: I think recognize that there’s not one size fits all, that there are a number of techniques you can use. Be creative, be pragmatic – they’re not difficult to use.

Secondly, stakeholders do really matter, so engage with them, identify them, work with them. Thirdly understand that the bottom line is that training is a partnership between the trainee, the training and the workplace. You need to understand all those three things to understand whether training has been effective or not. But most important is do it. The best way to try and do it is to do it and we need to know more about the effectiveness of training – the return on that investment – because if not there is a real risk there that we will see that optimism we’ve got, that positivity we’ve got about training, start to ebb away, and that would be an absolute tragedy.

Rachel Salaman: Richard Griffin, thank you very much.

Richard Griffin: Thank you.

Rachel Salaman: The name of Richard’s book again is “Complete Training Evaluation: The Comprehensive Guide to Measuring Return on Investment” and it’s a very clear, practical guide to the complete training evaluation process. There’s more information about return on investment in learning and development on the Mind Tools site. Thanks for listening.