Accounting has four types of measures — revenue, expense, assets and liabilities — and three standard statements — income or profit and loss, balance sheet and cash flow. What do we have in learning and development that is similar?
Five years ago, a group of industry leaders came together to consider a standard framework for learning. The group included Kent Barnett, then at Knowledge Advisors; Tamar Elkeles, then at Qualcomm; Jack Phillips from the ROI Institute; Jac Fitz-enz from the Human Capital Source; Josh Bersin from Bersin & Associates; Cedric Coco from Lowe’s; Karen Kocher from Cigna; and a host of others — 30 in all, including myself. After 24 revisions, we all agreed on a framework that we believed would provide the type of standards accountants have enjoyed for some time and that would help us become more professional.
These leaders recommended a framework consisting of three types of standard measures and three standard reports — three and three vs. the accountant’s four and three. The three types of standard measures are effectiveness, efficiency and outcomes. The three types of reports are operations, program and summary. The goal was to keep the framework simple and easy to use. We also wanted to build on the excellent work done in our profession over the past 70 years, especially that done by the Association for Talent Development, then called ASTD.
Effectiveness measures address the quality of a learning program or initiative. In learning, we are fortunate to have the four levels popularized by Don Kirkpatrick and the fifth level, ROI, popularized by Jack Phillips. Level 1 measures participant or sponsor satisfaction with or reaction to the learning — certainly an initial measure of quality. Level 2 measures the transference of skills or knowledge, and Level 3 measures the application of that knowledge or change in behavior. If they didn’t learn anything, or if they don’t apply what they did learn, we don’t have a quality program. This may reflect a lack of engagement or reinforcement from the sponsor, but we still have a problem. Level 4 measures impact or results; if there are no results it’s hard to argue in favor of program quality. Lastly, Level 5, ROI provides a return on investment, a final confirmation of quality assuming we have properly aligned learning to organizational needs and achieved high quality on the previous four levels. Most organizations have level 1 and 2 measures but few measure at the higher levels.
The most common efficiency measures include the number of courses, participants and classes as well utilization rates, completion rates, costs and reach. Typically, these do not tell us whether programs are efficient. We need to compare them with something else, such as last year’s numbers, historical trends, benchmark data, or the target we put in place at the beginning of a program. Now we have a basis to decide if we are efficient and if there is room to improve, to become more efficient.
All organizations have efficiency measures. The most common are number of participants, classes, hours as well as some cost measures like cost per learner or cost per hour. That leaves outcome measures. Unlike effectiveness and efficiency measures, most organizations are not measuring outcomes and few even talk about them. That is unfortunate because outcome measures are the most important of the three types, especially to senior leaders who make the funding decisions for learning.
In accounting, this would be like reporting expense and liabilities but never talking about revenue or assets. No one would have a complete picture of what we do, and it would be hard for anyone to understand why we have expenses and incur liabilities. These all-important outcome measures represent the impact or results learning has on an organization’s goals or needs.
Suppose a needs analysis indicated your salesforce would benefit from a consultative selling skills program and product features training. You and the head of sales agree it makes sense, and the two of you further agree on program specifics, including mutual roles and responsibilities, including how the sponsor will reinforce learning and hold her own employees accountable. Now, what sales impact can we expect from this training? How much of a difference can training make? A lot? A little? Enough to be worthwhile? This is the critically important realm of outcome measures that can sometimes — but not always — be subjective.
Sometimes the Level 4 results from the list of effectiveness measures will do double duty as an outcome measure. That’s OK; sometimes the same happens in accounting. Or, other measures will be selected. In any case, with outcome measures we are at last focused on how we align with corporate goals or needs, and what type of impact we can have, and this is what your senior leaders have been waiting for.
Next month, we’ll look at the three reports and how the three types of measures populate them. In the meantime, let’s start talking about these three types of measures. It would be big step forward if we could adopt a common language — the precondition to be a true profession.
