Learning Analytics: Long Live Smile Sheets

Despite persistent emphasis on more sophisticated learning analytics, post-learning satisfaction surveys — often derisively called “smile sheets” — continue to be the most common method of acquiring feedback on learning initiatives.

A majority (78 percent) of learning executives surveyed indicated they continue to use satisfaction surveys for post-learning assessment, according to a report analyzing survey data from the Chief Learning Officer Business Intelligence Board (BIB) set for release next week.

In January, the editors surveyed the BIB, a group of nearly 1,500 professionals in the learning and development industry, to assess and benchmark learning analytics practices. The full analysis of the results will be released in “Focus on Learning Analytics,” a Chief Learning Officer report.

In addition to smile sheets, other common post-learning assessments include formal testing, reported by 41 percent of the BIB, and anecdotal observation, reported by 36 percent. While these remain the most used methods, the BIB reports their use actually decreased from 2009 to 2010. In 2009, 86 percent of survey respondents indicated they used satisfaction surveys; 49 percent used formal testing; and 45 percent used anecdotal observation.

Many practitioners and consultants in the learning and development industry, including this magazine, have lobbied for more sophisticated measurement methods that go beyond use and satisfaction to the effectiveness of learning, its impact on employee productivity or its ultimate impact on organizational goals. According to BIB survey data, that movement remains a work in progress.

A majority of respondents (61 percent) report learning output — such as number of courses, number of students and hours of training — to executives in order to demonstrate learning’s impact on the enterprise. Nearly 49 percent reported student or stakeholder satisfaction, but only 9 percent reported impacts on employee productivity.

Output and satisfaction are relatively easy to measure, which is why many enterprises frequently use them. But their continued use has not significantly improved overall satisfaction with learning measurement efforts. When BIB members were asked how satisfied they are with their organization’s L&D measurement, a minority said they were satisfied or very satisfied (37 percent). Nearly 40 percent indicated that they were dissatisfied or very dissatisfied with measurement.