Learner Engagement: Measurement of your efforts-Part 3

So now that we have touched on the importance of Learner Engagement and a few strategies to obtain stronger learner engagement, how do we measure all of this? In an outcome based environment, we need to be sure that are efforts are not for nothing and that we are accomplishing what we set out to do...or at least we are on the right path.

According to ATD (2016), when they surveyed organizations (not necessarily healthcare organizations) they found that 65% of those surveyed did not measure the business results of their learning investments. Those learning investments can vary from time investments, investments in human or tech resources, etc. Sharing the value of learning is becoming increasingly more important, especially as it pertains to healthcare but as Nurse Educators, we tend not to take the time to look at how our training and development impacts the organizations or the schools bottom line. In Nursing academia, one way to measure this is by NCLEX passing rates but are we looking beyond that? Are we looking at how our Nursing students that have graduated, are faring in the practice arena?

Going back to measuring engagement, this can be tricky because it is very challenging to quantify a learners body language or attitude as it relates to their learning experience. We need to keep in mind that it is not necessarily all about the quantitative results, quality matters as well. In the learning and development industry (and this includes Nursing Education), it is standard [and typically recommended] to use the Kirkpatrick Evaluation Model, developed by Dr. Donald Fitzpatrick in 1954. Still relevant some 65 years later, the Kirkpatrick model provides a vigorous model that examines the impact of learning, looking at both qualitative and quantitative outcomes. In 2010, Dr. Jim Kirkpatrick and Wendy Kirkpatrick augmented the four levels of the model to embody employee engagement.

  • Level 1: Reaction of the learner: What does the learner think about the education? To what degree did they find it engaging, favorable and relevant? Augments to this level were called "New World Additions" and Engagement was added to this level-the degree in which learners are actively involved and contributing to the learning experience- and Relevance-the degree in which learners will use and apply what they learned; transferring that knowledge obtained, to practice.

  • Level 2: Learning: The result of increased knowledge, skills, and attitude, based on their participation in the learning. The New World Additions to this were Confidence and Commitment

  • Level 3: Behavior: The amount in which the the learners applies what they learned when on the job; implementation, application. The New World Addition was Required Drivers: the processes that are in place to reinforce, encourage and reward improved job performance.

  • Level 4: Results: The effects on the business, outcomes that occur as a result of the training. The New World Addition was Leading Indicators: what observations were made suggesting that the learner is on track to making a positive impact

Another evaluation tool, created by Dr. Jack Phillips, takes Kirkpatricks 4 levels and adds a 5th level for ROI-Return on Investment-which is measuring the business impact of training and education. I will go into ROI in another blog but if you are in the Nursing Education and Professional Development industry, you should be all to aware that ROI is becoming huge for us.

In game based learning environments within Nursing Education, analytic reports from the digital gaming platforms were used, in addition to narratives, which hold just as much weight as the numbers, from the midterm and final course evaluations. In academia, achievement of learning outcomes is typically measured by the students grade in the course. But is that enough?

In an elearning environment, how do we measure engagement there? According to Shank (2019), while it may not be perfect, "proxies" can be used. Keystrokes can be recorded, websites that staff /students are visiting while completing elearning- a way of assessing "time on, time off" data, looking at the length of time the learner is watching your embedded video-did they watch the entire video, then take the post test or did they skip the video and go straight for the test questions?

To that end, it is extremely challenging to isolate the effects of the education and development we are providing, so we can correlate that to staffs' improved productivity and performance. Take a moment to ask yourselves how are you able to tell that the education you are providing is actually working? That its not because of other factors, in addition to the education provided, that has shown improvement in content retention and patient outcomes? Continue the conversation with leaders and staff and lets shift in the direction needed to see that theory to practice gap begin to narrow.

ProDevo can be a value added investment in your organizations Nursing professional development and education plan. Let's speak about how we can strategize and work together to improve your learners engagement and see the return on investment with improved staff satisfaction with their education, knowledge retention and knowledge transfer to patient care. Contact me for a free phone consultation*

*Free consultations are limited to 30 minutes



Davidson, S., Candy, L. (2016). Teaching EBP Using Game Based Learning: Improving the Student Experience. Worldviews on Evidence-Based Nursing, 13 (4), 285-293.

Desire2Learn (2016). The ROI of Learning: Why Measuring learning is key in a new world of work. [Whitepaper]. Retrieved from https://www.d2l.com/resources/assets/the-roi-of-learning/

HCPRO Staff (2017, June 22). Measuring the Effects of Nursing Education. HealthLeaders. Retrieved from https://www.healthleadersmedia.com/nursing/measuring-effects-nursing-education

Shank, P. (2019). Does Video Improve Engagement and Learning? Retrieved from https://elearningindustry.com/engagement-and-learning-does-video-improve

1 view