Measuring Engagement in Online Learning
People have mixed feelings about online learning. Many people have found useful content and built real skills online and even herald it as the future of all learning. Others have found online classes to be lackluster or even, frankly, terribly designed and developed.
When considering how to improve the eLearning experience, one concept that rises to the top is engagement. In an effort to avoid dry, dull learning experiences, we want to develop opportunities for learners to engage with the content in a positive way.
Engagement, however, is a bit of an abstract concept and can be difficult to measure. Some of the KPIs traditionally be used for measurement may not be as connected to engagement as once thought. It is worthwhile to look at these with a critical eye and think about how we can really tell if learners are engaged with online learning.
Time on Task
The first metric that came to mind when thinking about measuring engagement was “time on task” - or the time the learner was spending on the class itself. This is, in fact, one metric that has been used in the past. However, I soon realized that there are many reasons that a learner can spend more time on a learning experience, and many of these have no connection to engagement such as:
Slow pace of instruction
Complexity
Distraction
Confusion, such as needing to reread text or watch video clips several times
Poor fit of difficulty level for the learner
If the users are taking a class or training voluntarily, perhaps these factors would cause them to exit early or start skimming the material. However, if the training or course is required, they may spend extra time trying to overcome these barriers. That extra time should not be counted toward engagement. Though the learner may be engaged with trying to figure out confusing content, that it not the type of engagement that we are hoping for when designing learning experiences.
Similarly, there are reasons that a learner might spend less time in a course, that are actually positive:
Efficient learning
Concepts made easy to digest
Concise language
Learner background knowledge
Therefore, time on task does not seem to be a good indicator of student engagement, at least not the kind of engagement we want as instructional designers. So what other possibilities can we look at?
User Surveys
One way we can determine a learner’s engagement is to just ask them! With a carefully worded survey, we may be able to gain a lot of insight into how engaging the learning experience was for the users. Of course, this is somewhat subjective. Some users may rate the training highly because they enjoy the subject or because it had a lot of fun bells and whistles. Still, compared to the other options, it is a strong contender.
Sharing
Another indication of an engaging learning experience is how much the user shares their experience with others. This could include sharing their accomplishment, recommending the course to others, or referencing the course in other contexts.
Follow-up
Did the learner sign up for the next course? Did the user come back and consult the training at a later date? Follow up would be a good indication of the quality of the learning experience. This might not be the same as engagement, but I believe that engagement and quality usually go hand-in-hand. The exception would be if there is no other resource for a particular skill or concept, then follow up might just show that the learner had no other option. However, now that the internet is replete with information, it is fairly unlikely for one course to be the only available source of learning.
Behavior Change
At least in the case of training, the ultimate goal is to change behavior after the learning experience. Metrics can be gathered on how the learners are using what they learned in the workplace. Managers can be contacted or polled to find out if they are seeing a difference in employee behavior after the training.
All of the Above
Because there is no one metric to look at to determine the engagement of users of an online learning experience, it makes sense to look at many metrics. To truly determine engagement, it makes sense to gather data, feedback, and opinions to get a full picture. Only with multiple metrics can we get a balanced look at engagement. Once we do, it is up to us to use that information to develop more engaging courses in the future.