People are notoriously bad at self-evaluation 

As learning professionals, our ultimate goal for employee training is behavioral change. We want to develop the behaviors that drive performance and increase competitive advantage while discouraging the behaviors that reduce productivity, increase risk exposure or damage culture. But on-the-job behavior is hard to measure, and observing people as they work can cause behavioral changes on its own (what organizational psychologists call the Hawthorne effect¹).

As a result, many talent professionals turn to self-evaluation questions such as: “How often did you use the new training in the last week?” or “How relevant is the training for your day-to-day work?” to evaluate whether an initiative actually changed behavior and delivered a positive business outcome.

Unfortunately, people are notoriously bad at self-evaluation, this is particularly true with a soft skills assessment. People new to a skill are highly prone to overestimate their abilities: organizational psychologists refer to this as the Dunning-Kruger effect².

Even if they do have an accurate understanding of their skills, leading questions (and questions that suggest the desired answer) cause employees to consciously or unconsciously distort their answers to comply with the lead. Social psychologists designing experiments refer to these as demand characteristics³. For example, a question that asks “How relevant was the training?", contains an embedded assumption that the training was somewhat relevant.

In addition, people’s conscious and unconscious desire to see themselves, and present themselves to others, as highly competent and skilled workers, inflates people’s estimates of their abilities. Social psychologists refer to this as the social desirability effect⁴.

A perfect storm of distortion

When employees are asked to assess their own proficiency in soft skills, all of these factors (Dunning-Kruger, demand characteristics, and social desirability) combine in a perfect storm of distortion.

This is illustrated with a snapshot of data from 27,728 people who completed 83,932 ETU simulation sessions. Before these particular simulations, people were asked to predict how well they would perform on various soft skills and tasks. They then completed the simulation, and their behavior was automatically measured according to those same soft skills.

Figure 1 shows how peoples’ self-reported skills (their predicted performance) corresponds to their demonstrated skills in the simulations.

Out of 431,871 prediction/behavior pairs, only 23.2% of simulation performances were within 5 points of the prediction.

Figure 1. Ability based on behavior and self-assessed skills.
The grey line represents accurate self-assessment.

Overconfidence drops as skills increase

As Dunning and Kruger predicted, overconfidence drops as skills increase - this is very clear from Figure 2. Someone with low ability (who scored 20 in a simulation) will overestimate their skills by a median of 189%. Someone with high ability (who scored 70 in a simulation) will still overestimate, but by only 12%.

Most interestingly, people who have mastered a skill (score of 100 in the simulation) actually tend to underestimate their skills by 12%.

Figure 2. Self-assessments become more accurate (closer to ability
as measured via simulation decisions) as abilities improve.

When training is focused on developing skills or behaviors, self-reported measurement is inaccurate at measuring transfer, impact or change in response to the training.

Measuring behavior change is critical to successfully evaluating and shaping our learning programs and initiatives and ultimately driving our organizations to succeed and excel in a competitive world. The facts are clear: to truly measure the impact of learning in our organizations we have to accurately assess skills and behaviors. ETU can help accurately measure skills and behaviors in an objective, standardized and scalable way.

¹ Hawthorne Effect
When employees know they are being observed, their behavior changes in ways that can affect performance.

² Dunning-Kruger Effect
Ability is inversely related to self-perceived competence.

³ Demand Characteristics
Word choices and behavior by assessors away what answer is desired or expected.

⁴ Social Desirability Effect
Behaviors driven by the desire to conform with the expectations of others.

This research article is an amended extract from an earlier white paper from ETU, authored by Dr. Bethany Kok.