Good Data or Bad Data: The Idiosyncratic Rater Effect2018-08-15T09:15:01+00:00

Good Data or Bad Data: The Idiosyncratic Rater Effect.

Read Marcus’ article in HBR to learn how to solve for IRE

Performance management redesign. It may sound boring, but it’s actually everything. Every employee is affected by your performance management system. It determines how they get paid, promoted, evaluated. It touches everybody.

We all know this, and it’s why in recent years many companies have taken radical steps to improve their performance management. One bold step many companies have tried is getting rid of ratings altogether. Recently a CEB report indicated that companies who did so are doing worse than companies who have kept them. But the whole debate about ratings vs. no ratings is a complete red herring. The question isn’t ratings or no ratings. It’s do you want good data or bad data?

Ratings generate bad data. The reason why has been known for decades. It’s called the “idiosyncratic rater effect.” Big term, but it simply means that we human beings are unreliable raters of other human beings. All the data show that when I rate you, over 60% of your rating is about me, and not you. Which is a big problem, because we are paying, promoting, training, and deploying you based on that rating — which doesn’t reflect you. So ratings are a terrible performance measurement tool, because they claim to measure one thing but actually measure another.

So, by all means, throw out your current rating system. But don’t do it because people hate ratings. Do it because ratings are bad data. It comes down to a measurement question, and nobody seems to be trying to answer it: how do we reliably reveal the true range in performance in an organization so that we can invest differentially in our people? That’s the question to ask.

2 Comments

  1. Berta Serrano February 21, 2018 at 3:56 PM - Reply

    Thanks for this post, very Interesting!
    But now what? performance measurement is a must, what we can´t be measured, can´t be improved!
    Do you have any suggestions?

    • Meredith Bohling February 22, 2018 at 8:37 AM - Reply

      Hi Berta,

      Great question! The thing is, people can be reliable raters – but only of their own experiences. Definitely check out our Data Fluency Series, especially How to Tell Good Data from Bad Data, Do Current HR Tools Give Us Good Data and the Case Study which goes through a tool that does provide good, reliable data on talent – but only by allowing people to measure their own thoughts, feelings, and experiences. Enjoy! And please comment or let us know if you have more questions!

Leave A Comment

Leadership, HR, and LOVE+WORK content every week. British accent included.