Share This Story

Good Data or Bad Data: The Idiosyncratic Rater Effect.

Read Marcus’ article in HBR to learn how to solve for IRE

Performance management redesign. It may sound boring, but it’s actually everything. Every employee is affected by your performance management system. It determines how they get paid, promoted, evaluated. It touches everybody.

We all know this, and it’s why in recent years many companies have taken radical steps to improve their performance management. One bold step many companies have tried is getting rid of ratings altogether. Recently a CEB report indicated that companies who did so are doing worse than companies who have kept them. But the whole debate about ratings vs. no ratings is a complete red herring. The question isn’t ratings or no ratings. It’s do you want good data or bad data?

Ratings generate bad data. The reason why has been known for decades. It’s called the “idiosyncratic rater effect.” Big term, but it simply means that we human beings are unreliable raters of other human beings. All the data show that when I rate you, over 60% of your rating is about me, and not you. Which is a big problem, because we are paying, promoting, training, and deploying you based on that rating — which doesn’t reflect you. So ratings are a terrible performance measurement tool, because they claim to measure one thing but actually measure another.

So, by all means, throw out your current rating system. But don’t do it because people hate ratings. Do it because ratings are bad data. It comes down to a measurement question, and nobody seems to be trying to answer it: how do we reliably reveal the true range in performance in an organization so that we can invest differentially in our people? That’s the question to ask.

Share This Story


  1. Berta Serrano February 21, 2018 at 3:56 PM - Reply

    Thanks for this post, very Interesting!
    But now what? performance measurement is a must, what we can´t be measured, can´t be improved!
    Do you have any suggestions?

    • Meredith Bohling February 22, 2018 at 8:37 AM - Reply

      Hi Berta,

      Great question! The thing is, people can be reliable raters – but only of their own experiences. Definitely check out our Data Fluency Series, especially How to Tell Good Data from Bad Data, Do Current HR Tools Give Us Good Data and the Case Study which goes through a tool that does provide good, reliable data on talent – but only by allowing people to measure their own thoughts, feelings, and experiences. Enjoy! And please comment or let us know if you have more questions!

  2. Rick Holbrook May 28, 2019 at 7:15 PM - Reply

    I wonder if the indisyncratic rater effect holds true for 360 assessments for purposes of behavioural coaching improvement? Anyone know of any such research?

    • Alex October 9, 2019 at 5:22 PM - Reply

      Hi Rick,

      Yes he discusses the 360 assessments, they are even worse because they simply increase the number of bad data inputs, leading to even more confusing and incorrect information.

      They have amazing examples of how badly people assess potential in other people too, and how disengaging it can be encouraging people to build on their weaknesses rather than following their strengths, so this will likely be relevant to behavioral coaching. I strongly suggest you read his book.

  3. Liz A March 30, 2021 at 11:08 AM - Reply

    I think ratings are good as long as there is an anonymous component (where the person evaluated is evaluated by someone distant, without physical characteristics).
    What I am trying to say is that people should be evaluated without a previous knowledge of the person evaluated (maybe after a system of coding), and absolutely no physical description (or photo of employee, so that they could not look it up on social media or tools used, like Chat etc).
    I think when there is a pictured representation of the people evaluated, it is when people are prisoners of their own biases.
    Looks and nepotism are what rules our present world and makes us take bad decisions.

    This would require a bit more organization but would make the system more reliable and accurate.
    When there is a will there is a way.

  4. Teodora Takacs January 21, 2022 at 6:22 AM - Reply

    This is brilliant! I have been a manager for many years and it’s the first time I look at ratings as “data points”. When I think about ratings and performance reviews I’ve done, I now understand why I was sometimes doubting myself and had a feeling of “who am I to give this rating”. So I usually look at other data points to measure people’s performance in terms of their progress on specific projects and objectives. All in all, a super difficult exercise when you want to do it right and unfortunately many managers, me included, don’t allocate enough time in this area, because of all the other functional responsibilities and ongoing work. Insightful video, thanks!

Leave A Comment