The odds are you use psychometric tests as part of your interview process. If you don’t, experts probably tell you should. Unless you understand their assumptions and interpret them correctly then you will hire the wrong people. Here’s why.
Firstly, these tests are based on a number of assumptions. They assume that, fundamentally, we have a constant personality that predicts our behaviour. This might seem like common sense, but it’s good to question common sense occasionally. Do you behave the same when relaxed and when stressed? With strangers and with friends? At work and at home?
The tests also assume that we can all be measured against similar yardsticks. We are all points on a graph with axes corresponding to universal traits.
You need to be aware of how these tests are developed. Researchers ask thousands of people to describe themselves and others and then examine the words used. Assuming that words used correspond to characteristics, they search for correlations between words. If there are strong correlations, then one of the characteristics is eliminated. For example, if ‘nice’ always corresponds to ‘kind’ then both describe the same trait so one is discarded. Of course, there’s never a perfect correlation so information is lost.
Personality is therefore distilled down to between 3 and 16 factors. This results in gross over-simplification. Carry out the same exercise with physical characteristics and you’d end up describing everybody in terms of their height, weight and skin colour. That broad-brush picture would give us some information, but it’s the lost nuances that are important.
You should note that these tests ignore some realms of personality. Some of the missing traits are important to the workplace, some are not. These tests rarely tell you if you’re an asshole, evil, gullible, attractive, intelligent, a hard worker, promiscuous or religious. Although not always relevant to work, these are important characteristics that people have.
The last point is more subtle and is about interpreting the results. When evaluating the world, our minds use certain heuristics and take certain short cuts. Normally, this works well but it can introduce some biases. One of the most notorious is that we tend to ignore base rates. For example, say a doctor tests you for a disease. The disease affects one person in ten thousand and the test is 99% accurate. You test positive. How likely is it that you have the disease? Take a moment to think about it.
Most people – and this includes doctors and statisticians – will say 99%. This is the wrong answer. This becomes clear if you think in concrete rather than abstract terms. Take ten thousand people. One person is ill. The test is 99% accurate so 1% of people will have false positives. That means that 100 people test positive. Since only one of those people is actually ill, the odds of you being ill given a positive test result is actually only 1% (ignoring the negligible effect of false negatives). The important point here is that the base rate is important. The underlying incidence of the disease massively affects the probabilities.
Your mind can take you down a similar blind alley with psychometric test results. You can see this from the diagram below. Say you’re trying to hire a developer. Obviously you want to hire an outstanding one, and they are rare (the small circle). You’ve noticed that the outstanding developers you know almost all share similar traits (the red area) so you decide to test for that trait.
You find a developer with the required trait. This group is represented by the large circle. However, because many average developers may also have the required trait, you are much more likely to have picked somebody in the blue area (average developer with the trait) than the red area (outstanding with the trait). You have been misled by the base rate – the number of developers in the general developer population with the trait. In other words, if you hire on the basis of the test results, you will likely hire the wrong person.
In conclusion, by all means use psychometric tests but understand their assumptions and interpret them carefully.
Great points, enjoying the blog…
I think it is important to point out that this kind of bayesian attention to false positives is only relevant when you are dealing a random sample of the group. Maybe better said, the initial assumption about 1 in 10000 is incorrect if the person in the doctor’s office isn’t randomly selected (they are there because they feel sick, rather than randomly there for a disease screening program). For example, only one in 1000000 people have an axe in their head on any given day. A doctor’s quick visual inspection is 99.9% accurate. Yet if a patient comes in with an axe in his head, he probably has an axe in his head. If the guy at the Halloween party has an axe in his head, then it depends more on whether they are serving Scotch.
The same goes for an interview. If you are looking for certain skills in abstract thinking, or maybe even signs of slight autism, you shouldn’t be thinking about the general population statistics. Your starting point is “the general population that programs software for a living”, which includes all kinds of weirdos.
I know that is picky, but before throwing out conventional wisdom, I think we should be on really solid ground. The hard part of Bayesian logic is getting the initial assumptions right, and that is all about the picky, and indeed sometimes impossible details : ).
Robin,
Thanks for the comment. You make some very good points. You’re right – I had the population of developers on the job market in mind rather than the general population and was considering random medical screening (eg for Down Syndrome or the worried well). It hadn’t occurred to me (thanks for pointing it out) that the population of people appearing before the Dr isn’t a random sample. It’s odd how Bayesian logic is so hard for humans, yet so simple in theory.
– Neil