In chapter 20, “The Illusion of Validity” we continue to explore “Overconfidence” with Daniel Kahneman’s  book Thinking Fast and Slow (2011). Validity of a test or a measure, simply put, is testing what you think you are testing… or measuring, or predicting. Often times, we think we have found something to measure, that will predict something else but often times we are wrong and not measuring what we think we are measuring. For example, testing students on a subject under serious time constraints. You may think you are measuring their knowledge of the topic but you may be just getting a measure of how well they work under extreme stress, high cortisol levels, or how fast they are at regurgitating information on the topic – rather than they how much they actually know about a topic.

Then there is reliability, the other side of the coin but equally important. It is the ability to show the same results with repeated use of the measurement. If you test a student on the same subject matter, repeatedly, and they come up with the same results consistently – you have good reliability and the measurement is a reliable measure. Although, if given the opportunity to learn from their mistakes – they may improve with each re-test. Also, you can have a very reliable measure which turns out to have very low validity. That is, they consistently perform the same on the measure but it turns out the measure isn’t measuring what you thought it was measuring. So, you can see why validity is very important and any measure without it is pointless.

Kahneman gave an example of analyzing a groups of cadets’ performance under atificial conditions to predict their performance in officer academy and ultimately their performance as a real officer in the field/real world. They found their measures of the kadets’ performance in the artificial environment was completely unpredictive of their sucess in officer academy and conceptually therefore, the real world. Nevertheless, they continued to conduct these evaluations and to report their findings with an air of confidence. Why? Well, this is the complexity of the “illusion of validity” and which Kahneman describes as the discovery of his first “cognitive illusion” (p. 211).

Despite Kahneman and his colleagues knowledge of the lack of predictability and validity of their measure, they confidently continued to measure subsequent cadets’ in the same manner because, as Kahneman states, “Confidence is a feeling which reflects the coherence of the information and the cognitive ease of processing” (p.212). He states that these were errors of “substitution” and the “representativeness heuristic” (p. 212). He went on to warn against trusting in people who have high confidence because “declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind,  not necessarily that the story is true” (p. 212).

He illustrated this with examples about political pundits predictive skills and those in the financial field who are supposed to have “stock-picking” skills (p. 212). Kahneman explained that, “The illusion that we understand the past fosters overconfidence in our ability to predict the future” and recalled Nassim Taleb’s Black Swan Theory (discussed in previous chapters)  (p. 218).  Kahneman cited various studies which found highly experienced or in demand political pundits and people in the financial industry who were entrusted and paid bonuses based on their ability to beat the stock-market, were found to be not much more reliable than chance. Yet, when presented with this information, these people ignored the information and confidently continued. Although, they often became a little more defensive about their skills. This has been illustrated by the current political administration as well, who has been described as – “doubling down” when their competence or performance came into question. Kahneman described this “illusion of skill” as an “individual aberration… deeply ingrained in the culture” –  professional, religious, or national (p. 216). He goes on to explain that “facts that challenge… [or] threaten people’s livelihood and self-esteem” cannot usually be “absorbed” or “digested” by the individual in question (p. 216).

Furthermore, Kahneman stated that, “Cognitive illusions can be more stubborn than visual illusions” which are undeniable and often impossible not to see even when you know you are not seeing what is true (e.g., Müller-Lyer‘s optical illusion (p. 217).  With a visual illusion, one can often take a ruler to the object in question and prove to themselves that the two lines are the same length, despite one’s perception of one line being shorter than the other. Whereas, in cognitive illusions people are often “ignorant of their ignorance”. Their “subjective confidence” is a “feeling” that they can’t change, can be very strong, and basically just feels better to believe. He goes on to state that such cognitive illusions can be supported by “powerful” cultures (e.g., professional, religious, national, etc.) and create an “unshakable faith” no matter how “absurd, when they are sustained by a community of like-minded believers” (p. 217). Especially, when it makes them feel happier, superior in skill, talent, power, and/or confidence.

Kahneman cited some personality types from “Isaiah Berlin’s essay on Tolstoy ” – “hedgehogs” and “foxes” (p. 220). Hedgehogs were described as those who “know one big thing”, “have a theory about the world”, ” account for particular events within a coherent framework, bristle with impatience toward those who don’t see things their way, and are confident in their forecasts. They are also especially reluctant to admit error… and are opinionated and clear, which is exactly what television producers love to see on programs” (p. 220).  Whereas, foxes are “complex thinkers” who “recognize reality”, including “luck” and “unpredictable outcomes” (p. 220) and states that foxes are less likely to of interest to television producers.