Calibration Is Hard

People want to be safe more than right. The benefits of being right are not cut-and-dry, so it has a marketing problem.

Confirmation bias rules everything around me (C.R.E.A.M.)

People seek confirmation, not truth. We are ego protectors
Many calibration errors are not errors; they are motivated reasoning. Everyone's talking their own book. Especially on matters that definitely don’t have right answers that reasonable people disagree on.
SlatestarCodex: Of the fifty-odd biases discovered by Kahneman, Tversky, and their successors, forty-nine are cute quirks, and one is destroying civilization. This last one is confirmation bias - our tendency to interpret evidence as confirming our pre-existing beliefs instead of changing our minds. This is the bias that explains why your political opponents continue to be your political opponents, instead of converting to your obviously superior beliefs. And so on to religion, pseudoscience, and all the other scourges of the intellectual world.
Rohit Krishnan: “human ability is normally distributed but the outcomes are power law distributed”. What this means is that just because someone builds a company that produces extraordinary outcomes, 10000x the average, doesn’t mean that they were 10000x as capable. Achievements are created from multiplicative outcomes of many different variables. So if you’re investing in a “10x founder” it doesn’t mean that they themselves are 10x the capability of everyone else, but what it means is that their advantage, combined with everyone else’s advantage, can get you to a 10000x outcome. Which means the adulation we pour on top of some folks creates its own gravitational field, and makes others susceptible to falling in love.
Risk-conscious hard work and discipline can lead someone to achieve a comfortable life with a very high probability. Beyond that, it is all randomness: either by taking enormous (and unconscious) risks, or by being extraordinarily lucky. Mild success can be explainable by skills and labor. Wild success is attributable to variance. Nassim Taleb

A good filter reduces the chain of coin flips you must win before you reach your goal

“the world doesn't shrink to your size just because you lack the depth to understand it”

No matter how much we say ‘correlation is not causation’ the admonition remains underrated. An example I see everywhere: Someone says “x makes us good at y” but usually what makes us good at x also makes us good at y.

Post-hoc fallacy: we don’t know the difference between “because of” and “in spite of”

Berksons Paradox and tail divergence are horribly underappreciated. Just listen to how anybody speaks about unconventional outliers.

Survivorship bias —> overfitting —> cargo-culting

The mark of smart is asking good questions. This is a good test for understanding.

From Slatestarcodex “fake” graduation speech:

The system is not your friend. The system is not your enemy. The system is a retarded giant throwing wads of $100 bills and books of rules in random directions while shouting “LOOK AT ME! I’M HELPING! I’M HELPING!” Sometimes by luck you catch a wad of cash, and you think the system loves you. Other times by misfortune you get hit in the gut with a rulebook, and you think the system hates you. But either one is giving the system too much credit.

Improving decisions requires:

better benchmarking and accounting for counterfactuals
Recruit others into your dilemmas

Daniel Kahneman from Thinking, Fast and Slow: “It is easier to recognize other people’s mistakes than our own.”
The best antidote to cognitive bias is getting perspective from others. We can’t see our own blindspots.
Julia Galef’s also believes “peer pressure” can combat bias if people prefer to flatter themselves as “scouts” rather than cling to something they want to believe but think is wrong. This makes sense to me because a culture of being well-calibrated will nudge you towards that goal (SIG, LessWrong community)

There’s also an argument I find unsatisfying in Slatestar’s review of The Scout Mindest

It reminds me of C.S. Lewis - especially The Great Divorce, whose conceit was that the damned could leave Hell for Heaven at any time, but mostly didn’t, because it would require them to admit that they had been wrong. I think Julia thinks of rationality and goodness as two related skills: both involve using healthy long-term coping strategies instead of narcissistic short-term ones.

I know some rationalists who aren’t very nice people (I also know others who are great). There are lots of other facets of nice-person-ness beyond just an ability to acknowledge your mistakes (for example, you have to start out thinking that being mean to other people is a mistake!) But all these skills about “what tests can you put your thoughts through to see things from the other person’s point of view?” or “how do you stay humble and open to correction?” are non-trivial parts of the decent-human-being package, and sometimes they carry over.

In one sense, this is good: buy one “rationality training”, and we’ll throw in a “personal growth” absolutely free! In another sense, it’s discouraging. Personal growth is known to be hard. If it’s a precondition to successful rationality training, sounds like rationality training will also be hard. Scout Mindset kind of endorses this conclusion. Dan Ariely or whoever promised you that if you read a few papers on cognitive bias, you’d become a better thinker. Scout Mindset also wants you to read those papers, but you might also have to become a good person…Here Scout Mindset reaches an impasse. It’s trying to train you in rationality. But it acknowledges that this is closely allied with making you a good person. And that can’t be trained

(in case this is starting to sound too touchy-feely, Julia interrupts this section for a while to mercilessly debunk various studies claiming to show that “self-deluded people are happier) [Kris: that’s another argument for being well-calibrated!]

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so. “

Munger: “It’s not the bad ideas that do you in. it’s the good ideas. And you may say, ‘That can’t be so. That’s paradoxical.’ What he [Graham| meant was that if a thing is a bad idea, it’s hard to overdo. But where there is a good idea with a core of essential and important truth, you can’t ignore it. And then it’s so easy to overdo it. So the good ideas are a wonderful way to suffer terribly if you overdo them.”

This is why you lose the most with the second-best hand
For bad ideas to spread there has to be a nugget of truth in them. Many will point to the nugget of truth as justification when it’s really just an explanation of why it spread