In Zen And The Art Of Motorcycle Maintenance, Robert Pirsig’s bonkers-but-brilliant philosophical novel that turns 40 this year, he describes “the old South Indian Monkey Trap”. (I’m pretty sure it was never used to trap monkeys, but that’s par for the course with Pirsig; he doesn’t teach you much about motorbikes, either.) The trap “consists of a hollowed-out coconut, chained to a stake. The coconut has some rice inside which can be grabbed through a small hole”. The monkey’s hand fits through the hole, but his clenched fist can’t fit back out. “The monkey is suddenly trapped.” But not by anything physical. He’s trapped by an idea, unable to see that a principle that served him well – “when you see rice, hold on tight!” – has become lethal. I’m not the first to note what a great metaphor this is for our paralysis in the face of climate change: we’re so rigidly attached to a certain notion of progress that we can’t let go when it turns against us. “The difficulty,” as Keynes put it, “lies not in the new ideas, but in escaping from the old ones.”

One striking example of Keynes’s point is what psychologists call the Einstellung effect, describing the way preconceptions can blind us, in almost a literal way, to better ways of doing things. In clever experiments by Merim Bilalic and his colleagues, chess masters were presented with a board arranged to offer them two paths to victory: a well-known five-move option, and a more obscure one, requiring three moves. Even these superb players often couldn’t see the best way to win, because the one they knew better colonised their minds. Later, those who’d missed the three-move solution were shown a similar board on which it was the only way to win. Now none had any problem seeing it.

The Einstellung effect seeps into every corner of existence, with potentially grave effects. When doctors make errors, it’s been demonstrated, it’s likely to be because they jump to conclusions based on past patients, not because they lack medical knowledge. If you were choosing between doctors, you’d surely never choose the less experienced one. Yet sometimes it’s precisely experience that gets in the way.

It’s tempting to despair. If amateurs make mistakes, and experts just make different mistakes, what hope is there of eliminating mistakes? But those chess experiments suggest a different conclusion. “At the highest level of expertise – Grand Master – it gets much harder to trip players up with the Einstellung effect; you can do it, but the less obvious solution has to be really difficult to find. It’s as if there were two kinds of expertise: one in which people are simply guided by experience, and a superior kind in which they’re hyper-aware of the limits of expertise itself. Zen Buddhists famously speak of “beginner’s mind”, and it can sound as if they’re recommending a state of dopey ignorance. But perhaps beginner’s mind is better thought of as this expertise that sees past the edges of expertise, to the ignorance beyond – which, if we don’t stay aware of it, can make a monkey out of any of us.

• oliver.burkeman@theguardian.com. Follow Oliver on Twitter.