Similarities and differences between stage magicians purporting to do close reading, and p-hackers purporting to do real science
It all started with this 5-hour video
Jason Pekos writes:
I ran into this five hour long youtube video the other day that you might be interested in. On some level it seems a bit presumptuous to send someone a five hour video, so some context: The video attempts to unpack some of the “mentalism” grift. This is a genre of performance where the presenter claims to uncover detailed information from a mark via psychological tells, e.g. body language, or alternatively, the presenter claims to inject subconscious cues into the mind of the mark via subliminal messaging.
This is distinct from a magic show in that the mentalist portrays themselves as “genuinely performing mentalism,” that is, reading / incepting the mind of the mark, as opposed to a magician, who presents themselves as, like, “falsely performing magic” by “actually performing sleight of hand.”
The video pretty convincingly demonstrates that almost no actual mentalism occurs. Instead, mentalists … just do essentially what magicians do, e.g. sleight of hand and prop gimmicks, just under the facade of something else.
Pekos continues:
What’s interesting here are the parallels between the video and some aspects of modern behavioural science. The longer video highlights this shorter video at the start, and it struck me that this was exactly a priming exercise. Similarly, watching the rest of the longer video, even without knowing the *way* the trick was performed, it seems like most of the examples are deflated by a consideration of the apparent effect size. For example, a mentalist incepts an entire crowd to write down “Will Smith” on a sheet of paper, without a single defector. Even without knowing the trick, this is obviously impossible given the scope of the treatment (referring to “a man in black” and mentioning the word “slap” in his preamble. At the very least, aren’t there competing primes here – billboards outside the studio, ads, etc – which should induce at least some audience members to respond differently?).
The characters even feel familiar–the main focus of the video is Oz Pearlman (recently spoke at the whitehouse correspondents dinner). The video points out that he has a book (timestamp), and looking it up, it seems to be in exactly the same genre as other books from other characters. He even has a Ted talk!
Ok, in some sense, these people aren’t the same. The mentalists seem more egotistical, maybe more fraudulent. But towards the end of the video there’s the quote from James Randi:
I want to be, if I can, as sure of the world, the real world around me as is possible. The way I think about it is that the more incorrect things you believe about reality, then the greater the chances that you are going to bump into reality in ways that aren’t […]
I think we can place a bit of blame on the psychologists here, right? If you go through your life believing in the invisible piranhas of social science – hungry judges, power posing, infinite soup response – I think you’re basically committing yourself to a worldview where something like mentalism is possible. Especially if you really take the implied effect sizes seriously, nothing Oz does should be too surprising. The audience may react in disbelief, but we all know “disbelief is not an option.”
This stuff has been ubiquitous for a while. I remember being shown Sapolsky videos by an enthusiastic teacher in high school. A sports coach had us all power posing. I don’t think this is a unique experience for someone growing up in the 2000s. Dan Ariely had a TV show. Given this, the amount of credulity granted to Oz by the media and his audience doesn’t seem that surprising. Maybe if you’ve read Kahneman you actually derive a smug sense of superiority from other people’s surprise; you already knew the world worked this way.
Anyways, thought you might appreciate the video. I ended up listening to it audio-only while I unpacked my apartment. I think it basically works as a long podcast.
I took a look at the video–not the full 5 hours but quite a bit!–and have a bunch of thoughts. Rather than trying to organize these into an essay, I’ll just reel them off in the order that they come to me:
The 1970s and the 2000s.
Pekos’s remark, “I don’t think this is a unique experience for someone growing up in the 2000s,” makes me think about what were the silly things we were told in the 1970s. There was a lot of ESP, also the ancient astronauts, Noah’s ark, the Bermuda triangle, and the idea that dolphins can talk to each other. Bigfoot, the Loch Ness monster, and UFOs were more of a joke. Oh, and there was astrology, which my friends and I always thought of as a joke too, but many celebrities believed in it, and it was a big part of the culture.
Meta-deception.
The video is about Oz Pearlman, a stage magician who I’d never heard of, but who is apparently a big deal. He’s been on lots of TV shows and podcasts, he has a bestselling book (or so it says) with a blurb from sports owner Mark Cuban. He (Pearlman) is also a liar, in that he does his tricks using sleight of hand, video editing, and other tricks of the magic trade–nothing wrong with that, magic is a form of entertainment!–but then falsely claims that he does it by close reading of people. He’s also a convincing liar, having bamboozled Katie Couric and Adam Grant, according to the blurbs on his book page. Or maybe Couric and Grant are in on the scam, I don’t know.
My favorite of all the blurbs is this one:
“Oz Pearlman is truly a master of his craft.” — Tony Robbins
That’s delightfully true: Pearlman does appear to be a master of his craft–and this craft is not “mentalism” or “cold reading”; it’s doing magic tricks and passing it off as if he’s using these tools.
The guy in the video, Stevie Baskin, explains how Pearlman does his magic tricks–and also how Pearlman does the “meta-deception” of getting people to think that he’s doing it by careful observation of them, rather than via sleight of hand and other standard tools used by stage magicians.
I agree with Jason Pekos (author of the email quoted at the top of this post) that the idea of meta-deception comes up in NPR/Ted/PNAS-style junk science. We’ll return to this in a bit.
As we’ve discussed before, we’re now in an era in which the grifters have moved from the carny circuit to academia. 1970s ESP icon Uri Geller was a model and the Chariots of the Gods guy was a hotel manager; neither seems to have ever gone to university. Current high-profile scammers are much more upscale: Oz Pearlman graduated from the University of Michigan, and Dan Ariely, of course, has a Ph.D. They look more “boardroom” and less “street.”
Liar . . . or performer?
As noted above, Oz Pearlman lies all the time. He lies about details of his tricks, he lies about how he does the tricks, he’s lying to whoever buys his books, etc.
But should we call him a “liar,” just because he’s knowingly telling untruths? You could also call him a performer who’s always “on,” who never takes the mask off.
And should we say the same of Brian Wansink and Dan Ariely? Sure, they publish papers based on data that have never existed (at least not in the form by which they became famous, as they say in Twenty Questions); sure, there’s a lot of healthy doubt about whether that modified soup bowl or that modified shredder ever existed–but maybe it would be better to characterize them as performers who are always “on.” Recall that, even now, Wansink and Ariely present their careers with entirely straight faces, never breaking the fourth wall.
Is he really fooling people?
Is Pearlman actually fooling people with his tricks? I suspect that he is and he isn’t. The tricks are well executed–if I’d seen them straight up without Baskin’s explanations, I wouldn’t have been able to figure out how he did them. And many of the people in these TV shows and podcasts seem to really think that Pearlman obtains his effects using some very effective form of cold reading. On the other hand, his act is full of stage-magic razzle-dazzle, picking random cards out of the deck, etc., so a lot of this could be a sort of joyful suspension of disbelief. People are fooled here in large part because they want to be fooled, and it’s fun and costless. Even buying his book, which would seem to me to be a real sucker play, is still only $24.95 or whatever.
Meta-deception in stage magic and in academic science.
Now on to the connection to junk science. In junk science, the “effect” (to use the term from stage magic) is apparently incontrovertible evidence in favor of some implausible or seemingly impossible claim. In stage magic, the effect could be the appearance of the performer floating apparently unsupported in the air, or the assistant apparently being sawed in half, or the card you have secretly picked showing up in the middle of a cake, etc.
As in a magic show, so with science: there are many different ways of obtaining the effect.
– Most directly, you could just do it–levitation, sawing someone in half, reading someone’s mind, figuring out what they’re thinking by observing subtle bodily cues, demonstrating social priming, tallying up the births and finding that beautiful parents really are 36% more likely to have girls, counting and finding that single women really are 20 percentage more likely to support Barack Obama during a certain time of the month, etc.
The trouble is, these effects are not real, or are orders of magnitude smaller than what it would take to produce the claimed effects. So the direct approach won’t work.
– The natural alternative is to do a magic trick, using sleight of hand, misdirection, costumes, mirrors, whatever, to create the effect. The equivalent for junk science is to manufacture statistical significance by p-hacking or faking data or whatever, then also to come up with some bogus social-science theory to match your findings. That last bit is so easy it hardly counts as a skill–it’s on the level of a magician’s patter–but it’s still necessary. The fake result won’t fly without the fake explanation to go with it.
Again, I assume that researchers would be fine demonstrating their theories using legit science. It’s just not possible. In the same way that a stage magician would love to be able to draw a rabbit out of an empty hat for reals.
Oz Pearlman’s meta-deception was that he does not claim to have supernatural powers; he just claims to be really good at reading people. In essence, he characterizes himself as Batman, not as Superman. Batman doesn’t exist either, but he doesn’t contradict the laws of nature like Superman does, or at least not in such an obvious way.
In junk science, the meta-deception is when researchers claim they’re doing the hard work of theory, measurement, and analysis, when they’re actually just making up numbers, p-hacking, and telling stories.
Here’s a funny thing, though. As Baskin says, Oz Pearlman is a talented stage magician, and without that expertise, he’d have difficulty pulling off the meta-deception. In contrast, people like Wansink and Ariely have no particular scientific talent: they’re just charming guys who can attract hardworking students and collaborators, they’re good at writing in the style of scientific papers, and they have the special superpower of being unscrupulous. If you’re willing to cut corners in science you can go far. The other day I was chatting with a psychology researcher who told me that Dan Ariely is a wonderful adviser and that his students love him. I guess that helps. If you’re going to be publishing collaborative work, it helps to inspire loyalty. So that’s not nothing. Just as Pearlman has strong magic skills that serve as the foundation for his meta-deception, Wansink, Ariely, etc., have solid interpersonal skills, without which they could not have achieved the career heights that they did. The ability to talk students and collaborators into cheating for you is not as impressive as the ability to do sleight of hand, but it’s still a thing.
Conscious and unconscious meta-deception.
OK, we’ve discussed the frauds. But what about the true believers? Researchers who use the same data-hiding, p-hacking tricks to manufacture evidence out of nothing (in the immortal words of Simmons, Nelson, and Simonsohn, “undisclosed flexibility in data collection and analysis allows presenting anything as significant”) but who believe they’re doing this in the service of true science? Not the stone-cold cynics but the p-hackers for truth, as they see it?
I’m thinking about the sociologist who reported that beautiful parents were 36% are more likely to have girls, or the psychologists who reported that women were 3 times more likely to wear red or pink shirts during certain times of the month when it wasn’t too cold outside [it’s hard to type that last bit without laughing; it’s such an extreme example of forking paths — ed.], or the other psychologists who reported that unpartnered women were 20 percentage points more likely to support Barack Obama during certain times of the month, or the psychologist who sifted through a mountain of data to report that Cornell students have extra-sensory perception, or the political scientists who reported that losing an election for governor took 5 to 10 years off politicians’ lives . . . so many examples. I don’t know any of the people involved in any of these studies, and my impressions of their motivations are just speculations based on what they’ve written; that said, all these researchers seem sincere to me. Even if at some level they recognize that they’re not following ideal practices of scientific measurement and data analysis, I have the impression that they all think they have discovered real effects; they’re misled by a combination of wishful thinking and statistical fallacies, what Tversky and Kahneman called belief in the law of small numbers.
The interesting thing is: They’re doing meta-deception too; they just don’t realize it. The meta-deception is that they’re claiming to have achieved these amazing effects through careful experimentation and analysis, while actually they’re doing it through the sleight of hand of inaccurately or misleadingly describing their results. But . . . they think they’re doing it right!
You know the saying: The first step in any con is to fool yourself. If you can fool yourself, you can fool anyone.
OK, we can’t take that literally–these people aren’t fooling me–but there’s a kernel of truth to that saying. I do think the sincerity of many of the perpetrators of junk science is one of their strengths.
And here’s the connection to Oz Pearlman. As Stevie Baskin explains in that video, Pearlman stays in character all the time. At the very end of these podcasts, he doesn’t shake everyone’s hand, laugh, and tell them it was all magic tricks. It’s gotta be exhausting to stays in character all the time–you have to wonder whether he ever lets the mask slip, even when he’s out with his friends or at home with the wife and kids–but it gives him real payoffs: I guess he makes a lot of money from public speaking based on the fiction that he’s the Batman of close readings.
The perpetrators of junk science stay in character all the time too–but it’s easier for them, if they don’t even realize they’re cheating.
Getting the same effect two different ways.
Getting back to the magic-show thing, Baskin shows a clip from Penn and Teller to demonstrate the idea of producing the same effect twice, using two different methods. Stage magicians can do this easily enough. For example, there are lots of ways to force a card.
This tactic works in junk science too!
For example, consider the research on mind-body healing discussed by Nicholas Brown and me in this paper from 2024. The article under discussion presented three sources of evidence: direct data from an experiment they just did, reports from previous experiments conducted by their research group, and references to experiments by others in the literature.
The trouble is, as we discuss in our paper, all three of these apparently strong pieces of evidence disintegrates under scrutiny. The apparent statistical significance is explained by large numbers of uncontrolled researcher degrees of freedom, various alternative explanations for the phenomena are not actually ruled out, and the literature is described inaccurately in a way that exaggerates the evidence from past experiments.
But you can see how those claims could be convincing, as they appear to back each other up. As with the stage magicians, each trick produces the same effect but using different methods.
Did the authors of the mind-body healing paper know they were using bad methods? Did they intentionally misrepresent their data? Did they explicitly p-hack by trying lots of comparisons until they found something statistically significant? Did they purposely misinterpret the published literature? I have no idea, and I think they could well have done all these things by accident. Remember Clarke’s Law: it’s really easy to do this kind of thing without trying.
The motivation of Steven Levitt, Sean Carroll, etc.
An interesting aspect of the Oz Pearlman story is now many people in the news and entertainment media play along. He’s had Ted talks and a TV show, his book gets celebrity endorsements, etc. Some of those people have got to know that he’s faking it, that he’s doing stage magic and then lying about it. Or, to put it politely, that he’s performing, that he’s staying in character.
Is that OK, for the media people to do this? I don’t think so. From one perspective, it’s a harmless bit of entertainment, kind of like interviewing Miss Piggy or whatever. From another perspective, though, it’s a problem. Pearlman is just a performer, entertaining people and making money, but he’s doing so under the cover of science. You could say that Oz Pearlman is to science as Elmer Gantry is to religion. And I am concerned that there’s a through line from B.S. numbers in junk science to B.S. numbers coming from the government.
Now let’s return to junk science. As we’ve discussed earlier, the above-discussed unsupported claims of mind-body healing have been hyped by economist Steven Levitt and physicist Sean Carroll on their podcasts.
What’s their motivation? I guess they want to have entertaining podcasts, mind-body healing is cool, so why rock the boat? And, hey, if Kermit called me up and offered to be go on our blog, I’d interview the green guy–why not? It’s hard for me to believe that Levitt and Carroll actually think it’s ok to promote junk science. Rather, my guess is that they want to run these fun, credulous interviews, and so they decide to blur their critical facilities so to give themselves a kind of plausible deniability–that way they don’t have to admit to themselves that they’re party to a (possibly inadvertent) deception.
People lie.
The Pearlman story is also a good reminder that performers will blatantly and deliberately lie as part of their performance. When we write about bad science, we have a way of giving everyone the benefit of the doubt. Even mega-cheater Brian Wansink is so affable and seems so helpless . . . maybe he was just confused about things, maybe he was too trusting of research assistants who were under pressure to find good results, whatever. Maybe Satoshi Kanazawa, the beauty-and-sex-ratio guy, was perfectly sincere in using different methods of analyses for different datasets so as to keep finding what he was looking for. Etc etc. I don’t know. Could be. It’s just good to remember that, even if we want to assume the most innocuous explanation, sometimes people do just lie.
Pre-show.
In his five-hour video, Baskin shows how Pearlman achieves some of his effects using pre-show preparation that is not revealed to the viewer. Watching the TV show, you get the impression that Pearlman is doing the act from scratch, but actually he’s already obtained information from the other people involved in his trick. It’s an old idea from stage magic, that key steps are being done when the audience isn’t looking, indeed has no opportunity to look.
It struck me that something similar happens in a lot of bad science. Not just fraudulent science or even “junk science,” but run-of-the-mill studies such as the one discussed here that was conducted by two Nobel prize winners. What happens is that the flaws are hidden in the measurements, and no amount of arguing about the data analysis will get to that point. That’s one reason why it’s important, when looking at a research paper, to look back at the pre-show period and consider what is being measured.
Incongruous mixing of fake-mentalism with magic tricks.
To me, the most jarring thing about Pearlman’s performances is how he mixes fake “mentalism” (purported cold reading) with hokey magic-show bits like asking people to pick a card, throwing dollar bills in the air, etc. In the parts of the video I saw, he didn’t saw anyone in half or pull a rabbit out of a hat, but I wouldn’t have been surprised if he had.
The reason this seemed weird to me was that Pearlman puts sooooo much effort into staying in character and insisting that his magic tricks are not actually magic tricks–but then he basically gives the game away with all this magic-show hoo-hah. I mean, if you could really do cold reading the way Pearlman claims, you could just do it. You wouldn’t need to pull out decks of cards and all the rest. It just seems like a dead giveaway that he’s doing bog-standard magic tricks, albeit skillfully.
This genuinely puzzles me. First, that the suckers don’t get it, but, second, that Pearlman does it that way at all. I mean, sure, he has to do magic tricks–he’s not actually “Batman” and can’t actually do “mentalism”–but why not do it in a more sedate way? Why have the answer appear on shuffled playing cards, which practically screams, “I’m doing a magic trick using sleight of hand”??
I just don’t know. The only answer I can think of is that Pearlman’s good at card tricks, he’s been practicing them for decades, and people love it. I think his cell phone trick was more impressive, just because it didn’t look so obviously like a magic trick. The other half of it is that the hosts of these shows are like Steven Levitt and Sean Carroll: they want to be part of the show themselves so they work hard to look away from the obvious explanation.
Still, it seems weird to me, in the same way that it seems weird that Arthur Conan Doyle was fooled by fairy photos that were so obviously faked; nobody even went to the trouble of blurring the photos to make them look more mysterious.
Dramatic ruling out of alternative possibilities.
Pearlman does the standard magician’s shtick of ruling out alternative explanations for his effects: there’s nothing up my sleeve, that sort of thing. These can help convince in their own right and also can distract from how the trick is actually done.
The same thing happens in junk science. And you have to look carefully at what’s being said. For example, in the above-discussed paper on mind-body healing, the authors wrote:
If a person who does not exercise weighed themselves, checked their blood pressure, took careful body measurements, wrote everything down, maintained their same diet and level of physical activity, and then repeated the same measures a month later, few would expect exercise-like improvements. But in a study involving hotel housekeepers, that is effectively what the researchers found. Half of the housekeepers were informed their work meets the federally recommended guidelines for daily physical exercise and half were not. The group that received that information lost weight, had lower blood pressure, and slimmer waist-to-hip ratios compared to the group that did not, despite no change in workload, exercise habits, overall physical activity, or diet in either group.
Actually, no. The cited paper offers no evidence that there was “no change in workload, exercise habits, overall physical activity, or diet.” See section 3.3 of our paper for details.
They did that ruling-out thing! In this case maybe it was confusion on their part rather than deliberate deception, but it was the same form of reasoning.
The moral question.
I find myself annoyed with Oz Pearlman. He’s knowingly telling untruths with an attempt to deceive. I’d be ok with such behavior if it were for a larger cause that I respect (for example, you might falsely tell a friend that you like their new haircut, out of a praiseworthy desire to build their confidence) or if it were part of an enclosed system in which lying was ok (poker!) or in some combination of those settings (as in politics, where some amount of insincerity and even lying is expected, but I’d want it done for some larger goal, not just winning for its own sake). In Pearlman’s sake, the goal is presumably for him to gain fame and fortune, which to me doesn’t seem like a good reason to lie. And it’s not like fiction, where untruths are told but without intent to deceive. It’s more like that popular literary genre, the fake memoir. On the other hand, Pearlman is providing entertainment for millions, so maybe that’s a good enough reason? And, unlike his namesake, Mehmet Oz, he’s not promoting dubious medical treatments. Of the two celebrity Ozes, Pearlman is doing less damage to society.
In the above-linked five-hour video, Stevie Baskin doesn’t seem angry at Pearlman; his view seems to be an amused toleration, even a grudging admiration of Pearlman for pulling of the scam so and sticking with the bit so consistently. In contrast, the person who made the long anti-Sopolsky video really seems to hate the guy.
I can’t say I hate Sapolsky–I’ve never met the man and I don’t know much about him–but I will say he bothers me more than Pearlman does. Pearlman’s an annoying faker who goes on TV; Sapolsky’s an annoying faker who’s a decorated Stanford professor. They’ve both given Ted talks, but Sapolsky’s the one who’s pooping in my sandbox.
How can Robert Sapolsky justify his propensity to bullshit with an air of authority?
Buuuuut, to take the other side for a moment, if you ask Sapolsky whether he thinks it’s immoral for him to be spewing fake statistics to his many audiences, he might reply that, hey, he’s just throwing out provocative ideas, and it’s the responsibility of his students, readers, and viewers to judge them. We’re all grownups here, right? The marketplace of ideas. I don’t buy this argument myself; I think that if you’re presented as an authority figure and you give a public lecture, you should tell the truth, and, if you want to bullshit, you should label that part of your talk as such. Yes, the marketplace of ideas, but who asked you to bring a bunch of rats into the market? Some people will get bitten and other people will have to waste their time chasing the rats away. But that’s the best argument I can see in favor of Sapolsky’s approach. That, or the argument that he’s a busy man, and any time he might spend fact-checking his lectures (or money spent hiring a fact checker) would be a distraction from his important research work.
How can science podcasters justify their uncritical broadcasting of junk science?
Finally, there’s the moral question of how can Steven Levitt and Sean Carroll justify their puffing of mind-body-healing pseudoscience?
I see the moral justification for the researchers who actually do the mind-body pseudoscience: I assume they’re true believers, so even if they’re clued enough to realize that they’re cutting corners, they see this as just a way to make a more persuasive argument and get closer to the ultimate truth.
But I doubt that Levitt and Carroll are true believers in mind-body healing. (I don’t even think Levitt is a true believer in climate denial. His promotion of that, I think, is coming from an enemy-of-my-enemy political reasoning: he enjoys opportunities to give a poke in the eye to sanctimonious do-gooders. Not being a sanctimonious do-gooder seems like a key part of his persona.)
I don’t think Levitt and Carroll can justify their promotion of junk mind-body healing based on their advocacy of the topic. Part of it is simply that celebrity scientists stick together, and it can be fun to do a softball interview. But that’s an explanation, not a justification. I think their best justification would be a general open-mindedness: all things are possible, bad ideas will eventually be discredited, so let a thousand flowers bloom.
I don’t buy that justification, for two reasons. First, these guys don’t always do the open-mindedness thing. I doubt that Levitt would devote a podcast episode to an uncritical proponent of Marxist economics, and Sean Carroll has been known to express some skeptical views himself. So their open-mindedness is selective. That’s fine–my open-mindedness is selective too, that’s inevitable–; my point here is just that open-mindedness by itself isn’t a good justification for uncritical promotion of some stupid stuff. Second, if you’re a scientist and you’re doing a science podcast, then why not think like a scientist in your interviews? Turning off your brain when you’re podcasting . . . what’s the point of that?
Summary.
Oz Pearlman has some similarities and some differences with researchers and promoters of junk science.
The similarity is they’re all doing meta-deception, Pearlman by claiming he’s doing close reading when he’s really just using magic tricks, and the junk scientists by claiming they’re doing careful scientific measurement and analysis when they’re really just p-hacking and misrepresenting the research literature. The meta-deception can distract many people from seeing the problems.
The difference is that Pearlman knows exactly what he’s doing–obviously he can’t think he’s doing “mentalism” when what he’s really doing is peeking at people’s cell phones or whatever–whereas I have the impression that the junk scientists mostly think they’re doing the real thing.
Oh, yeah, one other similarity is how all these people just give the game away. Pearlman works so hard to create the illusion of being a “mentalist,” and then he performs standard pick-a-card-any-card shtick, just mixing it in with the serious stuff. And the junk scientists will occasionally lift the mask to give full-throated defenses of p-hacking. It’s funny how obvious it is what’s going on, once you know where to look.