Emma Beckett: This is a story about a data detective. An unassuming doctor from the southwest of England.
John Carlisle: I got some email saying to agree to not disclose what I was about to get emailed about. So it was all cloak and dagger stuff, I didn’t know what they were going to say.
Emma Beckett: Who, armed with spreadsheets and statistical tests, ended up blowing holes in one of the world’s largest nutrition trials.
Gideon Meyerowitz-Katz: The researchers had built their careers on this study. The New England Journal of Medicine had published the biggest and in many ways most important nutrition study ever.
Emma Beckett: It’s a story with the potential to damage the public’s opinion of science.
Gideon Meyerowitz-Katz: We have this house of cards built on trust. People based their opinions a lot on trusting the researchers and trusting that everything went well.
Emma Beckett: And influence the future of research.
John Carlisle: It’s gonna get easier and easier to manufacture data.
Emma Beckett: But it’s not about something controversial like cloning or stem cells. It’s about a diet, and a really popular one.
Evangeline Mantzioris: Packed full of nutrients, fibre, antioxidant, the works.
Emma Beckett: For Science Friction, you’re listening to Cooked, our series here on Radio National about the nuance of nutrition, the foods we eat, and how to make sense of the headlines about them, as you navigate the ever-growing bombardment of diet advice online.
I’m food and nutrition scientist, Dr. Emma Beckett. Today, join me for a mystery in the Mediterranean.
This is a story about the Mediterranean diet. But we’re not starting with the Mediterranean diet. We’re starting with … anaesthetics.
John Carlisle: So my name’s John Carlisle. I’m an anaesthetic doctor. I work for the NHS in the UK. I’ve worked as an NHS anaesthetist for about 25 years.
Emma Beckett: Years back, John Carlisle was reading the very latest medical research. Not an uncommon pastime for a doctor. And one author’s work really stood out to him. A Japanese researcher in anaesthesia named Yoshitaka Fujii.
Fujii had published many papers looking at how to prevent vomiting and nausea after surgery. And this was something of a focus for John too. But the findings Fujii was making were neat. Tidy. Almost … too perfect.
John Carlisle: In experiments done by human beings, we expect some sort of variation between what is intended to be a very similar trial. So two different trials, intended to be very similar to each other, you expect the results to be similar but not identical.
Emma Beckett: Imagine you have 10 friends. Six of them have brown hair and four of them have blonde hair. You randomly split them into two groups. It’s the most likely outcome that you’ll get three people with brown hair and two people with blonde hair in each group.
But if you repeat the random split a hundred times, you’ll eventually get all sorts of differences. Uneven numbers of brown and blonde hair across our two groups. We expect that the data won’t be perfectly symmetrical, that there’ll be natural variation. This is important to understand because it’s the basis of how John dug into Fujii’s work.
John Carlisle: So the bit of the study I looked at was the characteristics of the people before anything different had happened to them. So for instance, how old they were, how tall they were, how heavy they were before the experiment started.
Emma Beckett: This data is captured in what’s typically referred to as Table 1 of a medical research paper. This is the boring stuff that most people gloss over before they get to the juicy findings. But John Carlisle isn’t most people.
John Carlisle: The variation you’d expect to follow a particular pattern, one determined by random variation, which means that we can then, or I could, apply common statistical techniques to just look at those.
Emma Beckett: This is our brown hair, blonde hair example, but over tens or hundreds of characteristics and applying classic tests of statistics to those data. To be clear, John wasn’t inventing a whole new approach to statistics, he was actually using methods that have been around for a good while, that you’ll learn in any university statistics course.
But applying them to that boring Table 1 data, the part most people skip over, that was a pretty novel approach.
John Carlisle: And that’s what I did for Fujii, and it resulted in a likelihood that was very, very, very, very small. So, the conclusion was, something was wrong with his papers. And over the next year, year and a half, Fujii’s work was investigated and they published their report.
It basically said, it looks like he sat down at a desk and just made up the papers without any patients being involved. It was remarkable.
Emma Beckett: Fujii was fired. He was found to have falsified the data, to have simply made it up, in many of his published papers. And as we make this episode, he has the second highest number of retractions as a scientific researcher, according to the leaderboard of the organisation Retraction Watch.
172 of his papers have been retracted.
So, how does this all fit into the Mediterranean diet? After finding the fraud in Fujii’s work, John started to look elsewhere. Partly out of curiosity, and partly because his colleagues were starting to get, well, a little annoyed.
John Carlisle: They came up to me over coffee and said, John, um, enjoyed the work on Fujii. Amazing and interesting.
But it’s about an anaesthetist. When we look at Scientists who have the most papers retracted, we see quite a few anaesthetists listed there. We’re a bit worried that anaesthetists are getting a bad name. So, um, do you think you could look at some people who aren’t anaesthetists and see whether you can identify problems with their work?
Emma Beckett: John had proven his methods could find scientific fraud and error in ways that hadn’t really been done before, and he was about to test them on a much bigger scale. John set out to analyse more than 5,000 papers, including those from some of the biggest medical journals in the world. After combing through the thousands of papers, a handful stood out as having issues.
And one of those was called the PREDIMED study, one of the largest nutrition trials in the world. Only, John didn’t know that yet.
John Carlisle: The PREDIMED study, for me, wasn’t the PREDIMED study, for me, it was, in fact, let me just, whilst we’re chatting, it was paper 529 of the 958 trials from the New England Journal.
It was only later on that I became aware that it was for many people, an important trial about diet, I hadn’t really appreciated that at the time.
Emma Beckett: At the time, this was the study into the Mediterranean diet. It played a role in shaping nutrition guidelines and the kind of advice you get when you step into your doctor’s office.
Make no mistake, PREDIMED was big. Researchers had randomly assigned more than 7,000 people in Spain to eat either a low-fat diet or two different variations of the Mediterranean diet, one focused on nuts and the other on olive oil. So there were three groups in total. Then they were followed for almost five years to see how their health tracked over time.
And a Mediterranean diet, in case you don’t know, is basically a plant-rich diet that also includes lots of olive oil.
Gideon Meyerowitz-Katz: I assume that he knew that by looking at the New England Journal of Medicine which is the biggest medical journal in the world, he would include some groundbreaking research in his data set.
Emma Beckett: Dr Gideon Meyerowitz-Katz is an epidemiologist and senior research fellow at the University of Wollongong. He’s been part of research teams investigating systems for detecting problematic research data.
Gideon Meyerowitz-Katz: The New England Journal of Medicine had published the biggest and in many ways most important nutrition study ever. The researchers had built their careers on this study.
Emma Beckett: The results published in 2013 in the New England Journal of Medicine found that those groups that followed the Mediterranean diet, whether it was nut focused or olive oil focused, they had about a 30 per cent lower risk for developing heart attack or stroke compared to those on the low fat diet.
Gideon Meyerowitz-Katz: That is monumental. A 30 per cent reduction in heart disease and heart disease deaths was one of the most impressively big reductions for anything that we’ve ever found, at least in terms of heart disease. Statins, which are very commonly prescribed therapy for high cholesterol, reduce your risk of dying of a heart attack by less than this study found.
Emma Beckett: Yet, four years later, John’s analysis had flagged PREDIMED as having problems.
So, what went wrong? Well, the statistics John applied couldn’t tell us what the issue was, just that Table 1 looked pretty funky. He published his findings on PREDIMED and other papers and heard nothing for about six months. Then …
John Carlisle: In the first half of 2018, I got some emails from the office of the New England Journal saying to confirm my identity. And then to agree to not disclose what I was about to get emailed about. So it was all cloak and dagger stuff, I didn’t know what they were going to say.
Emma Beckett: The journal had done their own investigation after John flagged problems with the paper. And they found some issues. Remember, people were meant to be assigned to one of three groups. Nutrition education about a low-fat diet, or a Mediterranean diet with a free supply of either mixed nuts or olive oil.
But in some cases, researchers would assign one person in a household, say the husband, to one arm of the study, like the group that were given olive oil. Then they’d ask other people in the household to share that diet. They’d include them in the study as though they’d been randomly assigned to the olive oil.
But they really hadn’t. Another issue the investigation discovered is that a researcher at one of the 11 clinical centres in the trial worked mostly in small villages. People there complained that their neighbours were getting free olive oil while they got nothing. So the investigator decided to give everyone in each village the same diet.
He never told the research leads what he had done. You might be thinking, well, whatever. Who cares if they randomised people or not?
But this randomisation is so important to what they found in the study. It’s what made the study so rare and exciting. It’s almost never done at scale in nutrition because of how difficult it is.
And it’s part of the reason they were confident the reduction in heart attacks and strokes they saw was something real, not just a blip in the data. Now, that was in question.
John Carlisle: Then I got an email from the editor in chief at the time saying that they had taken on board my analyses and they had contacted the authors of the PREDIMED study.
And the PREDIMED authors had then found various problems with their paper. Sufficiently severe that they decided to retract the PREDIMED studies. They decided that it was sufficiently flawed that they’d have to retract it.
Gideon Meyerowitz-Katz: This is where the story gets, I would say, it’s like small p politics. There were many people who’d been students or early career researchers when the study was first being conducted, who now had published over a hundred papers on this data set and so they took the unusual step after finding out that there were serious issues in their data and in the way the study had been conducted of retracting it.
Retraction is normal in this case, of taking it out of the literature and then republishing it as an observational study. I can see why that may have happened. It means that it’s very difficult now to trust any of the results that have come out of this piece of research.
Emma Beckett: An observational study is more limited than a randomised controlled study.
You can no longer confidently say that things are cause and effect. Errors made by the researchers, undetected until John Carlisle showed up, meant a full fifth of the people involved, more than 1,500 participants, had to be excluded when the paper was republished. For many in the field, that was a big hit to its credibility.
Gideon Meyerowitz-Katz: We have this house of cards built on trust. People base their opinions a lot on trusting the researchers and trusting that everything went well. And I think if we look at other industries, like if we look at the Enron shenanigans of the early noughties, when questions were raised about their books, someone else came in and audited them and an independent person, who was very suspicious of them, came and had a look at everything that they did.
But in science, that doesn’t happen. It very rarely happens. I’ve been involved in cases where it does happen, but I would say the majority of these cases, the people who are reviewing the data and who are telling us what happened, what went wrong in a study, are other researchers themselves. And there’s an obvious conflict of interest there.
Emma Beckett: For Science Friction, you’re listening to Cooked with me, Dr. Emma Beckett. Today, a data detective on the trail of one of the biggest nutrition trials ever conducted.
These mistakes were a bombshell for the nutrition research community, with the results of the trial suddenly more shaky than ever before. But in the wake of all this zigging and zagging on PREDIMED, the publishing, the retraction, the republishing, does that mean we should give up on the diet altogether?
Evangeline Mantzioris: I think there is a lot of confusion about what the Mediterranean diet is, and that confusion could be fuelled by what we see in restaurants, what we see in cookbooks and recipes that are put up. And I think the common misconception is that it contains a lot of meat.
Emma Beckett: This is Dr Evangeline Mantzioris. She’s Program Director of Nutrition and Food Science at the University of South Australia and an accredited practicing dietitian and sports dietitian.
Evangeline Mantzioris: When we talk about the traditional Mediterranean diet, and it was defined, you know, in the 50s and 60s, it’s a diet that’s incredibly high in plant food and lower in animal-based food. It doesn’t mean it doesn’t have it in there, but the amounts are quite low.
Emma Beckett: Put simply, eat foods, mostly plants … and add some olive oil.
Evangeline Mantzioris: Eat lots of plant-based foods, make sure you get all your veggies in, use extra virgin olive oil, and limit your meat to about twice a week, preferably fish or chicken. But, I guess, the thing that defines the Mediterranean diet compared to any other type of healthy dietary pattern is the use of extra virgin olive oil.
And that is really high. It was used as the fat. They didn’t use butter or cream or any other sort of animal fat in their production. It was always olive oil and about 50 to 60mls a day.
Emma Beckett: Perhaps as important as the food itself is the how and why of eating it. The sharing of meals means social connection, which a growing body of research suggests is crucially important, especially as we age.
Evangeline Mantzioris: It’s important for reducing the risk of a lot of neurocognitive disorders, depression, a lot of things. And so eating a meal together with someone is an incredibly important part of it. I’m going to sound like I come out of My Big Fat Greek Wedding now, but the word diet in Greek actually comes from the word theeta, which actually means the way you choose to live your life.
It’s not about just the food. So if we think about the way you choose to live your life, it includes your exercise, your rest. Who you socialise with, who you eat with, perhaps who you cook meals with, how your family and friends are involved in all of that.
Emma Beckett: When it comes to assessing the evidence base as a whole, Evangeline says that while PREDIMED is important, it’s not the only piece of the puzzle.
Evangeline Mantzioris: In science, we don’t look at one study. In isolation, because if we did, there is a risk of errors, either by mistake or intentional, whatever it is. And that’s why we always look at the body of evidence. And I think if the PREDIMED study was the only one that showed a decline in heart disease rates and the other improvements they saw in diseases, you’d go, oh, well, it’s only that studies that’s shown it.
Everyone else has shown nothing’s happened. And the Mediterranean diet is just the same as the American Heart Association diet. then that would be a concern.
Emma Beckett: Because yeah, if you look at the most recent reviews, even the ones that exclude PREDIMED, they suggest a link between following the Mediterranean diet and better heart health.
Evangeline Mantzioris: I am confident that the Mediterranean diet research is fairly accurate. I’m also confident that maybe not the level of adherence in all of the studies is as high as we would potentially like it to be. But that’s the nature of nutrition studies, right? And that’s why, you know, there’s sometimes conflicting messages.
And it’s always seems to be coffee, right, that gets the brunt of someone shows coffee does this, but no, someone else shows coffee does that. And looking at the overall evidence and going, okay, well, what sort of evidence did it come from? Was it, you know, observational studies? Was it intervention? And interpret it and piece it together.
And I think that’s where the difficulty comes in trying to piece it together as a recommendation for people. By and large, we know that the more plant foods you eat, the better off you’re going to be. The less ultra processed foods you eat, the better off you’re going to be. And you are better off consuming white meat over red meat, and you’re definitely better off consuming red meat over processed red meat.
Gideon Meyerowitz-Katz: I don’t know if the situation with PREDIMED is necessarily an indictment of nutrition research as a field. I mean, it’s so broad. We’re talking about millions of researchers worldwide. I think it tells us more about problems with trust in academia, because as I said, this should have been picked up. There are so many places along the way that someone should have noticed that something was off about the study.
Emma Beckett: But to give this a positive framing, it shows that peer review in science continues after publication. And someone did eventually notice. That’s a good thing.
Gideon Meyerowitz-Katz: We should talk about it as a lesson, what not to do in randomised studies, the failings of scientific research. But citing this as evidence for the Mediterranean diet is, in my opinion, problematic. And that does happen.
Emma Beckett: So where does that leave us on the Mediterranean diet? Did one study retraction undermine all of this health advice? No. We know from the breadth of the evidence that the diet is linked to positive health outcomes generally. It’s rich in all of the food groups we know are good for our health.
What I do think the PREDIMED story reveals is that nutrition, like all sciences, isn’t simply a set of facts. It’s a process undertaken, interpreted and applied by people. And people are fallible. They make mistakes and they have beliefs. Even when they’re scientists.
And that replicating results and interrogating research, even after it’s published, is vital. One piece of research in isolation isn’t enough. It’s the collective body of evidence that makes researchers and clinicians confident in this tricky area of science.
John Carlisle: It’s not easy science to do, very difficult science to do, and it won’t take much in such a trial that’s going to be very sensitive to how well it’s done for the results to be skewed in ways that are probably not known. So it’s not just a window on science, it’s a window on your own biases because if you believe strongly in an intervention.
Your belief will not be shifted very easily. So, me personally, well, I like olive oil, and I like oily fish, and I like nuts, but I don’t eat them because of the PREDIMED study. I eat them because I like the taste.
Emma Beckett: It’s almost like you have created a new, a new field of inquiry, a new way of scientific investigation. How do you see your work in this area?
John Carlisle: I’m aware of other people’s work that has preceded mine, I’m aware that I’ve almost used off the shelf tests, that I didn’t invent any tests, I just applied them in to Table 1, which most people don’t usually do. So, yes, I’m pleased with myself, I suppose. I’m content with what I’ve done.
I’ve been part of a group of people who use different techniques, so techniques other than mine, to start to generate this process of really looking at papers that are published and to question how many trials and other types of papers that are published that have flaws will never get identified. That is a really important question. There’s no absolute way of knowing for sure, but I think it’s quite a lot.
The particular techniques I’ve used, I think will have had their day fairly shortly. I can generate Table 1 with 32 variables like the PREDIMED Study, that I can make up, and I can evade, I can get a result that would be uninteresting, so it would go below the radar, and I could do that because I can program, but nowadays, why not just ask your local AI to do it for you.
So it’s going to get easier and easier to manufacture data. So how, Emma, are we going to improve scientific papers, make them more trustworthy? People have talked about various solutions. They’d all really come down to making the source of those studies less vulnerable to human messing around with. So, we need to get systems in place in universities that means fewer studies are done.
Which will allow the fewer studies that are done to be monitored very closely, that they’re being conducted appropriately, because there’s so much research going on at the moment, nobody has got the resources to keep an eye on it. So we need to do less research, make it better, and make it less vulnerable to human intervention.
Peer review is also going to have to be distributed. So the best type of study is one where you publish your protocol. So you haven’t actually done the experiment yet. You publish your plan for how am I going to do this experiment? And people can comment on that and improve on the methods. Then you implement the methods and then you publish, or pre-publish what you’ve done so that people can check your analysis. And that’s how things, I think, should work in the future. So you’ve got more eyes on the problem.
Emma Beckett: So a bit more open.
John Carlisle: Very open.
Emma Beckett: Science done more openly. I love that.
Dr. John Carlisle. He’s an anaesthetist and data detective from Torbay in the United Kingdom. You also heard from epidemiologist Dr. Gideon Meyerowitz-Katz and Program Director of Nutrition and Food Science at the University of South Australia, Dr. Evangeline Mantzioris. Thanks to producer Carl Smith, senior producer James Bullen, and sound engineer Angie Grant.
For Science Friction, this has been Cooked. I’m Dr Emma Beckett. Catch you next time.
Dining and Cooking