Deep dive The journal Science is preparing to remove an editorial expression of concern that cast doubt on a five-year-old Microsoft quantum computing research paper.
The notice is expected to be replaced by a correction that says researchers didn’t present a full description of how they tuned the relevant devices and didn’t present a full catalog of the data measured.
The decision simultaneously delights and frustrates Charles Marcus, professor of physics at the University of Washington, a former scientific director of Microsoft Quantum Lab at the University of Copenhagen, and one of the paper’s authors.
It follows a determination issued last year by the University of Copenhagen’s Committee for Responsible Research Practices that Marcus – affiliated with the university at the time – “has not engaged in questionable research practices in this matter.”
“So it feels great,” Marcus explained in a phone interview. “It’s nice to be exonerated. It’s nice to have both Science Magazine and the National Board and the Practice Committee of the University of Copenhagen and my own department back in Copenhagen, all exonerated, and the authors of the paper. But I want those four years back.”
At the same time, the resolution of the matter has dumbfounded Sergey Frolov, professor of experimental physics at the University of Pittsburgh, who initially raised concerns about the paper and maintains the work should be retracted.
This is the first time that a journal will roll back a positive action they previously took to alert other scientists about the pattern of problematic research, by replacing an EEoC with a correction
“This is the first time that a journal will roll back a positive action they previously took to alert other scientists about the pattern of problematic research, by replacing an EEoC [editorial expression of concern] with a correction,” Frolov told The Register. “I have not seen the draft of the correction and will not see it until it is published, but regardless of the content, problems in the paper undermine the conclusion and retraction would be the better course.”
The four-year-old dispute echoes current concerns about whether Microsoft researchers really created a topological superconductor capable of using Majorana particles for quantum computing, as announced in February. And it underscores the challenge of maintaining civil peer review norms in the social media era.
The dispute is clearly personal to those involved. But it’s also corporate. It matters immensely to Microsoft and to other companies in the quantum computing business – because it’s not clear there really is a quantum computing business.
“While other companies have working qubits that certainly exist – as in based on proven physics principles – they are still far from a product that they could really sell, from something that can do something useful that you can verify by buying it and using it,” Frolov said. “You don’t think that your microwave oven is a fraud because you can put your mac and cheese in it and it will cook.”
That’s not possible with any existing quantum computer, said Frolov, so companies can claim whatever they want because those claims are fundamentally unverifiable.
The skeptics who doubt Microsoft’s results
An editorial expression of concern (EEoC) is a way for editors or publishers of academic journals to indicate potential problems with a published article. They have become more common in recent years, according to a 2017 study of biomedical literature. To the extent problematic claims can be investigated, an EEoC may end up being removed or replaced by a correction or a retraction, depending on the outcome of the inquiry.
The disputed article, “Flux-induced topological superconductivity in full-shell nanowires,” was published on March 27, 2020, and placed under an EEoC on July 30, 2021. It describes how nano-scale wires can be applied to a semiconductor substrate to create a topological quantum computing device. The technology, subject to a Microsoft patent, is a precursor to Microsoft’s Majorana 1 quantum chip, the validity of which has also been challenged.
Majoranas are elementary particles that researchers believe can be used for quantum computing. If they can be harnessed, they should reduce the need for quantum error correction – a major obstacle to creating a useful quantum computer. Microsoft claims to have taken a step in this direction with its Majorana 1 chip, but doubt remains in the scientific community.
Other quantum computing papers from Microsoft-backed researchers have been hit with EEoCs. As noted recently in Science, “Nature retracted two papers from the [Microsoft-funded] Delft lab claiming to find evidence of the Majorana particles. In 2022, Science retracted another Majorana paper from the University of California, Davis, for data irregularities, and several others have received expressions of concern or faced replicability issues.”
The nanowire paper EEoC followed from concerns raised by Frolov and Vincent Mourik, an experimental physicist with German research institute Forschungszentrum Jülich. As the EEoC explained, the pair “expressed a joint concern that the tunneling spectroscopy data published in the original paper are not representative of the entirety of the data released in association with this project.”
Frolov told The Register in a phone interview and email exchanges that he and Mourik kicked off the experimental study of the Majorana particle with a paper published in 2012.
“We have been investigating issues in Microsoft’s research on Majorana for over five years now,” said Frolov. “Our work has so far resulted in two Nature retractions, several corrections, and two expressions of concern.”
Frolov also noted that other scientists have expressed skepticism about Microsoft’s quantum claims, notably Henry Legg from the University of St Andrews. And The Register understands that Simone Severini, when he was head of Amazon’s quantum technology group (before he joined Google), told Amazon CEO Andy Jassy that Microsoft’s research paper on the Majorana 1 failed to demonstrate its claims.
Regarding the nanowire paper EEoC, Frolov explains that in 2019 Science received a joint submission involving two papers: A theoretical paper from Microsoft and the University of Copenhagen, and an experimental one from Microsoft, the University of Copenhagen, the University of California Santa Barbara, and Yale.
Science, he said, asked him to review the papers and initially he was impressed. But upon closer examination, he concluded the data was unlikely to be representative, and Science rejected both. The authors, he said, then combined their papers into one and resubmitted it. Science accepted it in 2020 and a Microsoft press release heralded the new discovery.
During that period, Frolov said Mourik and he identified other problems in Microsoft papers on Majorana.
“Given the strange rejection/merger/acceptance history of the Science paper, and my responsibility to this process as a referee, I reached out to the Microsoft authors, Lutchyn and Marcus, to ask for additional data,” said Frolov. “They refused the requests for a few months until [Science editor] Jelena Stajic stepped in. At that point the authors shared 50 MB of additional data, which I would estimate to be perhaps a day worth of data-taking.”
Based on this additional data, Frolov and Mourik concluded that the claims in the 2020 Science paper were not supported.
“The authors had selected data to represent a Majorana signal, but their evidence was so weak that even a small amount of additional data undermined their discovery claim,” said Frolov. “We wrote an extensive analysis and sent it to Science as a complaint. The complaint was initially confidential, but a version is now online here.”
That was the basis for the EEoC that is now being replaced with a correction.
There were other signals that the Microsoft research was flawed, Frolov says.
“In parallel to our work on the Microsoft-Copenhagen article, another group, from Austria, worked with the same wires from the same chip; like studying grass from the same lawn. They repeated the experiments and saw similar phenomena. They shared 10x more data with us than the Microsoft group, about 600 MB in total. From the larger volume of data, it was clear that they did not see Majorana.”
Frolov said Science and Nature both published data from these non-confirmatory experiments. “This move is almost unprecedented in the history of publishing, when two of the highest impact journals go out of their way to debunk a claim,” he said.
The only difference, we believe, is that Microsoft cherry-picked what they showed in their paper, a pattern they exhibit in other publications
“These publications could be called ‘negative replications’. But in fact, the data were fully replicated. The only difference, we believe, is that Microsoft cherry-picked what they showed in their paper, a pattern they exhibit in other publications. The cherry-picking is extreme, to the level that it alters how an expert would interpret the data. It is not only a question of exaggerating or highlighting a nice result, but data selection in support of unjustified claims of new fundamental discoveries.”
Frolov said the selective use of data is the same problem that Mourik and he flagged in two retracted Nature papers from Microsoft.
“It’s unfortunate to see Science making a different decision here,” he said, with regard to the expected removal of the EEoC on the nanowire paper.
According to Frolov, Henry Legg is negotiating with Science to publish concerns about the Microsoft Majorana claims from February.
Marcus argues that Legg’s concerns reflect his background as a theoretician, rather than an experimentalist who has to deal with the complexity of quantum hardware, which involves configuring devices using 15 separate controls that define where to look for Majoranas.
But Legg, in response, told us via email, “Instead of taking the outcomes of [the Science investigation] to heart, it seems like the authors who still work for Microsoft have doubled down and are now trying to tell the world they have built a topological qubit based on what looks like gross cherry picking of data.”
Legg said, “I may only be a humble theorist, but every high-schooler learns that in science you should not cherry-pick data to fit a narrative. This is a lesson Microsoft has repeatedly failed to learn, which is why it’s no surprise that after hundreds of millions of dollars, Microsoft and I have built the exact same number of topological qubits: zero.”
A crusade against quantum computing?
According to Marcus, Science only offered the option of a correction to resolve the matter, but he considers it vindication.
“There are no incorrect statements in the paper,” Marcus said. “Not one single incorrect statement or figure or data set or conclusion or interpretation that was wrong in that paper. The only thing that the expression of concern [raised was] you didn’t describe in detail how you tuned up the device. You didn’t define what the tunneling regime is. That’s now been added information in case anybody wants to look at that. So it’s a total clearing of the paper, and I’m grateful for Science for figuring that out.”
Marcus contends Frolov and Mourik have been conducting a harassment campaign against his work, calling conferences, writing to grant managers, texting students at conferences, and lobbying on social media.
“They really want to shut down a field,” he said.
“Frolov seems to have made this into his profession,” Marcus added. “And he’s not publishing very many papers anymore. Mostly what he’s doing is social media stuff and critiquing me, and it seems to me basically attacking anyone who has anything to do or had anything to do with Microsoft.”
Michael Manfra, director of Purdue Quantum Science and Engineering Institute and scientific director of the Microsoft Quantum Lab, West Lafayette, told The Register in a phone interview that the complaints raised about the nanowire paper were unfounded.
No paper is flawless, but they found no evidence of malfeasance. No evidence of any data manipulation
“No paper is flawless,” he said, “but they found no evidence of malfeasance. No evidence of any data manipulation. And at the end of the day the conclusions were substantiated after this very rigorous review process. Science is doing the right thing by lifting the expression of concern.”
Acknowledging his ties to Microsoft, Manfra said the criticism Marcus and his coauthors received went beyond the bounds of professional norms.
“This is personal,” he said. “You don’t use social media to launch ad hominem attacks against individuals. That’s not within the rules of normal scientific inquiry and disagreement.
“All of this has been focused on Charlie Marcus, and was there misconduct. You could turn this around and say, ‘Okay, now that the dust has settled, did the accusing parties act in good faith and were they interested in getting at scientific truth or something else?’”
The concerns raised by Frolov and Mourik cannot be entirely discounted. Marcus recalled how Frolov and Mourik succeeded in having a 2018 Nature paper, “Quantized Majorana conductance,” retracted.
One of the paper authors was Leo Kouwenhoven, a quantum physics professor at TU Delft in the Netherlands and a former partner manager and principal researcher at Microsoft.
In that instance, the concerns raised by Frolov and Mourik proved to be valid. According to TU Delft’s news outlet Delta, Kouwenhoven acknowledged mistakes during an appearance on the Atlas podcast in 2022.
But Marcus contends there was a personal connection as well. “Leo Kouwenhoven was Frolov and Mourik’s advisor,” he explained. “These guys had a lot of fun getting their advisor’s paper retracted, and they were successful, and they did it.
“And then they went after the next big guy in the field, who’s me,” said Marcus, noting that he too was affiliated with Microsoft.
“So they picked a paper of mine,” he said. “Actually, they picked seven papers of mine, and all about Majoranas. They began by saying, please send us all of your data, everything, lab notebooks, PowerPoint files. Everything. And you have to. You signed up with Science to publish it. …I’ve never gotten a letter like this before in 30 years of being a professor. I’ve never gotten somebody saying to me, ‘I demand all of your files and notebooks.’”
Marcus says that Frolov’s and Mourik’s debunking efforts have chilled researcher enthusiasm in the field.
“For the last four years, every time I talked to a graduate student, or we went somewhere, they said, ‘Oh, yeah, well, we hear from Frolov that you’re hiding your data and you’re cheating for whatever.’ No other group wants to have that on their head. So basically in the public sector, that field has died. To that extent, Frolov and Mark have been successful in killing anybody who would be interested in working on that problem. Nobody will touch it because they don’t want to get attacked.”
Marcus also believes Microsoft deserves credit for throwing resources at a very hard scientific problem.
“I think it’s a very courageous thing,” he said. “It’s a very hard problem. I don’t know whether it’s too hard or not too hard, or whatever. But it’s a gutsy thing to try to do a really end run on quantum error correction, which, by the way, is its own nightmare.”
“The problem with Microsoft is that, okay, maybe the press release was a little bit over the top or whatever,” he said. “I don’t think it matters that much. They’re trying something really hard. And they’re reporting progress outward. …I don’t want to argue whether the press release was accurate or not. I’m not interested in that. What I’m interested in is: is this a hard physics problem, which is extremely interesting? And the answer is yes. Will they crack this nut? I don’t know.”
Discourse and skepticism are all part of the scientific process
For its part, Microsoft welcomed Science’s resolution of the concerns.
“It’s our understanding that the authors have been working with Science through the journal’s standard process to address any concerns,” a Microsoft spokesperson said in an emailed statement. “It will be great to see this matter resolved. Discourse and skepticism are all part of the scientific process. That is why we are committed to continued open discussion and engagement on our research, so that everyone can build on what others have discovered and learned.”
Undaunted by the criticism, Microsoft continues publishing Majorana-related research.®