Who wants the skin of the Sigaps system? All researchers, or almost. For years and in particular since the Covid-19 crisis, the System for querying, managing and analyzing scientific publications has been in the sights of many specialists in scientific ethics. On Wednesday March 20, it was even the subject of a conference-debate within the High Council for the Evaluation of Research and Higher Education (Hcéres), in Paris. Soberly titled “What future for the Sigaps score?”, the event took place in the cozy atmosphere befitting measured exchanges of experts. However, the criticisms were not very friendly.
Initiated at the CHRU of Lille in 2002, the Sigaps project was imposed a few years later on all health establishments carrying out research, with the ambition of providing the government with a reliable and objective evaluation tool to measure the production of scientists. How it works is quite simple: when researchers publish their work in scientific journals, they obtain Sigaps points. The number of points awarded depends on the ranking of the journal (between A + and E) and the order of signatures: the first and last author being the best rewarded, then the second and penultimate, etc.
These Sigaps points are then used to measure part of the funding that medical research establishments – mainly University Hospital Centers (CHU) – can receive from the Ministry of Health. Nothing trivial, since the amounts are most often in the millions, or even tens of millions of euros. And if the Sigaps system was not originally designed for this, these points can also determine the recruitment of researchers or access to new, better-paid positions. Thus, establishments can require that a researcher collect at least 200 Sigaps points before applying for a position as University Lecturer-Hospital Practitioner (MCU-PH), 400 points for University Professor-Hospital Practitioner (PU-PH). ). The process is surprising in that the prerequisite has no legal status. It was decided a few years ago by “a ministerial advisor who scribbled them on a piece of table”, slip a few scientists who refuse to name the inventor. But the result is there: researchers and establishments have every interest in accumulating as many points as possible.
“Biased and manipulated”
“At the time of its launch, Sigaps was progress, but in 20 years, the world of scientific publication has profoundly evolved, it has gone from that of care bears to a swimming pool full of sharks,” believes Bruno Riou, dean of the Faculty of Health of the Sorbonne University, invited to the conference. “The tool is biased and manipulatable,” says another researcher, less bluntly. The main criticism is that Sigaps measures the quantitative and not the qualitative aspect of scientific production. During the conference, Patrick Devos, statistician at the CHRU of Lille and head of the Sigaps project, wanted to defend his baby by maintaining that he still takes into account the quality of the journals. But the argument is brushed aside by other researchers: the scientific world has witnessed in recent years an explosion in the number of scientific journals, including “predatory” and “opportunistic” ones which publish any work with little or no proofreading, provided that the authors pay. “The MDPI and Frontiers journals publish at a pace that is not compatible with scientific requirements,” illustrates Thierry Damerval, CEO of the National Research Agency. Other journals can also be exploited by unscrupulous researchers.
The best example is the system set up at the IHU in Marseille by Didier Raoult. The former director of the Marseille establishment and several of its members have directed scientific journals (such as New microbes and new infections, Travel Medicine and Infectious Disease Or International Journal of Antimicrobial Agents) in which they published hundreds of studies signed by their colleagues or themselves. This work was sometimes reread and validated in record time, up to less than 24 hours, whereas this process normally takes months.
Worse, the interest of dozens of these works has been judged to be more than relative by many experts: some only consist of one, two or three pages and relate to a handful of patients. A practice that can be explained by two hypotheses. Either the IHU teams carried out around ten large-scale studies and divided them into a multitude of small publications, a method called “salami-slicing”, or they published a study at each time they took a sample from a patient. In any case, these “tricks” allowed the IHU to obtain “10 million euros per year” for years. However, as the “Sigaps cake” is not scalable (its annual budget is set at around 1.6 billion euros), those who obtain a large share of it using dubious techniques penalize others. Enough to provoke some anger.
But the IHU of Marseille is not the only establishment to have engaged in such practices. 23% of Inserm researchers questioned during a survey conducted by the Sorbonne Sociological Analysis Methods Study Group (Gemass) acknowledged including a colleague as the author of their study even if he or she did not not contributed to their work. “There is a sort of informal economy that operates in certain research units, with exchanges around these signatures,” explains sociologist Michel Dubois, research director at the CNRS. The most common is for a team leader to sign the work of his subordinates, even if he did not participate in it. Enough to explain how Didier Raoult was able to author thousands of studies during his career. An expert preferring to remain anonymous sums up the problem: “With the right contacts in the right journals, it is possible to publish easily, add loved ones to the bylines and boost their career.”
“In addition to publication in friendly journals and salami, we can also be concerned about articles written by artificial intelligence,” explains Louise Benoit, head of clinic in gynecological oncological surgery at Georges Pompidou Hospital. The researcher again recalls that it is difficult to publish studies which present negative or insignificant results. “However, if we discover that a medication does not work, it is very important to say so,” she insists. However, by pushing for publication, the Sigaps system would divert researchers from this type of research.
Remove or reform?
Faced with these excesses, the world of research is therefore imagining the future of Sigaps. Especially since “French biomedical research is in decline”, insists Guillaume Gellé, president of France Université. After all, working for seven years on a research project and publishing your results in Nature Or The New England Journal of Medicine – the most prestigious for biomedical research – today earns as many Sigaps points as publishing a dozen studies of poor quality in minor journals. “However, it is not comparable,” says Alain Fischer, president of the Academy of Sciences. Good research takes time, and it must be valued.
The majority agrees on one priority: abolishing the use of points allowing one to obtain MCU or PU-PH positions. “A quantitative concept for developing a researcher’s career is, from my point of view, not acceptable,” further indicates Alain Fischer. Alain Riou agrees: “concerning individual assessment, it is clear that Sigaps must be discarded and replaced by a qualitative index”. The question of maintaining Sigaps for financing research establishments seems more complex. The debate is still open, even if it seems to be moving at least towards the addition of a bibliometric tool allowing the quality of scientific production to be evaluated as a priority. Invited to the conference, representatives of the Ministry of Labor, Health and Solidarity as well as the Ministry of Higher Education and Research said they were attentive to the researchers’ debates. Some would undoubtedly have preferred that they take the lead and sanction, at a minimum, the most obvious abuses.
.