Quantitative research assessment: using metrics against gamed metrics.

John P A Ioannidis, Zacharias Maniadis
Author Information
  1. John P A Ioannidis: Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, SPRC, MSOB X306, 1265 Welch Rd, Stanford, CA, 94305, USA. jioannid@stanford.edu. ORCID
  2. Zacharias Maniadis: SInnoPSis (Science and Innovation Policy and Studies) Unit, Department of Economics, University of Cyprus, Nicosia, Cyprus.

Abstract

Quantitative bibliometric indicators are widely used and widely misused for research assessments. Some metrics have acquired major importance in shaping and rewarding the careers of millions of scientists. Given their perceived prestige, they may be widely gamed in the current "publish or perish" or "get cited or perish" environment. This review examines several gaming practices, including authorship-based, citation-based, editorial-based, and journal-based gaming as well as gaming with outright fabrication. Different patterns are discussed, including massive authorship of papers without meriting credit (gift authorship), team work with over-attribution of authorship to too many people (salami slicing of credit), massive self-citations, citation farms, H-index gaming, journalistic (editorial) nepotism, journal impact factor gaming, paper mills and spurious content papers, and spurious massive publications for studies with demanding designs. For all of those gaming practices, quantitative metrics and analyses may be able to help in their detection and in placing them into perspective. A portfolio of quantitative metrics may also include indicators of best research practices (e.g., data sharing, code sharing, protocol registration, and replications) and poor research practices (e.g., signs of image manipulation). Rigorous, reproducible, transparent quantitative metrics that also inform about gaming may strengthen the legacy and practices of quantitative appraisals of scientific work.

Keywords

References

  1. PLoS One. 2018 Sep 26;13(9):e0195773 [PMID: 30256792]
  2. J Empir Res Hum Res Ethics. 2018 Apr;13(2):187-195 [PMID: 29345178]
  3. Science. 2018 Mar 2;359(6379): [PMID: 29496846]
  4. PLoS Biol. 2019 Aug 12;17(8):e3000384 [PMID: 31404057]
  5. Account Res. 2017;24(2):80-98 [PMID: 27797590]
  6. Nature. 2019 Dec;576(7786):210-212 [PMID: 31827288]
  7. J Med Ethics. 2013 Aug;39(8):509-12 [PMID: 22865926]
  8. JAMA. 2023 Feb 28;329(8):637-639 [PMID: 36719674]
  9. JAMA. 1997 Aug 20;278(7):579-85 [PMID: 9268280]
  10. Eur J Clin Invest. 2019 Sep;49(9):e13151 [PMID: 31206647]
  11. Anaesthesia. 2021 Apr;76(4):472-479 [PMID: 33040331]
  12. Scientometrics. 2023;128(5):3171-3184 [PMID: 37101975]
  13. JAMA. 2023 Apr 18;329(15):1253-1254 [PMID: 36939740]
  14. Nature. 2022 Aug;608(7921):135-145 [PMID: 35732238]
  15. JAMA. 2023 Aug 22;330(8):702-703 [PMID: 37498593]
  16. PLoS Biol. 2021 Mar 1;19(3):e3001107 [PMID: 33647013]
  17. Eur J Cancer. 2016 Oct;66:1-8 [PMID: 27500368]
  18. Med J Aust. 2020 Apr;212(6):247-249.e1 [PMID: 32017115]
  19. Proc Natl Acad Sci U S A. 2018 Mar 13;115(11):2557-2560 [PMID: 29487213]
  20. BMJ. 2007 Feb 3;334(7587):223 [PMID: 17272539]
  21. Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72 [PMID: 16275915]
  22. J Med Ethics. 2014 May;40(5):346-8 [PMID: 23955369]
  23. J Clin Epidemiol. 2021 Oct;138:40-48 [PMID: 34182146]
  24. BMJ. 2022 Nov 28;379:e071517 [PMID: 36442874]
  25. BMJ. 1994 Dec 3;309(6967):1482 [PMID: 7804054]
  26. CMAJ. 2007 Jan 2;176(1):41-6 [PMID: 17200389]
  27. PLoS Biol. 2020 Oct 16;18(10):e3000918 [PMID: 33064726]
  28. PLoS Biol. 2016 Sep 06;14(9):e1002541 [PMID: 27599104]
  29. J Psychosom Res. 2015 Jan;78(1):7-11 [PMID: 25466321]
  30. Science. 2023 Jan 27;379(6630):313 [PMID: 36701446]
  31. Sci Eng Ethics. 2018 Apr;24(2):655-668 [PMID: 28397175]
  32. mBio. 2016 Jun 07;7(3): [PMID: 27273827]
  33. Nature. 2019 Aug;572(7771):578-579 [PMID: 31455906]
  34. BMJ. 1994 Dec 3;309(6967):1456-7 [PMID: 7804037]
  35. Nature. 2015 Apr 23;520(7548):429-31 [PMID: 25903611]
  36. Radiology. 2014 May;271(2):472-8 [PMID: 24475845]
  37. Nature. 2021 Mar;591(7851):516-519 [PMID: 33758408]
  38. FEBS Lett. 2021 Jul;595(13):1751-1757 [PMID: 34180058]
  39. J Clin Epidemiol. 2010 Jun;63(6):660-4 [PMID: 20097531]
  40. J Clin Epidemiol. 2020 Dec;128:13-19 [PMID: 32781115]
  41. Nature. 2018 Sep;561(7722):167-169 [PMID: 30209384]
  42. BMJ. 2011 Oct 25;343:d6128 [PMID: 22028479]
  43. Radiology. 2011 May;259(2):479-86 [PMID: 21386051]
  44. Anaesthesia. 2021 Apr;76(4):444-447 [PMID: 33124075]
  45. J Korean Med Sci. 2023 Oct 23;38(41):e333 [PMID: 37873630]
  46. Fertil Steril. 2023 Sep;120(3 Pt 1):412-414 [PMID: 36842709]
  47. PLoS Biol. 2021 Nov 23;19(11):e3001133 [PMID: 34813595]
  48. Br J Dermatol. 2018 Jun;178(6):1464-1465 [PMID: 29663321]
  49. BMJ Open. 2020 Feb 9;10(2):e035561 [PMID: 32041864]
  50. Account Res. 2014;21(3):176-97 [PMID: 24325212]
  51. Arch Phys Med Rehabil. 2014 Mar;95(3):418-28 [PMID: 24215989]
  52. PLoS Med. 2007 Jan;4(1):e19 [PMID: 17227134]
  53. Scientometrics. 2011 Apr;87(1):85-98 [PMID: 21472020]
  54. Science. 2023 Feb 24;379(6634):740-741 [PMID: 36821673]
  55. Account Res. 2011 Mar;18(2):76-90 [PMID: 21390872]
  56. Gigascience. 2019 Jun 1;8(6): [PMID: 31144712]
  57. Nature. 2013 Aug 29;500(7464):510-1 [PMID: 23985850]
  58. Nature. 2017 Oct 17;550(7676):303 [PMID: 29052636]
  59. Sci Eng Ethics. 2019 Jun;25(3):771-789 [PMID: 29460082]

Grants

  1. 857636/H2020 Excellent Science

MeSH Term

Humans
Bibliometrics
Journal Impact Factor
Publishing
Authorship

Word Cloud

Created with Highcharts 10.0.0gamingmetricspracticesresearchmayauthorshipquantitativewidelymassiveQuantitativeindicatorsgamedperish"includingpaperscreditworkfactorspuriousalsoegsharingbibliometricusedmisusedassessmentsacquiredmajorimportanceshapingrewardingcareersmillionsscientistsGivenperceivedprestigecurrent"publish"getcitedenvironmentreviewexaminesseveralauthorship-basedcitation-basededitorial-basedjournal-basedwelloutrightfabricationDifferentpatternsdiscussedwithoutmeritinggiftteamover-attributionmanypeoplesalamislicingself-citationscitationfarmsH-indexjournalisticeditorialnepotismjournalimpactpapermillscontentpublicationsstudiesdemandingdesignsanalysesablehelpdetectionplacingperspectiveportfolioincludebestdatacodeprotocolregistrationreplicationspoorsignsimagemanipulationRigorousreproducibletransparentinformstrengthenlegacyappraisalsscientificassessment:usingBibliometricsCitationsFraudGamingGiftImpactResearchassessmentSelf-citations

Similar Articles

Cited By