Are numerical scores important for grant assessment? A cross-sectional study.

Ivan Buljan, David G Pina, Antonija Mijatović, Ana Marušić
Author Information
  1. Ivan Buljan: Department of Psychology, Faculty of Humanities and Social Sciences in Split, University of Split, Split, Croatia. ORCID
  2. David G Pina: European Research Executive Agency, European Commission, Brussels, Belgium. ORCID
  3. Antonija Mijatović: Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, Medical School of Split, Split, Croatia.
  4. Ana Marušić: Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, Medical School of Split, Split, Croatia. ORCID

Abstract

In the evaluation of research proposals, reviewers are often required to provide their opinions using various forms of quantitative and qualitative criteria. In 2020, the European Commission removed, for the Marie Skłodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) funding scheme, the numerical scores from the individual evaluations but retained them in the consensus report. This study aimed to assess whether there were any differences in reviewer comments' linguistic characteristics after the numerical scoring was removed, compared to comments from 2019 when numerical scoring was still present. This was an observational study and the data were collected for the Marie Skłodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) evaluation reports from the calls of 2019 and 2020, for both individual and consensus comments and numerical scores about the quality of the research proposal on three evaluation criteria: Excellence, Impact and Implementation. All comments were analyzed using the Linguistic Inquiry and Word Count (LIWC) program. For both years, the comments for proposal's strengths were written in a style that reflects objectivity, clout, and positive affect, while in weaknesses cold and objective style dominated, and that pattern remained stable across proposal status and research domains. Linguistic variables explained a very small proportion of the variance of the differences between 2019 and 2020 (McFadden R =0.03). Removing the numerical scores was not associated with the differences in linguistic characteristics of the reviewer comments. Future studies should adopt a qualitative approach to assess whether there are conceptual changes in the content of the comments.

Keywords

References

  1. Elife. 2021 Jan 13;10: [PMID: 33439120]
  2. BMJ Open. 2015 Jul 02;5(7):e008380 [PMID: 26137884]
  3. Scientometrics. 2018;117(1):313-329 [PMID: 30220747]
  4. PLoS Comput Biol. 2022 Aug 18;18(8):e1010371 [PMID: 35980892]
  5. Res Integr Peer Rev. 2022 Mar 4;7(1):2 [PMID: 35246264]
  6. Res Integr Peer Rev. 2020 Apr 30;5:6 [PMID: 32368354]
  7. PLoS One. 2018 May 11;13(5):e0196914 [PMID: 29750807]
  8. Acad Med. 2015 Jan;90(1):69-75 [PMID: 25140529]
  9. J Empir Res Hum Res Ethics. 2023 Oct;18(4):250-262 [PMID: 37526052]
  10. PLoS One. 2012;7(9):e46054 [PMID: 23029386]
  11. F1000Res. 2021 Jun 15;10:471 [PMID: 34394917]
  12. Res Integr Peer Rev. 2023 Jul 24;8(1):10 [PMID: 37488628]

Word Cloud

Created with Highcharts 10.0.0numericalcommentsscoresevaluationresearch2020MSCAstudydifferences2019LinguisticusingqualitativeremovedMarieSkłodowska-CurieActionsInnovativeTrainingNetworksITNindividualconsensusassesswhetherreviewerlinguisticcharacteristicsscoringproposalprogramstyleproposalsreviewersoftenrequiredprovideopinionsvariousformsquantitativecriteriaEuropeanCommissionfundingschemeevaluationsretainedreportaimedcomments'comparedstillpresentobservationaldatacollectedreportscallsqualitythreecriteria:ExcellenceImpactImplementationanalyzedInquiryWordCountLIWCyearsproposal'sstrengthswrittenreflectsobjectivitycloutpositiveaffectweaknessescoldobjectivedominatedpatternremainedstableacrossstatusdomainsvariablesexplainedsmallproportionvarianceMcFaddenR=003RemovingassociatedFuturestudiesadoptapproachconceptualchangescontentimportantgrantassessment?cross-sectionalCharacteristicsPeerReview

Similar Articles

Cited By