OpenLB
Open Library of Bioscience
Advanced Search
Scientific reports
11 04, 2024;14 (1): 26626.

A mixed-methods survey and focus group study to understand researcher and clinician preferences for a Journal transparency Tool.

Jeremy Y Ng, Henry Liu, Mehvish Masood, Jassimar Kochhar, David Moher, Alan Ehrlich, Alfonso Iorio, Kelly D Cobey
Author Information
  1. Jeremy Y Ng: Centre for Journalology, Methodological and Implementation Research Program, The Ottawa Hospital Research Institute, Ottawa, Canada. ng.jeremyy@gmail.com. ORCID
  2. Henry Liu: Centre for Journalology, Methodological and Implementation Research Program, The Ottawa Hospital Research Institute, Ottawa, Canada. ORCID
  3. Mehvish Masood: Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ONs, Canada. ORCID
  4. Jassimar Kochhar: Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ONs, Canada. ORCID
  5. David Moher: Centre for Journalology, Methodological and Implementation Research Program, The Ottawa Hospital Research Institute, Ottawa, Canada. ORCID
  6. Alan Ehrlich: EBSCO Information Services, Ipswich, MA, USA. ORCID
  7. Alfonso Iorio: Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ONs, Canada. ORCID
  8. Kelly D Cobey: School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada. ORCID
PMID: 39496735 DOI: 10.1038/s41598-024-77790-z

Abstract

Transparency within biomedical research is essential for research integrity, credibility, and reproducibility. To increase adherence to optimal scientific practices and enhance transparency, we propose the creation of a journal transparency tool (JTT) that will allow users to obtain information about a given scholarly journal's operations and transparency policies. This study is part of a program of research to obtain user preferences to inform the proposed JTT. Here, we report on our consultation with clinicians and researchers. This mixed-methods study was conducted in two parts. The first part involved a cross-sectional survey conducted on a random sample of authors from biomedical journals. The survey asked clinicians and researchers about the inclusion of a series of potential scholarly metrics and user features in the proposed JTT. Quantitative survey items were summarized with descriptive statistics. Thematic content analysis was employed to analyze text-based responses. Subsequent focus groups used survey responses to further explore the inclusion of items in the JTT. Items with less than 70% agreement were used to structure discussion points during these sessions. Participants voted on the use of user features and metrics to be considered within the journal tool after each discussion. Thematic content analysis was conducted on interview transcripts to identify the core themes discussed. A total of 632 participants (5.5% response rate) took part in the survey. A collective total of 74.7% of respondents found it either 'occasionally, 'often', or 'almost always' difficult to determine if health information online is based on reliable research evidence. Twenty-two participants took part in the focus groups. Three user features and five journal tool metrics were major discussion points during these sessions. Thematic analysis of interview transcripts resulted in six themes. The use of registration was the only item to not meet the 70% threshold after both the survey and focus groups. Participants demonstrated low scholarly communication literacy when discussing tool metric suggestions. Our findings suggest that the JTT would be valuable for both researchers and clinicians. The outcomes of this research will contribute to developing and refining the tool in accordance with researchers and clinicians.

Keywords

Clinician Health literacy Journal metrics Journal transparency tool Researcher Transparency

References

  1. WMA - The World Medical Association n.d. https://www.wma.net/ .
  2. Robishaw, J. D., DeMets, D. L., Wood, S. K., Boiselle, P. M. & Hennekens, C. H. Establishing and maintaining research integrity at academic institutions: Challenges and opportunities. Am. J. Med. 133, e87-90. https://doi.org/10.1016/j.amjmed.2019.08.036 (2020). [DOI: 10.1016/j.amjmed.2019.08.036]
  3. Zhaksylyk, A., Zimba, O., Yessirkepov, M. & Kocyigit, B. F. Research integrity: Where we are and where we are heading. J. Korean Med. Sci. 38, e405. https://doi.org/10.3346/jkms.2023.38.e405 (2023). [DOI: 10.3346/jkms.2023.38.e405]
  4. Lee, C. J. & Moher, D. Promote scientific integrity via journal peer review data. Science 357, 256–257. https://doi.org/10.1126/science.aan4141 (2017). [DOI: 10.1126/science.aan4141]
  5. Brezgov, S. Google Scholar is filled with Junk Science (2019). https://scholarlyoa.com/google-scholar-is-filled-with-junk-science/ .
  6. Hughes, B., Joshi, I., Lemonde, H. & Wareham, J. Junior physician’s use of Web 2.0 for information seeking and medical education: A qualitative study. Int. J. Med. Inf. 78, 645–655. https://doi.org/10.1016/j.ijmedinf.2009.04.008 (2009). [DOI: 10.1016/j.ijmedinf.2009.04.008]
  7. Duran-Nelson, A., Gladding, S., Beattie, J. & Nixon, L. J. Should We Google It? Resource use by internal medicine residents for point-of-care clinical decision making. Acad. Med. 88, 788–794. https://doi.org/10.1097/ACM.0b013e31828ffdb7 (2013). [DOI: 10.1097/ACM.0b013e31828ffdb7]
  8. Weng, Y. et al. Information-searching behaviors of main and allied health professionals: A nationwide survey in Taiwan. J. Eval. Clin. Pract. 19, 902–908. https://doi.org/10.1111/j.1365-2753.2012.01871.x (2013). [DOI: 10.1111/j.1365-2753.2012.01871.x]
  9. Boeker, M., Vach, W. & Motschall, E. Google Scholar as replacement for systematic literature searches: Good relative recall and precision are not enough. BMC Med. Res. Methodol. 13, 131. https://doi.org/10.1186/1471-2288-13-131 (2013). [DOI: 10.1186/1471-2288-13-131]
  10. Haddaway, N. R., Collins, A. M., Coughlin, D. & Kirk, S. The role of google scholar in evidence reviews and its applicability to grey literature searching. PLoS ONE 10, e0138237. https://doi.org/10.1371/journal.pone.0138237 (2015). [DOI: 10.1371/journal.pone.0138237]
  11. Manca, A., Moher, D., Cugusi, L., Dvir, Z. & Deriu, F. How predatory journals leak into PubMed. Can. Med. Assoc. J. 190, E1042–E1045. https://doi.org/10.1503/cmaj.180154 (2018). [DOI: 10.1503/cmaj.180154]
  12. Journal Transparency Tool. Cent Journalology n.d. http://www.ohri.ca/auditfeedback .
  13. Dopp, A. R., Parisi, K. E., Munson, S. A. & Lyon, A. R. A glossary of user-centered design strategies for implementation experts. Transl. Behav. Med. 9, 1057–1064. https://doi.org/10.1093/tbm/iby119 (2019). [DOI: 10.1093/tbm/iby119]
  14. Olson, G. M. & Olson, J. S. User-centered design of collaboration technology. J. Organ Comput. 1, 61–83. https://doi.org/10.1080/10919399109540150 (1991). [DOI: 10.1080/10919399109540150]
  15. Ricketts, A., Lalu, M.M., Proulx, L., Halas, M., Castillo, G., Almoli, E., et al. Establishing patient perceptions and preferences for a journal authenticator tool to support health literacy: A mixed-methods survey and focus group study. In review (2021). https://doi.org/10.21203/rs.3.rs-875992/v1 .
  16. Ng, J. et al. Publisher preferences for a journal transparency tool: A Delphi study protocol. https://doi.org/10.17605/OSF.IO/UR67D (2022).
  17. Open Science Framework (OSF). Title Page n.d. https://osf.io/ .
  18. Ng, J. Y. et al. Researcher and clinician preferences for a journal transparency tool: A mixed-methods survey and focus group study. medRxiv. https://doi.org/10.17605/OSF.IO/AS3CY (2023).
  19. Eysenbach, G. Improving the quality of web surveys: the checklist for reporting results of internet E-surveys (CHERRIES). J. Med. Internet Res 6, e34. https://doi.org/10.2196/jmir.6.3.e34 (2004). [DOI: 10.2196/jmir.6.3.e34]
  20. Tong, A., Sainsbury, P. & Craig, J. Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. Int. J. Qual. Health Care J. Int. Soc. Qual. Health Care 19, 349–357. https://doi.org/10.1093/intqhc/mzm042 (2007). [DOI: 10.1093/intqhc/mzm042]
  21. SurveyMonkey n.d. https://www.surveymonkey.com/ .
  22. Aria, M. pubmedR: Gathering Metadata about Publications, Grants, Clinical Trials from “PubMed” Database (2020). https://cran.r-project.org/web/packages/pubmedR/index.html . Accessed 16 March 2024.
  23. Willis, J. V. et al. Knowledge and motivations of training in peer review: An international cross-sectional survey. PLoS ONE 18, e0287660. https://doi.org/10.1371/journal.pone.0287660 (2023). [DOI: 10.1371/journal.pone.0287660]
  24. Cobey, K. D. et al. Editors-in-chief perceptions of patients as (co) authors on publications and the acceptability of ICMJE authorship criteria: A cross-sectional survey. Res. Involv. Engagem. 7, 39. https://doi.org/10.1186/s40900-021-00290-1 (2021). [DOI: 10.1186/s40900-021-00290-1]
  25. Guest, G., Namey, E. & McKenna, K. How many focus groups are enough? Building an evidence base for nonprobability sample sizes. Field Methods 29, 3–22. https://doi.org/10.1177/1525822X16639015 (2017). [DOI: 10.1177/1525822X16639015]
  26. Joffe, H., & Yardley, L. Research methods for clinical and health psychology. Content Themat. Anal. 56–68 (2003).
  27. Mullen, R., Kydd, A., Fleming, A. & McMillan, L. A practical guide to the systematic application of nominal group technique. Nurse Res. 29, 14–20. https://doi.org/10.7748/nr.2021.e1777 (2021). [DOI: 10.7748/nr.2021.e1777]
  28. Hsieh, H.-F. & Shannon, S. E. Three approaches to qualitative content analysis. Qual. Health Res. 15, 1277–1288. https://doi.org/10.1177/1049732305276687 (2005). [DOI: 10.1177/1049732305276687]
  29. Hsu, C.-C. & Sandford, B. The Delphi technique: Making sense of consensus. Pract. Assess. Res. Eval. 12, 10 (2007).
  30. Krueger, R. A. Focus Groups: A Practical Guide for Applied Research (SAGE, 2014).
  31. Saunders, B. et al. Saturation in qualitative research: Exploring its conceptualization and operationalization. Qual. Quant. 52, 1893–1907. https://doi.org/10.1007/s11135-017-0574-8 (2018). [DOI: 10.1007/s11135-017-0574-8]
  32. Rice, D. B., Skidmore, B. & Cobey, K. D. Dealing with predatory journal articles captured in systematic reviews. Syst. Rev. 10, 175. https://doi.org/10.1186/s13643-021-01733-2 (2021). [DOI: 10.1186/s13643-021-01733-2]
  33. Battineni, G. et al. Factors affecting the quality and reliability of online health information. Digit Health 6, 2055207620948996. https://doi.org/10.1177/2055207620948996 (2020). [DOI: 10.1177/2055207620948996]
  34. Swanberg, S. M., Thielen, J. & Bulgarelli, N. Faculty knowledge and attitudes regarding predatory open access journals: A needs assessment study. J. Med. Libr. Assoc. JMLA 108, 208–218. https://doi.org/10.5195/jmla.2020.849 (2020). [DOI: 10.5195/jmla.2020.849]
  35. Jebb, A. T., Ng, V. & Tay, L. A review of key likert scale development advances: 1995–2019. Front. Psychol. https://doi.org/10.3389/fpsyg.2021.637547 (2021). [DOI: 10.3389/fpsyg.2021.637547]
  36. Saman, E.G.Z. Promoting Awareness, Reflection, and Dialogue to Deter Students’ Predatory Publishing. Predatory Pract. Sch. Publ. Knowl. Shar., Routledge (2023).
  37. Dale, J. & Craft, A. R. Professional applications of information literacy: Helping researchers learn to evaluate journal quality. Ser. Rev. 47, 129–135. https://doi.org/10.1080/00987913.2021.1964337 (2021). [DOI: 10.1080/00987913.2021.1964337]
  38. Otike, F., Bouaamri, A. & Hajdu, B. Á. Predatory publishing: A catalyst of misinformation and disinformation amongst academicians and learners in developing countries. Ser. Libr. 83, 81–98. https://doi.org/10.1080/0361526X.2022.2078924 (2022). [DOI: 10.1080/0361526X.2022.2078924]
  39. Ciro, J. B. & Pérez, J. H. Pedagogical strategy for scholarly communication literacy and avoiding deceptive publishing practices. J. Librariansh Inf. Sci. https://doi.org/10.1177/09610006231187686 (2023). [DOI: 10.1177/09610006231187686]
  40. Power, H. predatory publishing: how to safely navigate the waters of open access. Can. J. Nurs. Res. 50, 3–8. https://doi.org/10.1177/0844562117748287 (2018). [DOI: 10.1177/0844562117748287]
  41. Teixeira da Silva, J. A. et al. An integrated paradigm shift to deal with ‘predatory publishing’. J. Acad. Librariansh 48, 102481. https://doi.org/10.1016/j.acalib.2021.102481 (2022). [DOI: 10.1016/j.acalib.2021.102481]
  42. Richtig, G., Berger, M., Lange-Asschenfeldt, B., Aberer, W. & Richtig, E. Problems and challenges of predatory journals. J. Eur. Acad. Dermatol. Venereol. 32, 1441–1449. https://doi.org/10.1111/jdv.15039 (2018). [DOI: 10.1111/jdv.15039]
  43. Braun, V. & Clarke, V. One size fits all? What counts as quality practice in (reflexive) thematic analysis?. Qual. Res. Psychol. 18, 328–352. https://doi.org/10.1080/14780887.2020.1769238 (2021). [DOI: 10.1080/14780887.2020.1769238]
  44. Wang, X. & Cheng, Z. Cross-sectional studies: Strengths, weaknesses, and recommendations. Chest 158, S65-71. https://doi.org/10.1016/j.chest.2020.03.012 (2020). [DOI: 10.1016/j.chest.2020.03.012]

Grants

  1. IT32200/MITACS Accelerate Industrial Award (EBSCO Health)

MeSH Term

Humans
Research Personnel
Surveys and Questionnaires
Focus Groups
Cross-Sectional Studies
Biomedical Research
Periodicals as Topic
Male
Female
Journal Article

Word Cloud

Created with Highcharts 10.0.0surveytoolresearchtransparencyJTTpartusercliniciansresearchersmetricsfocusjournalscholarlystudyconductedfeaturesThematicanalysisgroupsdiscussionJournalTransparencywithinbiomedicalwillobtaininformationpreferencesproposedmixed-methodsinclusionitemscontentresponsesused70%pointssessionsParticipantsuseinterviewtranscriptsthemestotalparticipantstookliteracyessentialintegritycredibilityreproducibilityincreaseadherenceoptimalscientificpracticesenhanceproposecreationallowusersgivenjournal'soperationspoliciesprograminformreportconsultationtwopartsfirstinvolvedcross-sectionalrandomsampleauthorsjournalsaskedseriespotentialQuantitativesummarizeddescriptivestatisticsemployedanalyzetext-basedSubsequentexploreItemslessagreementstructurevotedconsideredidentifycorediscussed63255%responseratecollective747%respondentsfoundeither'occasionally'often''almostalways'difficultdeterminehealthonlinebasedreliableevidenceTwenty-twoThreefivemajorresultedsixregistrationitemmeetthresholddemonstratedlowcommunicationdiscussingmetricsuggestionsfindingssuggestvaluableoutcomescontributedevelopingrefiningaccordancegroupunderstandresearcherclinicianToolClinicianHealthResearcher

Similar Articles

Cited By