INTRODUCTION: Previous studies about the replicability of clinical research based on the published literature have suggested that highly cited articles are often contradicted or found to have inflated effects. Nevertheless, there are no recent updates of such efforts, and this situation may have changed over time.
METHODS: We searched the Web of Science database for articles studying medical interventions with more than 2000 citations, published between 2004 and 2018 in high-impact medical journals. We then searched for replications of these studies in PubMed using the PICO (Population, Intervention, Comparator and Outcome) framework. Replication success was evaluated by the presence of a statistically significant effect in the same direction and by overlap of the replication's effect size confidence interval (CIs) with that of the original study. Evidence of effect size inflation and potential predictors of replicability were also analyzed.
RESULTS: A total of 89 eligible studies, of which 24 had valid replications (17 meta-analyses and 7 primary studies) were found. Of these, 21 (88%) had effect sizes with overlapping CIs. Of 15 highly cited studies with a statistically significant difference in the primary outcome, 13 (87%) had a significant effect in the replication as well. When both criteria were considered together, the replicability rate in our sample was of 20 out of 24 (83%). There was no evidence of systematic inflation in these highly cited studies, with a mean effect size ratio of 1.03 [95% CI (0.88, 1.21)] between initial and subsequent effects. Due to the small number of contradicted results, our analysis had low statistical power to detect predictors of replicability.
CONCLUSION: Although most studies did not have eligible replications, the replicability rate of highly cited clinical studies in our sample was higher than in previous estimates, with little evidence of systematic effect size inflation. This estimate is based on a very select sample of studies and may not be generalizable to clinical research in general.
Lancet. 2016 Apr 9;387(10027):1540-1550
[PMID:
26712084]
Epidemiology. 2008 Sep;19(5):640-8
[PMID:
18633328]
Medicine (Baltimore). 2018 Feb;97(7):e9909
[PMID:
29443766]
N Engl J Med. 2011 Jun 9;364(23):2187-98
[PMID:
21639811]
PLoS One. 2020 Feb 19;15(2):e0229179
[PMID:
32074131]
N Engl J Med. 2009 Sep 3;361(10):947-57
[PMID:
19692680]
BMJ. 2011 Nov 24;343:d6898
[PMID:
22115901]
N Engl J Med. 2008 Jul 24;359(4):378-90
[PMID:
18650514]
JAMA. 2018 Mar 6;319(9):883-895
[PMID:
29509864]
N Engl J Med. 2005 Sep 22;353(12):1209-23
[PMID:
16172203]
Lancet. 2005 Mar 26-Apr 1;365(9465):1159-62
[PMID:
15794971]
JAMA. 2007 Dec 5;298(21):2517-26
[PMID:
18056905]
Expert Rev Clin Pharmacol. 2019 Aug;12(8):815-824
[PMID:
31287333]
Science. 2015 Aug 28;349(6251):aac4716
[PMID:
26315443]
N Engl J Med. 2017 Jan 26;376(4):383-391
[PMID:
28121511]
N Engl J Med. 2004 Jan 15;350(3):221-31
[PMID:
14724301]
Proc Natl Acad Sci U S A. 2020 May 19;117(20):10762-10768
[PMID:
32366645]
Nature. 2019 Nov;575(7782):E1-E2
[PMID:
31723284]
N Engl J Med. 2010 Aug 26;363(9):809-19
[PMID:
20818844]
Oncotarget. 2016 Nov 8;7(45):73068-73079
[PMID:
27683031]
N Engl J Med. 2012 Jun 28;366(26):2455-65
[PMID:
22658128]
Lancet Oncol. 2009 Jan;10(1):25-34
[PMID:
19095497]
N Engl J Med. 2014 May 8;370(19):1790-8
[PMID:
24678937]
PLoS Med. 2005 Aug;2(8):e124
[PMID:
16060722]
Int J Cancer. 2017 Feb 15;140(4):948-958
[PMID:
27813059]
PLoS Biol. 2021 Mar 1;19(3):e3001107
[PMID:
33647013]
BMJ. 2021 Sep 15;374:n1864
[PMID:
34526355]
N Engl J Med. 2009 Mar 26;360(13):1320-8
[PMID:
19297566]
Br J Psychiatry. 2015 Oct;207(4):357-62
[PMID:
26159600]
BMC Med. 2010 Mar 24;8:18
[PMID:
20334633]
Ann Intern Med. 2017 Oct 03;167(7):449-455
[PMID:
28869989]
Cochrane Database Syst Rev. 2021 Mar 18;3:CD010383
[PMID:
33734432]
PLoS One. 2016 Sep 19;11(9):e0162874
[PMID:
27644090]
JAMA. 2005 Jul 13;294(2):218-28
[PMID:
16014596]
Medicine (Baltimore). 2016 Dec;95(49):e5591
[PMID:
27930578]
N Engl J Med. 2009 Jul 9;361(2):123-34
[PMID:
19553641]
N Engl J Med. 2012 Jun 28;366(26):2443-54
[PMID:
22658127]
N Engl J Med. 2010 Jun 24;362(25):2380-8
[PMID:
20573926]
Mem Inst Oswaldo Cruz. 2020 Oct 23;115:e200328
[PMID:
33111750]
Nat Hum Behav. 2018 Sep;2(9):637-644
[PMID:
31346273]
PeerJ. 2017 Jul 7;5:e3544
[PMID:
28698825]
J Vasc Surg. 2002 Jul;36(1):194-5
[PMID:
12096281]
N Engl J Med. 2015 Jan 1;372(1):11-20
[PMID:
25517348]
N Engl J Med. 2015 Mar 12;372(11):1019-30
[PMID:
25671798]
N Engl J Med. 2016 Nov 10;375(19):1823-1833
[PMID:
27718847]
Sci Adv. 2021 May 21;7(21):
[PMID:
34020944]
N Engl J Med. 2011 Jun 30;364(26):2507-16
[PMID:
21639808]
N Engl J Med. 2015 Mar 12;372(11):1009-18
[PMID:
25671797]
N Engl J Med. 2008 Jun 12;358(24):2545-59
[PMID:
18539917]
Proc Natl Acad Sci U S A. 2023 Feb 7;120(6):e2208863120
[PMID:
36716367]
N Engl J Med. 2015 May 21;372(21):2018-28
[PMID:
25891174]
Proc Natl Acad Sci U S A. 2017 Apr 4;114(14):3714-3719
[PMID:
28320937]
BMJ. 2016 Apr 18;353:i1754
[PMID:
27091337]
Trials. 2019 Feb 14;20(1):118
[PMID:
30760329]
Eur Neuropsychopharmacol. 2013 Feb;23(2):118-25
[PMID:
22633617]
Immunotherapy. 2017 May;9(6):471-479
[PMID:
28472905]
J Am Heart Assoc. 2015 May 05;4(5):
[PMID:
25944874]
Ann Intern Med. 2001 Apr 17;134(8):663-94
[PMID:
11304107]
JACC Cardiovasc Interv. 2017 Sep 25;10(18):1899-1901
[PMID:
28935083]
Proc Natl Acad Sci U S A. 2018 Mar 13;115(11):2628-2631
[PMID:
29531051]
Lancet. 2012 Jun 23;379(9834):2364-72
[PMID:
22632907]
J Clin Oncol. 2015 Jan 20;33(3):244-50
[PMID:
25366685]
J R Stat Soc Ser A Stat Soc. 2009 Jan;172(1):137-159
[PMID:
19381330]
JAMA Intern Med. 2015 Dec;175(12):1992-4
[PMID:
26502403]
Diab Vasc Dis Res. 2010 Apr;7(2):119-30
[PMID:
20382775]
BMJ. 2018 Sep 05;362:k3519
[PMID:
30185521]
BMC Med. 2018 Feb 21;16(1):26
[PMID:
29463308]
N Engl J Med. 2015 Jul 2;373(1):23-34
[PMID:
26027431]
BMJ. 2013 Nov 08;347:f6625
[PMID:
24212107]
N Engl J Med. 2013 Jun 20;368(25):2385-94
[PMID:
23724913]
N Engl J Med. 2013 Jul 11;369(2):122-33
[PMID:
23724867]
N Engl J Med. 2009 Mar 5;360(10):961-72
[PMID:
19228612]
N Engl J Med. 2005 Oct 20;353(16):1659-72
[PMID:
16236737]
Psychol Methods. 2023 Apr;28(2):438-451
[PMID:
34928679]
Biostatistics. 2014 Jan;15(1):1-12
[PMID:
24068246]
PLoS Biol. 2015 Oct 13;13(10):e1002273
[PMID:
26460723]
PLoS One. 2015 Aug 05;10(8):e0132382
[PMID:
26244868]
N Engl J Med. 2017 Jun 8;376(23):2277-2279
[PMID:
28581902]
Int J Cardiol. 2016 Sep 1;218:50-58
[PMID:
27236108]
Nat Hum Behav. 2024 Feb;8(2):311-319
[PMID:
37945809]
Lancet. 2016 Apr 23;387(10029):1723-31
[PMID:
26898852]
JAMA. 2014 Sep 10;312(10):1024-32
[PMID:
25203082]
BMJ. 2019 Oct 7;367:l5460
[PMID:
31591158]
Lancet Oncol. 2012 Mar;13(3):239-46
[PMID:
22285168]
Lancet. 2019 May 4;393(10183):1819-1830
[PMID:
30955977]
PLoS Biol. 2020 Mar 27;18(3):e3000691
[PMID:
32218571]
N Engl J Med. 2008 Jun 12;358(24):2560-72
[PMID:
18539916]
Evid Based Med. 2016 Aug;21(4):125-7
[PMID:
27339128]
N Engl J Med. 2013 Jul 11;369(2):134-44
[PMID:
23724846]
N Engl J Med. 2008 Sep 25;359(13):1317-29
[PMID:
18815396]
