GRAPHICAL ABSTRACT

INTRODUCTION

Systematic reviews could provide the best insights into the state of a field by summarizing the available evidence when conducted with adherence to rigorous standards. With the growing importance of systematic reviews in clinical and public health, methodological guidelines have addressed important issues about the standards for conducting systematic reviews.

One of the issues is protocol registration before conducting a systematic review. Prospectively registering the protocol on free public-access websites such as the International Prospective Register of Systematic Reviews (PROSPERO) or publishing it in a journal has many benefits such as preventing duplicated research and selective reporting of findings, promoting completeness of research, improving the transparency and accuracy of the research process, and enhancing the reliability of the findings [1]. Therefore, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), a reporting guideline for systematic reviews, considers protocol registration an important reporting item [2].

The grading of recommendation, assessment, development and evaluation (GRADE) system is a tool to evaluate the certainty of evidence and assess the overall evidence by considering factors that decrease the quality of evidence (e.g., study design and inconsistency) and those that increase it (e.g., a dose-response relationship) [3].

Recently, we evaluated the prevalence and methodological quality of systematic reviews published in medical journals in Korea [4]. We inferred that the methodological quality of the 126 systematic reviews published in the last 4 years from 2018 to 2021 was low to very low according to A MeaSurement Tool to Assess Systematic Reviews 2 (AMSTAR 2), a quality assessment tool for systematic reviews [5]. AMSTAR 2 consists of a total of 16 items (7 critical and 9 non-critical), explicitly considers protocol registration as a critical aspect of high-quality systematic reviews, and regards the domains of the certainty of evidence assessment (such as risk of bias, inconsistency, and publication bias) as critical domains for high-quality systematic reviews. Despite the endorsement of methodological guidelines and international trends, we found that prospective protocol registration was insufficient and the certainty of evidence reporting was limited.

Although these findings could indicate the authors’ ignorance, they could also indicate the need for training for journal editors and reviewers. The editorial policies of a journal reflect its scope and publishing standards. The authors are expected to adhere to these policies to publish their studies. Therefore, journal editorial policies could affect studies’ methodological standards and reporting quality. Statements describing the editorial policies of journals are typically found in the “Instructions for Authors,” indicating requirements or guidelines for authors to adhere to when submitting a manuscript to be considered for publication.

This study aimed to present the status of systematic reviews published in major Korean journals, focusing on protocol registration and the use of GRADE, based on our previous review. We additionally reviewed the author guidelines of the journals included in the review to identify guidelines related to systematic reviews.

MATERIALS AND METHODS

We used previously collected data to investigate the methodological quality of systematic reviews in medical journals in Korea. The review protocol was registered (CRD42021283214), a full description of the study’s review methods and findings were made available [4]. The primary study process is presented below.

Search and inclusion criteria

We searched major medical journals on the KoreaMed website (http://koreamed.org), a service provided by the Korean Association of Medical Journal Editors on October 18, 2021. We selected the top 15 journals with the highest Korean Medical Citation Index (KoMCI)—a measure of citations within a database maintained by the Korean Academy of Medical Science: Allergy Asthma Immunol Res, Asian Oncol Nurs, Cancer Res Treat, Clin Psychopharmacol Neurosci, Diabetes Metab J, J Bone Metab, J Korean Acad Nurs, J Korean Acad Nurs Adm, J Korean Med Sci, J Korean Soc Clin Toxicol, J Menopausal Med, J Stroke, Korean J Radiol, Perspect Nurs Sci, and Ultrasonography.

We reviewed all publications of these 15 journals from 2018 to 2021, since AMSTAR-2 was published in 2017, and identified all systematic reviews. We regarded a study as a systematic review when authors explicitly reported that they conducted a systematic review using terms such as “systematic” or “systematically.” Moreover, we considered studies as systematic reviews when authors described some steps of systematic review standards such as study searches and study selection processes that could be reproducible.

Study selection and data extraction

Teams of 2 reviewers each performed reference screening and data extraction. At each stage, we conducted a calibration exercise that was documented to ensure that the reviewers understood the study objective and inclusion criteria in detail.

We used pre-piloted data extraction forms and collected study characteristics and GRADE use from the systematic reviews. Additionally, we reviewed the author’s guidelines, “Instructions for Authors” of each journal, and whether the journal recommended or mandated protocol registration.

The study characteristics of the systematic reviews comprised the following details

First author, international collaborative authorship (no, yes), publication year, language of publication (Korean or English), type of question (intervention/therapy or others), study design of primary studies (randomized controlled trials [RCTs], non-RCTs, or both), number of primary studies, number of participants, type of intervention or exposure, type of outcome, meta-analysis (yes or no), whether PRISMA guidelines were followed (yes or no), and whether the overall certainty of evidence was rated (yes or no).

The journal characteristics from the “Instructions for Authors” of the journals comprised the following details

Language of publication (Korean only, English only, or both), Science Citation Index (Expanded) listing (yes or no), whether systematic reviews were mentioned in the author guidelines (yes or no), endorsement or enforcement of reporting guidelines of a systematic review article (yes or no, if yes, the name of the reporting guideline), protocol registration (yes or no), and rating the certainty of evidence (yes or no).

For each of the abovementioned guidelines, the level of endorsement or enforcement was documented as required or mandatory (an essential criterion for manuscript acceptance), recommended (usage encouraged, but not mandatory), and unclear (guidelines were mentioned, but the necessity of their inclusion in the manuscript remained ambiguous).

Statistical analysis

We presented the general characteristics of the journals with descriptive statistics. Furthermore, we presented the proportion of protocol registration or publication of systematic reviews and compared them to the characteristics of the studies using the chi-square test. Subsequently, we provided descriptions of the journals’ guidelines regarding systematic reviews. All statistical analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

Ethics statement

Ethics approval is not required because we only used data from published papers and journals’ websites.

RESULTS

Protocol registration of systematic reviews

Only 14.3% (18 of 126) of systematic reviews registered their protocol. However, the proportion of registration increased from 6.7% in 2018 to 33.3% in 2021. Reviews with international collaborative authorship exhibited a higher registration rate than those by Korean authors (Table 1). Supplementary Material 1 presents the detailed characteristics of the 126 systematic reviews included.

Grading of recommendation, assessment, development and evaluation adoption by systematic reviews

Five studies [6-10] rated the certainty of evidence, and all 5 reviews used the GRADE system to do so. All of these studies were published in English, and 4 studies were interested in intervention questions. The number of primary studies ranged from 11 to 30, and 4 reviews conducted meta-analyses; they reported that they adhered to reporting guidelines. None of these studies registered their protocol (Table 2).

Author guidelines of the journals

We found that the author guidelines of 6 of the 15 journals mentioned systematic reviews. Six journals mentioned PRISMA as their reporting guidelines for systematic reviews; however, the level of endorsement or enforcement was required or mandatory in 2, recommended in 3, and unclear in 1. No instructions mentioned protocol registration and certainty of evidence assessment in the main body of the authors’ guidelines (Table 3). Supplementary Material 2 presents the detailed characteristics of 15 journals included.

DISCUSSION

Main findings

We found that 14.3% of systematic reviews registered their protocol and only 5 studies used the GRADE method to rate the certainty of evidence. In addition, the authors’ guidelines in the journals rarely provided specific guidelines for systematic reviews.

Strength and limitations

Our study primarily addresses the status quo of protocol registration and GRADE use in systematic reviews in major medical journals in Korea. Our study adhered to the standards of this area, including duplicate and independent reference screening and data extraction, in addition to calibration exercises to ensure accuracy and reliability.

This study has limitations. First, we evaluated protocol registration based on the studies’ reports and did not check the open-access online database of systematic review protocols such as PROSPERO. However, we believe that it would be unlikely for papers to register their protocol but not mention it. Second, due to the low number of studies that registered their protocol and used the GRADE system, we could not conduct a multivariable analysis to identify factors related to protocol registration or a comparative analysis to investigate study characteristics associated with GRADE use. Third, since this study identified the characteristics of journals from only 15 journals that published systematic reviews included based on a quality assessment, we could not provide comprehensive information on trends in the policies or characteristics of Korean journals related to systematic reviews.

Relation to previous works

Several studies have investigated the protocol registration of systematic reviews. On one such study, out of 284 included reviews, 60 (21%) protocols were registered. The proportion of registration increased from 5.6% in 2009 to 27% in 2015 (p for trend <0.001) [11]. Out of 495 systematic reviews in dentistry, 162 (32.7%) reported registering the systematic review protocol or working from a previously established protocol; moreover, it must be noted that previously registering a protocol positively influenced the final report quality of systematic reviews in dentistry [12]. Although the registration proportion of reviews in this study was low compared to previous studies, it appears to have increased from 2018 to 2021. In a survey of global researchers, 44.2% did not register their protocols prior to publishing their systematic reviews, and the main reasons for not registering their protocol were lack of knowledge about protocol registration (44.9%), it was not mandatory (43.0%), and the benefits of registration were unknown (35.0%) [13]. Therefore, there is a need to make prior protocol registration mandatory, as well as raising researchers’ awareness of the benefits of protocol registration.

Only 5 reviews (4.0% of 126) used GRADE to rate the certainty of evidence in this study. In previous studies, the proportions of GRADE use in systematic reviews substantially varied depending on the research area or publishing institution. For example, out of 800 systematic reviews in high-impact-factor nutrition journals, 55 (6.9%) used GRADE to rate the certainty of evidence, with a slight increase in the GRADE use by publication year [14]. Among Cochrane systematic reviews of traditional Chinese medicine, 86 (38.1%) used the GRADE approach to rate the certainty of evidence [15]. In the included systematic review, the authors presented the distribution of domains determining the certainty of evidence, such as the risk of bias and imprecision, and emphasized the wider adoption of GRADE for transparent and evidence-based decision-making. Due to the limited number of studies using GRADE in our review, we could not reveal any trends and characteristics related to GRADE use; therefore, we recommend finding support for our findings using a larger sample.

Previous studies have investigated the authors’ guidelines of journals to examine the journals’ policies on systematic reviews. In a sample of 146 journals that published systematic reviews indexed in the 2009 Journal Citation Index, the PRISMA statement was referred to in the instructions for authors in 27% (40 of 146) of journals, 50% in general and internal medicine journals (7 of 14), and 25% in specialty medicine journals (33 of 132) [16]. Among 134 surgery journals indexed in the 2009 Journal Citation Report, the Instructions for Authors of only 1 journal mentioned registration for systematic reviews [17]. Out of 80 dermatology journals, 38.8% mentioned systematic reviews in their author guidelines, while only 2 (2.5%) required PROSPERO registration in their author guidelines [18]. Aligning with previous studies, journals did not provide guidelines for systematic review authors or endorsement of relevant guidelines to encourage improving in reporting and registration.

Implications

Most reviews did not register their proposals prospectively, and journals did not provide relevant guidelines for the authors. Authors should be aware that prospective registration of the protocol is necessary for the transparency and credibility of their review. In addition, journals and editors should specify guidelines that reflect the latest high-quality standard guidelines for systematic reviews, inform authors, and educate reviewers accordingly.

Systematic reviews have been recognized as tools that can systematically appraise and summarize the available evidence on a health issue. Assessing the certainty of evidence acknowledges the methodological limitations of the evidence found and provides an accurate view of how confident we are in it. The GRADE approach is the best tool available to assess the certainty of evidence and facilitates adherence to methodological standards, comparison of results between studies, and application of results to the field.

In conclusion, notifying authors, reviewers, and editors of the importance of prospective protocol registration and evaluating the certainty of evidence using GRADE could improve the transparency and credibility of systematic reviews. Based on this, our study presented the status quo of Korean journals and found that the prospective registration of systematic review protocols was low and that GRADE use was very low in medical journals in Korea. Our results could be used to revise journal guidelines, educate authors, reviewers, and editors, and ultimately improve the quality of systematic reviews in Korea.

SUPPLEMENTARY MATERIALS

NOTES

CONFLICT OF INTEREST

The authors have no conflicts of interest to declare for this study.

FUNDING

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education (2021R1I1A3041301) and the Ministry of Science and ICT (NRF-2022R1A5A2030454).

AUTHOR CONTRIBUTIONS

Conceptualization: Han MA, Kim SJ, Hwang EC, Jung JH. Data curation: Han MA, Kim SJ, Hwang EC, Jung JH. Formal analysis: Han MA. Funding acquisition: Han MA. Methodology: Han MA, Kim SJ, Hwang EC, Jung JH. Project administration: Han MA, Kim SJ, Hwang EC, Jung JH. Visualization: Han MA. Writing – original draft: Han MA. Writing – review & editing: Han MA, Kim SJ, Hwang EC, Jung JH.

ACKNOWLEDGEMENTS

REFERENCES

• 1. Nagendrababu V, Duncan HF, Dummer PM. International Endodontic Journal policy on mandatory prospective (a priori) protocol registration for clinical trials and systematic reviews. Int Endod J 2021;54:1685-1686.ArticlePubMedPDF
• 2. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71.ArticlePubMedPMC
• 3. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64:383-394.ArticlePubMed
• 4. Kim SJ, Han MA, Jung JH, Hwang EC, Kim HR, Yoon SE, et al. Prevalence and methodological quality of systematic reviews in Korean medical journals. Epidemiol Health 2023;45:e2023017.ArticlePubMedPMC
• 5. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008.ArticlePubMedPMC
• 6. Hsu CY, Chiu SW, Hong KS, Saver JL, Wu YL, Lee JD, et al. Folic acid in stroke prevention in countries without mandatory folic acid food fortification: a meta-analysis of randomized controlled trials. J Stroke 2018;20:99-109.ArticlePubMedPMCPDF
• 7. Park SC, Choi MY, Choi J, Park E, Tchoe HJ, Suh JK, et al. Comparative efficacy and safety of long-acting injectable and oral second-generation antipsychotics for the treatment of schizophrenia: a systematic review and meta-analysis. Clin Psychopharmacol Neurosci 2018;16:361-375.ArticlePubMedPMC
• 8. Kulthanan K, Subchookul C, Hunnangkul S, Chularojanamontri L, Tuchinda P. Factors predicting the response to cyclosporin treatment in patients with chronic spontaneous urticaria: a systematic review. Allergy Asthma Immunol Res 2019;11:736-755.ArticlePubMedPMCPDF
• 9. Ashe MC, Santos IK, Edward NY, Burnett LA, Barnes R, Fleig L, et al. Physical activity and bone health in men: a systematic review and meta-analysis. J Bone Metab 2021;28:27-39.ArticlePubMedPMCPDF
• 10. Kim PH, Suh CH, Kim HS, Kim KW, Kim DY, Lee EQ, et al. Immune checkpoint inhibitor with or without radiotherapy in melanoma patients with brain metastases: a systematic review and meta-analysis. Korean J Radiol 2021;22:584-595.ArticlePubMedPDF
• 11. Tsujimoto Y, Tsujimoto H, Kataoka Y, Kimachi M, Shimizu S, Ikenoue T, et al. Majority of systematic reviews published in high-impact journals neglected to register the protocols: a meta-epidemiological study. J Clin Epidemiol 2017;84:54-60.ArticlePubMed
• 12. Dos Santos MB, Agostini BA, Bassani R, Pereira GK, Sarkis-Onofre R. Protocol registration improves reporting quality of systematic reviews in dentistry. BMC Med Res Methodol 2020;20:57.ArticlePubMedPMCPDF
• 13. Tawfik GM, Giang HT, Ghozy S, Altibi AM, Kandil H, Le HH, et al. Protocol registration issues of systematic review and meta-analysis studies: a survey of global researchers. BMC Med Res Methodol 2020;20:213.ArticlePubMedPMCPDF
• 14. Werner SS, Binder N, Toews I, Schünemann HJ, Meerpohl JJ, Schwingshackl L. Use of GRADE in evidence syntheses published in high-impact-factor nutrition journals: a methodological survey. J Clin Epidemiol 2021;135:54-69.ArticlePubMed
• 15. Wang Q, Xiao Y, Guo T, Zhu H, Li J, Lai H, et al. The use of GRADE approach in Cochrane reviews of TCM was insufficient: a cross-sectional survey. J Clin Epidemiol 2022;142:1-9.ArticlePubMed
• 16. Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals’ instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One 2011;6:e27611.ArticlePubMedPMC
• 17. Smith TA, Kulatilake P, Brown LJ, Wigley J, Hameed W, Shantikumar S. Do surgery journals insist on reporting by CONSORT and PRISMA? A follow-up survey of ‘instructions to authors’. Ann Med Surg (Lond) 2015;4:17-21.ArticlePubMedPMC
• 18. Osto M, Edriss M, Hamzavi IH. Prospective protocol registration should be required for systematic reviews in dermatology literature. Clin Dermatol 2022;40:225-227.ArticlePubMed

Figure & Data

References

Citations

Citations to this article as recorded by  

• Transparency of Systematic Reviews on Interventions for Multiple Sclerosis: Adherence to the GRADE Approach – Meta-Research
  Adriane Carvalho de Meneses, Giullia Carvalho Mangas Lopes, Letícia Barbosa de Lima, Giovanna Marcilio Santos, Elaine Marcílio Santos, Andrea de Carvalho Anacleto Ferrari de Castro, Ana Luiza Cabrera Martimbianco
  Neuroepidemiology.2025; : 1.     CrossRef