Reporting quality of systematic reviews with moxibustion
Chinese Medicine volume 15, Article number: 104 (2020)
Moxibustion is one of the major interventions of Chinese medicine (CM). The systematic reviews (SRs) are essential references for evaluating the efficacy and safety of moxibustion interventions. This study aimed to assess the reporting quality of these SRs, particularly whether necessary information related to moxibustion was adequately reported.
Seven databases (including four English and three Chinese databases) were systematically searched for SRs of moxibustion that were published up to 31 December 2019. The primary analysis was to assess their reporting quality based on 27-item of the Preferred Reporting Items for SRs and Meta-Analyses (PRISMA) and 14-item of moxibustion-related information designed according to CM theory and the STandards for Reporting Interventions in Clinical Trials Of Moxibustion (STRICTOM). Descriptive statistics were also used to analyze their baseline characteristics.
A total of 97 SRs of moxibustion were identified from 2011 to 2019. For 27-item of PRISMA, except item 5, 8, 16 and 23, the remaining 23 items had the reporting compliances higher than 55%, of which 2 items (item 20 and 26) were fully reporting (100%). However, for moxibustion-related information, 69.1% (67/97) SRs did not provide the specific type of moxibustion, 39.2% (38/97) lacked details regarding the materials, procedure and technique used for moxibustion, 67.0% (65/97) did not report the selection criteria of acupoints for moxibustion, 28.9% (28/97) did not provide the number or duration of treatment sessions, 69.1% (67/97) did not provide any information about safety evaluation, and 94.8% (92/97) SRs did not report the treatment environment. For 51 (55.4%) of 92 SRs that included meta-analysis, it was impossible to assess whether meta-analysis had been properly conducted due to inadequate reporting of moxibustion interventions.
The reporting quality of SRs of moxibustion need further improvements in terms of adequate reporting of moxibustion interventions and of moxibustion-related rationales. Reporting guideline of “PRISMA extension for moxibustion interventions” should be developed thus to improve their quality.
Systematic reviews (SR) can help practitioners to keep abreast of the medical literature by summarizing large bodies of evidence and explaining differences among studies on the same question. SRs can be used to inform medical decisions and to establish clinical policy. A meta-analysis (MA) is a type of SR that uses statistical methods to combine and summarize the results of several primary studies . SR/MA can help clinicians to keep up to date with their field and help policymakers to judge the risks and benefits of health care behaviors. They provide a starting point for clinical practice guideline developers and summaries for funders seeking new research to support . But inadequate reporting occurs in some field, such as surgery and traditional medicine etc. [3,4,5]. The suboptimal reporting quality of SR/MA led to the development of the Quality Of Reporting Of Meta analyses (QUOROM) Statement and its updated revision named Preferred Reporting Items for SRs and Meta-Analyses (PRISMA), published in 1999 and 2009, respectively [6, 7]. The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram, with an explanation and elaboration for each checklist item also published in 2009 . The PRISMA checklists are used to guide authors of SR/MA to improve reporting quality. It is also a universal standard to assess the reporting quality of available SR/MA publications [9,10,11].
With a history of over 2000 years, the CM practice offers natural, safe and effective therapies and cures for many diseases with much less side effects. CM takes a unique theoretical and practical approach to health. For treatments, it includes the use of various interventions, such as moxibustion and acupuncture. The moxibustion technique is commonly used in clinical practice, including direct moxibustion with a traditional moxa stick, and indirect moxibustion. Mugwort leaves (Artemisia vulgaris, moxa) are the most commonly material used for moxibustion. Other Chinese herbal medicines could also be used in combination with mugwort. Mugwort is considered to be warm, acidic and bitter. It can promote better circulation of qi and blood by warming meridians. According to CM theory, the meridians are the channels inside the human body that circulate vital energy (in Chinese called qi and blood). Although it includes adverse effects, such as allergic reactions, burns and infections, sometimes, the moxibustion intervention is always believed as a safety therapy [12, 13].
As an increasing number of randomized controlled trials (RCTs) and SRs of moxibustion published in recent years, the importance to improve reporting quality of moxibustion studies has been highlighted by both researchers and users of moxibustion evidence [14,15,16]. In 2013, our working group has published The STandards for Reporting Interventions in Clinical Trials Of Moxibustion (STRICTOM), which aims to improve the reporting quality of RCTs with moxibustion . However, no previous study has assessed the reporting quality of moxibustion SRs. Thus, this study was designed to: (i) summarize the general characteristics of all included SRs of moxibustion, (ii) assess the reporting quality of these SRs based on the PRISMA checklist, (iii) evaluate whether necessary information related to moxibustion is adequately reported, and (iv) assess whether these SRs are properly conducted in terms of synthesis of results (e.g., meta-analysis).
Inclusion and exclusion criteria
This study included all SR/MAs of moxibustion published in seven databases, including Embase, Cochrane Library, MEDLINE, Allied and Complementary Medicine (AMED), CNKI, VIP and Wanfang, up to 31 December 2019. Moxibustion interventions may have been given alone or in combination with other interventions or complementary alternative medicine. There were no limitations in the participants, comparisons, and outcomes. We excluded the following SR/MAs: repeat publications, research on acupuncture or other treatments as the main intervention, protocols, and withdrawal SR/MAs.
The databases were searched on 13 Feb 2020 for all SR/MAs of Moxibustion that had been published up to 31 December 2019. The search terms included “meta-analysis,” “systematic review,” “moxibustion,” “mora,” “Jiu,” “vesiculation,” “Sanfu or dog days,” “Fapao or blister,” “Suanni or Garlic”, “Baijiezi or Bai Jie Zi or White mustard seed”, “Maogen or Japan Buttercup”, “acupuncture and moxibustion”, “Hanlian or Eclipta”, “Tiannanxing or Araceae or Arisaema”, etc. were included. The detailed search strategy is given in Additional file 1: S1.
The titles and abstracts of the SRs were independently screened by two researchers (SY-L and RT) based on inclusion and exclusion criteria, and the full-texts of potentially suitable articles were retrieved for further assessment. Disagreements were resolved by a third reviewer (XZ).
There were three pre-designed forms for data collection: (1) General characteristics form, including publication year, information of the authors, and descriptive information of included SRs. (2) PRISMA assessment form, including 27 items of the checklist. (3) Moxibustion-related information form, which was designed according to (a) CM theory and (b) STRICTOM guideline. The details are presented in Table 1.
Two reviewers (RT and SY-L) used the PRISMA and special-designed moxibustion-related list to independently assess the reporting quality of included SRs. Each item was given a “1” score if fully reported or “0” if incompletely reported or absent. Disagreements were resolved by discussion or consultation with a third researcher (XZ). All data were collected and recorded in Microsoft Office Excel 2016. Categorical data is presented as a number (n) and percent (%).
Our initial literature search identified 3182 records. Preliminary screening excluded 3070 SRs due to duplication or focus on non-moxibustion interventions. After examination of the full texts of 112 articles, a total of 97 SRs was eligible for inclusion in this study (Fig. 1).
General characteristics of included SRs
The first included SR of moxibustion was published in 2011, and the number of publications has been continued increasing (Fig. 2). The general features of these SRs are presented in Tables 2 and 3. The three most commonly examined conditions were diseases of musculoskeletal system or connective tissue (30.9%), diseases of the digestive system (12.4%), and diseases of the respiratory system (10.3%), respectively.
PRISMA score of included SRs
As presented in Table 4, the total reporting rates of 27 items varied from 12.4% to 100%. Two items, including item 20 and 26, were fully reported (100%). Thirteen items, namely item 1, 2, 4, 6, 7, 9, 10, 12, 13, 14, 17, 18, and 24 were reported in more than 80% of all SRs. Eight Items, including item 3, 11, 15, 19, 21, 22, 25, and 27, were reported in more than 55% SRs. The least reported items including 5, 8, 16, and 23 were lower than 55%.
Reporting score of moxibustion-related information
As presented in Table 5, fourteen items of Moxibustion-related information were assessed. Of 97 SRs, 25.8% reported the specific name of studied moxibustion in the title, but no title included studied CM syndrome. In the included articles, few included CM syndrome-related content, such as diagnostic criteria (6.2%) and outcomes (3.1%). Relevant CM theory and rationale was provided in the Introduction among only 25.8% of included SRs. For the details of moxibustion, except one item of the materials and techniques used for moxibustion was reported in 60.8% of included SRs, the other items are seriously inadequate reporting, such as treatment environment (5.2%), the number and frequency of treatment (27.8), the safety assessment of moxibustion (30.9%), the selection of acupoints and meridians (33%), and the duration of treatment (43.3%). Besides, few SRs (27.8%) considered subgroup analysis. Based on the inadequate reporting, it is impossible to evaluate whether the synthesis of results had been appropriately conducted in more than half (55.4%) of 92 SRs with meta-analysis.
General characteristics of included Moxibustion SRs
In this study, we included 97 Moxibustion SRs from 2011 to 2019 and described the baseline characteristics. Some problems have been identified. Firstly, most SRs (94.8%, 92/97) included the meta-analysis but more than half (69.1%, 67/97) did not report the specific type of studied moxibustion(s). Generally, choosing a wide range of moxibustion interventions for one disease in an SR requires rigorous methodology techniques in data analysis. Without transparent reporting, it is hard to judge the reasonable rationale for results synthesis. Secondly, very few SRs (2.1%, 2/97) have updated on time. Although it is well known that results from SRs are most useful when they are current, this study found that all included SRs published in Chinese journals did not mention any update. Based on the features of moxibustion SRs, this review is the first to systematically assess the reporting quality of moxibustion SRs with two instruments of the PRISMA checklist and moxibustion-related items. The additional moxibustion-related checklist was designed to identify the important moxibustion details in the procedure of SRs’ design, conduct, and analysis.
PRISMA assessment of included moxibustion SRs
In this review, most SRs (84.5%, 82/97) were published in Chinese medical journals. As Chinese journals usually did not require the registration and protocol for SRs, thus the reporting of these items was very low. Consistent with previous studies, the reporting compliance of these information in Chinese SRs, particularly in the interventions of Chinese herbal medicines, acupuncture, and massage (Tuina) was seriously inadequate [18,19,20]. In addition, the reporting of “Search” (item 8) was unsatisfactory because more than half of the included SRs only provided some keywords for search instead of a comprehensive search strategy according to the PRISMA requirement. In comparison, the moxibustion SRs published in English journals usually have a better reporting for this item. Honestly, the inadequate reporting of “Search” could be related to the word limitation of Chinese journals, and most such journals did not provide an online appendix choice. Although most Chinese medical journal includes the endorsement of the PRISMA statement, strictly compliance and application is the key to improve the reporting of moxibustion SRs . Besides, the items of “additional analysis”, either in the section of Methods or Results, were reported relatively less frequently. According to the characteristics of moxibustion, whether the additional analyses (e.g., subgroup analysis) were properly designed is the key for assessing the value of summary results . Similar to previous reports, however, SRs with CM interventions (e.g., Chinese herbal medicines) usually prefer to include many different types of studied interventions [23, 24]. As the proportion of wide intervention ranges rises, and the rates of unreasonable synthesis of results more easily appeared. Thus, an appropriately pre-design of subgroup analysis is much important.
Moxibustion-related information assessment of included SRs
For moxibustion SRs, the specific reporting items of interventions are not provided in the PRISMA checklist, so there might be a gap between the international reporting guideline and specific reporting of moxibustion SRs. Thus, we have further assessed the reporting of moxibustion-related information based on a self-designed checklist. Obviously, some deficiencies have been discovered. Firstly, we found that the inadequate and absent reporting of moxibustion details and CM related rationales are needed to be improved urgently. For SRs, whether original data items were extracted entirely and accurately is closely related to the synthesis of results. In terms of moxibustion details, the following elements are critical to be extracted: type of moxibustion, materials and technique used for moxibustion, selection of acupoints and location for moxibustion, number, frequency and duration of treatment sessions, treatment environment, and safety assessment . As a result, in this study, the least frequently reported information was the number and frequency (27.8%), safety assessment or adverse effect (30.9%), the types of moxibustion (30.9%), names of acupoints and meridians (33.0%), and the duration of the treatment (43.3%). Relatively, the materials and techniques were reported in 60.8% of the included SRs. Secondly, due to the inadequate reporting of intervention details, more than half (72.2%, 70/97) SRs did not conduct subgroup analysis based on different features of moxibustion interventions (e.g., type, material, dosage). Moreover, for 92 SRs with meta-analysis, 55.4% (51/92) was impossible to assess whether data synthesis had been conducted properly. The synthesis of results includes statistical, methodological, and clinical considerations. Although the former two factors are perhaps more technical and evidence-based, the clinical considerations should be highly valued, especially for the moxibustion therapy used in clinical practice . Relevant CM rationale, such as syndrome differentiation, should be proposed appropriately . This content was also rarely mentioned in the included SRs, although most of them studied CM-based moxibustion.
Improvement measures and suggestions
As some deficiencies of reporting were identified in this study, specific improvements are urgently needed. In agreement with previous findings, guidelines do help improve the quality of reporting [27, 28]. Therefore, except for continuously improved the completeness reporting of the PRISMA checklist, a series of standard reporting items especially for moxibustion SRs should be developed as an extension for the PRISMA. With the PRISMA extensions for acupuncture and Chinese herbal medicines have been published [29, 30], the specific guideline for moxibustion is helpful to supplement the SRs with CM interventions. Our group has registered this reporting guideline on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) under the item of “PRISMA for traditional Chinese medicine”, and has initiated the related work . We wish to publish the Guideline soon.
This study has some limitations. Firstly, this review identified moxibustion SRs published up to 31 December 2019 in the targeted seven main databases. Any records which had not been included in these databases by that cut-off period have not been included. Secondly, the assessment scoring (“1” or “0”) did not allow partial information to be used. All incomplete reporting (e.g., partial and absence) were given as “0”, which decrease the reporting rate of some items. These limitations mean that the results of the study may not necessarily be comprehensive. We do however believe that the general trends indicated by the analysis of the information we did use, even if incomplete, are valid.
Moxibustion SRs summarize evidence relating to efficacy and safety of moxibustion interventions—but they are valuable only if done accurately and reliably. Due to the inadequate reporting, it is a challenge to assess whether the data synthesis had been properly conducted in moxibustion SRs. Such situation seriously affects the readers’ judgments about the conclusion of these SRs and further compromises the values of moxibustion therapy. Therefore, the development of a reporting guideline for moxibustion SRs, as an extension of the PRISMA, could be an effective measure to improve the current situation.
Availability of data and materials
The data used for this study are included in the manuscript and supplementary file.
Preferred Reporting Items for Systematic reviews and Meta-Analyses
STandards for Reporting Interventions in Clinical Trials Of Moxibustion
Quality Of Reporting Of Meta analyses
Randomized controlled trial
Allied and complementary medicine database
Enhancing the QUAlity and Transparency Of health Research
Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical ecisions. Ann Intern Med. 1997;126(5):376–80.
Villar J, Carroli G, Belizán JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet. 1995;345(8952):772–6.
Anttila H, Malmivaara A, Kunz R, Autti-Rämö I, Mäkelä M. Quality of reporting of randomized, controlled trials in cerebral palsy. Pediatrics. 2006;117:2222–30.
Chan S, Bhandari M. The quality of reporting of orthopaedic randomized trials with use of a checklist for nonpharmacological therapies. J Bone Joint Surg. 2007;89:1970–8.
Zhang JH, Shang HC, Gao XM, Zhang BL, Xiang YZ, Cao HB, et al. Methodology and reporting quality of systematic review/meta-analysis of traditional Chinese medicine. J Altern Complement Med. 2007;13:797–805.
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354(9193):1896–900.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.
Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. A PRISMA assessment of the reporting quality of systematic reviews in orthodontics. Angle Orthod. 2013;83(1):158–63.
Tan WK, Wigley J, Shantikumar S. The reporting quality of systematic reviews and meta-analyses in vascular surgery needs improvement: a systematic review. Int J Surg. 2014;12(12):1262–5.
Li JL, Ge L, Ma JC, Zeng QL, Yao L, An N, et al. Quality of reporting of systematic reviews published in “evidence-based” Chinese journals. Syst Rev. 2014;3:58.
Cheng X, Deng L. Chinese acupuncture and moxibustion. Beijing: Foreign Languages Press; 1987.
Shi XM, Zhou JZ. Shi Xue-min’s comprehensive textbook of acupuncture and moxibustion. Beijing: People’s Medical Publishing House; 2007.
Liu JP, Xia Y. Quality appraisal of systematic reviews or meta-analysis on traditional Chinese medicine published in Chinese journals. Zhongguo Zhong Xi Yi Jie He Za Zhi. 2007;27(4):306–11 (Article in Chinese).
Xiong J, Zhu D, Chen R, Ye W. Report quality of randomized controlled trials of moxibustion for knee osteoarthritis based on CONSORT and STRICTOM. Zhongguo Zhen Jiu. 2015;35(8):835–9 (Article in Chinese).
Kim SY, Chae Y, Lee SM, Lee H, Park HJ. The effectiveness of moxibustion: an overview during 10 years. Evid Based Complement Altern Med. 2011;2011:306515.
Cheng CW, Fu SF, Zhou QH, Wu TX, Shang HC, Tang XD, et al. Extending the CONSORT Statement to moxibustion. J Integr Med. 2013;11:54–63.
Liu Y, Yang S, Dai J, Xu Y, Zhang R, Jiang H, Yan X, Yang K. Risk of bias tool in systematic reviews/meta-analyses of acupuncture in Chinese journals. PLoS ONE. 2011;6(12):e28130.
Ma B, Guo JW, Qi GQ, Li H, Peng J, Zhang Y, et al. Epidemiology, quality and reporting characteristics of systematic reviews of traditional Chinese medicine interventions published in Chinese journals. PLoS ONE. 2011;6(5):e20185.
Zhang XT, Hu YY, Zhang XY, Chen HQ, Wang XL, Li XY, et al. An overview of systematic reviews on Tuina from 2013 to 2017. Chin J Evid Based Med. 2019;19(3):361–7.
Tian R, Zhao C, Cheng CW, Lam WC, Yang Z, Wu TX, et al. Constructing the Chinese medicine clinical curative effects research report system. Chin J Evid Based Med. 2018;18(7):651–3 (Article in Chinese).
Hu D, Kang DY, Hong Q. Heterogeneity analysis of systematic reviews on traditional Chinese medicine. Chin J Evid Based Med. 2010;10(4):488–91 (Article in Chinese).
Zhang X, Aixinjueluo Q, Li S, Song L, Lau C, Tian R, et al. Reporting quality of Cochrane systematic reviews with Chinese herbal medicines. Syst Rev. 2019;8:302.
Liu Y, Zhang R, Huang J, Zhao X, Liu D, Sun W, et al. Reporting quality of systematic reviews/meta-analyses of acupuncture. PLoS ONE. 2014;9(11):e113172.
Han M, Wang YY, Mu YJ, Liu ZJ, Li Q, Li XX, et al. Reporting specification of systematic reviews on Chinese medicine and methodological evaluation. Zhongguo Zhong Xi Yi Jie He Za Zhi. 2012;32(7):872–4 (Article in Chinese).
Liu JP, Han M, Li XX, Mu YJ, Lewith G, Wang YY, et al. Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study. BMJ Open. 2013;3(7):e002968.
Kim KH, Kang JW, Lee MS, Lee JD. Assessment of the quality of reporting in randomised controlled trials of acupuncture in the Korean literature using the CONSORT statement and STRICTA guidelines. BMJ Open. 2014;4(7):e005068.
Wang JL, Sun TT, Lin YW, Lu R, Fang JY. Methodological reporting of randomized controlled trials in major hepato-gastroenterology journals in 2008 and 1998: a comparative study. BMC Med Res Methodol. 2011;11:110.
Wang X, Chen Y, Liu Y, Yao L, Estill J, Bian Z, et al. Reporting items for systematic reviews and meta-analyses of acupuncture: the PRISMA for acupuncture checklist. BMC Complement Altern Med. 2019;19:208.
Zhang X, Tian R, Lam WC, Yao L, Wang XQ, Cheng CW, et al. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) extension for Chinese herbal medicines 2020 (PRISMA-CHM 2020). Am J Chin Med. 2020;48(6):1–35.
Enhancing the QUAlity and Transparency Of health Research (EQUAOR) Network. http://www.equator-network.org/library/reporting-guidelinesunder-development/#38. Accessed 26 May 2020.
The authors thank the financial support from Fang’s Charitable Foundation and Strategic Development Fund of HKBU, Hong Kong.
This work is supported by Fang’s Charitable Foundation and Strategic Development Fund of HKBU, Hong Kong SAR, China. The funder had no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Tian, R., Zhang, X., Li, SY. et al. Reporting quality of systematic reviews with moxibustion. Chin Med 15, 104 (2020). https://doi.org/10.1186/s13020-020-00385-z
- Systematic review
- PRISMA statement
- Reporting quality
- Chinese medicine