The effect size in the pooled analysis of balanced studies ie, studies in which major clinical indicators are similarly distributed between arms was 1. The effect size in the pooled analysis of only randomized studies was 1. The trend of these results correlates with the first of the four typical patterns described above.
In other words, the hypothesis of this meta-analysis radiochemotherapy is significant in reducing disease-free survival after D2 gastrectomy is strongly supported. The trend in which the effect size increases from considering all studies to considering only balanced or randomized studies strengthened the validity of the hypothesis.
The results of observational studies may have underestimated the effect size relative to the true effect due to the influence of confounders eg, patients assumed to have greater risk of recurrence underwent radiochemotherapy. Furthermore, the low heterogeneity in the analyses of balanced and randomized studies suggests that the pooled results of those studies are reliable and well designed, and they are less affected by possible confounders.
The second study was on the benefit of local treatment for oligometastases. In the recent literature, it was proposed that local treatment for oligometastatic foci could prolong cancer survival [ 28 , 29 ].
Several randomized studies have been published, but the number of patients recruited is generally insufficient [ 30 ]. In addition, because the studies in the literature were published according to the type of primary cancer, it was difficult to comprehensively analyze the oncologic benefit of local treatment on general oligometastases.
Therefore, we attempted to prove the hypothesis that local treatment for oligometastases will increase overall survival in a meta-analysis [ 22 ]. In the analysis of all studies, the pooled effect size was 3. In the analysis of balanced studies, the pooled effect size was 2.
The trend of these results correlates with the fourth of the four typical patterns Figure 4. In other words, the hypothesis of this meta-analysis is true, referring to the analysis results of randomized studies. However, unlike the pattern seen in the meta-analysis of gastric cancer, the change in the effect size or P value does not increase the validity of the hypothesis. Observational studies may have been affected by a confounder, and the results may have been larger than the true effect size.
Of note, in many studies, local treatment arms had a lower number of metastatic foci than control arms, although the difference was not statistically significant. Unlike the low heterogeneity in the pooled analysis of randomized studies, the high heterogeneity among observational studies suggests the possible effects of confounders. However, such a pattern does not necessarily indicate that the result is weak and not useful.
Therefore, the authors concluded that although local treatment for oligometastases is beneficial, patients must be carefully selected with consideration of the type of disease or metastatic burden, and the design of future observational studies needs to be improved.
The number of meta-analyses in the literature that include observational studies has been steadily increasing [ 1 ]. In actual clinical fields, the decisions that can be fully supported by blinded, randomized studies are limited. It is difficult to assemble a sufficient number of patients free from ethical considerations when the benefits of an intervention are expected to be significant due to observational studies [ 31 ].
The treatment methods applied to the intervention and control groups should be of the same type in terms of what the patient perceives. As the understanding of a disease increases and treatment options diversify, it will become increasingly necessary to obtain assistance for therapeutic decisions from meta-analyses of observational studies [ 17 ]. Shrier et al [ 11 ] described the clinical necessity of meta-analyses including nonrandomized studies; they discussed the practical limitations of randomized studies and explained that well-designed observational studies obtained similar results to those of randomized studies.
Vandenbroucke [ 32 ] suggested that the reliability of the results can be improved by meta-analyzing observational studies selected in terms of the subject, design, and analysis method. Other previous publications have discussed the justification for including observational studies in meta-analyses or how to select studies with valid qualities. The Cochrane Handbook for Systematic Reviews of Interventions [ 12 ], which previously had a conservative perspective on including nonrandomized studies in systematic reviews, added a chapter in the most recent edition on how to assess and interpret these studies in a meta-analysis.
Of note, the handbook asserted that only observational studies without a high risk of bias should be included in the meta-analysis. It was also pointed out that there is still no established model that can evaluate how bias or confounders of observational studies affect estimates. However, little is known about how observational and randomized studies should be integrated and analyzed to yield actual clinical decisions.
They include fundamental flaws such as inappropriate eligibility criteria, flawed measurement of exposure, inadequate follow-up, and inadequate control of confounders. In the presence of these limitations, it is suggested that the evidence grade should be lowered by one or two steps.
Although it is agreeable to evaluate the validity of observational studies in stages, a practical methodology for integrating randomized studies with low- and high-grade observational studies into a formal meta-analysis has not been sufficiently introduced.
Indeed, many clinical practice guidelines use GRADE to analyze the grade of evidence and recommendations; those analyses, including observational studies, often rely on narrative reviews. In summary, the necessity to include observational studies in systematic reviews and evaluate their quality has been highlighted in recent literature analytics. However, obtaining clinically useful information by complementing the results of randomized studies with information from observational studies has not been sufficiently suggested.
Recently, the integration of different studies into designs in the field of network meta-analysis has been discussed.
In a network meta-analysis, direct and indirect evidence should be analyzed and integrated. A methodology integrating randomized and observational studies has also been studied in the process of synthesizing evidence with different levels of validity [ 33 , 34 ]. Efthimiou et al [ 35 ] classified the proposed integrated analysis methods in the literature to date into three categories.
These are design-adjusted analyses, in which all trials included in the network meta-analysis involve estimates adjusted according to possible bias and overprecision based on expert opinions ; using informative priors, in which meta-analysis of randomized trials is performed based on priors formulated from meta-analyzing observational studies Bayesian approach ; and three categorical models, in which a meta-analysis is performed for each design, and consequently, the overall effect is acquired by synthesizing all design-specific estimates.
The methods suggested in the field of network meta-analysis and the method of the present study are similar in principle. That is, the results are integrated into a differential consideration of the validity of the evidence. On the other hand, the model of this study is distinct from those suggested in network meta-analysis, in that it is a clinically logical model that analyzes the trend of the synthesized results after differential analysis by considering study quality.
In addition, the model proposed in this study is less difficult to apply because it does not require additional statistical analysis or software use. It also has the advantage that clinical interpretation is easy and intuitive, even for physicians without mathematical expertise, because it is based on clinical logical flow.
These distinctive features and practical merits provide a summary of the significance of the stepwise hierarchical model, which is a novel method suggested for integration of nonrandomized and randomized studies in frequentist or classical meta-analyses.
The limitations of this study are as follows. The four typical patterns described in this study cannot explain all possible patterns and their variations. For a detailed interpretation of clinical decisions, indicators of heterogeneity and publication bias should be interpreted as well. Researchers who are accustomed to making bidirectional decisions based on a specific P value of. Therefore, quantitative and qualitative interpretation are necessary.
Cooperation between a clinician and a biostatistician with sufficient experience in meta-analysis is recommended to successfully use our model. The conclusion empowered by the main results as well as the subgroup results of our second example study can serve as a reference of cooperative interpretation.
We expect future meta-analysis studies to use our model and interpret their results, including diverse variations to strengthen the utility of the model and resolve current limitations. We recommend using the stepwise-hierarchical pooled analysis approach as a model for interpreting meta-analyses involving randomized and observational studies in a synergistic manner. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Edited by G Eysenbach; submitted Skip to Main Content Skip to Footer. Article Authors Cited by Tweetations Metrics. Original Paper. Related Article This is a corrected version. Introduction In the literature, the number of meta-analyses that include observational studies has steadily increased in recent decades [ 1 ].
Methods Identifying Limitations to Overcome Meta-analyses are performed to aid clinical decision-making in intractable oncologic situations in which a single standard modality has not been established.
Confounders in Observational Studies When comparing intervention and control groups, the randomization of participants has the advantage of evenly distributing both known and unpredictable confounders [ 11 ].
View this figure Clinical Logic Flow in the Gray Zone Physicians should make clinical decisions by using studies with different designs in gray zone situations. However, this unsystematic method should be avoided, and a recommended clinical logic flow of interpretation may be as follows: The pooled results from RCTs determine the direction of the clinical hypothesis and the representative effect size. Rationale of Stepwise-Hierarchical Pooled Analysis Stepwise-hierarchical pooled analysis is a method of interpreting the pooled results of studies categorized according to their design and validity.
Results Descriptive Interpretation The descriptive interpretation of the four representative patterns Figure 2 is as follows: The effect size and statistical significance increase gradually: The results of the randomized study analysis are statistically significant, and the effect size gradually increases, strengthening the support for the hypothesis.
Therefore, the probability is high that the hypothesis is true and strongly positive. The effect size in the observational studies will be lower than the true effect, and if confounders are controlled for, the effect size can be increased. The results of the pooled analyses of observational studies with confounders may not be statistically significant. The effect size gradually increases and the results are statistically significant at all stages: The results of the randomized study analysis are statistically significant, and the pattern of increasing effect size gradually strengthens the reliability of the hypothesis.
The effect size of observational studies is lower than that of the true effect. Confounders may have a negative effect on the results of observational studies, but because they show statistically significant results, this effect is assumed to be smaller than that in pattern 1. The effect size and statistical significance decrease gradually: The target hypothesis is rejected because the results of the randomized study analysis are not statistically significant.
The effect size and statistical significance of the observational studies are not trustworthy. The effect size gradually decreases and the results are statistically significant at all stages: The target hypothesis is judged to be true because the results of the randomized study analysis are statistically significant.
However, the pattern of the effect size gradually decreases, which lowers the reliability of the hypothesis. The effect size of observational studies is larger than the true effect. Once again, out of the above patterns, the hypothesis is true if the effect sizes are similar in the pooled analyses of both randomized and observational studies, and both analyses are statistically significant.
In contrast, if the results of the randomized and observational studies contradict each other, the pooled results of the randomized studies should be weighted more heavily and further investigation of this contradiction should be performed. The stepwise-hierarchical method may not be highly necessary for these situations. Interpretation of the four representative patterns of stepwise-hierarchical pooled analysis. OBS: observational studies. View this figure Examples of Clinical Interpretation Our team recently published two meta-analyses that used the stepwise-hierarchical method [ 21 , 22 ].
A clinical meta-analysis example of the ascending pattern in the stepwise-hierarchical method based on our previous meta-analysis evaluating the benefits of adjuvant radiochemotherapy after D2 gastrectomy as compared to chemotherapy alone [ 21 ]. The forest plots are newly drawn from the raw data obtained by the authors. ES: effect size; CRT: chemoradiotherapy.
A clinical meta-analysis example of the descending pattern in the stepwise-hierarchical method based on our previous meta-analysis evaluating the benefits of local treatment on oligometastatic disease [ 22 ]. ES: effect size; LCT: local consolidative treatment. View this figure Discussion Principal Considerations The number of meta-analyses in the literature that include observational studies has been steadily increasing [ 1 ]. Limitations The limitations of this study are as follows.
Conclusions We recommend using the stepwise-hierarchical pooled analysis approach as a model for interpreting meta-analyses involving randomized and observational studies in a synergistic manner. Conflicts of Interest None declared. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA Apr 19; 15 Meta-analysis and the science of research synthesis. Nature Mar 07; Overlapping meta-analyses on the same topic: survey of published studies. Evidence for health decision making - beyond randomized, controlled trials.
N Engl J Med Aug 03; 5 Updating perspectives on meta-analyses in the field of radiation oncology. Cancer statistics in Korea: incidence, mortality, survival, and prevalence in Efficacy and safety of sorafenib in patients in the Asia-Pacific region with advanced hepatocellular carcinoma: a phase III randomised, double-blind, placebo-controlled trial.
The Lancet Oncology Jan;10 1 Sorafenib in advanced hepatocellular carcinoma. N Engl J Med Jul 24; 4 Comparison of radiation therapy modalities for hepatocellular carcinoma with portal vein thrombosis: a meta-analysis and systematic review. Radiother Oncol Oct; 1 A survey on transarterial chemoembolization refractoriness and a real-world treatment pattern for hepatocellular carcinoma in Korea.
Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles. Am J Epidemiol Nov 15; 10 Ch GRADE handbook. Causal diagrams for epidemiologic research. Epidemiology ;10 1 A definition of causal effect for epidemiological research. A meta-analysis of the efficacy of postoperative adjuvant radiotherapy versus no radiotherapy for extrahepatic cholangiocarcinoma and gallbladder carcinoma.
The simple pooling of data is often used to provide an overall summary of subgroup data or data from a number of related studies. In simple pooling, data are combined without being weighted. Therefore, the analysis is performed as if the data were derived from a single sample. This kind of analysis ignores characteristics of the subgroups or individual studies being pooled and can yield spurious or counterintuitive results.
In meta-analysis, data from subgroups or individual studies are weighted first, then combined, thereby avoiding some of the problems of simple pooling.
0コメント