Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Systematic Review Service @ ECU: Appraisal in a Systematic Review

Assessing Studies for Bias

In any systematic review or meta-analysis, a critical question is: are the results of the studies selected trustworthy enough to be included as part of the findings? Answering this question requires planned, thorough appraisal of included studies.

Any issues with the design and conduct of individual studies may compromise the validity of the study findings. In systematic reviews and meta-analyses, the design and conduct flaws of included studies can undermine the strength of conclusions or recommendations for the systematic review/meta-analysis as a whole. It is therefore important to assess the validity of all studies that are included in the systematic review or meta-analysis. This is typically done using some type of Risk of Bias assessment tool that is explicitly described, consistently used for all studies under consideration, and ideally available to readers of the fully published systematic review or meta-analysis.

 

Risk of Bias Tools

Selecting a Risk of Bias tool that will be used in a systematic review or meta-analysis is a critical step during the creation of a protocol. Whatever Risk of Bias tool is chosen must be explicitly named and described within the manuscript and subsequently consistently applied to all studies included in the review.

The choice of tool is determined by the types of studies included in the review. The tool used to assess risk of bias in randomized controlled trials will necessarily differ from that used for observational studies. The list of tools below is meant to provide a set to choose from when selecting a risk of bias tool or checklist.

 

Randomized Controlled Trials:

Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J., Gavaghan, D. J., & McQuay, H. J. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17(1), 1–12.

Non-Randomized Studies:

Bench Research:

Qualitative Research:

Other resources for Critical Appraisal:

  • The Joanna Briggs Institute provides a list of critical appraisal tools for different study types, with additional study types not listed above (such as economic evaluations). That list may be accessed here: https://jbi.global/critical-appraisal-tools

Deciding to Exclude Studies Based on Design Flaws

Typically, studies at high risk of bias are still fully included in a systematic review or meta-analysis. The best practice is to provide a table or chart that provides the risk of bias scores for each article included so that readers may decide the degree to which a given study's results are or are not credible and the ways that may influence the synthesized evidence.