Systematic Review Service @ ECU: Appraisal in a Systematic Review

In any systematic review or meta-analysis, a critical question is: are the results of the studies selected trustworthy enough to be included as part of the findings? Answering this question requires planned, thorough appraisal of included studies.

Any issues with the design and conduct of individual studies may compromise the validity of the study findings. In systematic reviews and meta-analyses, the design and conduct flaws of included studies can undermine the strength of conclusions or recommendations for the systematic review/meta-analysis as a whole. It is therefore important to assess the validity of all studies that are included in the systematic review or meta-analysis. This is typically done using some type of Risk of Bias assessment tool that is explicitly described, consistently used for all studies under consideration, and ideally available to readers of the fully published systematic review or meta-analysis.

Selecting a Risk of Bias tool that will be used in a systematic review or meta-analysis is a critical step during the creation of a protocol. Whatever Risk of Bias tool is chosen must be explicitly named and described within the manuscript and subsequently consistently applied to all studies included in the review.

The choice of tool is determined by the types of studies included in the review. The tool used to assess risk of bias in randomized controlled trials will necessarily differ from that used for observational studies. The list of tools below is meant to provide a set to choose from when selecting a risk of bias tool or checklist.

Randomized Controlled Trials:

• The Cochrane Collaboration's tool for assessing risk of bias: https://handbook-5-1.cochrane.org/chapter_8/table_8_5_a_the_cochrane_collaborations_tool_for_assessing.htm
• RoB 2 tool: a revised Cochrane risk of bias tool for randomized trials: https://www.riskofbias.info/welcome/rob-2-0-tool/current-version-of-rob-2
• Jadad scale (the Oxford quality scoring system): available in the appendix of the article: 

Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J., Gavaghan, D. J., & McQuay, H. J. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17(1), 1–12.

Non-Randomized Studies:

• Newcastle-Ottowa Scale: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
• Downs and Black Instrument (instrument in appendix of article): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1756728/pdf/v052p00377.pdf
• ROBINS-I tool (Risk Of Bias in Non-Randomized Studies - of Interventions): https://www.riskofbias.info/welcome/home/current-version-of-robins-i
• Tool to Assess Risk of Bias in Cohort Studies: http://methods.cochrane.org/sites/methods.cochrane.org.bias/files/public/uploads/Tool%20to%20Assess%20Risk%20of%20Bias%20in%20Cohort%20Studies.pdf
• Tool to Assess Risk of Bias in Diagnostic Test Accuracy: http://training.cochrane.org/resource/primer-cochrane-diagnostic-test-accuracy-reviews 
• Tool to assess risk of bias in case control studies: https://www.evidencepartners.com/wp-content/uploads/2017/09/Tool-to-Assess-Risk-of-Bias-in-Case-Control-Studies.pdf

Bench Research:

• ARRIVE Guidelines Checklist: https://www.nc3rs.org.uk/sites/default/files/documents/Guidelines/NC3Rs%20ARRIVE%20Guidelines%20Checklist%20(fillable).pdf

Qualitative Research:

• Critical Appraisal Skills Program (CASP) Checklist for Qualitative research: https://casp-uk.net/wp-content/uploads/2018/01/CASP-Qualitative-Checklist-2018.pdf

Other resources for Critical Appraisal:

• The Joanna Briggs Institute provides a list of critical appraisal tools for different study types, with additional study types not listed above (such as economic evaluations). That list may be accessed here: https://jbi.global/critical-appraisal-tools

Typically, studies at high risk of bias are still fully included in a systematic review or meta-analysis. The best practice is to provide a table or chart that provides the risk of bias scores for each article included so that readers may decide the degree to which a given study's results are or are not credible and the ways that may influence the synthesized evidence.