Subject & Study Guides: Systematic Review Guide: Critical Appraisal / Data Extraction and Synthesis

Critical Appraisal / Data Extraction and Synthesis

Examples of Quality Checks	Examples of Bias
Ethical approval was obtained? Is there a Protocol? Registered with clinical trial registry? Reported in accordance with reporting standard e.g. CONSORT? Was there a clear statement of the aims of the research Does this study address a clearly focused question? Did the study use valid methods to address this question? Conflicts of Interest, affiliations of researchers etc? Was the study participant recruitment strategy appropriate Were the relationship between the recruiters and the participants considered? Are the valid results of this study important? Were all important outcomes assessed Were Outcomes measured the same way for all groups What percentage of participants completed the study? Was there full 'follow-up' Are the analysis or statistical methods well described and appropriate	Selection Bias: Differences between groups to be compared Performance Bias: Unequal provision of care Detection Bias: Unfair assessment of outcomes Reporting Bias: Only statistically significant differences are reported Attrition Bias: Loss of outcome data

Examples of Quality Checks

Examples of Bias

Ethical approval was obtained?
Is there a Protocol?
Registered with clinical trial registry?
Reported in accordance with reporting standard e.g. CONSORT?
Was there a clear statement of the aims of the research
Does this study address a clearly focused question?
Did the study use valid methods to address this question?
Conflicts of Interest, affiliations of researchers etc?
Was the study participant recruitment strategy appropriate
Were the relationship between the recruiters and the participants considered?
Are the valid results of this study important?
Were all important outcomes assessed
Were Outcomes measured the same way for all groups
What percentage of participants completed the study?
Was there full 'follow-up'
Are the analysis or statistical methods well described and appropriate

Selection Bias: Differences between groups to be compared
Performance Bias: Unequal provision of care
Detection Bias: Unfair assessment of outcomes
Reporting Bias: Only statistically significant differences are reported
Attrition Bias: Loss of outcome data

Some Quality Assessment Resources

Randomised Control Trials

(CASP) Critical Appraisal Skills Programme; Effective Practice and Organisation of Care Reviews (EPOC); Cochrane risk of Bias; Centre for Evidence Based Medicine critical Appraisal tools (CEBM); SURE for Randomized Controlled Trials and Other Experimental Studies; (SIGN) Scottish Intercollegiate Guidelines Network; (NHLBI) National Heart, Lung and Blood Institute; Joanna Briggs Checklist for Randomized Controlled Trials; PEDro Scale.

Observational Studies

(CASP) Critical Appraisal Skills Programme; Effective Practice and Organisation of Care Reviews (EPOC); Cochrane ROBINS ; (SURE) Specialist Unit for Review Evidence all quality appraisal tools; Joanna Briggs Checklists; (SIGN) Scottish Intercollegiate Guidelines Network; (NHLBI) National Heart, Lung and Blood Institute

Qualitative

(CASP) Critical Appraisal Skills Programme;; (SURE) Specialist Unit for Review Evidence all quality appraisal tools;; Joanna Briggs Checklists; Centre for Evidence Based Medicine critical Appraisal tools (CEBM)

AMSTAR checklist for Systematic Reviews What is AMSTAR AMSTAR stands for A MeaSurement Tool to Assess systematic Reviews. There has been a proliferation of systematic reviews as one of the key tools for evidence-based health care. This has presented both opportunities and risks. The opportunities being that it creates an environment where researchers can base decisions on accurate, succinct, credible, comprehensive and comprehensible summaries of the best available evidence on a topic thereby minimising error and bias. The risks include variation in quality and empirical validation.
Equator The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network is an international initiative that seeks to improve the reliability and value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines.
ICMJE recommendations International Committee of Medical Journal Editors developed these recommendations to review best practice and ethical standards in the conduct and reporting of research and other material published in medical journals, and to help authors, editors, and others involved in peer review and biomedical publishing create and distribute accurate, clear, reproducible, unbiased medical journal articles. The recommendations may also provide useful insights into the medical editing and publishing process for the media, patients and their families, and general readers
How to make a Forest Plot in Excel
Quality Assessment and Risk of Bias Tool Repository A resource for finding and selecting a risk of bias or quality assessment tool for evidence synthesis projects. This project was created and is maintained by Duke University's Medical Center Library & Archives.
ROBVIS risk of bias tool robvis is a web app designed to for visualizing risk-of-bias assessments performed as part of a systematic review. The tool creates: “traffic light” plots of the domain-level judgements for each individual result; and weighted bar plots of the distribution of risk-of-bias judgements within each bias domain. The figures are of publication quality, and are formatted according the risk-of-bias assessment tool used to perform the assessments (e.g. RoB 2, ROBINS-I, or QUADAS-2).
The Systematic Review Toolbox The Systematic Review Toolbox is a web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. It was orignally developed by Dr Christopher Marshall and Anthea Sutton, launching in 2014.
Systematic Reviews and Meta-Analyses - How to Interpret the Results

Example of a more detailed Data Extraction Template from Cochrane here

In most systematic reviews of quantitative studies data extraction is a relatively linear process. Key items for data extraction are specified in advance in a data extraction template, based on the participants, interventions, comparisons and outcomes of interest. This template is then applied to each included study.

Programs like Excel or Google Spreadsheets may be the best option for smaller while specialist systematic review tools provide more detailed and comprehensive support.
It is recommended that to pilot your data extraction tool, especially if you will code your data, to determine if fields should be added or clarified, or if the review team needs guidance in collecting and coding data.

Extraction Templates and Tools

Systematic Review Toolbox

The Systematic Review Toolbox is a web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. It was orignally developed by Dr Christopher Marshall and Anthea Sutton, launching in 2014.

Excel

Excel is the most basic tool for the management of the screening and data extraction stages of the systematic review process. Customized workbooks and spreadsheets can be designed for the review process..

SRDR Systematic Review Data Repository

The Systematic Review Data Repository (SRDR) is a powerful and easy-to-use tool for the extraction and management of data for systematic review or meta-analysis. It is also an open and searchable archive of systematic reviews and their data.

Covidence
Covidence is a software platform built specifically for managing each step of a systematic review project, including data extraction.

RevMan
RevMan is free software used to manage Cochrane reviews..

Distiller (needs subscription)

EPPI Reviewer (subscription, free trial)

Example of a Cochrane Data Extraction Template

Data Extraction Form (Cochrane Review group)

Data Extraction Tips: Blog post from UK research group on issues to consider regarding data extraction.

Information to be extracted may include:

Bibliographic information (author, year);

Study type: systematic reviews of randomized controlled trials; randomized controlled trials; systematic reviews of non-randomized or cohort studies; cohort studies; case control studies; case series; case report

Participant information: (age, gender, sample size, whether from a high risk group, etc)

Intervention information (number of techniques used, use of multiple sessions, duration of intervention (weeks), format of delivery (individual, group, or mixed), source of delivery (medically trained health professional, non-medically trained health professional or non-health professional), theoretical background);

Methodological information (attrition, outcomes, how outcome was validated, length of follow up, study design);

Effect size information (mean, standard deviation, statistic type, value of statistic, p-value, direction of effect, number of responders).

Other possible categories:

Location setting (school, family, community wide, environmental), country);

Behavioural targets of intervention (physical activity, health eating)

Economic information: costs of the intervention, existing evaluation of cost-effectiveness, cost-benefit;

NB: Differentiate between dichotomous and continuous outcomes

Example of an outcomes data extraction table in a systematic review.

Exercise‐based rehabilitation programmes for pulmonary hypertension by Norman R MorrisFiona D KermeenAnne E Holland (2017)

Evidence Synthesis

An specific analytical method to be used should be proposed at the protocol stage

Main analytical methods are: Meta-analysis; Narrative synthesis; Mixed methods; and Qualitative evidence synthesis.

**Data Synthesis methods:**
Narrative Synthesis	refers to an approach to the systematic review and synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis Format/ Structure example: Key question(s) to answer Group studies (Can be by: Intervention, by Theme, by Outcomes, by Population) Summarize results Discuss limitations Further research
Mixed Methods	"The mixed methods approach to conducting systematic reviews is a process whereby (1) comprehensive syntheses of two or more types of data (e.g. quantitative and qualitative) are conducted and then aggregated into a final, combined synthesis, or (2) qualitative and quantitative data are combined and synthesized in a single primary synthesis." (The Joanna Briggs Institute 2014 Reviewers Manual)
Meta-analysis	A meta-analysis is the use of statistical methods to summarize the results of the studies selected as significant in a Systematic Review
Qualitative analysis	A qualitative evidence synthesis (commonly referred to as QES) can add value by providing decision makers with additional evidence to improve understanding of intervention complexity, contextual variations, implementation, and stakeholder preferences and experiences.... One type of Qualitative synthesis is Mata-ethnography, see: Meta-ethnography in healthcare research: a guide to using a meta-ethnographic approach for literature synthesis Methods for qualitative evidence synthesis are complex and continue to develop. Authors should always consult current methods guidance at methods.cochrane.org/qi.

A meta-analysis is a statistical combination of quantitative results from similar studies.

[For a detailed overview see: Cochrane Handbook: Chapter 10 ]

The existence of comparable quantitative data across included studies is necessary for a statistical meta-analysis.
In meta-analysis's the distinction between dichotomous and continuous outcomes is important

Outcome types for analysis:

Dichotomous: Yes/No; alive/dead; success/failure:

Continuous: BMI; blood pressure; psychological scale etc

In the case of dichotomous outcomes the four pieces of data required for a meta-analysis are:

Control total number (data input 1) and event number (data input 2) and Experimental total number (data input 3) and event number (data input 4).

In the case of continuous outcomes an (SD) standard deviation is required, this can sometimes be inferred from other data.

Other statistical relevant calculations can be inferred from the control (total and event numbers) and Experimental (control and event numbers) and include:

Odds	Odds(event) = Prob(event)/Prob(Not event)
Odds Ratio	Odds(event) Control group /Odds(event)Treatment group
Relative Risk	Prob(event) Control group / Prob(event) Treatment group

Example of a date outcome statistics figure in a systematic review from which 'Experimental ' and 'Control' total and standard deviation numbers can be extrapolated

Exercise‐based rehabilitation programmes for pulmonary hypertension by Norman R MorrisFiona D KermeenAnne E Holland (2017)

Statistical packages such as RevMan (free from Cochrane) can be used to preform the statistical analysis and produce a forest plot.

RevMan5 showing statistical analysis options and input tables to allow production of a forest plot.

Note: As is shown in the 'Statistical Input in RevMan diagram only 4 pieces of data input are required (input boxes outlined in green), to produce a forest plot!

Statistical Options in RevMan	Statistical Input in RevMan

A forest plot is:the graphical display of individual study results and, the weighted average of studies included in a systematic review. A forest plot is one way of summarizing the review's results for a specific outcome.

Forest plot example:

Example of a forest plot figure in a systematic review from Cochrance: (Extension of figure four from the studyshown already)

Exercise‐based rehabilitation programmes for pulmonary hypertension by Norman R MorrisFiona D KermeenAnne E Holland (2017)

AMSTAR checklist for Systematic Reviews What is AMSTAR AMSTAR stands for A MeaSurement Tool to Assess systematic Reviews. There has been a proliferation of systematic reviews as one of the key tools for evidence-based health care. This has presented both opportunities and risks. The opportunities being that it creates an environment where researchers can base decisions on accurate, succinct, credible, comprehensive and comprehensible summaries of the best available evidence on a topic thereby minimising error and bias. The risks include variation in quality and empirical validation.
CADIMA CADIMA is a free web tool facilitating the conduct and assuring for the documentation of systematic reviews, systematic maps and further literature reviews.
Colandr
Covidence Covidence streamlines evidence synthesis with a gold standard process for creating high-quality systematic reviews. Note; Covidence is not free. There are individual and Package pricing options
JBI SUMARI JBI SUMARI facilitates the entire systematic review process from protocol to report, and includes team and contributor management for effective and efficient collaboration
Rayyan Rayyan is a free web-tool (Beta) designed to help researchers working on systematic reviews and other knowledge synthesis projects, and dramatically speeds up the process of screening and selecting studies.
RevMan 5 RevMan Web is the online platform recommended for Cochrane intervention and flexible reviews. Log in to RevMan Web to access your review.
SR toolbox The Systematic Review Toolbox is a web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. It was orignally developed by Dr Christopher Marshall and Anthea Sutton, launching in 2014. The toolbox aims to help researchers and reviewers find the following: Software tools Quality assessment / critical appraisal checklists Reporting standards Guidelines
Covidence information videos