Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

TUS Logo

Systematic Review Guide

Overview of the Systematic Review process with links to resources

Critical Appraisal / Data Extraction and Synthesis

Examples of Quality Checks Examples of Bias
  • Ethical approval was obtained?
  • Is there a Protocol?
  • Registered with clinical trial registry?
  • Reported in accordance with reporting standard e.g. CONSORT?
  • Was there a clear statement of the aims of the research
  • Does this study address a clearly focused question?
  • Did the study use valid methods to address this question?
  • Conflicts of Interest, affiliations of researchers etc?
  • Was the study participant recruitment strategy appropriate
  • Were the relationship between the recruiters and the participants considered?
  • Are the valid results of this study important?
  • Were all important outcomes assessed
  • Were Outcomes measured the same way for all groups
  • What percentage of participants completed the study?
  • Was there full 'follow-up'
  • Are the analysis or statistical methods well described and appropriate
  • Selection Bias: Differences between groups to be compared
  • Performance Bias: Unequal provision of care
  • Detection Bias: Unfair assessment of outcomes
  • Reporting Bias: Only statistically significant differences are reported
  • Attrition Bias: Loss of outcome data

 

Example of a more detailed Data Extraction Template from Cochrane here

In most systematic reviews of quantitative studies data extraction is a relatively linear process. Key items for data extraction are specified in advance in a data extraction template, based on the participants, interventions, comparisons and outcomes of interest. This template is then applied to each included study.

Programs like Excel or Google Spreadsheets may be the best option for smaller while specialist systematic review tools provide more detailed and comprehensive support.
It is recommended that to pilot your data extraction tool, especially if you will code your data, to determine if fields should be added or clarified, or if the review team needs guidance in collecting and coding data.

Extraction Templates and Tools

Systematic Review Toolbox

The Systematic Review Toolbox is a web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. It was orignally developed by Dr Christopher Marshall and Anthea Sutton, launching in 2014.

Excel

Excel is the most basic tool for the management of the screening and data extraction stages of the systematic review process. Customized workbooks and spreadsheets can be designed for the review process..

SRDR Systematic Review Data Repository

The Systematic Review Data Repository (SRDR) is a powerful and easy-to-use tool for the extraction and management of data for systematic review or meta-analysis. It is also an open and searchable archive of systematic reviews and their data.

Covidence
Covidence is a software platform built specifically for managing each step of a systematic review project, including data extraction. 

RevMan
RevMan is free software used to manage Cochrane reviews..

Distiller (needs subscription)

EPPI Reviewer (subscription, free trial)

Example of a Cochrane Data Extraction Template

Data Extraction Tips: Blog post from UK research group on issues to consider regarding data extraction.

Information to be extracted may include:

Bibliographic information (author, year);

Study type: systematic reviews of randomized controlled trials; randomized controlled trials; systematic reviews of non-randomized or cohort studies; cohort studies; case control studies; case series; case report

Participant information:  (age, gender, sample size, whether from a high risk group, etc)

Intervention information (number of techniques used, use of multiple sessions, duration of intervention (weeks), format of delivery (individual, group, or mixed), source of delivery (medically trained health professional, non-medically trained health professional or non-health professional), theoretical background);

Methodological information (attrition, outcomes, how outcome was validated, length of follow up, study design);

Effect size information (mean, standard deviation, statistic type, value of statistic, p-value, direction of effect, number of responders).

Other possible categories: 

Location setting (school, family, community wide, environmental), country);

Behavioural targets of intervention (physical activity, health eating)

Economic information: costs of the intervention, existing evaluation of cost-effectiveness, cost-benefit;

NB: Differentiate between dichotomous and continuous outcomes

 

Example of an outcomes data extraction table in a systematic review.

Exercise‐based rehabilitation programmes for pulmonary hypertension by Norman R MorrisFiona D KermeenAnne E Holland (2017)

Evidence Synthesis

An specific analytical method to be used should be proposed at the protocol stage

Main analytical methods are: Meta-analysis; Narrative synthesis; Mixed methods; and Qualitative evidence synthesis.

 

Data Synthesis methods:
Narrative Synthesis

refers to an approach to the systematic review and synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis

Format/ Structure example:  
Key question(s) to answer

  • Group studies (Can be by: Intervention, by Theme, by Outcomes, by Population)
  • Summarize results
  • Discuss limitations
  • Further research
Mixed Methods "The mixed methods approach to conducting systematic reviews is a process whereby (1) comprehensive syntheses of two or more types of data (e.g. quantitative and qualitative) are conducted and then aggregated into a final, combined synthesis, or (2) qualitative and quantitative data are combined and synthesized in a single primary synthesis." (The Joanna Briggs Institute 2014 Reviewers Manual)
Meta-analysis meta-analysis is the use of statistical methods to summarize the results of the studies selected as significant in a Systematic Review
Qualitative analysis

A qualitative evidence synthesis (commonly referred to as QES) can add value by providing decision makers with additional evidence to improve understanding of intervention complexity, contextual variations, implementation, and stakeholder preferences and experiences....

One type of Qualitative synthesis is Mata-ethnography, see: Meta-ethnography in healthcare research: a guide to using a meta-ethnographic approach for literature synthesis

Methods for qualitative evidence synthesis are complex and continue to develop. Authors should always consult current methods guidance at methods.cochrane.org/qi.

 

A meta-analysis is a statistical combination of quantitative results from similar studies.

[For a detailed overview see: Cochrane Handbook: Chapter 10 ]

  • The existence of comparable quantitative data across included studies is necessary for a statistical meta-analysis.
  • In meta-analysis's the distinction between dichotomous and continuous outcomes is important

Outcome types for analysis:

Dichotomous: Yes/No; alive/dead; success/failure:   

Continuous: BMI; blood pressure; psychological scale etc

In the case of dichotomous outcomes the four pieces of data required for a meta-analysis are:

Control total number (data input 1) and event number (data input 2) and Experimental total number (data input 3) and event number (data input 4).

In the case of continuous outcomes an (SD) standard deviation is required, this can sometimes be inferred from other data.

Other statistical relevant calculations can be inferred from the control (total and event numbers) and Experimental (control and event numbers) and include:

Odds Odds(event) = Prob(event)/Prob(Not event)

Odds Ratio

Odds(event) Control group /Odds(event)Treatment group
Relative Risk Prob(event) Control group / Prob(event) Treatment group

Example of a date outcome statistics figure in a systematic review from which 'Experimental ' and 'Control' total and standard deviation numbers can be extrapolated

Exercise‐based rehabilitation programmes for pulmonary hypertension by Norman R MorrisFiona D KermeenAnne E Holland (2017)

Statistical packages such as RevMan (free from Cochrane) can be used to preform the statistical analysis and produce a forest plot.

RevMan5 showing statistical analysis options and input tables to allow production of a forest plot.

Note: As is shown in the 'Statistical Input in RevMan diagram only 4 pieces of data input are required (input boxes outlined in green), to produce a forest plot!

Statistical Options in RevMan Statistical Input in RevMan

A forest plot is:the graphical display of individual study results and, the weighted average of studies included in a systematic review. A forest plot is  one way of summarizing the review's results for a specific outcome.

Forest plot example: 

Example of a forest plot figure in a systematic review from Cochrance: (Extension of figure four from the studyshown already)

Exercise‐based rehabilitation programmes for pulmonary hypertension by Norman R MorrisFiona D KermeenAnne E Holland (2017)