The explosion of systematic reviews – have we gone too far?

Posted by Simon Knight on September 14, 2016

A fascinating recent paper from Professor John Ioannidis from Stanford University, published in the Milbank Quarterly, has examined the explosion in the publication of systematic reviews and meta-analyses over recent years. Publications between 1991 and 2014 increased by 2,728% for systematic reviews, and 2,635% for meta-analyses – far in excess of the rate of publication of new randomised controlled trials. Of concern, meta-analyses frequently overlap in scope, published close together and without any additional or novel information. Sometimes these studies have identical conclusions, sometimes differences in inclusion criteria lead to differing conclusions resulting in confusion to the reader. Futhermore, Ioannidis estimates that up to 20% of meta-analyses go unpublished, and 20% are flawed beyond repair. He estimates that only around 3% are of decent quality and are clinically useful.

A number of ways in which reviews can be improved are discussed, including standardised reporting, prospective registration of review protocols, improvements in trial design and consistency of outcome reporting and greater availability of individual-patient data. The problem of duplication could be addressed by the use of collaborative tools that allow for continuously updated prospective meta-analyses as new data are published (such as open collaborative meta-analysis). Use of network meta-analysis tools would even allow the addition of data regarding new treatments for a condition as they become available.

One area where I disagree slightly with Ioannidis is his assertion that a high proportion (17%) of reviews are non-informative and hence not useful. Often, a systematic review will conclude that there are a paucity of quality data regarding an intervention, and that further high-quality studies are required. I have written previously about the Cochrane approach to such topics which often results in empty reviews (due to the the lack of randomised evidence). Whilst this approach is undoubtedly non-informative, a well performed narrative systematic review will not only inform the reader of the lack of evidence (an important finding in itself), but also examine the existing (often flawed) evidence to help inform the design of future trials.

Despite the reservations expressed in his paper, Ioannidis still sees value in properly-performed systematic reviews and meta-analyses performed by experienced teams in the absence of commercial bias and other interests. The challenge is to maintain the reputation of these important studies in the face of increasing dilution from biased and flawed reviews.

July 2016 Transplant Trial Watch now available

Posted by Simon Knight on July 1, 2016

The July 2016 edition of the Transplant Trial Watch is now available.

This month, trials include long-term outcomes from BENEFIT-EXT, treatment of asymptomatic bacteriuria in renal transplant recipients and and in depth analysis of the VICTOR trial.

Join the discussion: recipient obesity and outcome after renal transplantation

Posted by Liset Pengel on November 5, 2015

Two recent systematic reviews were published on this important topic. The review by Lafranca and colleagues entitled “Body mass index and outcome in renal transplant recipients: a systematic review and meta-analysis”(BMC Medicine 2015;13(1):111) included 56 studies. The authors concluded that “Several of the pooled outcome measurements show significant benefits for ‘low’ BMI (<30) recipients. Therefore, we postulate that ESRD patients with a BMI >30 preferably should lose weight prior to RT”. The review by Hill and colleagues entitled “Recipient obesity and outcomes after kidney transplantation: a systematic review and meta-analysis” (Nephrol Dial Transplant 2015;30(8):1403-11) included 17 studies. They concluded that “Despite having a much higher likelihood of DGF, obese transplant recipients have only a slightly increased risk of graft loss and experience similar survival to recipients with normal BMI”. Both authors have left commentaries in the Transplant Library (

Join the discussion and leave your opinion in the Transplant Library. Login with your personal account details or through your society. Contact us if you have problems logging in or would like a free trial account.

Sirolimus associated with less malignancy but an increased mortality after kidney transplantation

Posted by Peter Morris on January 6, 2015

A group from Ottawa, led by Greg Knoll, have published an important systematic review of risk of cancer in patients who had sirolimus included with their immunosuppression de novo as well as those who were converted at some stage after transplantation to sirolimus. The study also included a meta-analysis of individual patient data. This was available from 5,676 patients from 21 randomised trials. Sirolimus was associated with a 40% reduction in the risk of malignancy and particularly in the reduction of the risk of a non-melanoma skin cancer, compared with patients who did not receive sirolimus. The most marked effect was seen in patients who were converted to sirolimus at some time following the transplant procedure.

However sirolimus was associated with a 43% increased risk of death compared with controls. The causes of death in sirolimus patients were either infection or cardiovascular. The authors suggest that although there is a reduction in the risk of malignancy, the increased risk of death in patients receiving sirolimus does not justify its use. This may well be true but this is a critically important study and no doubt will trigger others to look specifically at mortality in their patients who are receiving, or have received, an mTOR inhibitor.

22 Pages of Nothing

Posted by Simon Knight on August 23, 2013

I was very amused (and slightly horrified) by the Cochrane Hepato-Biliary group’s most recent offering regarding surgical resection versus liver transplantation for hepatocellular carcinoma.  The 22-page review details elegantly the background to the review, gives an in depth search strategy and description of excluded studies and then concludes:

“There are no randomised clinical trials comparing surgical resection and liver transplantation for hepatocellular carcinoma treatment.”.

Recent evidence suggests that a chocolate teapot may me marginally more useful.

The whole idea of a systematic review is to summarise the best available evidence for an intervention.  Whilst the ideal outcome would be to identify a large body of RCT data, this is usually not the case but that does not necessarily mean that there is no value to the evidence that is available.  Knowing what work has been previously performed in the area of interest, even if studies are not randomised, can sometime provide useful information about a potential treatment effect, and even if not may provide useful data upon which to base future studies.  Indeed, the Cochrane Collaboration’s own GRADE approach allows for the upgrading of non-randomised evidence if effect sizes are large and there is no discernable bias.

To my mind, if a literature search identifies no randomised controlled trials relevant to the intervention under investigation, then the next available level in quality of trial should be appraised.  Often, firm conclusions will be impossible to make, in which case rather than uselessly stating “no trials were found”, the available evidence can be used to suggest a specific and realistic trial that could be performed in order to fill the identified gap in the evidence.

Incidently, our own group reviewed the evidence for transplantation in hepatic malignancy back in 2007, identifying 5 retrospective studies comparing resection and transplantation in patients with Childs A cirrhosis and HCC.  These trials supported the idea that disease free survival is improved with transplantation, but that there is no great evidence for an improvement in overall survival.  The suggestion is that the benefits in terms of removing the diseased liver tissue are offset by the surgical risk and risks of lifelong immunosuppression with transplantation.  We recommended an RCT of transplantation versus resection for patients with normal hepatic function, utilising an ITT analysis to account for progression whilst on the transplant waiting list.

Multiple reviews of machine perfusion find reduced delayed function of kidneys but no effect on graft survival

Posted by John O'Callaghan on May 24, 2013

A group from Sun Yat-sen University, Guangzhou, China has recently published a systematic review comparing the outcomes in patients receiving kidneys from donation after cardiac death (DCD). Deng et al conducted a meta-analysis using the results from four randomised controlled trials (351 kidneys).  Machine perfusion reduced the rate of delayed graft function compared to static cold storage (odds ratio=0.56, p=0.008).

Another group, from the University of Sydney, Australia, Lam et al, published a review late last year, including DCD and (donation after brain-death) DBD kidneys (1353 kidneys), also finding a reduced risk of delayed graft function (relative risk=0.83, p=0.01).

Our group has recently conducted a systematic review and meta-analysis addressing the same issue, presented at the Transplantation Society Congress, Berlin 2012, and now published in the British Journal of Surgery. We conducted a meta-analysis including all donor types (1475 kidneys) as well as subgroup analysis by donor type (DCD versus heart-beating donation or DBD). Whilst in the overall meta-analysis of all donor types we found a significant reduction in delayed function with machine perfusion (relative risk=0.81, p=0.002) we did not find a significant reduction with kidneys from cardiac death alone (relative risk=0.8, p=0.094). This could be due to two key factors:

1) We did not include data from the study by Matsuno et al, as we felt it was not a randomised controlled trial. It is relatively small however, and may not have contributed greatly to the overall meta-analysis by Deng et al.

2) I-squared testing showed heterogeneity of 40%, we therefore used random effects meta-analysis, which will result in wider confidence intervals. Deng et al used fixed effects meta-analysis given the lower heterogeneity between their included studies.

Both of these factors will result in less power to demonstrate a significant effect in our subgroup analysis of DCD kidneys. Overall though the effect of machine perfusion was similar on DBD and DCD in our study. Overall we would agree with the conclusions of Deng et al.

In comparison to the study by Lam et al, our results are very similar, including the point estimate of the summary effect and the 95% confidence intervals. (relative risk=0.83, 0.72-0.96 and relative risk=0.81, 0.71-0.92). These slight differences can be attributed to two small differences between the studies:

1) We were fortunate to receive extra information from the steering committee of the Eurotransplant study, which allowed us to incorporate results from the study extensions of DCDECD, and senior program kidneys without overlap.

2) We did not include data from the paper by Jaffers and Banowsky 1989, as we felt it was a retrospective study and did not meet our inclusion criteria. We instead included a study by Van der Vliet et al.

These slight differences in methods lead to the small differences in summary effects and confidence intervals. Overall we would agree with the conclusions of Lam et al.

Interestingly all three reviews did not find evidence for a difference in graft survival, Deng et al and Lam et al using meta-analysis, and our group  in narrative review. Further weight is added to this discussion by the recent review of UNOS data by Cannon et al at the University of Louisville, USA. Across multivariate analysis, paired kidney analysis, and propensity-matched comparisons, Cannon et al found a consistent reduction in delayed function associated with machine perfusion, but no difference in graft survival.





Alemtuzumab as induction therapy in renal transplantation: differing results from systematic reviews.

Posted by John O'Callaghan on January 3, 2013

Re: Alemtuzumab induction in renal transplantation: A meta-analysis and systemic review. Zhang X, Huang H, Han S, Fu S, Wang L. Transpl Immunol. 2012 Oct;27(2-3):63-8

A recent systematic review and meta-analysis has been published in Transplant Immunology. The review by Zhang et al assesses the evidence for the use of Alemtuzumab as induction therapy in renal transplantation. It is compared to “traditional antibodies” including Rabbit Antithymocyte Globulin (rATG) and Interleukin-2 Receptor Antibodies (IL2RAs). The authors conclude that Alemtuzumab is superior to traditional antibodies in the prevention of acute rejection. Last year Morgan et al published a review of this topic, reaching slightly different conclusions to Zhang et al.

Zhang et al concluded that Alemtuzumab reduced acute rejection in comparison with “traditional antibodies” which included a combination of agents. They found specifically that Alemtuzumab reduced acute rejection compared to rATG but did not have enough studies to compare Alemtuzumab to IL2RAs alone.

In contrast to this, we concluded that Alemtuzumab reduces rejection events in comparison with IL2RAs (RR=0.54, p<0.01) but not in comparison with rATG (RR=0.79, p=0.28).

There are three key reasons why these differing conclusions were presented by the two papers:

  1. Zhang et al combined results from studies comparing a variety of agents to Alemtuzumab. Given the different mechanisms of action we did not do this and only compared it to each class separately.
  2. Our method of analysis was facilitated by inclusion of data from studies that Zhang et al did not include. One of these studies is currently unpublished but additional data was kindly provided by the authors on request.
  3. We used data from the later report of one study, rather than the earlier report used by Zhang et al. The later report had longer follow-up and included more patients (180 versus 77).

The differences in methodology lead to rather different conclusions between the two reviews. In order to minimize biases in reviews it is important to identify unpublished studies and to request data from these if possible. Requesting additional data from the authors of published studies also allows reviewers to take extra steps in making the best use of study data.

Research fellow awarded MSc in Surgical Sciences

Posted by Simon Knight on August 24, 2012

Philip Macklin

Philip Macklin

Philip Macklin was awarded an MSc in Surgical Sciences jointly from the University of Edinburgh and the Royal College of Surgeons in June of this year, with distinction. Working under the supervision of the CET, his dissertation was entitled “A Systematic Review of the Use of Rituximab in Renal Transplantation”. He is now preparing manuscripts from his work for publication.

Philip is now starting clinical training as a pathologist.

Systematic Review of Preservation Solutions for Static Cold Storage of Kidneys

Posted by John O'Callaghan on October 5, 2011

We have recently completed a systematic review of preservation solutions for the static cold storage of kidney allografts (this was presented at the recent ESOT meeting and has been accepted for publication subject to satisfactory revision). We used MEDLINE, EMBASE, the Cochrane Library and the Transplant Library to perform the literature search. Searches combined MeSH and EMTREE keywords with free-text aliases for the preservation solutions. No date or language limits were applied. Inclusion criteria specified any comparative, prospective study of preservation solutions for cadaveric renal allografts.

15 trials with a total of 3,584 kidneys were included (10 RCTs and 5 Non-RCTS). We found that the risk of DGF is increased with EC stored kidneys when compared to both UW and HTK in the largest, best quality RCTs. On the basis of the 3 RCTs that compared UW with Celsior and 2 that compared UW with HTK, we concluded that these 3 solutions are associated with a comparable risk of DGF. We were able to combine the results for comparisons of UW and Celsior in a fixed effects meta-analysis to demonstrate that the risk of DGF was equal (RR=0.97, CI= 0.76-1.23, p=0.79).

There does not appear to be support for the use of one of these 3 solutions over another on the basis of cost as prices per litre are similar, as are the recommended flush volumes. Most surprisingly, there has been no RCT comparing hypertonic citrate (HOC) for renal preservation to any of the solutions in use before its development, or to newer solutions such as UW and HTK. Overall, the level of available evidence was low and a limited amount of prospective evidence was available. Furthermore the methodological quality of most of the trials that met the inclusion criteria was poor. We are now examining the evidence for preservation methods in non-renal abdominal organs.

Prospective registration of systematic reviews – the PROSPERO intiative

Posted by Simon Knight on April 23, 2011

In the world of randomised controlled trials (RCTs), a number of initiatives have existed for some time to attempt to improve methodological quality and reporting.  Amongst the most important of these have been the CONSORT statement for the reporting of RCTs, and initiatives for the prospective registration of clinical trials such as the ISRCTN.  More recently, efforts have extended to the reporting of systematic reviews, with the PRISMA statement mirroring the CONSORT statement for the reporting of systematic reviews.  The Cochrane Collaboration have been providing tools and support for health care researchers undertaking systematic reviews for some time, including the registration of review protocols, but until now there has been no attempt to mirror the ISRCTN database in prospectively registering all systematic reviews, whether Cochrane reviews or otherwise.

Prospero logo

Now the Centre for Reviews and Dissemination at the National Institute for Health Research have begun an initiative to prospectively register systematic reviews in an international register (PROSPERO).  Registration is free and there is a structured minimum dataset required for registration of a review protocol that ensures that standards are met.  Registered trials are given a unique identifier.

Prospective registration of systematic reviews is important for a number of reasons.  Most importantly, it means that any deviation for the published protocol can be questioned and must be explained, reducing the risk of bias caused by modification of the review protocol during the course of the review process.  It also provides an online database of reviews in progress to prevent duplication of efforts or to enable other researchers to identify in-progress reviews relevant to their field.  The structured format of the protocol helps to ensure that the finished review will adhere to the PRISMA guidelines for reporting of reviews.

We will be using the PROSPERO site in future to prospectively register our reviews here at the CET, and would encourage all researchers in the field of evidence based medicine to support this initiative.  It is entirely possible that at some point in the future, as with RCTs, journal editors will require evidence of a prospectively published public protocol for the review before publication is considered.

Next Page »