Subscribe X
Back to Top

Learn

January 7, 2019 by Melanie Newman | The BMJ

Has Cochrane lost its way?

(The BMJ) – Dissent over growing centralisation culminated in the expulsion of one of Cochrane’s founding members. Melanie Newman reports on the organisation’s internal struggles

The dust is not yet settling on Cochrane after it expelled one of its most high profile scientists and founding fathers. Peter Gøtzsche’s sacking and the resignation of four fellow Cochrane board members in protest has been held out by some as a symptom of a wider malaise at the heart of the international network. Cochrane, they say, has lost its way, its members increasingly disenfranchised from a corporate centre focused on income generation and “message control.”

The Cochrane Collaboration was founded by Iain Chalmers in Oxford in 1993 as a loose knit, international network of 77 researchers1 to help clinicians and others make informed decisions about drugs, surgery, and other interventions. It aimed to do this through “high-quality, relevant, accessible” systematic reviews of randomised controlled trials. Unpaid scientists would produce the reviews, governed by 10 principles, including “fostering good communications, open decision-making, and teamwork” and “minimising bias.” The questioning of orthodoxies and opposition to centralised control was fundamental to Cochrane’s ethos: Chalmers wore a T shirt bearing the words, “Challenge authority.” The international collaboration, he said, should be “committed to opposing any tendency for it to become dominated by any nation, institution, or individual.”

A quarter of a century later and Cochrane seems to be thriving. Membership is at 12 500 people2 and growing. Its income has doubled in the past four years to more than £8m (€9m; $10m). New Cochrane centres are opening in Asia and South America, expanding the collaboration’s global reach. The Cochrane Library boasts 7500 reviews, half of which are accessible without charge (up from 0.05% in 2013 and increasing by 1% a month), with the entire library free to 3.6 billion people in lower income countries. And more people are using the reviews: they were downloaded 12.5 million times in 2017 (a 28% increase on the year before).

Quantity not quality

But Cochrane’s critics contend this growth is not necessarily something to be celebrated. “In healthcare, more does not mean better,” argues former Cochrane board member David Hammerstein. “The core business of Cochrane is its systematic reviews, yet in the past decade Cochrane has dragged its heels in response to insistent concerns that they are largely synthesised information from industry sponsored studies.” He is unimpressed with progress on open access. “After one year of moratorium behind a paywall almost all publications become open access anyway.”

Tom Walley, who until recently made decisions on funding Cochrane in his role as a National Institute for Health Research (NIHR) director, agrees the organisation’s priorities need to change. “It has become a machine, churning out reviews,” he says. The NIHR is partly responsible for this: for years it evaluated Cochrane on how many reviews it produced rather than their impact. “Cochrane should focus less on quantity and more on methodologically high quality reviews in areas of importance to patients,” he advises. “It needs to be more iconoclastic, more challenging, and more of an advocate for evidence based medicine.”

Tellingly, though, the pair disagree on how these improvements in quality are to be achieved. For Hammerstein and many of Gøtzsche’s supporters, Cochrane’s growing central executive and the tighter control it is exerting on the network’s activities, are antithetical to cutting edge science. Hammerstein argues that two opposing views are emerging within Cochrane: that of a collaboration “not afraid of publicly questioning some of the basic social, economic, and scientific premises of our current medical research model” set against a “centralised, functionalist, conformist, and conservative approach.” The leadership’s adoption of the centralised approach has isolated it “intellectually and professionally,” he maintains.

Walley’s view, in contrast, is that more cultural and structural change is required, not less, starting with Cochrane’s review groups, which “were created in the nineties, based on the enthusiasm of people who have now retired.” The central team’s recent introduction of networks to set research priorities in areas such as emergency medicine and cancer is a step in the right direction, he says, but do not go far enough. “Cochrane needs to become more professional,” he adds.

“There is a fundamental tension between those who prioritise individual interests—which were vital to getting the whole organisation off the ground—and those who can see that Cochrane needs to be managed as a whole, with clear aims.”

Internal divisions

While Gøtzsche’s sacking is a first for Cochrane, it is not the first time it has found itself bitterly divided. Its history is punctuated with debates and criticism over conflicts of interest, research quality, and structure.

In the 1990s, when Cochrane was set up, governance was minimal. Lisa Bero, who has chaired a US Cochrane Centre and sat on Cochrane’s board for more than a decade, recalls the way the steering committee operated back then. “Somebody would say, ‘This person is doing some good research, let’s give him about £100 000,’ and it would be agreed.”

Industry influence, however, was taken seriously. Reviewers were asked to consider the harms of interventions on patients, and some members pushed to include unpublished data in reviews.

The more radical fringes of Cochrane campaigned for access to raw trial data and clinical study reports, looking for evidence beyond industry funded trials and analysis.

Within a decade the organisation had fundamentally helped to change the way healthcare decisions were made. Cochrane’s methods drove a collective, international, move towards evidence based decision making. That is not to say its work was infallible. A 1998 assessment of 53 reviews found “major problems” in 29%, with all the problematic conclusions giving too favourable a picture of the experimental intervention.3

Despite Cochrane’s relatively strong position on industry relations, divisions emerged early on over its conflicts of interest policy. By its 10th birthday the collaboration was “at a crossroads” over drug company sponsorship, according to an article in The BMJ.4 Cochrane’s rulebook stipulated that “direct funding from a single source with a vested interest in the results of the review is not acceptable.” Yet, TheBMJ reported the Cochrane library already contained two reviews of drugs funded by their manufacturer.4 The company had agreed to make all data available to reviewers, so Cochrane decided to waive the rule.

Peter Gøtzsche, a founder member of the Cochrane Collaboration and director of the Nordic Cochrane Centre, was one of the strongest opponents of the waiver. A former industry insider whose academic career has focused on bias in clinical trials, Gøtzsche proposed a prohibition on industry sponsored reviews at 2003’s Cochrane Colloquium, an annual open conference. After lengthy debate Cochrane agreed a policy banning funding of Cochrane reviews or review groups by “commercial sources with financial interests in the conclusions”5. However, no consensus was reached on funding of Cochrane centres, and employees of manufacturers of drugs or medical devices were still allowed to write and propose reviews.

Success stories

The year of Cochrane’s 10th birthday was significant for other reasons. The year 2003 saw the return of amodiaquine to the World Health Organization’s essential drugs list for the treatment of malaria—evidence of Cochrane’s growing influence on global health policy.6 The drug had been banned because of case reports of side effects but was reintroduced after a Cochrane review that included unpublished reports showed it was as safe and more effective than the WHO approved chloroquine.

As the next decade progressed Cochrane continued to live up to its aim to be “the most reliable source of evidence healthcare.” Tom Jefferson and colleagues’ finding in 2009 that the anti-flu drug oseltamivir (Tamiflu) offered no clear advantage over aspirin,7 overturning the findings of his own earlier review,8 sealed this status. WHO had recommended oseltamivir in response to fears of a flu pandemic, leading many countries to stockpile the drug.9

Cochrane’s reputation prevailed against a background of methodological debate and a steady but low key flow of criticism about the quality of reviews,10 the reviews’ sometimes esoteric nature, the time it took to produce and update them,11 and continued debate over reviewer12 and trial13 conflicts of interest.

Nevertheless, while it could sometimes be defensive, the organisation continued to positively embrace criticism, setting up an annual award—the Ben Silverman prize—for the best constructive evaluation of its own work. By 2008 Cochrane had more than 50 review groups with some 20 000 contributors supported by a dozen independently funded centres, But despite this enormous growth, the collaboration largely retained its original values and organising structure: that of a grassroots organisation, led from the bottom up.

Corporatisation

That was to change. A strategic review was carried out in 2009 which made 26 recommendations, including an increase in central support for the expanding organisation.

Jeremy Grimshaw, who was co-chair of the Cochrane board from 2010 to 2014 recalls opposition to increased central spending. “There were people who argued that we should be splitting the money between the review groups fractionally, according to their contribution,” he says. “If we’d have done that each group would have gained a small amount that would not have helped their own sustainability and left Cochrane’s central team at risk.”

In November 2012, a new chief executive was appointed: Mark Wilson, a former journalist who had worked at the International Federation of the Red Cross but had no clinical or science background. His first task was to turn the recommendations into strategy.14 “We are a vast organisation, still being managed in an ad hoc hand-to-mouth sort of way. To be ready for the next 20 years, we need to be transformed,” he said at the time.15

The strategy was deeply unpopular in some quarters.

“There was challenge at that time from some of the people who are shouting now,” Wilson says. Shortly after he arrived, he recalls, a video was sent around comparing the then Cochrane leadership to “Hitler and his high command.”

Wilson describes his plan, which was approved unanimously by the board, as “laser-like focused” on both review production and on ensuring Cochrane evidence was used in policy and practice. One plank of the new strategy was greater unity of brand and message across the network. In 2015 the Cochrane Collaboration was renamed16 Cochrane. (Members were told, in a 93 page edict,17” that “you can talk about us as a collaboration, using a small ‘c’”).

Founder member Hilda Bastian, who by that time had already left the organisation over its growing centralisation, felt the change sent the wrong signal. “Embracing a collaboration-less public identity is an important signal, for an organization now explicitly aiming to draw people to a brand with products to sell, rather than to the best research to answer their questions,” she wrote in a blogpost.18

A new “spokesperson policy” was also issued, prompted in part by concerns about Peter Gøtzsche, who as director of the Nordic Cochrane Centre had published a book in 2014 that described the pharma industry as “organised crime,”19 and a Lancet paper arguing that antidepressants can cause more harm than good.20 Gøtzsche’s campaigns resonated among many Cochrane members.

In the same year a fresh Cochrane review of oseltamivir including hitherto unseen industry data had confirmed the 2009 finding.21 These data, whose volume far eclipsed that in the public domain, had been released as a result of a five year concerted campaign by The BMJ and Cochrane members, prompting accusations of industry cover-up. In this environment many viewed Gøtzsche as a champion of truth and research quality, and if some thought he sometimes went too far, many others admired his uncompromising stance.

Response to “spokesperson policy” was equally mixed. Bero saw the policy as uncontroversial. “I am used to working at universities and other organisations where it is understood that only the leadership speaks on behalf of the organisation,” she said. “When individuals hide their own views behind an organisation’s name, it exposes the organisation to liabilities.”

But at Cochrane’s 2015 annual general meeting the spokesperson policy was questioned by Carl Heneghan, one of the authors of the final Cochrane review on oseltamivir.22 When discussing the review with the media he had described himself as part of the Cochrane Collaboration. He would not now do so, he told the meeting, and said he was “confused as to whether Cochrane was still a collaboration.”23

Meanwhile some of the centres were feeling sidelined. An internal document passed to The BMJ reveals that the director of the French centre, Philippe Ravaud, was thinking of resigning in 2015 after he learned the central team was bidding for funding on a translation project jointly with the French centre’s partners. Cochrane, Ravaud wrote, was evolving from a scientific collaboration dedicated to a common idea and shared values to a “pyramidal” organisation “led by power-hungry technocrats.” He also complained that Cochrane was not doing enough to improve the quality of primary evidence.

Cochrane’s central executive also caused ructions in the US, such as when it approached a funder without notifying one of the US Cochrane centres, an action for which Wilson has since apologised.

Further missives were issued from Cochrane’s central team in 2016 as part of a review of the functions of centres and review groups. Centre directors were advised that they were directly answerable to the chief executive24 and new “collaboration agreements” were drawn up.25 Review groups were later organised into eight new networks that set priorities.

The aim was to improve the quality of reviews, Wilson says. “Given the limited resources, we had to make sure choices on which reviews are produced, how they are produced, and how they are disseminated were being done in the most coherent and coordinated fashion.”

Just as the strategy had not been universally welcomed, so its implementation met with resistance. “Are all of the 52 coordinating editors across Cochrane happy that their powers to decide what review to do, when and how they work is being lessened and they’re being encouraged and indeed made to work more collaboratively together?” asks Wilson. “No. You wouldn’t expect them to be, especially as some of them have been doing their own thing for 20 years and regard any change as an impertinence.”

Changes were also made to the Cochrane board in 2016. External members were appointed for the first time and internally elected members ceased to represent specific groups of contributors, standing instead as individuals.26 “Some of our critics feel that has taken away power from certain groups. The impact of that change is still being grappled with,” Wilson says.

By the year end 201627 Cochrane’s income had grown to £6.8m, up from £4.4m in 2014. Most of this income came from royalties on reviews. Staff costs had also more than doubled, from £1.2m to more than £3m. For many organisations an increase in income would be a source of pride. Within Cochrane, resentment at the share being swallowed up by the central executive in London—which had not directly earned it—was growing, alongside frustration at the slowness with which its reviews were becoming open access. Gerd Antes, who was a co-director of Cochrane Germany until recently, challenged the central spending at the 2016 Cochrane AGM.28 He wants central costs cut by 80%.

Board conflicts

For Hammerstein, who joined the board in 2017 the “large expensive London staff” was not only failing to improve research quality but militating against it. High costs mean income must be maintained. “Many long-time Cochrane members are calling for the production of fewer reviews of greater scientific quality, credibility and independence,” he says. “But, of course, greatly slowing down the ‘systematic review assembly line production’ would reduce revenue.” At year end 2017, £6.5m of Cochrane’s £8.6m income came from royalties from review sales.29

Making more reviews freely available would also reduce income, he said.

He says these issues were not discussed sufficiently by the board, which served mainly to “rubber stamp” central executive team decisions, he says.

Nancy Santesso, deputy director of Cochrane Canada, who joined the Cochrane governing board in early 2017 also had concerns over the way the board was run. “Proposals were put forward without enough information to allow an informed decision; there was 10 or 15 minutes’ discussion and then a vote.”

A third board member, Gerald Gartlehner, director of Cochrane Austria, did not initially share Santesso’s and Hammerstein’s serious doubts. He tells TheBMJ that while he had concerns over Cochrane’s change from an academic to a corporate structure and its “message control,” these were “for the most part not beyond anything that might be expected within such a large organisation.”

Like Hammerstein and Santesso, Peter Gøtzsche had rejoined the Cochrane governing board in early 2017. He had stood on a platform that made clear his opposition to the organisation’s direction of travel. He also continued to push for stronger action on conflict of interest. Cochrane’s policy had been strengthened in 2014 to require reviews to have a majority of non-conflicted authors and a non-conflicted lead author,30 with an outright ban on authors employed by holders of patents relevant to the trial intervention. As Bero has argued,31 this goes far further than most journals. But it still allows reviewers to review their own studies and for up to half of the authors in a review team to have conflicts of interests with the company that makes the product they are evaluating.

At a board meeting in March 201832 Gøtzsche lost an argument for tougher measures to be adopted as soon as possible. The central team had proposed a review process, which would also look at non-financial conflicts. “We argued for consultation because there are a wide range of views within the organisation,” Wilson says. The process, which has been described by Hammerstein as a “tortuous” and bureaucratic procedure, was agreed by majority vote and is still ongoing.

Gøtzsche clashed with the Cochrane leadership more publicly when a critique,33 led by him, of Cochrane’s review of the HPV vaccine was published.34 Gøtzsche had previously been accused by Wilson of breaching Cochrane’s spokesperson policy by using the organisation’s letterhead on a complaint to the European Medicines Agency about its evaluation of possible harms of HPV vaccines.

Cochrane instructed lawyers to examine complaints against Gøtzsche going back years. Cochrane has not put the preliminary legal report submitted in September into the public domain. The same month Cochrane board members voted six to five with one abstention that he should be expelled for “bad behaviour”—not only from the board but from all his roles within the organisation, which was unprecedented. Four of the five board members who had voted for Gøtzsche to remain, including Santesso, Hammerstein, and Gartlehner, then walked out in protest.

Documents and statements subsequently released by both Gøtzsche and Cochrane stoked the flames of controversy. For an organisation so concerned with messaging the result was not positive, with media coverage as far afield as India.

The Cochrane Iberoamerican group called for an independent inquiry35 and the Independent Society of Drug Bulletins for the dissolution of the remaining board.36

Gartlehner says: “The issues have become much larger than Peter Gøtzsche. It’s now also about how Cochrane has handled the crisis.” He believes an entirely new board would be best to overcome the rift within the organisation. “Many people think the current rump board does not have the moral authority to carry on.”

The affair became a lightning rod for discontent from many quarters on the issues Gøtzsche had come to embody: opposition to conflicts of influence, improvements in research quality, open access, and outspoken advocacy in relation to all three.

Caroline Struthers is a 2018 Cochrane award winner and a trainer with the Equator Network, an international initiative aimed at improving research quality. Having once enthused, “My aim in life is to help make the Cochrane Collaboration the household name it deserves to be,”37 she became one of many longstanding members to publicise their disillusionment.

Cochrane places “way too much emphasis on ever more sophisticated review methods and the shoehorning of dodgy data into meta-analyses,” she tells The BMJ.

It is impossible to quantify discontent among Cochrane’s 13 000 members without a poll. After news emerged that Gøtzsche’s hospital job was under threat, 8000 people signed a petition38 to the Danish health minister against his dismissal.

A group of Cochrane members led by Jos Verbeek, coordinating editor of the work review group, then set up a petition39 asking Cochrane for action in four areas40: open discussion without recriminations; more financial support for review production and methods development; more member involvement in the organisation’s governance; and increased open access to Cochrane reviews. This petition secured around 600 signatures—not all from members.

Where next?

Even Grimshaw, who believes Gøtzsche could have “avoided” his expulsion, says Cochrane now needs to “establish better working relationships.” “It’s a wake-up call,” he says. “It needs to listen and respond to concerns.”

To date the Cochrane leadership has shown little sign of listening. The board has resisted the calls for it to resign. Board co-chair Martin Burton insists that the majority of Cochrane supported the decision to expel Gøtzsche.

“The anti-group has been very vocal, but this is a community of 13 000 people and way over 90% of those people want to move on with the job in hand,” he says.

This sentiment is echoed by Mark Wilson. “The empty drum sounds loudest,” he tells The BMJ. “If we were seeing that vast numbers of people were leaving Cochrane then you could quite rightly say our strategy is failing. That’s not the case. The community has grown hugely.”

To Cochrane’s critics Gøtzsche’s expulsion represents the latest step on a downward path away from its founding principles. To its supporters it is a blip in Cochrane’s maturation into a more powerful global force. The debate over Cochrane’s future seems unlikely to be resolved quickly or easily. Signatories to Verbeek’s petition include some of Cochrane’s most loyal foot soldiers: scientists who have devoted many hours of unpaid labour to a cause they viewed as bigger than them. These are not employees, who can be forced to toe the line. Retaining the enthusiasm of the volunteer cadre will require more than guidelines. It will require diplomacy, a willingness to admit fault, and good internal communication—the absence of which arguably caused the crisis in the first place.

Cochrane’s structure

Cochrane is made up of several types of entities.2 There are 52 Cochrane review groups (CRGs) incorporated into 8 networks; 17 methods groups; 11 thematic fields; and 20 centres, with 34 associate centres and affiliates in 44 countries.41

CRGs conduct systematic reviews within topic areas set and prioritised by the networks, with the other entities, such as the Cochrane centres, supporting the review groups in various ways.

Despite Wilson’s efforts to expand the numbers of centres—five opened in 2017—some members have questioned the leadership’s commitment to their survival. Wilson scotches any suggestion they will disappear.

“The role of Cochrane Centres is fundamentally to be the representatives of Cochrane in that geographical space and to drive Cochrane evidence into policy and practice in that country or region—we can’t do that from London,” he says.

Numbers of affiliate or associate centres are set to increase substantially with the formation of the Cochrane China Network, he adds. “We have also just opened the formal invitation for organisations and institutions in the US to apply to join or support the new Cochrane US Network.”

The Cochrane board (formerly known as the steering group) sets the organisation’s strategy, and the central executive team manages its day to day operations.

Cochrane finances

Income in 2017 was £8.6m, of which £6.5 million came from royalties on the Cochrane Library.29 Other sources of income included fundraising from trusts and foundations, consultancy services, and events. In 2016 Cochrane received a grant of $1.15m from the Bill and Melinda Gates Foundation, which was still being spent in 2017.

Cochrane’s expenditure was £8.1m, lower than budgeted because of delays in project spending. This included:

  • Costs of publishing (direct IT and editorial support): £1.9m

  • People related costs of central executive team: £2.4m

  • Future technology project costs: £530 000 (Gates money was spent here)

  • Training and learning for Cochrane collaborators: £400 000

  • New product and service development: £332 000

  • Methods development: £259 000

  • Translation of Cochrane evidence: £162 000

  • Fundraising: £170 000

  • Premises costs (UK, Denmark, and Germany): £332 000

  • Governance costs £237 000

  • Support for the Global Evidence Summit: £564 000

Footnotes

  • Competing interests: We have read and understood BMJ policy on declaration of interests and have no relevant interests to declare.

  • Provenance and peer review: Commissioned; externally peer reviewed.

References

BMJ.com/content/364/bmj.k5302

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.


Related Blogs

  • Dr. David Healy

    Dr. David Healy

    Dr. Healy is a professor of psychiatry at Cardiff University in Wales and an author on the history of pharmaceuticals and government regulation.
    READ BLOG
  • Mad In America: Robert Whitaker

    Mad In America: Robert Whitaker

    Journalist and author Bob Whitaker distills the latest in pharmaceutical and mental health research.
    READ BLOG
  • Selling Sickness

    Selling Sickness

    Creating a new partnership movement to challenge the selling of sickness.
    READ BLOG
  • Kathy Brous

    Kathy Brous

    A serial of Kathy's recovery journey as an adult with attachment disorder.
    READ BLOG
  • Nev Jones

    Nev Jones

    Exploring the intersections of psychiatry, philosophy, neuroscience, cultural theory, critical community psychology and the mad/user/survivor movement.
    READ BLOG
  • 1boringoldman

    1boringoldman

    Retired psychiatrist and raconteur offers insightful analysis of the day's events from the woods of Georgia.
    READ BLOG