“The basic structure of peer review is the best we’ve ever invented. But it is showing signs of strain.”

Dr Finkel gave the opening keynote address at the 6th World Conference on Research Integrity in Hong Kong on Sunday 2 June 2019.

The full speech is available below, or can be downloaded as a PDF.

*************************

Looking around the room today, I’m reminded that research truly is a human pursuit: it thrives on face-to-face connections.

It’s easy to forget that, when you’re a student, and it’s late at night, and you’re the last person left in the lab – again.

So, every so often, it’s worth pausing to remember just how many people are out there, working hard, gathering data – just like you.

Worldwide, there are more than eight million researchers.

Every year, we produce well over a quarter of a million new PhDs.

China alone has added more than a million people to its research workforce since 2011.

Not all of these researchers will work in academia – but those who do are highly productive.

They publish in the order of four million academic journal articles every year, spread across more than 40,000 journals.

And all of that traffic is routed through a single critical bridge. The publication process.

Picture that bridge.

We know it so well. It’s stood there for centuries. And in that time, it’s developed from a simple footbridge – with a handful of pedestrians – into a triple-decker multi-lane high-speed monster freeway.

It’s still fundamentally sound. The basic structure of peer review is the best we’ve ever invented. Every day, I see trucks on that bridge carrying outcomes that even Einstein thought would never arrive.

The detection of gravitational waves.

Devices that can translate brain signals into speech.

Atomic clocks that can mark a second with precision in the parts per quintillion.

This great bridge that holds up civilisation has served us well. It is not about to collapse.

But it is showing signs of strain.

Start with the fact that there are now more than 20,000 retracted papers in the Retraction Watch database.

Does that catch 50% of the times that the quality assurance process failed? 10% of the times? We can’t say. But we know enough to be concerned.

There was the 2015 analysis, conducted by the US Federal Reserve, of 67 economics papers published in reputable academic journals.

Only a third of the findings could be independently replicated.

There was the 2018 analysis of 100 psychology papers, also published in reputable journals.

Only 2 in 5 could be independently replicated, at the level of significant results.

There was the 2015 survey of about 400 statisticians on their interactions with collaborating researchers.

Almost half had been asked to report results before the data had been cleaned and validated.

A quarter had been asked to remove some of the data.

More than ten had been explicitly directed to falsify the statistical significance; some of them, on more than ten occasions.

Whatever your field, you’ll have your own examples.

Put them together, and we have more than enough evidence to conclude that we cannot write off these lapses as the occasional bit of bad driving.

The evidence says: we haven’t built the optimal bridge.

The people who pay for the petrol and rely on the safe delivery of the cargo – the taxpayers and governments – are no longer prepared to take us on trust.

They want actions to shore up the bridge.

So we are gathered at this conference to be the civil engineers.

***

Now I’m an engineer by training.

I am also Australia’s Chief Scientist.

And in that capacity I wanted to understand what we could do to strengthen the bridge.

I acknowledge, it’s an enormous topic – I can’t even list all of its dimensions, let alone disentangle them.

My focus was practical: what we could do to make a material difference, with a focus on the overarching framework from which other important measures could flow.

So in October last year, I organised a workshop on research quality and the publication process in my office in Canberra.

We invited the editor of the Springer-Nature group, Sir Phillip Campbell, along with the heads of our research funding agencies, leaders of our research institutions, and experts in the field of publications.

The full list of names is available on my office website, along with a subsequent article that I published in Nature.

I want to share with you today some of the practical measures we considered, and how we are pursuing some of them at the national level in Australia.

Then, I want to turn to the global infrastructure we would need to consider if we’re going to standardise good practices across the world.

And in the spirit of fair attribution, let me note that my reflections today are informed by our workshop, but the recommendations are my own.

***

So, let me start as our workshop did, and as all engineers are trained to do: defining the problem.

The publication bridge is fundamentally sound and it’s critical to keep it open.

But quality assurance is weakening.

We’ve got trucks arriving with rotten cargo that has to be retracted.

Sometimes – but not that often – we’ve got trucks arriving with contraband cargo and forged transit documents.

We’ve got trucks arriving with useless cargo that nobody wants to purchase.

And we’ve got drivers speeding madly to make as many trips as possible.

The traffic backs up at the toll gates, because the good peer reviewers are overloaded.

We’ve got smugglers – predatory publishers – dodging the toll gates entirely.

And we’re got increasingly frustrated researchers looking for alternatives, jumping off the bridge and into the wild waters of open science below.

We talk constantly about these problems – and still, they remain.

All of the participants in our workshop agreed: there are many thought-leadership organisations; there are excellent and widely acknowledged guidelines; but that’s not enough when the incentives in the system run the opposite way.

We know from the mining sector: if the safety incentives are set correctly, the safety record dramatically improves.

We know the opposite from the finance and banking sectors: if the incentives are set incorrectly, appalling practices prevail.

It’s exactly the same in research. It doesn’t matter how many times we say we want quality-over-quantity in theory, if we keep rewarding quantity in practice.

We’ve all got to take responsibility for bringing the theory and the practice into line.

***

Let’s look first at the people directing the trucks and the drivers – our research institutions.

For centuries, we’ve relied on an apprenticeship model of training – just like the way we used to teach our teenagers to drive. Put them in the car with an experienced driver.

That made sense in a world where senior researchers were publishing less frequently and had the time to give to perhaps a handful of students.

It’s reckless in a world where there’s much less time to give, and many more students wanting to share it.

Even in my day, forty years ago, the pressures were showing.

My PhD supervisor, Steve Redman, sat firmly in the school of quality. He averaged about two papers a year. He expected a lot of his students, but he was generous with his time.

I realise now just how hard it must have been for Steve to stick to his principles.

As if in confirmation, two months ago, I received an email from one of Steve’s contemporaries, recalling that every time he sat on a panel assessing Steve, the beauty of Steve’s papers was lost in the clamour about his production rate.

That was a senior researcher, an acknowledged superstar, conducting research of the highest quality, with undeniable impact, decades in the past… under constant pressure to accelerate.

Let’s just think about the intensity of that pressure on a PhD student and a research supervisor today.

How much is that student really going to learn from that supervisor by osmosis?

A far more reliable mechanism is explicit instruction: structured, formal teaching in research integrity and professional expectations.

Research institutions should make that instruction mandatory, not just in student training programs, but for every one of their existing researchers.

And if we’re going to put the time into training, then we should have agreed minimum standards for the modules.

As a starting point, accredited research integrity courses should probably cover the material from the Singapore Statement and the Montreal Statement issued after the 2nd and 3rd World Conferences on Research Integrity.

At the same time, we shouldn’t expect mentors to be good mentors by instinct. Their institutions should train them in good mentorship, and make that training a condition for any post where they’re supervising staff.

And instead of judging a senior researcher’s performance by the number of students on their books, we should ask for impact statements on, say, two of their former PhDs, at least one of them female: how they were mentored, and what they went on to achieve.

That’s not the only change I’d like see in an academic CV.

Think for a moment about how a so-called competitive CV looks today – pages and pages and pages of article and authorship credits. No reviewer has the time to evaluate those lists to gauge the quality, so quantity prevails.

Imagine how it would look in a system that made quality the focus.

We would opt for a model such as the Rule of Five.

Candidates present their best five papers over the past five years, accompanied by a description of the research, its impact and their individual contribution.

The exact number of years or papers that institutions opt to consider isn’t important. On both counts, it could be anything up to ten.

What matters is the emphasis on the significance of the research – and the message it clearly sends.

***

How do you shift the behaviour in research institutions – hundreds if not thousands of institutions?

There is a principle known as “follow the money”.

And in this case, the money trail from research institutions leads straight back to the agencies that supply the grants.

If we want to motivate change, at scale, then those national granting agencies are key.

My recommendation is that for investigators to be funded by a national granting agency they should be required to prove that they have undertaken an accredited course in research integrity. Without that proof, the grant would fail to get through the first stage of administrative review.

In addition, national granting agencies should evaluate investigators’ publication records from a Rule of Five perspective, with total publications and H-indices pushed to the background as secondary considerations.

***

Now I acknowledge that I am building on existing ideas. My ambition is for granting agencies to take the leadership role in supporting best quality research beyond the grant itself.

One of those agencies is Australia’s National Health and Medical Research Council, or NHMRC, led by Professor Anne Kelso.

If my principle is “follow the money”, Professor Kelso’s is “use the power of the funder”. And do so thoughtfully, and deliberately, to keep the focus on quality, where it belongs.

Already, we’ve seen some significant steps forward.

The Rule of Five is now in place for some NHMRC grant schemes – and in future, it may well be extended.

Further, for two of the major schemes, the impact of the investigator’s past research is now an explicit part of their track record assessment.

Impact – be it on knowledge, on health, on the economy, or on the community – is judged on case studies. Not just numbers: explanations.

This is the beginning of the NHMRC’s quality agenda – not the end.

Professor Kelso is looking comprehensively at the NHMRC’s role in supporting high quality research through all of its processes: policies, guidelines, peer review, the lot.

They will be working with research institutions to recognise and spread good practice.

The expectation is clear: research institutions have to be more explicit in conveying the message to their research staff that quality counts.

To verify the commitment, the NHMRC is calling for regular self-assessment by accountable leaders in research institutions, of their institutional policies and reforms.

Another notable example is the Responsible Conduct of Research requirement in the United States.

Major granting agencies, including the National Science Foundation and the National Institutes of Health, require every institution that applies for grants to provide appropriate research conduct training.

However, at present, the requirement applies only to postgraduate students and postdoctoral fellows involved in a project.

My recommendation – to make grant funding conditional on every investigator providing proof they have completed a course in research integrity – goes further, in recognition of the fact that we aren’t just looking to support a new generation.

We’re still playing catch-up with the generation before – as well as absorbing researchers from countries where the training isn’t required.

There is some progress. For example, I learned this afternoon from the head of the Irish Health Research Board that they have recently implemented a strict requirement that every investigator be able to prove that they have completed a research integrity training course. And the Wellcome Trust is also implementing broad quality and integrity policies.

***

So, as we can see, there are pockets of progress.

The bigger challenge remains: how can we scale up and standardise good practice, right across the globe?

Today, we simply lack the systems for a collective approach.

In particular, we lack real oversight of journals.

We have seen some jurisdictions take action against the worst of the predatory publishers through fraud law.

But the reach of those laws is limited.

And the standard we want for journals isn’t – “not criminal”.

It’s – “best practice”.

Journals are not simply players in a knowledge market. They are knowledge custodians, with all the prestige and privilege that affords.

We have to be united in our expectations: if journals are to retain their position as knowledge custodians then they have a responsibility to be more than scrupulous.

They also have to be accountable and transparent.

Where exactly a particular journal fits on the continuum between “criminal fraud” and “agreed best practice” is rarely clear.

Of course, there are outliers at both ends – but there are tens of thousands of journals in the middle.

That’s not good for the journals that do commit to best practice, because we’ve got very few ways to verify their claims.

It is extremely good for the journals that don’t commit to best practice, because we’ve got very few ways to save junior researchers and journalists and even policy makers from being duped.

To date, it’s fair to say that even reputable journals have not welcomed greater scrutiny.

But scrutiny doesn’t have to come as an imposition.

Let me give you an analogy from my time in industry.

I was the founder and CEO of a company called Axon Instruments.

We made research instruments, but we also made medical devices – including a product that inserted an electrode three inches deep into human brains during surgery to treat the symptoms of Parkinson’s disease.

No-one would purchase that product unless they knew it was safe, so we undertook ISO 9000 certification. This international quality assurance program is like a superset of Good Manufacturing Practices – GMP – required to register products with the FDA in the United States.

The ISO 9000 standards are extremely demanding.

They apply to the company, the production process and the product itself.

Compliance is verified by a combination of internal and external audits.

To my surprise, I found that as a manufacturer, those standards became my best friend – because they told my customers that we were selling a trustworthy product.

They also kept the market clear of low-quality producers who would first, steal my customers, and second, destroy the whole industry’s reputation.

So imagine if we had something equivalent for the publication process. I’m calling it PPQA – Publication Process Quality Assurance.

Compliance with PPQA would indicate to researchers, research institutions and granting agencies that the journal followed internationally accepted guidelines.

And granting agencies would only consider research that has been published in a PPQA compliant journal when judging applications.

Now I want to be absolutely clear: PPQA is not akin to an impact factor.

What I’m talking about is quality assurance, to ensure that journals implement an agreed minimum standard for their publication processes.

We could start with the guidelines developed by organisations like COPE – the Committee on Publication Ethics.

Higher levels of PPQA could pick up on the Transparency and Open Promotion guidelines, known as the TOP guidelines, compiled by the Centre for Open Science, or the Reproducibility and Replicability in Science recommendations published this year by the National Academies of Science, Engineering and Medicine.

These guidelines form a tremendous body of work, by deeply knowledgeable people, who have reflected on these issues for many years. Let’s use it.

Who would audit and accredit that each journal title meets the standards?

It could be an existing body like COPE, but they would need funding.

It could be a new entity.

Or, as happens with the ISO standards, it could be credentialed private companies.

However the audit for accreditation is done, we would require a central global body to hold the list of successful journals, open for checking by granting agencies, institutions, journalists, venture capital funders, everyone.

There would obviously be costs – so the inevitable question is “who should pay?”

Turn the question around – “who has a reason to be invested?”

Journals, for one. Granting agencies, for another. Large philanthropic bodies with an interest in high-quality research, for a third.

Some provision would have to be made to ensure that small society journals are not overburdened by audit costs.

All of these questions would require careful deliberation on the model, through discussions involving libraries, publishers, grant agencies and research institutions.

And the agreed model that emerged would have to be tested through a pilot to see what works and what might go wrong.

It can’t happen without global forums: global bodies with the networks and credibility to speak as the collective voice of science.

The International Science Council would be an obvious candidate – as would the Global Research Council.

But we should not continue and extend the good discussions of the past without a matching commitment to action.

Since granting agencies provide the keystone research funding, they have the greatest capacity to push for a shift in behaviour. They should set a timetable for the deliberations.

Finally, my recommendation to the granting agencies is that they should turn the results of the deliberations into actions by setting the date after which new papers can only be included in a grant application if they were published in a journal that is shown to comply with PPQA.

***

Now, as I acknowledged at the beginning, there are many issues wrapped up in research integrity– and we’ll have the opportunity to dive into them this week.

But my focus today is on the practical, in the firm conviction that we have a system that is fundamentally sound – but can undoubtedly be improved.

To recap, based on the principle of “follow the money”, these are my recommendations:

One, granting agencies should make proof of research integrity training a requirement for applying for a grant, applicable to all investigators listed on the application.

Two, granting agencies should require CV’s submitted for grant review to follow the Rule of Five, and

Three, granting agencies should only consider new publications from journals that have proven their compliance with PPQA – Publication Process Quality Assurance.

Ambitious, yes – but considering the stakes, I’d say a bit of ambition can be excused.

***

And I hope this is the spirit that we’ve all brought to this great global gathering today.

Think of that bridge this week.

It has served us well but it is creaking under the increased load and evolving driver behaviours.

If it were a physical bridge, there’d be no question. We’d fix it.

The research bridge is every bit as critical – because we make life and death decisions on the basis of the data that is trucked across.

It’s soldered onto the neural circuits of every engineer: there’s always a better way.

We can find that better way to do research.

THANK YOU