“In the case of the research sector here and abroad, we need to acknowledge that as good as the research system is, there is a problem. There are a significant number of papers that are poor quality, and should never have made it through to publication. In considering why this might be the case, I have found myself reflecting on the role of incentives in the research system.”

Dr Finkel gave the keynote address at the Peter Doherty Institute for Infection and Immunity Fifth Anniversary Celebrations in Melbourne on Thursday 12 September 2019.

The full speech is available below, or can be downloaded as a pdf.

***************************

A Relentless Commitment to Quality

It’s always a pleasure to reflect on the outstanding work of Australia’s research sector and to celebrate its achievements.

And what an achievement the Peter Doherty Institute is – five years on.

Sharon, as inaugural director of this Institute, I congratulate you on your stellar leadership. Having dedicated more than 25 years to tackling HIV and infectious diseases, you’re clearly not one to shy away from a challenge.

You’ve been described as a key player in the search for an HIV cure, both in the basic biology of HIV latency and in taking this knowledge from the bench to the bedside. An outstanding researcher, clinician, teacher and administrator.

A good friend. And now, director of a world-class research institute.

But, with no disrespect, Sharon, you are like me. You get the credit for what is clearly a team effort.

Like all organisations, as Aristotle put it so elegantly, the whole is greater than the sum of its parts.

And so it is with the Doherty Institute.

Whether it is elucidating the role of tissue resident T cells in the formation of immunological memory, the development of novel vaccines for influenza, or combatting antimicrobial resistance – to name just a few – the work being undertaken here at the Doherty Institute is outstanding.

But we all know, and as I learned from one of my heroes, Sir Isaac Newton: as scientists, we stand on the shoulders of giants.

And Professor Peter Doherty, what a giant.

Awarded the Nobel Prize in Physiology or Medicine in 1996 together with Rolf Zinkernagel for the fundamental discovery of how the immune system recognises an 'altered self’.

Revolutionary at the time. Now in every textbook.

This discovery has since provided an important basis for cutting edge therapies, such as promising new drugs that harness killer T cells for cancer therapy.

But Peter, you didn’t rest on your laurels after being awarded science’s top honour.

You certainly live by the Doherty Institute’s vision:

'Discover today, deliver tomorrow, inspire always’.

I recall seeing you maybe ten years ago now, at a baggage carousel in Melbourne airport.

We were both returning from international trips, and as we waited for our bags to come through, you rattled off your list of guest lectureships and talks that you had just delivered.

Peter, I was exhausted just listening to you. What a devoted person you are, giving back to the international community and showing such outstanding leadership.

The leadership of Peter, and Sharon, and many others around the country, all contribute to Australian research being recognised as being of a very high standard, nationally and internationally.

In the main, we are producing high quality research that is rigorous and reproducible, and makes a significant contribution towards scientific progress.

But just because something is going well, that doesn’t mean we can’t do it better.

My credo in life is that there is always a better way.

In the case of the research sector here and abroad, we need to acknowledge that as good as the research system is, there is a problem. There are a significant number of papers that are poor quality, and should never have made it through to publication.

In considering why this might be the case, I have found myself reflecting on the role of incentives in the research system.

Because incentives matter, as we have seen through the findings of Kenneth Hayne and the Royal Commission he led into the banking sector.

The Royal Commission has been an eye opener in the way it has shone a light on how the sector incentivises its employees.

You may wonder why I am referencing a Royal Commission into the banking sector in a speech to a room of medical researchers.

My reason is that my life experience has taught me that fundamental lessons are often transdisciplinary. In this case, I consider that the lessons arising from the Royal Commission are inherently relevant to the research sector.

Because there are some incentives within the research community that, in my view, need to be looked at.

It’s not that the system is broken.

It’s more that we may be inadvertently encouraging poor behaviour.

And to ensure research remains high-quality and trustworthy, we need to get the incentives right.

We can learn the lessons from the Royal Commission.

What we learned is that over the last decade or two, the banking sector moved from salary based remuneration to bonus based remuneration. But those bonuses have been mapping to the wrong values: to sales and profit instead of compliance with the law and net benefit to customers.

For example, mortgage brokers had, and in some cases still have, incentives to push borrowers into the largest loan possible, beyond their realistic capacity to repay, because the broker’s commissions increased with the size of the loan.

Staff were paid commissions when signing up vulnerable people, even dead people, who clearly had no ability to repay their loans. I’ll leave it up to others to determine if these were criminal acts, but they definitely fall below the standard that the community expects of our banks.

To quote Commissioner Hayne:

“Rewards have been paid regardless of whether the person rewarded should have done what they did.”

The question must be, should we do this? not, could we do this?

And it’s these kind of grey area behaviours that I want to protect against in the research sector.

As researchers, we are driven by a thirst for knowledge, to understand the what if or the how ….

We are seeking to make a difference through our work.

But we can’t ignore that there are many incentives pushing researchers to cut corners and lower their standards.

The competition for funding is fierce and is increasing every day.

The temptation to judge a researcher’s performance through simple metrics is strong. You know the ones I’m talking about.

These metrics are incentives and they are incentives that drive behaviour, not all of it good.

We all know of instances of poor research practice.

Selective publication of results to support a hypothesis.

HARKing: hypothesising after the results are known.

Manipulating data and research methods to achieve statistical significance, colloquially known as p-hacking, data dredging, or my personal favourite, torturing the data until it screams.

And apologies to those out there who are competing, but no single field has the monopoly on poor research practices.

Take economics, a field relevant to the Royal Commission. In 2015 the USFederal Reserve analysed 67 economics papers published in reputable academic journals.

Only a third of the findings could be independently replicated.

Your minds might immediately jump to fraud, but thankfully that is still rare. My focus is on reducing the level of flawed research that even though it is poor quality, is still being rewarded.

If we can focus on improving the quality of research in general, we can achieve broad and long-lasting benefits.

And I think the best way to do this is to look at the incentives.

We all know that publication is a principal criterion for scientific career advancement. And I don’t want to change that.

However, the institutionalisation of performance metrics has created incentives for researchers to publish as many papers as possible.

When you think about it, it’s quite similar to the mortgage broker’s incentive model.

Getting research published has, in some cases, become more important than getting the science right.

I am of the firm belief that there shouldn’t be an incentive for a researcher to salami slice their results into three or four separate publications, rather than one meaningful publication.

If the purpose of publication is to share your results in a way that can be built on by other researchers, this kind of practice completely defeats that purpose.

One model that places the focus on quality over quantity is the Rule of Five.

With the Rule of Five, a researcher’s performance for grant funding or promotion is judged on their best five publications over a five year period, accompanied by a description of its impact and the researcher’s individual contribution.

The exact number of publications or years that institutions opt to consider isn’t important. On both counts, it could be anything up to ten. As long as it is small.

And of course, there are disciplinary differences that may need to be taken into account.

But what matters is the emphasis on the significance of the research.

Australia’s granting agencies have already taken concrete steps to broaden their assessment of research performance. But their use of the Rule of Five is not yet comprehensive.

Instituting the Rule of Five as a requirement across the grant funding system would have a transformative effect.

Starting as a splash in the granting agencies, it will ripple through to appointment and promotion processes.

We need to put in place visible changes if we are to convince researchers that their performance will be assessed on research quality and a deeper view of its impact.

It requires a cultural change at all levels, and that leads us to the next issue: training.

Unlike other professions, there are no national competencies and no national recognition of research integrity education and training.

And while you may think scientists don’t need this kind of training, be aware that the bad behaviour pointed out by the Royal Commission was being perpetrated by financial advisors without any accredited training.

Now, while many institutions in Australia do provide training programs for their PhD students, these programs vary in quality, content and reach.

And, to the best of my knowledge, no Australian institutions have a training requirement for their existing research workforce.

I strongly believe that the overall quality of research in Australia would be strengthened by research integrity training for all researchers.

Training puts a spotlight on expectations for the whole community and encourages consistent behaviour.

It also removes that old chestnut of plausible deniability.

“Honest”, officer, “I didn’t know it was wrong!”

The training must be accredited, and must be high-quality. It should not be a tick the box exercise.

I am not saying it should be compulsory across the individual institutions. But I am saying that it is important. And if we circle back to incentives, the best way to encourage researchers to undertake the training is to tie it to grant funding.

What I am proposing is that you should not be able to be a named investigator on a grant application unless you can prove you have completed accredited research integrity training.

“It’s too hard” I keep hearing. Well, to those naysayers who say it will never happen, let me tell you that it already has. The Irish Health Research Board has recently implemented such a scheme. Every named researcher must undertake accredited research integrity training within six months of being awarded a grant.

Making proof of training a requirement for obtaining a grant will again have a ripple through effect and embed the expectation of what is or is not acceptable.

Finally, I am concerned that the incentives in the research system are not just driving bad behaviour for researchers, but are also creating a market for criminals to enter scholarly publishing.

What is motivating the crooks is the pay per page system that has come with the introduction of open access publishing.

Now, open access publishing has many benefits, as we all know, and I support the move to open access publishing. But I remain concerned that it has opened the door for predatory, evil, crooked journals.

It is just too easy to set up a journal and a website with a 'high-falutin’ title, and appropriate the biographies of leading researchers for the editorial board – without their knowledge or permission.

Before you know it, huge numbers of papers are being published without any rigour.

And there are researchers who are knowingly paying to publish in journals that have no peer review, even though they claim to. Journals that have no ethics. Not even an editorial team to consider the submitted paper.

That is worth repeating. There are researchers paying to publish in journals that have no peer review, no ethics, no editorial team. These researchers might just be naïve, but we have to acknowledge that the current incentives reward this behaviour.

While this is not a major problem in Australia, emerging research nations are really struggling with this. In India, for example, a regulation introduced by their University Grants Commission in 2013 requires postgraduate students to publish two research papers to receive a PhD.

This regulation, although well intended, has led to corruption. Thousands of students desperate for publications have become easy prey for the predatory journals.

In my conversations with senior research leaders around the world, they are looking for ways to improve performance metrics in a way that does not drive their researchers to these predatory journals.

My proposal is a rigorous quality assurance system. It is designed to inform stakeholders that a particular journal’s processes for assessing a paper meets agreed publishing standards.

I like to call it Publication Process Quality Assurance, or PPQA.

Compliance with PPQA would indicate to researchers, research institutions, libraries and granting agencies that the journal follows internationally accepted guidelines for the publication process.

Now, I want to be absolutely clear: PPQA is not akin to a journal impact factor.

It is also not a statement on the quality of the published research itself.

It’s about ensuring that the journal adheres to agreed publishing standards.

The standards could be modelled on existing guidelines developed by organisations such as the Committee on Publication Ethics.

Journal compliance with the standards would be externally assessed, and a central list of compliant journals would be made available.

“What’s that going to cost?” I can hear you all ask.

Personally, I think it would be worth bearing the cost … but it’s also possible to use existing processes that may dramatically limit the extra burden.

For example, we could use a commercial indexing service, such as the Web of Science Core Collection. In my conversations with the operator, Clarivate Analytics, I have been impressed with the rigour of their journal selection process – focussed on agreed standards, not citation impact.

Of course, there are potential issues with using a commercial provider, but they could be overcome. We already use Clarivate Analytics for the citation information for the ARC’s Excellence in Research for Australia assessment.

Extensive as it is, the Web of Science Core Collection does not yet include all of the journals that meet the agreed standards, but that can be built on over time.

I’ll be continuing to have conversations with granting agencies in Australia and around the world on how to destroy the business model of predatory publishers.

Why am I focused on granting agencies? Because granting agencies are best placed to provide the incentive for researchers to only publish in PPQA compliant journals by enforcing it through their grant application process.

You might have picked up by now a common thread; that in each of my three recommendations, I am looking to take the responsibility back to the granting agencies. It’s a concept referred to by others as “follow the money”.

If the granting agencies put in place these measures, they will ripple through into the research institutions.

Adopting all three proposals together, the Rule of Five, widespread integrity training and the PPQA, would mitigate the ongoing risks of poor quality research.

As a bonus, these measures would result in fewer total publications because researchers will be focusing on the quality of outcomes, rather than quantity.

There would therefore be less pressure on peer reviewers because the pipeline would be less clogged. They could actually take more time to properly review the papers before them.

It will change the culture and restore for the 21^st century research workforce, last century’s academic rigour.

And, as is the engineering way, we can continue to look at and improve how the incentives drive the best research practice.

It might make it tough for best practice groups such as yourselves, working in the Doherty Institute, if the rest of the world comes up to scratch, competing with you to make the next Nobel-winning discovery.

But I am sure you will agree it would be worth it.

I am delighted to be with you today – such an extraordinary institution with such exemplary researchers. I imagine that your discovery capacity is unbounded, so, if you come up with a no-risk recipe for the elixir of youth, with reproducible results, I look forward to being here for your 50th anniversary.

Please, invite me back to celebrate!

May the Force be with you.

Thank you!