EU open science expert gathered in Helsinki on 21st of October to discuss challenges with coordinating and advancing open science in Europe and beyond.
Evaluation, metrics, and rewards were present in every presentation and conversation. Participants saw these as the bottleneck and opportunity to transform open science into the new normal of science, or science 2.0 as Karel Luyben defined it.
Why is the current assessment culture a problem?
The experts paused to discuss why are we currently not using assessment to advance open science. Here is what was identified.
Activities related to open science (open access publishing, sharing data, citizen science, engaging with the society) are still often not seen as the ‘real job’ in research, but something additional added to the job burden. The notion of excellence has been largely constructed without consideration for Open Science practices. Sometimes even the notion of academic autonomy can be used to avoid engaging in a change towards a broader evaluation system.
To support this limited evaluation tradition, the current research metrics offering is based on narrow statistics and indicators. The traditional metrics are based dominantly on publication output and journal quality indicators like impact factor or H-index. These indicators are often political and difficult to change as they are widely used for national and competitive funding. Therefore the dominance of publication output prevails.
Fierce competition in the research world is further challenging changes in assessment. How to be brave to change if your existence as a top institution or researcher is at stake? Changes would need to be synchronised and global, while still have grassroot support and activity. The current climate was seen to be anticipatory - everyone is waiting for action, preferably by someone else.
From Vision to Action
The open science experts paused to consider what would be needed for a change in assessment what would help research to transform beyond publication metrics. The discussion identified four distinct activities.
The first activity is to create a shared vision and direction for changing current assessment culture. This would have to adopt a more holistic career evaluation viewing all aspects of research as part of a successful research career and research institution. It was suggested that we may even have to consider our degree structures as current publication based PhDs are setting a narrow standard on how to be a researcher.
Second activity is setting clear goals to achieve the vision. They need to capture the full spectrum of open science activities from open access to sharing data and engaging the community. The research community must then be held accountable in reaching the goals. The European Commission was seen to have an important role in supporting the community in vision creation and commitment to it.
This commitment includes analysis of evaluation processes at all levels for example by:
- research community members sitting in on evaluation panels as observers
- creating more transparent processes with feedback and explanation
- committing to self-audit against set evaluation criteria
- training evaluators to move beyond journal metrics
Third activity is to support leadership and role models for change. This requires bravery as the senior research ranks are people whose academic careers were made with the old tools and now new ones are required. The early career academics are often targeted with educational activities but they are not in a position to change the culture from bottom up alone.
Fourth activity is to create pilots and share examples of richer and more holistic evaluation practices. We need positive examples of measures supporting open science. Examples where multilingual research outputs are considered a merit and data sharing is seen as an essential part of responsible research.
The workshop was attended by members from Belgium, EU, Finland, France, Hungary, Ireland, Italy, Netherlands, Norway, Romania, Spain, Switzerland, UK. The workshop was led by Rebecca Lawrence (F1000) together with Ignasi Labastida (SPARC Europe), Laura Himanen (Tampere University) and Henriikka Mustajoki (Federation of Finnish Learned Societies).
Published 4.11.2019
You might also be interested in
