Good science: researchers cherish mutual differences but share the same concerns
Jeannette and Sonja went in search of how researchers themselves define good science and what they think is needed to conduct good scientific research. For three years, they investigated the everyday practices of five different disciplines: mathematics, chemistry, philosophy, anthropology and two types of medical sciences (epidemiology and projects for health care improvement). Their project “Achieving good science. A cross-disciplinary study” is part of the programme Fostering Responsible Research Practices that ZonMw realised together with NWO.
What are the most important differences between scientific disciplines?
Jeannette: ‘There are many differences in emphasis. More theoretical disciplines, such as theoretical physics, benefit far more from an exploratory research design. In contrast, in very large technical projects, such as a particle accelerator, everybody needs to know exactly what their task is. That means that you need to very accurately record in protocols what the next step must be. In the case of more conceptual research, you are searching for the phenomenon you want to analyse. And the concepts that could be linked to this. That difference clearly emerged from our research. In disciplines such as philosophy and anthropology, or the social sciences and humanities in general, the research route is less clearly defined. These are more interpretive disciplines.’
Uniform policy concerning open science and reusable data bring differences to the fore.
Sonja: ‘We saw that a range of disciplines, each in their own way, struggled with the science policy. Data management, for example, is seen as a way of promoting scientific integrity and facilitating the reuse of data. However, if you consider that from the perspective of various disciplines, you see that these concern completely different worlds. Anthropologists, in particular, object to the idea that they, as researchers, own the data. That is because the material acquired in interviews and observations is very difficult to anonymise. Then you cannot simply give everybody access to that empirical data. If you make uniform policy for all scientific disciplines, then the consequences for the social sciences, for example, are different than those for the medical sciences, which use statistics more. This is what we encountered in daily practice, and it is high time that something is done about it. The impact of centrally determined policy becomes particularly clear on the work floor and during projects. Then those structures and various research practices become visible.’
Jeannette Pols: ‘A good example: ethical committees examine more qualitative and ethnographic work in the social sciences. Medical ethics protocols, however, require a consent form with data about your study subjects. But if you are doing research in Rwanda during a period of social and political unrest, then you do not want to commit those data to paper due to the safety of the people involved. Then you need to ask the question: how can we anticipate such situations and what are the possible solutions? Aspects related to privacy can also differ. For example, it is almost impossible to anonymise detailed interviews and observations. Even if pseudonyms are used, the raw material can always be traced back to a person. Accordingly, it is not always possible to comply with the requirements of open science.’
Sonja: ‘As a researcher, you struggle with that. It is, of course, important that we deal with information and data in a responsible manner. But it requires so much extra work to discover a way of satisfying the uniform requirements researchers must fulfil. However, we also see that there exist other ways to satisfy the requirements of good science. Instead of uniform requirements, the ethical committee can also work with the researcher to determine what is worthwhile in a specific study and to consider the content of the research. That would also benefit the researchers, since they would receive feedback as part of this approach.’
Jeannette: ‘What we definitely do not want is an ethical committee that becomes a watchdog that blocks your progress, as is currently the case in the United Kingdom where it takes a year to obtain permission to do research, including social sciences research. Things are not that bad in the Netherlands. However, I am concerned about data and big data, which foster the idea that having as much data as possible will enable us to know more as well. Then we can also too easily forget the context in which the data were collected and what questions the data were intended to answer. You cannot simply blend all sorts of different data. We have witnessed that this can go wrong, in policies as well. A case in point is the Dutch childcare benefits scandal.’
Sonja: ‘The challenge is to retain the nuances in the information and empirical data that you collect and use. We should continue to find appropriate ways of doing that again and again. There are research practices where collecting a lot of data is worthwhile, can be justified, and a big data approach can be chosen. However, we must also carefully consider the detrimental effects of doing this.’
Jeannette: ‘The researchers interviewed by us told us this as well. One of the chemists we interviewed found the requirement to publish data a bit farcical: “Of course, I’ll publish the data on the web just like that; it is not as if anybody knows how to use it”. She felt it was a good way of preventing fraud but not a serious way to make reusable material available. Reusable data is, however, one of the goals of open science.’
Data and concerns about data therefore mean different things in different disciplines. Those nuances are lost when you produce a uniform policy.
Sonja: ‘In talking about those differences, it is good to add that it is not easy to define what those differences are in a comparable way. A clear finding from our research – and this is another result of those differences – is that data and concerns about data mean different things in different disciplines. Those nuances are lost when you produce a uniform policy. Such a policy effectively disrupts people’s work processes. We are still trying to find ways to discuss those differences. Especially how we can continue to cherish those differences in a constructive way.’
Researchers cherish their own traditions, but also share similar concerns
Sonja: ‘We were struck by just how much passion science harbours. And we have also noted the concerns that researchers have. For example, the fact that they must do things they would rather not do, such as continually having to compete with each other. The researchers themselves prefer to talk in terms of collegiality, about the connection between research and education, and bearing responsibility for research practice in the broadest sense of the word. That requires a considerable effort. We also heard many stories about burnout and about young researchers who no longer see a future in academia and decide to leave. We were initially interested in the differences between researchers and scientific domains. What surprised us is that researchers cherish their own traditions but that they also share similar concerns. That was a big surprise for us.
How do you view integrity issues within research?
Jeannette: ‘When we started our research, integrity was still very much framed in terms of individual researchers. On the one hand, that concerns the competitive researchers who strive for excellence. And on the other hand, there is the fraudulent researcher. However, researchers did not recognise themselves in that individualistic perspective. They are not particularly concerned about fraud and, of course, a major scandal, such as that concerning Diederik Stapel, does not occur that often. However, they did see a range of ongoing processes which sometimes result in having to take decisions that they would rather not take. We sometimes noticed, for example, that the authorship of publications was meddled with because it served a department’s financial or professional-strategic interest to mention the name of the head of department in an article or PhD thesis. In such cases, you make a trade-off between fair science and caring for your research group.’
Sonja: ‘We were really surprised to discover that concerns about bad science were clearly part of the public story, but that the constituents of good science are not voiced’
Good science is not an individual responsibility
Jeannette: ‘In our opinion, administrators, researchers and policymakers should focus on the terms and conditions under which research is conducted, and less on the integrity of individual researchers.’
Sonja: ‘That means we’re talking about an “ecological” issue. It is a collective issue that affects all aspects of academia. It also means that policy needs to take into account that good science is not an individual responsibility, but is shaped collectively. A collective research practice requires a considerable effort from many people.’
In our opinion, administrators, researchers and policymakers should focus on the terms and conditions under which research is conducted, and less on the integrity of individual researchers.
Jeannette: ‘Once you consider things from this perspective, the issue immediately becomes more complex. Because journals, funding structures and the policies of universities all play a role in this. But also the call for relevance from society. There are so many factors that have an influence on research practice. And that makes it really hard to say: this is what we should be doing! There is no blueprint to strengthen good research. It needs to be tackled on all these fronts.’
There is no blueprint for good science
Sonja: ‘We need to let go of the idea that there is a blueprint [for good science]. Everybody would like to have a blueprint but our research made clear that the strength of good researchers is that they persist and continue to puzzle things out. And the fact that something is unknown, does not scare them’. I think it would be fantastic if we could examine how we can approach this issue collectively. Just to return to the start of our research for a moment; it is very important that we continue to remind ourselves hat everybody we came across strove to deliver good science. And we should not lose sight of the strength of that. We need to try to find common ground without losing respect for the differences. That requires a collective effort. Realising good science is not just a job for a small group of policy officers or employees specifically tasked with that.’
Jeannette: ‘That was also the risk of scientific integrity, which had the sole objective of combating fraud. This objective seemed to imply that one single regulation would suffice to prevent fraud’. If you think like this, you have no eye for the conditions under which researchers work.’
Sonja: ‘I think things are different now. The language about integrity and good research has changed and that is a hopeful development. Now the discussion has shifted to good research and Recognition & Rewards. That constitutes a completely different approach. However, we must continue to guard against things becoming too rigid and standardised. Otherwise, before we know it, Recognition & Rewards will become yet another universal solution that will mainly be embraced by policymakers and not by researchers in their everyday practice.’
Jeannette: ‘That also elicits the question as to what you can and cannot prescribe with policy. Surely, good research and integrity should be cultivated from within research practice as well?’
There are so many factors that have an influence on research practice. And that makes it really hard to say: this is what we should be doing! There is no blueprint to strengthen good science.
What advice would you like to give to administrators and policymakers?
Jeannette: ‘I think that now we now need to pay attention to young researchers. It has often been said, and the data supported this, that the Netherlands belongs to the world top when it comes to scientific knowledge. And we might be able to hold on to this position a little bit longer. However, we cannot remain at the top without a new generation of researchers. Good employment conditions are important in this regard. So, we need to take more care of young researchers than is currently the case. The work pressure is high, the contracts are poor, and a lot of demands are made in a competitive system. The research conditions need to be less competitive with more room for reflection. It is okay for researchers to leave academia as the result of a positive choice to do something else. However, now it all too often seems to be a sort of tombola; who can best cope with the pressure, who dares accept temporary contracts for the longest period and who can afford the luxury of working abroad for a period? And you can also suffer the misfortune that your PhD research does not really yield anything. That means the end of a career without that necessarily saying anything about the qualities of the researcher concerned. And that is a real shame. Here and there, we can now see things changing in small ways. If you want to attract good people to your team, you need to start offering better contracts and employment conditions.
Your report was published a year ago. What strikes you now, looking back?
Jeannette: ‘I think that our research and report have contributed to a shift of the fraud paradigm towards a reflection on what constitutes good science. And that we increasingly understand that it is more complex than we think. New developments in science require us to continue to think about what would be the best approach, as in case of the promise held out by the use of big data, for example.
Sonja: ‘The ways in which we conduct research is not called into question often enough. However, there are genuinely different ways. That does not mean that science, in general, no longer has any meaning because it cannot yield a blueprint and does not have ready-made answers. Science has value because it offers the room that is needed, on a daily basis, to determine step-by-step what the relevant questions are. In the everyday research practices we studied, we found that researchers seek that space. They were open to the unexpected, posed questions and did not make assumptions too quickly. They slowed down to jointly reflect on how exactly you should think about a scientific puzzle. Researchers have good methods for doing that. That is not an individual characteristic. Rather, it is something you learn together when you do research.’
Jeannette: ‘We call that learning in practice. How could we cultivate that?’
Sonja: ‘Although it seems difficult, it is necessary in the long term because that will yield sustainable research.’
Which research into research do you think is still needed?
Jeannette: ‘We were talking about this when we got down from the train. Researchers are highly specialised. What they do is very specifically geared towards their discipline, and they are very well informed about how their discipline works. Science develops highly specialised knowledge. However, we still have a very poor view of the overall structure of academia with all the disciplines and methods it contains and its sometimes conflicting epistemologies. How exactly do things work and how do the different aspects relate to each other? I think that insights into the complexity and connections can be helpful to achieve good science. You need all those aspects of research to solve different problems. It would please me if we could provide more insight into that structure and complexity of science.’
Sonja: ‘And I do not think we should take a compartmentalised approach towards the phenomenon of what constitutes good science. For this project, “Good Science, a vision from the inside out”, I felt it was important that I, as a researcher, not only examined the situation from the outside, but from the inside as well, from the perspective of specific disciplines. ‘Research into good science shouldn’t be seen as just another specialism in which we are now deemed to be the experts who will tell everyone how to go about things. Instead, we need to learn from the diversity of good research practices and the common concerns voiced by researchers. I think that this is an important message. What constitutes good science is not determined by a few experts like us.’
Jeannette Pols (1966) is Professor of Anthropology of Everyday Ethics in Healthcare at the Department of Anthropology, Faculty of Behavioural & Social Sciences at the University of Amsterdam, and at the Department of Ethics, Law & Humanities of Amsterdam UMC. The chair is part of both faculties.
Pols’ chair comes with the mission to build bridges between research in medical ethics and medical anthropology. Her research and teaching further develop the ethnographic study of ethical questions. In comparative ethnographic analyses, her research provides insight into the practical and desirable ways in which these technologies shape care and societies, and the repertoires people develop as a result of these practices. The aim is to discover and develop normative directions for complex technological societies.
Prior to this appointment, Pols was Socrates Professor of Social Theory, Humanism and Materialities at the Department of Anthropology of the University of Amsterdam and Assistant Professor Medical Ethics at the AMC.
Sonja Jerak-Zuiderent is an Assistant Professor of Social Studies of Science, Technology and Care at the Department of Ethics, Law and Humanities of Amsterdam UMC. Her research focuses on a two-part question: (1) how “good care” is achieved in healthcare and research in its broadest sense; and (2) how accountability is provided for “good care”. She investigates this with a specific focus on the everyday work of people and how things function; aspects that are easily taken for granted and marginalised. Much of her research is ethnographic in nature. Her publications cover subjects such as infrastructures for accountability in healthcare (for example, guidelines and indicators); patient safety practices; embedding of other forms of care, such as spiritual counselling, in the healthcare structure; and practices for good research within and between disciplines.
Link to project (in Dutch).
Achieving good science. A cross-disciplinary study
Project leader: Jeannette Pols together with Sonja Jerak-Zuiderent, Jonna Brenninkmeijer and Amade M’charek.
Jeannette Pols (Professor of Anthropology of Everyday Ethics in Healthcare at the University of Amsterdam) and Sonja Jerak-Zuiderent (Assistant Professor of Social Studies of Science, Technology and Care at Amsterdam UMC) received funding from the programme Fostering Responsible Research Practices (ZonMw and NWO) in 2017. The outcome of the research was the report “Good Science: A vision from the inside out”.
In the project “Achieving good science. A cross-disciplinary study”, they investigated how researchers set to work in the everyday practice of scientific research. The first goal of the research was to realise a description from within academia as to what constitutes good science and what the problems of good science are. The second goal of the study was to provide statements, based on the research findings, about the responsible drawing up of science policy that aims to promote good and fair research practices. With their analysis, they wanted to recalibrate the problem of integrity in science in order to initiate a discussion with policymakers and researchers about the promotion of good science.