Systematic reviews of animal studies – Report of an international symposium

.


Introduction
Systematic reviews and other evidence syntheses are important tools for summarising and synthesising research from multiple locations into a single, accessible source. In addition to their traditional use in human medicine, these methodologies are increasingly applied in other disciplines, including animal studies. Systematic reviews are particularly important in animal studies due to small study sample sizes, bias associated with individual study populations, and publication silos associated with animal health literature (Manlove et al., 2016). In turn, the creation of systematic reviews helps to identify gaps where further research is needed and drives the use of standardised reporting guidelines so that future studies can more easily be compared with one another.
As more systems become available that incorporate evidence syntheses for veterinary practitioners and animal caregivers, it is crucial that there is a greater understanding of the underlying methodology used. This will enable proper evaluation of these resources for publication and application to policy and practice.
While animal studies share many similarities with human studies, there are distinct factors that need to be considered when conducting and evaluating systematic reviews and meta-analyses of animal studies. The growing interest in systematic reviews supported the creation of a Symposium on Animal Systematic Reviews held on 24 May 2022. Although the symposium focused on systematic reviews, other types of evidence syntheses were discussed. This article summarises discussion at the symposium with a goal of educating veterinary practitioners, animal researchers, systematic review authors, and journal editors and reviewers on current issues in animal evidence synthesis, and provides a curated list of related resources.

Methods
The Symposium on Animal Systematic Reviews brought together a diverse panel of experts from international organisations and institutions working to advance research, information systems, and syntheses that impact policies and practices affecting the health and welfare of the environment, humans, and animals. This 3-hour virtual symposium included introductions from invited experts, a panel discussion, and an audience question and answer session. The invitation to register for the symposium was promoted through professional organisation listservs, which included veterinarians, animal researchers, and information specialists.

Objectives
Symposium panelists and participants met with the following objectives: • To bring together a diverse group of organisations working on systematic reviews of animal studies into the same virtual space for introductions and discussion. • To define commonalities among groups working on systematic reviews of animal studies for differing purposes, e.g., drug development, toxicology, veterinary, animal welfare, food safety, sustainability and conservation. • To identify how systematic review methodology differs across organisations. • To discuss how information specialists, researchers and journal editors can create a culture that values the use of guidelines in primary animal studies and evidence syntheses. • To identify and document resources for systematic reviews of animal studies. • To identify and document common questions and unique problems that arise when searching for animal literature.

Panel members introduction
Ten panelists were invited to participate in the symposium.

Why systematic reviews of animal studies are needed
Systematic reviews of animal studies are needed to improve clinical decision-making, to assess how research is conducted, and to inform current practice and policy. Systematic reviews confirm, or conversely question, whether conclusions from any one primary study are generalisable and help to explore the heterogeneity between studies in a field. For animal research consumers (veterinary practitioners and policy-makers), this synthesis can serve as a short-cut for keeping up to date on current research; clinicians reading one well-conducted systematic review can gain insights from multiple primary studies and understand their results in context of one another.
In the preclinical research setting, systematic reviews can help advance understanding of human and animal disease and management, and help recognise when research can move from preclinical studies to the next stage of research (Wever et al., 2012). Systematic reviews also highlight areas where laboratory research can be improved or identify systemic barriers to the implementation of interventions Systematic reviews and related methodologies of veterinary studies provide critical appraisal of the quality of the available evidence surrounding a clinical question, increase external validity, reduce the risk of bias, and identify gaps in the research. Compared to human-directed research, there is relatively limited research support for agriculture, and even less for veterinary science and companion animal care, so there is an ethical obligation that researchers maximise the quality of primary studies.
Panelists noted an increasing need for reviews to be rapid and nimble, engaging policy-makers in the process, citing work by Chris Whitty, Chief Medical Officer for England and Chief Medical Adviser to the UK Government that examined the impact of evidence on real-time government decision-making during the pandemic (Whitty & Collet-Fenson, 2021).

Oversight and funding for systematic reviews of animal studies
Some worry that the growth in systematic review production may involve duplication of efforts and waste of limited time and resources. Panelists agreed on the need to focus future efforts on improving the quality and not the quantity of reviews, and to encourage researchers to ask if a review on that topic has already been done or if a protocol has been registered on their topic. In addition, researchers should demonstrate where their review fits into the canon of previous work on the topic.
Availability of funding, often influenced by institutional or country policies, can determine review topics, as well as whether researchers focus on preclinical, veterinary, global One Health issues, or the United Nations' Sustainable Development Goals. In The Netherlands, national funding exists for researchers to conduct mentored systematic reviews, and most applicants for this funding are PhD students. An impact study of this Dutch program showed that students' completion of a systematic review improved the quality of their future primary research, regardless of whether they participated in subsequent reviews (Menon et al., 2021).
While funding structures may exist for preclinical and One Health evidence synthesis projects, panelists observe less funding available for projects focused on veterinary questions, environmental factors, conventional housing, facilities, husbandry, morbidity and mortality, and background genomic factors of species, breeds and strains, except when veterinary studies are tied to industry (Wareham et al., 2017). One model that a large professional organisation can use to remedy this oversight is to adopt a systematic process whereby members or conference attendees are asked to submit questions, information specialists search to see what questions have already been answered, and working groups pursue those questions that remain.

Types of evidence synthesis
Types of evidence synthesis reviews have proliferated (Grant & Booth, 2009), with a 2019 article categorising 48 separate review types (Sutton et al., 2019). Some commonly used types include scoping reviews, systematic reviews, rapid reviews, umbrella reviews, and meta-analyses. Panelists shared their perspectives on the continuum of evidence synthesis types with which they have engaged, from systematic reviews to the less rigorous critically appraised topics (Brennan et al., 2020).
Preliminary stakeholder discussions about conducting higher levels of evidence synthesis like systematic reviews and meta-analyses can be challenging depending on the discipline, type of question, and study type availability. For instance, while randomised controlled trials are considered the most reliable type of evidence, this study design is not feasible or ethical in all fields of research. Researchers may lack or underestimate the resources needed to conduct a systematic review. While selling the idea of evidence synthesis is easy, learning the process and defining the scope may be difficult for researchers that are new to the methodology. The use of reporting guidelines for primary studies, that define the information that is necessary to include based on study design, is crucial for comparing one animal study to another in systematic reviews. • Selecting a review type Choosing the appropriate review type is a pivotal first step for researchers, as it helps to determine the scope and focus of the review, as well as the methodology and analysis that will be used. Cornell University Library's Evidence Synthesis Service Review Methodology Tree (Cornell University Library, n.d.), for deciding on a review type, can be a helpful tool to aid researchers in their decision. The flowchart helps researchers understand the basics of each review type and why one might be chosen over another. PROSPERO registers systematic reviews, rapid reviews or umbrella reviews of preclinical animal research studies that are of 'direct relevance to human health'. PROSPERO does not register scoping reviews. Developed with input from researchers, PROSPERO's preclinical form includes more mandatory fields than the clinical form. The PROSPERO team does not perform formal peer-review of protocols, but checks for transparency of methods and completeness of information.
Systematic review protocols of animal studies without a direct relevance to human health, such as those in veterinary or agricultural sciences, can be registered with SYREAF Protocols or Open Science Foundation (OSF) Registries (Center for Open Science, n.d.). The SYREAF team currently considers strict guidelines as a barrier to veterinary researchers submitting protocols, therefore few protocol elements are mandatory.
Some additional conversation among attendees highlighted researchers' hesitancy to pre-register protocols due to concerns related to intellectual property. Panel members felt that researchers fearful of protocol registration were in the minority. They asserted that pre-registration serves as a means to 'claim' a particular topic and that many journals and reviewers expect and reward this practice. Pre-registration of animal systematic review protocols also encourages the pre-registration of primary animal study protocols.

• Study types
Types of studies to include in animal evidence synthesis projects depends both on the available research in a particular discipline and the type of question being asked in the review. There may be an abundance of clinical trials in some disciplines, and in others the body of evidence may be much more qualitative or rely heavily on observational studies. For instance, preclinical research questions, focused on mechanisms of disease, therapeutic targets, and safety, translate best to randomised and controlled experimental studies, while research questions in clinical veterinary settings concerned with the aetiology, prevention, or prognosis of a disease translate best to observational studies (Veterinary Evidence 'Guidance for writing the clinical bottom line,' n. d.[b]). Broader One Health synthesis projects may include multiple questions with multiple variables, often incorporating mixed-method studies that address the role of human attitudes and behaviour. What might be evidence in one discipline does not always apply to all disciplines.
• Tools and resources Panelists noted that there are many tools for conducting systematic reviews. However, tools are only a means to an end and selecting an effective tool will depend on funding, familiarity, and unique needs of the review topic and / or field. Preclinical animal researchers, in particular, value traditional systematic review screening tools such as Covidence (Covidence, n.d.), Rayyan (Rayyan, n.d.), and DistillerSR (DistillerSR, n.d.). However, some screening tools designed for human studies cannot handle common animal study protocols. For example, many popular screening tools such as Covidence, do not support veterinary studies that include multiple primary studies in one publication. To facilitate the creation of preclinical in vivo systematic reviews, Systematic Review Facility (SyRF) was created to decrease the time needed for reviews and the need to use multiple software programs (Bahor et al., 2021).
A question was raised about whether the differences between animal and human systematic reviews are distinct enough to justify the need for different tools and resources. Panel member opinions varied about the degree to which different tools are required across preclinical, veterinary, and human studies, however all panelists agreed that, at this point in time, human-centric tools do not adequately support evidence synthesis of animal studies. This may be an important consideration as the number of One Health studies continues to grow.

Living systematic reviews
As the amount of evidence grows and as global questions become more complex and incorporate multiple variables, more organisations are pursuing continuously updated living systematic review platforms, which have an a priori plan to incorporate new evidence over time.
CAMARADES and SYREAF have both explored living systematic reviews made available via R Shiny web applications. CAMARADES has implemented living systematic reviews on transgenic mice models of Alzheimer's Disease and Covid-19 research (Hair et al., 2021b), and SYREAF has a living systematic review looking at the impact that multiple effects of animal production have on the health of surrounding communities (Fonseca & O'Connor, 2021 and SYREAF, 2023).
CAMARADES developed Systematic Online Living Evidence Summaries (SOLES) for different research areas by implementing automation and crowd sourcing (Hair et al., 2021b). Automation adds the ability to analyse data at a level that surpasses that of human screeners (Bannach-Brown et al., 2019). Living systematic reviews are greatly facilitated when workflows and platforms are created that minimise a need to transfer records from one tool to another.

Educating the next generation of practitioners on evidence synthesis
Panelists acknowledged a distinction between authors and consumers of systematic reviews and approach each audience differently. Practitioners and students may not conduct research or become authors of reviews, but they need to know about different review types and how to apply their findings to practice. Because of this, it is important to integrate instruction on using systematic reviews into the curricula of healthcare professions. The RCVS Knowledge EBVM Learning Course teaches veterinary practitioners how to apply evidence to answer clinical questions, which is important foundational knowledge when engaging in the review process. Students and first-time authors can engage first with 'mini reviews' called critically appraised topics (CATs), a type of rapid review (Brennan et al., 2020). Clinical examples include Knowledge Summaries published in Veterinary Evidence (RCVS Knowledge, n.d. [b]) and BestBETs for Vets (University of Nottingham, 2023) made available online through the CEVM. These are inexpensive, time efficient and simple ways of doing reviews that are useful for both learners and clinicians.
Depending on their availability, students across a variety of disciplines and levels of education can contribute significantly to a larger synthesis project. Involving a few undergraduates in phases of a synthesis review (as opposed to the entire process) can provide substantive experience and perhaps material for an honours degree thesis. Graduate students, or professional students enrolled in concurrent degree programmes such as public health, will dive deeper into scholarly writing as part of their curriculum. By forming a cohort, these students can peer-review each other's work to strengthen the learning process and to learn to apply reviews across different questions and topics. This then requires fewer one-on-one consultations by mentors or advisors. Building capacity for conducting systematic reviews among the next generation is vital to the continued evolution and improvement of review methods.

Artificial intelligence and the future of systematic reviews
Evidence synthesis projects rely increasingly upon machine learning and artificial intelligence (AI). It may become harder to perform evidence synthesis without automation tools that create and translate searches, citation search, deduplicate citations, screen both title / abstracts and full-text, extract data, assess risk of bias, and manage workflows. AI tools are needed to decrease the time required to complete a systematic review. Rapid completion of reviews is critical for policy change, as policy-makers will not often tolerate long timelines. In the health and social sciences, detailed manuals and handbooks, such as the Cochrane Handbook for Systematic Reviews of Interventions and the Campbell Collaboration's guidelines, contain rigorous methodological guidance on the process of conducting evidence syntheses. However, researchers using AI in evidence syntheses are likely to come from diverse research fields where detailed guidance may not exist. With this in mind, a network of organisations, the International Collaboration for the Automation of Systematic Reviews (ICASR), works to create open tools for automating evidence synthesis (Beller et al., 2018).
The panelists outlined some of the synthesis review tasks where AI tools have been deployed for evidence synthesis projects involving animal studies. Not all available tools were discussed in the symposium, and further recommendations for AI use in evidence synthesis are defined elsewhere (Bersani et al., 2022). Centre (EPPI-Centre) at University College London to classify studies based on title and abstract screening as 'primary' or 'other' (Hair et al., 2021b).

• Extraction
The SYREAF team and other panelists highlighted text extraction from PDFs as one of the most difficult automation steps. SyRF has worked on a graphical data extraction tool to aid in meta-analysis of preclinical studies (Bahor et al., 2021).
• Risk of bias assessment Automated tools that assess the actual risk of bias are difficult to develop, since performing a risk assessment is much more complex than extracting experimental variables. The reporting of measures to reduce bias, such as the term 'blinded experimenter' or 'random allocation' can be more readily assessed with automated techniques. The SYRCLE Risk of Bias (RoB) tool was developed to assess risk of bias for preclinical trials (Hooijmans et al., 2014). Evidence on the efficacy of AI tools in evidence synthesis for assessing quality of publications is lacking.

Discussion
Systematic reviews are useful tools for synthesising the results of studies, identifying research gaps, informing future research, and reducing research waste. As such, they are beneficial for health practitioners, researchers, and policy-makers. Systematic reviews based on studies of animals are increasingly being recognised as a critical part of knowledge translation frameworks by preclinical, veterinary and One Health groups that have not formally come together to discuss evidence synthesis.
What constitutes a high-quality systematic review is an evolving metric. Those working on synthesis of animal studies face common barriers of search strategy terminology and studies with small sample size, though the symposium uncovered distinct differences across disciplines in supporting infrastructures, required steps of the review, and included study designs. While much has already been done to define appropriate methodologies and create freely available tools, there are areas where development is still needed in order to further animal-focused evidence syntheses (e.g., inclusion of observational studies in reviews, AI). Automation of steps has the potential to make conducting systematic reviews more accessible to smaller research teams.
When looking at the breadth of topics under the 'evidence synthesis of animal studies' umbrella, recognising the value of having teams with individual skill sets involved in reviews (e.g., information specialists, subject expert researchers and clinicians) is key. Capturing discussions like this between various organisations, teams and individuals working to synthesise heterogenous studies contributes to a culture aiming to strengthen both evidence synthesis and primary research design and reporting.
The Symposium on Animal Systematic Reviews provided a unique opportunity for preclinical and clinical researchers, and information specialists, to collectively assess the state of systematic reviews focused on animals. While the underlying symposium objectives served to guide discussion, the time limited format did not allow for complete stakeholder representation or consensus building, particularly in the area of One Health. This was instead a forum for varied stakeholders, both panelists and attendees, to identify common challenges and opportunities unique to evidence synthesis of animal studies, including the use of existing resources or the need to develop new resources. Resources discussed in the symposium are compiled and shared in S1 Table 1.
Conversations and discussions should continue to further address some of the issues and barriers identified, including abstracts, indexing and reporting guidelines for primary studies, pre-registration, roles for AI, and types of studies to include, in order to improve the speed and quality of evidence synthesis of animal studies.

Supplementary materials
Supplementary Material S1 - Table 1: List of selected resources for evidence synthesis of animal studies