Datasets and codes for 'Elevating the importance of Risk of Bias assessment for ecology and evolution'

This dataset was collected as part of the manuscript entitled 'Elevating the importance of Risk of Bias assessment for ecology and evolution' Methods 1) Survey on the RoB awareness and use The survey was approved by the Ethics Committee of the Ruđer Bošković Institute, Zagreb, Croatia, ref. ZV/3218/1-2023. The survey was intended for ecologists and evolutionary biologists who have published at least one meta-analysis. It included non-identifying questions on familiarity with the concept of the Risk of Bias, awareness and use of RoB assessment, and also included general questions on familiarity with meta-analysis, field of research, and career stage. The survey was created in Google Forms and sent on the 11th September 2023 to the emails of corresponding authors of meta-analyses in ecology and evolution, via mailing lists (NC3 Collaborative Research Centre; Society for Open, Reliable, and Transparent Ecology and Evolutionary Biology; German Zoological Society; Sociedad Española de Etología y Ecología Evolutiv;, [e-mail]), slack channels (Big Team Science Conference Slack Channel, German Reproducibility Network Event Slack Channel, ESMARConf Slack Channel), Twitter posts, and the SORTEE newsletter. The survey was open to responses until the 15th October 2023. To determine the corresponding authors of meta-analyses, AST (author) searched for meta-analyses published in 300 journals via Web of Science (databases: SCI-EXPANDED, SSCI, AHCI, ESCI) on the 25th April 2023 (search string, which also provides codes for included journals, can be found in 'search_string.txt' ). This search retrieved 3,289 results (potential meta-analyses, BibTeX list of these available in this data package) published between 1945 and 2023. AST then extracted a list of all corresponding author email addresses from these articles using packages revtools 0.4.1 (Westgate 2019a, 2019 b) and stringr 1.5.0 (Wickham 2023) in R 4.2.3 (R Core Team, 2021). Code is available as '001_email_compilation.R'. This resulted in 3,346 email addresses; however, 789 emails (~24%) bounced back. The survey received 232 valid responses (i.e., 9.1% response rate). Because some of the responses were prone to subjective judgment on what the answer exactly meant, four assessors (AC, AST, ROD, and MG; authors) went through the answers of 188 respondents who had answered ‘YES’ to the question ‘Prior to receiving this survey invite, were you familiar with the concept of Risk of Bias (RoB)?’ or who had answered ‘NO’ but their remaining answers indicated otherwise. Each assessor provided an answer ‘YES’, ‘NO’, ‘Unsure’, or ‘NA’ to the following six questions: Has the respondent heard of RoB? Does the respondent have a correct interpretation of RoB? Does the respondent claim to have conducted a RoB assessment? Have they truly conducted RoB assessment? Respondent thinks RoB is publication bias Respondent has conducted publication bias, rather than RoB assessment We then compared the interpretations of the responses among all four assessors. When three or four assessors had the same interpretation, we chose the most common answer as the final one. When there was disagreement, we discussed the interpretations and agreed on whether the final answer should be ‘YES’, ‘NO’, or ‘NA’ (i.e., either the question is irrelevant given the previous answers, we could not agree on the interpretation, or agreed that the answer was too vague to interpret). For the analyses, we used this final data table containing both the original responses and each evaluator's scores ('Survey_scores_per_evaluator.csv') and the post hoc agreed scores ('Survey_final_scores.csv'). 2) Journals and RoB assessment Between 19th April 2025 and 12th May 2025, AC, OP, and ROD (authors) checked the websites of 275 journals that publish ecology and evolutionary biology research. The list of journals was taken from Ivimey-Cook et al. (2025). We checked each journal’s Aims & Scope, Author instructions, and Editorial policy sections to search for whether a journal accepts evidence synthesis, and whether it mentions Risk of Bias or any related concepts for authors of evidence synthesis articles. We first piloted our data extraction on 10 journals to adjust the data extraction questions and align responses. We used Google Forms for data extraction. The main questions included: Does the journal explicitly solicit some form of evidence synthesis? Does the journal specifically mention RoB or related assessment of primary literature in guidelines to authors of evidence synthesis, or is RoB/related assessment specifically mentioned in linked other guidelines (e.g., journal states something like ‘follow PRISMA guidelines when reporting MA’ but no further detail on RoB or related is mentioned)? What concept related to RoB (including RoB itself) does the journal or a linked guideline exactly mention? If the journal links or refers to external guidelines that mention RoB or related assessment, what are these guidelines? What specific RoB or related tool/checklist is mentioned in journal guidelines, or in linked guidelines? What is the strength of the journal’s policy on the use of RoB or related assessment? The following decisions were made in light of the pilot extraction: First, we followed this definition of evidence synthesis ‘‘Evidence syntheses are conducted in an unbiased, reproducible way to provide evidence for practice and policy-making, as well as to identify gaps in the research. Evidence syntheses may also include a meta-analysis, a more quantitative process of synthesising and visualising data retrieved from various studies’ (https://guides.library.cornell.edu/evidence-synthesis). Second, journals that explicitly solicit narrative reviews and similar (e.g. the Annual Review of Ecology, Evolution and Systematics solicits ‘essay reviews’ but not systematic or quantitative reviews) were scored as ‘NO’ for question (1) above, whereas journals that do not explicitly solicit evidence synthesis, but something more general (e.g. review articles, reviews and comprehensive synthesis, reviews) were scored as ‘Unsure’ for the same question. We divided journals across reviewers, and one reviewer checked each journal. The data table containing reviewer initials and their scores is 'Journals_risk_of_bias'.