In the first Delphi round (March/April 2018), respondents were asked to propose up to five topics for systematic reviews which they thought to be relevant. To accommodate the varying familiarity of the stakeholders with systematic reviews, we asked them to specify their proposed review topics according to the different components of the PICO format in a stepwise manner by providing a template for each component. We did not ask the respondents to suggest a comparator (C), as this was deemed too challenging for respondents, most of whom had never worked with systematic reviews.

We asked the respondents to propose review topics across the three pre-specified domains of public health, i.e., prevention, health promotion, and health services. Two people (DH and MM) coded and aggregated the proposed topics using content analysis in the software MAXQDA [31] while a third person (SL) provided feedback. The aggregation through the content analysis ensured that the large number of redundant or overlapping suggestions were transferred in a practicable amount of sufficiently distinct review topics. We counted how many review topics were proposed in each public health domain to understand which target groups (population), interventions, and outcomes were considered most frequently.

Additionally, respondents had to select criteria that they considered most important for assessing topics for systematic reviews. For establishing an initial list of criteria, criteria taken from literature were discussed within the research team and with the advisory board. Based on the initial list, respondents could propose further assessment criteria, which we aggregated, and if applicable merged, with the initial list of criteria. We then examined which assessment criteria were selected most frequently. But we also ensured that the criteria used for rating, measured distinct dimensions to be able to sufficiently discriminate in the assessment of the review topics. Table 1 shows the five stakeholder-refined assessment criteria that were considered most useful for rating review topics in public health.

Finally, we asked respondents to rate their previous experience with and their attitude towards systematic reviews.

All those nominated for participation in the first Delphi round were invited to participate in the second Delphi round (August/October 2020), independent of whether they had responded to the first round. Respondents were asked to rate the review topics based on the content analysis by applying the stakeholder-refined assessment criteria on a scale from 1 “totally agree” to 4 “totally disagree.” Each respondent had to assess approximately only one-third of the review topics, which were randomly assigned.

In this round, we additionally asked respondents to weigh the importance of the five stakeholder-refined assessment criteria on a 100-point scale.

We calculated the mean for each topic (named the “rating score”) for all five assessment criteria combined and for the five assessment criteria separately, to rank the topics accordingly [32]. Furthermore, the inclusion of a variety of stakeholder groups allowed us to compare the rating and ranking of the review topics and the criteria weighting by stakeholder group.

We used IBM SPSS Statistics (Version 23) software for data management and analysis.

Of the 215 nominees, 197 (91.6%) started the online survey for the first Delphi round and 133 (67.5%) completed it (see Table 2). On average, the respondents took 25.5 min to complete the survey for the first Delphi round. Of the 197 respondents, 160 (81%) responded in German, 29 (15%) in French and 8 (4%) in Italian.

A majority of respondents in the first Delphi round described their work as being primarily “research and/or teaching” (n = 77, 39.1%). Followed by “administration and/or politics” (n = 37, 18.8%), “health organisations representing certain target groups in the population” (n = 36, 18.3%), “representatives of healthcare professionals and institutions” (n = 17, 8.6%); “health insurers” (n = 1, 0.5%); and “other” (n = 11, 5.6%).

In total 167 (77.7%) individuals participated in the second round, of which 128 (76.6%) completed the online survey. The respondents took 19.5 min on average to complete the survey for the second Delphi round. Only four respondents (2%) mentioned that they had not participated in the first round. Most respondents indicated that their field of activity was in “research and/or teaching” (n = 69, 41.3%), then “health organisations representing certain target groups in the population” (n = 39, 23.4%); “administration and/or politics” (n = 33, 19.8%); “representatives of healthcare professionals and institutions” (n = 15, 9.0%); health insurers (n = 3, 1.8%); and “other” (n = 7, 4.2%).

Most of the respondents who answered the set of questions regarding their experience with systematic reviews (n = 138), stated some familiarity with systematic reviews and considered them essential for research: They knew what systematic reviews meant (n = 122, 88%) and had used them in their professional activities (n = 109, 79%). Two-thirds of the respondents relied on Cochrane for systematic review results (n = 91, 66%). A majority also agreed with the importance of systematic reviews in public health research (n = 84, 61%) and practice (n = 87, 63%); about half found the results of systematic reviews helpful for their own work (n = 69, 50%). Furthermore, approximately half of the respondents knew how to find relevant systematic reviews (n = 80, 58%) and less than half of the respondents (n = 59, 43%) indicated that they do not have difficulties understanding the essence of systematic reviews. Finally, less than half (n = 58, 42%) of the respondents had been involved in the process of producing a systematic review.

The respondents suggested a total of 728 review topics in the first Delphi round that were compiled in a comprehensive list of 245 review topics, which we further aggregated into 27 categories across the three domains (see Table 3). For example, in the domain “prevention,” most suggested topics (n = 32) were assigned to the category “Interventions for screening and/or monitoring measures,” comprising often mentioned suggestions such as “More early screening to detect developmental abnormalities.”

We excluded 421 suggestions for review topics because they were: 1) defined as open questions or general statements (e.g., “Minimal case numbers: Quality feature or incentive for overtreatment?”), 2) too unspecific or not presented in PICO format (e.g., Development of new task allocations), or 3) defined only as outcomes (e.g., “Reduction of infant and maternal mortality”).

Although there was some overlap between the suggestions in the different domains of public health, the majority of suggestions for review topics could clearly be allocated to a single domain.

The overall rating results for all review topics and the 15 best-rated review topics per criterion, can be found in Supplementary Files 1, 2. The 15 best-rated review topics overall can be found in both Table 4, 5.

Among the top 15 review topics, most focus on interventions providing education to different target groups, followed by interventions to increase access to specific healthcare services. Mental health and substance use were the public health topics prioritised most often. Finally, the target groups considered most frequently are children and adolescents, specific target groups, and vulnerable groups.

We found large differences in the rating of review topics when examining the five assessment criteria separately. Of the 15 best rated review topics, not a single one was ranked in the top 15 in all five criteria individually. For example, the overall top priority “Improved access to prevention services for specific target groups” was ranked 3rd, 2nd, 5th, and 2nd, respectively, according to the criteria “Improving the health of the population,” “Health equity,” “Insufficient research to date,” and “Potential for innovative action”; however, it was ranked only 48th according to the criterion “Effect on public health if successful.”

Even larger differences in the ranking per criterion can be observed for many review topics from the top 15 list (see Table 4). For example, only four review topics that are in the top 15 according to the criterion “Insufficient research to date” are also represented in the overall top 15 review topics. Furthermore, the highest ranked review topics for the criteria “Effect on public health if successful” and “Potential for innovative action” are not at all represented in the overall top 15 review topics list and are even ranked among the worst 15 review topics according to two other criteria (i.e., “Insufficient research to date” and “Effect on public health if successful”).

Likewise, large differences in the rankings can be observed when comparing the overall rating between different stakeholder groups (see Table 5). Our results show that only the review topic with the best overall rating was rated in the top 15 by each stakeholder group. For all other review topics in the overall top 15 we found at least one stakeholder group that did not rate it in their top 15 list. However, the differing stakeholder group was not for every topic the same one, so none of the individual stakeholder groups was solely responsible for the observed differences.

The importance of using different assessment criteria and different stakeholder groups is further highlighted by the weighting of the criteria (see Table 6). Across all respondents, the criterion “improving the health of the population” received the highest weight of 29.49 (out of 100). The criterion “effect on public health if successful” was weighted lowest with 12.28.

The high standard deviation of the overall weights per criterion (varying from 8.94 to 14.53) is mainly due to the large variability in the weightings of the different stakeholder groups. For example, (see also Table 6): Respondents from “research and/or education” (M = 29.65, SD = 1.89), “representatives of health professionals/institutes” (M = 36.43, SD = 3.82) and from “administration and/or politics” (M = 32.12, SD = 2.80) assigned the highest weight to the criterion “improving the health of the population”; whereas “Health insurers” weighted this criterion the lowest (M = 15.00, SD = 10.09). In addition, “health organisations representing certain target groups in the population” weighted the criterion “health equity” highest (M = 25.61, SD = 2.27), whereas “health insurers” rated the criterion “potential for innovative action” the highest (M = 30.00, SD = 7.95).

Although the main aim of our study was to develop and implement a structured approach for prioritising review topics in public health in general, the COVID-19 pandemic has had an impact on the results of our study: The first Delphi round was conducted in 2018 before the onset of the COVID-19 pandemic and the second Delphi round was conducted in 2020 in the midst of the pandemic. Therefore, it is likely that an updated run of the first Delphi round would yield some emerging topics that were not gathered during our first Delphi round. However, we do believe that many proposed and prioritised review topics, such as “Peer-to-peer education on health risks from drugs” and “Mental health resilience training,” are still considered highly relevant. Replicating this study would help to ascertain which topics are priorities for the short-term or for the long-term.

These long-term impacts and transformational potential of research can in general be a factor when it comes to the strength of RPS studies. It is often difficult to predict what impact particular research priorities are likely to have. Unilateral or short-term prioritisation could lead to long-term challenges or potentially transformative research being neglected. It is therefore important to carefully analyse the potential long-term impact of all the topics in the RPS [14] and to simultaneously be aware that prioritising specific topics can influence the long-term direction of research within a certain field. Our structured approach can be replicated easily and therefore has the potential to also investigate if the top priorities are considered important in the long-term.

Another factor that might influence the RPS study is that our approach has only been applied in Switzerland, with unclear applicability to other countries and world regions. Also, our RPS approach is focused on prioritising topics for systematic reviews along a PICO framework only. Further investigations are needed to find out if a similar structure is applicable for prioritising topics for other types of evidence or frameworks and if this would generate very different result.

A further limitation is that we used the PICO format which may be too narrow for those public health questions that aim at the evaluation of distal changes in multi-sectoral policies with indirect impacts on health [44, 45]. Hence, a potential extension would be to ask respondents additional and more detailed questions by expanding the PICO scheme by adding further dimensions, e.g., for setting or study design. However, this would increase the burden on the respondents participating in this study while also increasing the complexity of the analysis.

Finally, we have to stress that part of the differences found between stakeholder groups might be due to the differing, and in some cases small, group sizes. In general, the validity of our sample is constrained by the specific public health organisations we identified by purposeful sampling during our stakeholder search. This, for example, led to a large standard deviation of the criteria weighting for the stakeholder group “health insurers.” Our recruitment goal was not to have exact comparable group sizes, but to establish an extensive list of stakeholders from the field of public health. An alternative to this method would have been the use of an open call for participation in the Delphi stage. However, this comes with several disadvantages, such as uncertain participation rates, potential biases, and most of all unclear levels of respondent expertise. More research is needed to fully understand the distinct priorities by stakeholder groups and if patterns exist in how different stakeholder groups prioritise review topics.

Setting research priorities inherently involves value judgements and subjective decisions. These value judgements and subjective decisions may depend on the individual views, expertise and interests of the policymakers and other stakeholders [14]. The differences between stakeholder groups in the rating and ranking of the review topics and in the weighting of the assessment criteria that we found in our study clearly showed the differing interests of these groups. The contribution of a diverse range of stakeholder groups to a transparent RPS study is thus valuable in terms of content and should be encouraged through ongoing consultations with these groups in order to capitalise on the existing diversity [3].

Our applied modified Delphi technique allows for the inclusion of a large panel of respondents by presenting easy-to-conduct online surveys that reduces the workload for respondents to participate. The approach is designed to assist respondents in determining relevant systematic review topics in a PICO format without needing prior extensive knowledge on systematic reviews; thereby enabling the inclusion of all relevant stakeholders in the prioritisation process of topics for systematic reviews [46].

We believe that our RPS study can be replicated easily and likewise involve many relevant stakeholders in a prioritisation process. The online surveys and the rating results of this study are available in German, French and Italian, facilitating the adaptation and implementation of similar study designs in various settings.