Skip to main content


Int J Public Health, 20 March 2024

To Remind or Not to Remind During Recruitment? An Analysis of an Online Panel in Germany

Bianca KleeBianca Klee1Daniela CostaDaniela Costa1Thomas FreseThomas Frese2Anja KnoechelmannAnja Knoechelmann3Gabriele MeyerGabriele Meyer4Thorsten MeyerThorsten Meyer5Oliver PurschkeOliver Purschke1Jan SchildmannJan Schildmann6Anke SteckelbergAnke Steckelberg4Rafael Mikolajczyk
Rafael Mikolajczyk1*
  • 1Institute for Medical Epidemiology, Biometrics and Informatics, Interdisciplinary Centre for Health Sciences, Medical Faculty of the Martin Luther University Halle-Wittenberg, Halle, Saxony-Anhalt, Germany
  • 2Institute for General Medicine, Interdisciplinary Centre for Health Sciences, Medical Faculty of the Martin Luther University Halle-Wittenberg, Halle, Saxony-Anhalt, Germany
  • 3Institute of Medical Sociology, Interdisciplinary Centre for Health Sciences, Medical Faculty of the Martin Luther University Halle-Wittenberg, Halle, Saxony-Anhalt, Germany
  • 4Institute for Health and Nursing Sciences, Interdisciplinary Centre for Health Sciences, Medical Faculty of the Martin Luther University Halle-Wittenberg, Halle, Saxony-Anhalt, Germany
  • 5Institute for Rehabilitation Medicine, Interdisciplinary Centre for Health Sciences, Medical Faculty of the Martin Luther University Halle-Wittenberg, Halle, Saxony-Anhalt, Germany
  • 6Institute for History and Ethics of Medicine, Interdisciplinary Centre for Health Sciences, Medical Faculty of the Martin Luther University Halle-Wittenberg, Halle, Saxony-Anhalt, Germany

Objective: To explore the role of reminders in recruiting and maintaining participation in an online panel.

Methods: 50,045 individuals from five German federal states were invited by regular mail to participate in the online study “Health-Related Beliefs and Healthcare Experiences in Germany.” Those who did not respond to the first attempt received a postal reminder. Comparisons of sociodemographic characteristics and responses were made between first-attempt respondents and those who enrolled after the second letter.

Results: After the initial letter, 2,216 (4.4%, 95%CI: 4.3%–4.6%) registered for the study; after a reminder 1,130 (2.5%, 2.3%–2.6% of those reminded) enrolled. Minor sociodemographic differences were observed between the groups and the content of the responses did not differ. Second-attempt respondents were less likely to participate in subsequent questionnaires: 67.3% of first-attempt vs. 43.3% of second-attempt respondents participated in their fourth survey. Recruitment costs were 79% higher for second-attempt respondents.

Conclusion: While reminders increased the number of participants, lower cost-effectiveness and higher attrition of second-attempt respondents support the use of single invitation only for studies with a similar design to ours when the overall participation is low.


Participation in epidemiological studies has sharply declined in the last decades [1, 2]. While studies in the least developed countries can still include high percentages of those initially invited [3], participation in more developed countries is often low nowadays [4]. For example, the UK Biobank study was able to recruit 5.5% of the invited participants using two letters of invitation three weeks apart [5]. The German National Cohort (NAKO) achieved an overall rate of 17% at baseline [6], using telephone calls to potential participants with an available landline after the first invitation and up to three postal reminders for those who could not be contacted by telephone.

When conducting primary data collection, a large sample size implies high power and precision [7, 8]. Another point is to obtain a high level of participation among those who are invited; the effort to achieve this is justified in order to avoid selection bias and improve representativeness [911]. However, there is a growing understanding that low participation does not automatically lead to selection bias or compromise the validity of the study [12, 13].

Earlier observations, when participation was still high, indicated that those recruited with more effort (e.g., additional reminders) differed from those recruited earlier and were more similar to ultimate non-respondents [14]. Thus, some have argued that studying late respondents would help to inform about non-respondents, and their inclusion in the sample could help to prevent biased conclusions [1517]. However, this recommendation is based on findings from the 1980s, when representativeness was associated with 80% participation [18]. Given the current low participation rates, the same reasoning may no longer apply. A first insight is provided by a study with participants from NAKO, where reminders were used to increase participation, and showed that subjects recruited with more effort (two or even three reminders) and those who responded to the initial invitation, did not differ in their characteristics [19].

In addition to considerations related to single assessments, there is also a longitudinal perspective. Although further efforts during recruitment (e.g., through reminder letters, phone calls, and home visits) result in an increase in response proportions [5, 6, 19, 20], studies have indicated that participants recruited with more effort drop out earlier and are therefore of limited benefit for longitudinal studies [10, 18]. These participants may be different from those who remain in the study, but if they are lost, there is not much that can be done to control for this selection. Nonetheless, similar to low response at baseline, low response at follow-ups does not necessarily seem to imply the existence of selection bias [21, 22], but the investment at baseline is lost if follow-up is not continued.

Online studies have emerged in the past two decades as an alternative to traditional paper- or telephone-based data collection, and their use has increased in recent years, particularly during the COVID-19 pandemic [23]. This format presents many advantages but also brings new challenges [24, 25]. While online studies often suffer from an unknown sampling framework, based on volunteers (e.g., when participants are recruited through social media) [26], systematic population-based approaches to recruitment can also be used. To date, it is not clear whether the known strategies, such as repeated postal reminders, also work for online studies.

Using a practical example of an online panel with a population-based recruitment approach, we assessed the effects of sending postal reminders compared to a single postal invitation on the recruited fraction and continued participation.


Study Design

This analysis is based on the population-based study “Health Related Beliefs and Healthcare Experiences in Germany” (HeReCa) [2730]. The aim of this study is to assess perceptions and experiences regarding health-related topics. Participants receive an email invitation to complete a short online survey (each 10–15 min) three to four times per year. The Ethics Committee of the Medical Faculty of the Martin Luther University Halle-Wittenberg, Germany (No. 2019-044) approved the study, which was conducted in accordance with the Declaration of Helsinki.

Recruitment of Participants

We randomly selected 14–15 municipalities (over 5,000 inhabitants) or cities in each of five federal states (Saxony-Anhalt, Berlin, Schleswig-Holstein, Nordrhein-Westfalen, and Baden-Wuerttemberg) in Germany, and asked the relevant resident offices for a random sample of a defined number of persons aged 18–79 (a total of 10,000 per federal state). The municipalities and cities were stratified to represent the distribution of the population density (dense, medium, or sparse) in the given federal state. We mailed invitation letters to the selected residents (Figure 1). These letters contained information about HeReCa and a web link for registration. In order to finalize the registration process, it was crucial to provide the unique code specified in the accompanying letter and an email address. This email address was utilized for subsequent invitations to complete questionnaires, which were accessible via a provided web link. We sent a postal reminder six to twelve months (depending on the federal state) after the first postal invitation to those who had not yet registered and contacted us with a request to be actively removed from the study. Participants provided their informed consent online and did not receive any kind of allowance.

Figure 1

Figure 1. Flowchart of the study population (Health-Related Beliefs and Healthcare Experiences in Germany study, Germany, 2023). * Some individuals actively refused to be contacted and others had letters returned stating that they could not be reached at the provided address. Both groups were excluded from the reminder letters.

Data Collection

By March 2023, we had administered twelve different questionnaires in HeReCa. A multi-professional team of health scientists developed questionnaires covering contemporary health issues (e.g., transplantation law in Germany, knowledge about medical rehabilitation, various aspects of COVID-19, and perceptions related to climate change). Participants provided sociodemographic information as part of the first questionnaire. Given that participants were recruited from different federal states over time, some questionnaires were administered in fewer federal states. Starting in April 2021 all participants received the same questionnaires. After an invitation to a specific questionnaire, participants received two reminder emails, one after 2 weeks and another after 1 week, in case they had not completed the survey by then. The online questionnaires remained active for 2–3 weeks after the second reminder email.


The response was calculated as the ratio of the number of respondents to the number of persons invited to join the study (at baseline) or the number of respondents to a specific questionnaire among those invited to participate in the questionnaire (at follow-up).

Costs of Recruitment

We calculated the amount of money necessary to recruit the participants. We included the costs of materials (e.g., paper, flyers) and stamps used for mailings, and divided the total cost of the initial and reminder letters by the number of registrations received after the first invitation and reminder, respectively. Staffing and administrative costs, such as fees charged by resident offices, were not included.

Statistical Analysis

We compared the sociodemographic characteristics of participants recruited by the first invitation and the reminder letter using absolute and relative frequencies with 95% confidence intervals (CI) and means with standard deviation (SD). We also compared the content of the responses between these two groups using Chi-Squared tests, accounting for the false discovery rate (FDR). Finally, we assessed the probability of completing subsequent questionnaires in both groups. R software (Version 4.1.1) was used for all statistical analyses.


Comparison of Sociodemographic Characteristics of Participants Recruited by Initial Invitation or Reminder Letter

We sent out 50,045 postal invitations and received 2,216 registrations resulting in a response rate of 4.4% (95%CI: 4.3%–4.6%). These participants will be referred to as “first-attempt respondents” in the following analyses. In total, 45,877 subjects did not reply to the initial invitation within six to twelve months and were reminded about the registration with a follow-up postal letter. This way we recruited an additional 1,130 participants (2.5%, 95%CI: 2.3%–2.6% of those who received a reminder letter; 2.3%, 95%CI: 2.1%–2.4% of those who were initially invited). These participants will be referred to as “second-attempt respondents” in the following analyses (Figure 1). The two groups differed only slightly in sociodemographic characteristics (Table 1). Incomplete responses (missing category) were more common among “second-attempt respondents.” A comparison between the vocational training qualification data of the HeReCa panel and the micro census data in Germany showed a trend toward higher education attainment among HeReCa participants (Supplementary Table S1). We sent the first postal invitations in two federal states (Saxony-Anhalt, Berlin) before the COVID-19 pandemic (November 2019) and in the remaining three federal states (Schleswig-Holstein, Nordrhein-Westfalen, and Baden-Wuerttemberg) during the early phase of the pandemic (between March and May 2020). There was no indication of a higher response during the pandemic (Supplementary Table S2).

Table 1

Table 1. Comparison of sociodemographic characteristics between participants who registered for the study after the first invitation letter (first-attempt respondents) and after the reminder invitation letter (second-attempt respondents) (Health-Related Beliefs and Healthcare Experiences in Germany study, Germany, 2023).

Given the overall low response, the main cost component was postal letters. The cost per letter was approximately 0.83€ including printing of materials, resulting in recruitment costs of 19€ per first-attempt responder and 34€ per second-attempt responder (+79%).

Comparison of Response Content Between First- and Second-Attempt Respondents

We analyzed the answers to a questionnaire containing 81 questions about medical care during and before the COVID-19 pandemic. After adjusting for multiple tests using the False Discovery Rate, only one of the items in the questionnaire showed significant differences between the two groups. This question asked whether the participant had any further conditions that required continuous medical treatment, medication use, or a stay in a hospital in the year 2019. Second-attempt respondents showed reduced odds for this condition compared to first-attempt respondents (Odds Ratio = 0.7, 95%CI: 0.5–0.9). The preceding question in the questionnaire listed common diseases (e.g., hypertension, cardiovascular disease) and the participants could indicate whether they had a specific disease. This question was answered similarly in both groups suggesting that the only difference may be random rather than systematic.

Comparison of First- and Second-Attempt Respondents’ Participation in Subsequent Questionnaires

We selected the second, third, and fourth individual questionnaires for each participant in order to analyze how many participants stayed on the panel over time compared to the number of initial registrations. First, we studied the active withdrawal of consent (Table 2). The loss of participants due to active withdrawal was similar after each subsequent questionnaire and was less than 1 percentage point for the first respondents and approximately 2–3 percentage points for the second-attempt respondents per questionnaire.

Table 2

Table 2. Retained fraction of persons (in % related to the persons initially participating by recruitment group), considering only the active withdrawal of study participants (Health-Related Beliefs and Healthcare Experiences in Germany study, Germany, 2023).

In contrast, attrition by simply not responding to subsequent questionnaires was much more common among second-attempt respondents compared to first-attempt respondents (Figure 2). After the initial decline from baseline to the first follow-up questionnaire, first-attempt respondents displayed only a slight decline in participation. In contrast, second-attempt respondents showed a more pronounced decline in participation in subsequent questionnaires.

Figure 2

Figure 2. Participation in follow-up questionnaires by recruitment group (points indicate proportion estimates and whiskers the 95% Confidence interval) (Health-Related Beliefs and Healthcare Experiences in Germany study, Germany, 2023).


Sending a postal reminder after an online panel invitation resulted in an additional registration of 2.3% of invited individuals, increasing the initial response rate from 4.4% to 6.7% (i.e., +52%). Second-attempt respondents were similar to first-attempt respondents in terms of sociodemographic characteristics. Furthermore, the content of the responses was similar in both groups. Recruitment of second-attempt respondents was 79% more expensive than recruitment of first-attempt respondents (per recruited participant). While active withdrawal in subsequent rounds was only marginally different in both groups, attrition was much higher among those recruited with the reminder.

We did not detect any sociodemographic differences among first- and second-attempt respondents. Clearly, our study was not very large and in larger samples, some differences might appear. Still, the question is whether these differences are relevant. Undoubtedly, some specific groups may have been missed in our study, but it is not clear whether these groups - present in the population - could be recruited, even with more effort. Overall, it seems that the segment of the population accessible for recruitment is limited and within this segment achieving a higher response does not lead to qualitatively different results. Our participants were better educated than the general population, which is in line with other studies. There is clear evidence that participants in a cohort or cross-sectional study are more likely to be female, to have a high socioeconomic status and to have a healthier lifestyle compared to non-respondents [17, 31], and the reminder did not change this distribution in our study.

Our data come from an online study. In recent years, there seems to be a transition from paper-based to web-based surveys, with some studies already reporting higher response rates in web-based surveys than in paper-based surveys [32, 33]. Web-based studies have the advantage of fast and cost-effective data collection, and it has been repeatedly shown that there is no distortion in the results when comparing the two methods [3436]. Cohort studies with purely digital participation are not yet well established in Germany, but digitization processes and the extensive use of smart devices are becoming more and more integrated into daily life supporting an increasing use of digital data collection in the future. However, recruitment of participants for online studies remains challenging. The majority of online studies do not have a systematic framework for population-based recruitment but rather use advertising and social media. Validation of true participants is often difficult. In Germany, there is no population registry of email addresses that could be used for systematic sampling, therefore we had to resort to a mailed invitation. In the future, residents may potentially have an email address linked to their registration at local government offices, in addition to their home address. Another idea could be to use a hybrid approach that allows for non-systematic recruitment through general advertisement and social media and a systematic population-based sample through postal letters. These two groups could be compared in the analysis for validation. We have applied this recruitment strategy to our digital research platform DigiHero, which currently includes participants from nearly 100,000 households in Germany [37].

Because the HeReCa panel uses only web-based questionnaires, one might expect higher participation among younger participants, but this was not the case. Although evidence is still scarce, two studies showed lower participation for those born 1982–2003 compared to older cohorts in online surveys [38, 39], which is in agreement with our results. This may reflect the increasing internet literacy of older age groups in the population, counterbalancing the lower willingness to participate of younger age groups. In an earlier study, we showed that responses were similar in a population-based panel that allowed paper participation for those who wished, and in an online-only panel [40].

Strengths and Limitations

The strength of our study is the population-based sampling across five federal states in Germany. Furthermore, in contrast to studies that sent reminders after a short time period, where respondents to the different waves might overlap, we could fully separate the groups that responded to the initial invitation and the reminder. There are also limitations. Our study was conducted only in German, so it is limited to the native population and well-integrated immigrants (in terms of language). The content of our study is general perceptions and considerations related to health topics, so our findings may not be generalizable to investigations on different topics. The topics of the respective questionnaires were announced in the invitation email to complete the questionnaires and topic preferences could affect the decision to fill out a specific questionnaire.


A postal reminder increased the number of subjects recruited, but those recruited with more effort did not differ substantially in sociodemographic terms from first-attempt respondents. Moreover, first- and second-attempt respondents provided similar responses to the questionnaires presented. Given the higher recruitment costs and higher attrition when using reminders, our results support the use of a single invitation letter in a population-based online study, when participants need to be invited by regular mail.

Data Availability Statement

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Ethics Statement

The studies involving humans were approved by the Ethics Committees of the Medical Faculty of the Martin Luther University Halle-Wittenberg, Germany. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their informed consent online to participate in this study.

Author Contributions

BK conceptualized the study, carried out the analyses, interpreted the data, drafted the initial manuscript, critically reviewed, and revised the manuscript. DC drafted the initial manuscript and critically reviewed and revised the manuscript. RM conceptualized and designed the study, supervised the data analysis and interpretation and critically reviewed and revised the manuscript. OP contributed to the acquisition and analyses of the data, interpreted the data and critically reviewed and revised the manuscript. All the other authors—TF, AK, GM, TM, JS, and AS—interpreted the data and critically reviewed and revised the manuscript. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that they do not have any conflicts of interest.

Supplementary Material

The Supplementary Material for this article can be found online at:


NAKO, German National Cohort; HeReCa, Health-Related Beliefs and Healthcare Experiences in Germany; 95% CI, 95% confidence interval; SD, standard deviation; FDR, false discovery rate; ISCED 97, International Standard Classification of Education 1997.


1. Galea, S, and Tracy, M. Participation Rates in Epidemiologic Studies. Ann Epidemiol (2007) 17(9):643–53. doi:10.1016/j.annepidem.2007.03.013

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Mindell, JS, Giampaoli, S, Goesswald, A, Kamtsiuris, P, Mann, C, Männistö, S, et al. Sample Selection, Recruitment and Participation Rates in Health Examination Surveys in Europe--Experience From Seven National Surveys. BMC Med Res Methodol (2015) 15(1):78. doi:10.1186/s12874-015-0072-4

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Kengne, AP, Ntyintyane, LM, and Mayosi, BM. A Systematic Overview of Prospective Cohort Studies of Cardiovascular Disease in Sub-Saharan Africa. Cardiovasc J Africa (2012) 23(2):103–12. doi:10.5830/CVJA-2011-042

CrossRef Full Text | Google Scholar

4. Rindfuss, RR, Choe, MK, Tsuya, NO, Bumpass, LL, and Tamaki, E. Do Low Survey Response Rates Bias Results? Evidence From Japan. Demographic Res (2015) 32(1):797–828. doi:10.4054/demres.2015.32.26

CrossRef Full Text | Google Scholar

5. Allen, N, Sudlow, C, Downey, P, Peakman, T, Danesh, J, Elliott, P, et al. UK Biobank: Current Status and What It Means for Epidemiology. Health Pol Tech (2012) 1(3):123–6. doi:10.1016/j.hlpt.2012.07.003

CrossRef Full Text | Google Scholar

6. Peters, A, Greiser, KH, Göttlicher, S, Ahrens, W, Albrecht, M, Bamberg, F, et al. Framework and Baseline Examination of the German National Cohort (NAKO). Eur J Epidemiol (2022) 37(10):1107–24. doi:10.1007/s10654-022-00890-5

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Hajian-Tilaki, K. Sample Size Estimation in Epidemiologic Studies. Caspian J Intern Med (2011) 2(4):289–98.

PubMed Abstract | Google Scholar

8. Serdar, CC, Cihan, M, Yücel, D, and Serdar, MA. Sample Size, Power and Effect Size Revisited: Simplified and Practical Approaches in Pre-Clinical, Clinical and Laboratory Studies. Biochemia Med (2021) 31(1):010502. doi:10.11613/BM.2021.010502

CrossRef Full Text | Google Scholar

9. Keeble, C, Baxter, PD, Barber, S, and Law, GR. Participation Rates in Epidemiology Studies and Surveys: A Review 2007 – 2015. Internet J Epidemiol (2015) 14(1):1–14. doi:10.5580/IJE.34897

CrossRef Full Text | Google Scholar

10. Nohr, EA, and Liew, Z. How to Investigate and Adjust for Selection Bias in Cohort Studies. Acta obstetricia gynecologica Scand (2018) 97(4):407–16. doi:10.1111/aogs.13319

CrossRef Full Text | Google Scholar

11. Jöckel, K-H, and Stang, A. Cohort Studies With Low Baseline Response May Not Be Generalisable to Populations With Different Exposure Distributions. Eur J Epidemiol (2013) 28(3):223–7. doi:10.1007/s10654-013-9782-2

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Nohr, EA, Frydenberg, M, Henriksen, TB, and Olsen, J. Does Low Participation in Cohort Studies Induce Bias? Epidemiology (2006) 17(4):413–8. doi:10.1097/01.ede.0000220549.14177.60

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Bakke, PS. Non-Response in Epidemiological Studies - How to Cope With it? Respir Med (2010) 104(3):323–4. doi:10.1016/j.rmed.2009.11.018

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Lin, IF, and Schaeffer, NC. Using Survey Participants to Estimate the Impact of Nonparticipation. Public Opin Q (1995) 59(2):236–58. doi:10.1086/269471

CrossRef Full Text | Google Scholar

15. Morton, LM, Cahill, J, and Hartge, P. Reporting Participation in Epidemiologic Studies: A Survey of Practice. Am J Epidemiol (2006) 163(3):197–203. doi:10.1093/aje/kwj036

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Madigan, MP, Troisi, R, Potischman, N, Brogan, D, Gammon, MD, Malone, KE, et al. Characteristics of Respondents and Non-Respondents From a Case-Control Study of Breast Cancer in Younger Women. Int J Epidemiol (2000) 29(5):793–8. doi:10.1093/ije/29.5.793

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Keeble, C, Barber, S, Law, GR, and Baxter, PD. Participation Bias Assessment in Three High-Impact Journals. SAGE Open (2013) 3(4):215824401351126. doi:10.1177/2158244013511260

CrossRef Full Text | Google Scholar

18. Hoffmann, W, Terschüren, C, Holle, R, Kamtsiuris, P, Bergmann, M, Kroke, A, et al. Zum Problem der Response in Epidemiologischen Studien in Deutschland (Teil II). Gesundheitswesen (2004) 66(8-9):482–91. doi:10.1055/s-2004-813094

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Rach, S, Günther, K, and Hadeler, B. Participants Who Were Difficult to Recruit at Baseline Are Less Likely to Complete a Follow-Up Questionnaire - Results From the German National Cohort. BMC Med Res Methodol (2020) 20(1):187. doi:10.1186/s12874-020-01073-0

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Krist, L, Bedir, A, Fricke, J, Kluttig, A, and Mikolajczyk, R. The Effect of Home Visits as an Additional Recruitment Step on the Composition of the Final Sample: A Cross-Sectional Analysis in Two Study Centers of the German National Cohort (NAKO). BMC Med Res Methodol (2021) 21(1):176. doi:10.1186/s12874-021-01357-z

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Carter, KN, Imlach-Gunasekara, F, McKenzie, SK, and Blakely, T. Differential Loss of Participants Does Not Necessarily Cause Selection Bias. Aust New Zealand J Public Health (2012) 36(3):218–22. doi:10.1111/j.1753-6405.2012.00867.x

CrossRef Full Text | Google Scholar

22. Meterko, M, Restuccia, JD, Stolzmann, K, Mohr, D, Brennan, C, Glasgow, J, et al. Response Rates, Nonresponse Bias, and Data Quality. Public Opin Q (2015) 79(1):130–44. doi:10.1093/poq/nfu052

CrossRef Full Text | Google Scholar

23. Kumar, A, Chandran, S, and Somani, A. Conducting Online Web-Based Surveys at the Time of COVID-19 Pandemic: A Short Report. Arch Ment Health (2021) 22(2):158. doi:10.4103/amh.amh_44_20

CrossRef Full Text | Google Scholar

24. Daikeler, J, Bosnjak, M, and Manfreda, KL. Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates. J Surv Stat Methodol (2020) 8(3):513–39. doi:10.1093/jssam/smz008

CrossRef Full Text | Google Scholar

25. Van Gelder, MMHJ, Bretveld, RW, and Roeleveld, N. Web-Based Questionnaires: The Future in Epidemiology? Am J Epidemiol (2010) 172(11):1292–8. doi:10.1093/aje/kwq291

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Bethlehem, J. Selection Bias in Web Surveys. Int Stat Rev (2010) 78(2):161–88. doi:10.1111/j.1751-5823.2010.00112.x

CrossRef Full Text | Google Scholar

27. Schildmann, J, Nadolny, S, Führer, A, Frese, T, Mau, W, Meyer, G, et al. Gründe und Einflussfaktoren für die Bereitschaft zur Dokumentation von Präferenzen Bezüglich Organspende. Ergebnisse Einer Online-Umfrage. PPmP - Psychotherapie Psychosomatik Medizinische Psychol (2022) 72(08):354–61. doi:10.1055/a-1718-3896

CrossRef Full Text | Google Scholar

28. Führer, A, Frese, T, Karch, A, Mau, W, Meyer, G, Richter, M, et al. COVID-19: Knowledge, Risk Perception and Strategies for Handling the Pandemic. Z Evidenz, Fortbildung Qualitat im Gesundheitswesen (2020) 153-154:32–8. doi:10.1016/j.zefq.2020.06.002

CrossRef Full Text | Google Scholar

29. Krippl, N, Mezger, NCS, Fischer, H, Schildmann, J, Mikolajczyk, R, Danquah, I, et al. Climate-Sensitive Health Counselling: A Quantitative Survey on Addressing Climate Change. Eur J Public Health (2022) 32(Suppl. ment_3):2022. doi:10.1093/eurpub/ckac130.069

CrossRef Full Text | Google Scholar

30. Profilzentrum Gesundheitswissenschaften. Profilzentrum Gesundheitswissenschaften HeReCa Study (2023). cited 2023 Aug 20. Available from: (Accessed August 20, 2023).

Google Scholar

31. Shahar, E, Folsom, AR, and Jackson, R. The Effect of Nonresponse on Prevalence Estimates for a Referent Population: Insights From a Population-Based Cohort Study. Atherosclerosis Risk in Communities (ARIC) Study Investigators. Ann Epidemiol (1996) 6(6):498–506. doi:10.1016/s1047-2797(96)00104-4

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Van Den Berg, MH, Overbeek, A, Van DerPal, HJ, Birgitta Versluys, A, Bresters, D, Van Leeu Wen, FE, et al. Data on Fertility Issues Among Female Childhood Cancer Survivors: Differences in Response Characteristics. J Med Internet Res (2011) 13(3):1–12. doi:10.2196/jmir.1707

CrossRef Full Text | Google Scholar

33. McMaster, HS, LeardMann, CA, Speigle, S, Dillman, DA, Stander, V, Pflieger, J, et al. An Experimental Comparison of Web-Push vs. Paper-Only Survey Procedures for Conducting an In-Depth Health Survey of Military Spouses BMC Med Res Methodol (2017) 17(1):73–9. doi:10.1186/s12874-017-0337-1

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Rübsamen, N, Akmatov, MK, Castell, S, Karch, A, and Mikolajczyk, RT. Factors Associated With Attrition in a Longitudinal Online Study: Results From the HaBIDS Panel. BMC Med Res Methodol (2017) 17(1):132–11. doi:10.1186/s12874-017-0408-3

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Zuidgeest, M, Hendriks, M, Koopman, L, Spreeuwenberg, P, and Rademakers, J. A Comparison of a Postal Survey and Mixed-Mode Survey Using a Questionnaire on Patients' Experiences With Breast Care. J Med Internet Res (2011) 13(3):e68–13. doi:10.2196/jmir.1241

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Robles, N, Rajmil, L, Rodriguez-Arjona, D, Azuara, M, Codina, F, Raat, H, et al. Development of the Web-Based Spanish and Catalan Versions of the Euroqol 5D-Y (EQ-5d-Y) and Comparison of Results With the Paper Version. Health Qual Life Outcomes (2015) 13(1):72–9. doi:10.1186/s12955-015-0271-z

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Diexer, S, Klee, B, Gottschick, C, Xu, C, Broda, A, Purschke, O, et al. Association Between Virus Variants, Vaccination, Previous Infections, and Post-COVID-19 Risk. Int J Infect Dis (2023) 136:14–21. doi:10.1016/j.ijid.2023.08.019

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Bosch, OJ, Revilla, M, and Paura, E. Do Millennials Differ in Terms of Survey Participation? Int J Market Res (2019) 61(4):359–65. doi:10.1177/1470785318815567

CrossRef Full Text | Google Scholar

39. Revilla, M, and Höhne, JK. Comparing the Participation of Millennials and Older Age Cohorts in the Cross-National Online Survey Panel and the German Internet Panel. Surv Res Methods (2020) 14(5):499–513. doi:10.18148/srm/2020.v14i5.7619

CrossRef Full Text | Google Scholar

40. Rübsamen, N, Akmatov, MK, Castell, S, Karch, A, and Mikolajczyk, RT. Comparison of Response Patterns in Different Survey Designs: A Longitudinal Panel With Mixed-Mode and Online-Only Design. Emerging Themes Epidemiol (2017) 14:4. doi:10.1186/s12982-017-0058-2

CrossRef Full Text | Google Scholar

Keywords: response, participation, online study, bias, web-based survey

Citation: Klee B, Costa D, Frese T, Knoechelmann A, Meyer G, Meyer T, Purschke O, Schildmann J, Steckelberg A and Mikolajczyk R (2024) To Remind or Not to Remind During Recruitment? An Analysis of an Online Panel in Germany. Int J Public Health 69:1606770. doi: 10.3389/ijph.2024.1606770

Received: 27 October 2023; Accepted: 27 February 2024;
Published: 20 March 2024.

Edited by:

L. Suzanne Suggs, University of Italian Switzerland, Switzerland

Reviewed by:

Viktor Von Wyl, University of Zurich, Switzerland
Stefan Essig, University of Lucerne, Switzerland

Copyright © 2024 Klee, Costa, Frese, Knoechelmann, Meyer, Meyer, Purschke, Schildmann, Steckelberg and Mikolajczyk. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Rafael Mikolajczyk,

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.