The use of large trials to comprehensively evaluate psychosocial interventions is an established practice that can determine those interventions’ effectiveness and identify the practical considerations associated with delivering the interventions at scale. The high costs associated with such projects magnify researchers’ responsibility to design and implement successful and informative studies, as invalid results can be misleading and undermine adoption of an otherwise promising intervention. These projects often utilize randomized controlled trial (RCT) designs, which provide the strongest inferential basis for drawing causal conclusions about the relative effectiveness of the intervention on outcomes compared to one or more other treatments. Large studies of mental health interventions typically report smaller effect sizes than small studies [1]; while this can be due to the greater accuracy inherent in the larger sample sizes and wider range of settings, methodological problems can also reduce the apparent effect of interventions [2]. The history of Individual Placement and Support illustrates the opportunities and pitfalls that large and small RCTs face in facilitating the development, validation, and adoption of a widely implemented, evidence-based intervention.

The basic conditions for valid causal inference with an intent-to-treat analyses of RCTs—often referred to as internal validity--include a sample large enough to detect the chosen effect size, random treatment arm assignment, participation in assigned treatment, and minimal loss to follow-up [3]. External validity, a concept that includes generalizability, suitability, and transferability, describes the extent to which a study’s causal results hold over a range of people and settings beyond the those observed in the study [4]. Standards for evaluation of external validity have not been established and are therefore less likely to be systematically assessed for a given RCT [5]. External validity is typically assumed from the outset in the form of enrollment criteria and the selection of intervention implementation settings, which are usually selected to represent the desired population and their manner of service receipt. These assumptions face threats from recruitment errors yielding a sample that does not reflect the intended study population, overly restrictive inclusion criteria, and implementation of interventions in a manner not reflective of how they would be delivered in actual practice [6, 7]. Without careful preparation, these conditions can prove elusive.

Individual Placement and Support (IPS) is a form of supported employment that improves rates of competitive employment among those with serious mental illness (SMI). People with SMI experience substantial functional impairments associated with a diagnosed mental illness. Since publication of the first IPS RCT in 1996, researchers have established the effectiveness of IPS in over 30 RCTs and provided ongoing support to increasing numbers of real-world IPS programs and clients in more than 20 countries [8], including more than 1000 IPS programs in the United States [8–12]. Most of these RCTs have used samples of less than 200, and nearly all took place in real-world settings using typical employment specialists and participants with documented mental disabilities who wanted to pursue competitive employment. This is typical for psychosocial research, as the interventions are complex and must be embedded in the staff, clients, and local environment of the host organization.

Before conducting the first IPS RCT, researchers established the feasibility of IPS for people with serious mental illness [13]. This provided the necessary a priori understanding, usually acquired in the form of pilot studies or an equivalent research history, that the program would be both acceptable to the target population (e.g., participants would choose to participate and would remain in the program) and appropriate (e.g., address their needs and goals), and that any obstacles to its implementation would be identified [14]. Foreknowledge of feasibility and potential intervention effect can help investigators anticipate threats to validity when either scaling up an intervention or applying it to a new population (Table 1) [15, 16]. The developers of IPS followed the preferences of people with SMI for competitive employment and studied IPS implementation in a series of pilot studies involving conversions of traditional rehabilitation programs to supported employment programs [17–24]. These studies, relying on methods already developed for recruitment of people with mental illness to supported employment programs [25], drew samples of those with SMI receiving mental health services who did not have jobs but wanted to work and were of working age [13, 26, 27]. Investigators did not provide incentives to encourage participation in IPS, and clients not interested in employment received other supports. IPS researchers developed a thorough working knowledge of this population (adults with SMI who wanted to work) and the implementation context (community mental health programs).

From the earliest RCTs, the screening process included at least two informational sessions prior to consent. Researchers used those interviews to educate prospective recruits about the study (including IPS) and the expectations of them, answer questions, and confirm that recruits understood the study, were still interested in work, and were willing to participate in both the IPS program and the research assessments. Studies measuring IPS participation reported rates in assigned treatment above 90% [13, 27, 28] and minimal attrition from research. In the largest such study that enrolled adults with SMI and reported IPS participation rates, the multi-national EQOLIZE study (N = 213) [29], 87% of those randomized to IPS participated. Meta-analyses of IPS RCTs have reported that those participating in IPS were from 1.63 (95% CI 1.46, 1.82) (11) to 2.40 (95% CI 1.99, 2.90) times more likely to find employment than those in control conditions [30].

Due to the effectiveness of IPS across multiple RCTs and the absence of empirical support for other vocational interventions, clinical researchers have extended IPS to new populations, including people with autism [31–34], posttraumatic stress disorder [35, 36], substance use disorders [37, 38], common mental disorders such as anxiety and depression [39], chronic pain [40], and spinal cord injury [41]. These have typically been pilots and smaller studies that provide critical information regarding population responses to planned recruiting efforts, implementation feasibility, proposed instruments, obstacles to participant study completion, and the potential benefits of the intervention [42, 43]. Pilot findings can help researchers design a recruitment process that addresses the concerns and lifestyles of potential participants and minimizes dropout and non-participation by ensuring that enrollees are eligible and understand their responsibilities [25]; identify adaptations necessary for the intervention to maximize participation and fidelity; and plan how to maintain contact with recruits and incentivize their continued participation [44, 45].

The history of IPS research can serve as a case study in the development and implementation of an evidence-based practice. After a series of small RCTs comparing IPS to usual service controls, a series of large RCTs (N > 1,000) reported unexpectedly small effects. The purpose of this review is to examine threats to validity and their root causes in large IPS RCTs.

In this article, we examine threats to validity in large trials of IPS that are generally applicable across psychosocial intervention research. We used existing systematic reviews and a Google Search to identify large (N>=1,000) RCTs with IPS-based interventions compared with control groups receiving usual services and found three studies. We examined final reports and peer-reviewed articles to evaluate threats to valid causal effect estimates of treatment assignment (i.e, internal validity) and how errors in study design and execution can undermine external validity.

Two of three identified large RCTs of IPS suffered substantial threats to validity when they experienced low rates of treatment adherence and study retention. These flaws indicate poor planning. BBSD and SED researchers were unfamiliar with IPS in the proposed context and poorly informed regarding the targeted populations' attitudes toward employment and employment services, mental illness and mental health services, and ongoing participation in a research project. As a result, the small effect sizes that these studies reported cannot be taken at face value. That those results have not been published in peer-reviewed journals and have only been uncritically presented in government and agency reports distorts the scientific record and undermines the adoption of a potentially beneficial intervention.

Researchers should carefully consider how they would like a study to generalize [66, 67] and be explicit about whether their ITT analysis is intended to represent a per-protocol effect [2]. The study intervention’s integration into services should reflect its implementation in the real world to maximize generalizability, and both intervention fidelity and enrollee participation should be measured to confirm study assumptions and enable detailed analyses of the intervention and its engagement by participants. Each of these large studies assumed that ITT analyses would yield valid estimates of the effects of IPS participation: each attempted to enroll samples that would participate in job services; each funded the IPS services necessary to provide services to all of those participating in IPS; none of the studies included provisions for measuring participation or conducting adherence-based analyses of treatment effects. High rates of attrition only compounded the problems related to poor treatment adherence.

We recommend that investigators pause studies experiencing difficulties with adherence and attrition that lack immediate and obvious solutions. Investigators can study records, formulate potential responses, and consult with funders regarding the wisdom of continuing the trial. Researchers should not succumb to the temptation to draw on available information to understand shortcomings and optimize the enrollment or engagement process while the study is underway. Recruitment, intervention implementation, and data collection plans are extremely difficult to change midstream, threaten study validity, and cannot compensate for an absence of pilot data. Such remediation efforts are akin to attempting deferred maintenance from the driver’s seat while motoring down the highway. Researchers should also not continue the study under the assumption that problems with participation in assigned treatment and missing data can be addressed at the analytic stage. The statistical tools remaining at study completion to compensate for low rates of participation or follow-up are sufficient only to quantify the extent of the study’s failure to meet its own objectives, contextualize any published findings, and inform future researchers making a similar attempt. In other words, the failed study becomes the precedent it lacked at its commencement.

Differences between the investigators who conduct small versus large RCTs may offer a partial explanation for the flagrant methodological shortcomings of some large RCTs despite a copious literature [68, 69] describing psychosocial trials and threats to their validity. Small studies tend to be initiated by investigators with a primary interest and expertise in psychosocial treatment outcome research involving specific interventions and procedures. In contrast, funders and contractors with the capacity to conduct large scale evaluations may lack familiarity with the intervention and target population, and overestimate feasibility for the proposed study. Those designing large RCTs may also assume that such trials have sufficient statistical power to overwhelm problems with participation or attrition. They may also view participation in the interventions (or lack thereof) as one indicator of the suitability (and potential effectiveness) of the interventions for the target populations, without realizing that a high rate of participation in treatment is a precondition for an internally valid evaluation of the intervention.