The E¤ect of Job Coaching on the Employment Outcomes of Individuals with Mental Retardation: Could Cream-Skimming Help? Melayne M. McInnes

Orgul D. Ozturk Joshua Mann

Suzanne McDermott

Very Preliminary Please do not distribute or quote without permission. Abstract This paper examines the employment outcomes of the SC job coaching program for individuals with mental retardation to determine the average e¤ects of the training, who bene…ts most from the training, and the extent to which observed and unobserved characteristics of the applicants a¤ect their selection into the program and the extent to which they bene…t from job coaching. Our econometric approach follows Aakvik, Heckman, and Vytlacil (2005) in allowing the bene…ts from participation in the program to di¤er among individuals who are observationally equivalent and the decision to participate/to be selected in the program to depend in part on the anticipated di¤erential bene…t. While controlling for any sorting on gains or cream-skimming is important for correctly evaluating the program’s e¤ectiveness, we are most interested in …nding out whether there is a way to more e¤ectively target the program at the individuals who will most bene…t. As the availability of job coaches in South Carolina is limited, making the most e¢ cient use of the job coaches that remain becomes increasingly critical.

JEL Classi…cation: C50; H43; J24 Key words: Social program evaluation; Discrete-choice models; Policy simulations

1

1

Introduction

Employment is a stated goal of a large number of adults with mental retardation who have person-centered or habilitation plans (Blanck, 1998; Test, Carver, Ewers, Haddad, and Person, 2000). Person-centered plans are developed by facilitators who help individuals with severe disabilities coordinate available services, such as job coaching, to achieve the individual’s prioritized goals. Supported employment using job coaches was introduced in 1984 (PL 98-527) as a mechanism to achieve integrated employment for adults with severe disabilities (McGaughey, Kiernan, McNally, Gilmore and Keith, 1995; Wehman and Kregel, 1998; Rusch and Braddock, 2004). Supported employment is intended to move people from dependence on a service delivery system to independence via competitive employment, and it encourages social interaction, and integration. Recent evidence shows that the number of adults with mental retardation is approximately equal in supported employment and sheltered workshop programs (nationally 118,000 and 126,000 respectively) (Rasch and Braddock, 2004). The literature suggests that individuals with mental retardation who work in integrated settings have higher wages and bene…ts compared to those in segregated settings (Mank, Cio¢ , Yovano¤, 1998; Wehman, Revell, Brooke, 2003; West, Wehman, Wehman, 2005). Supported employment is designed to place individuals in jobs in integrated settings and to provide on-site training and support with the goal of the individual achieving independence and maintaining competitive employment. Recent studies indicate that the provision of on-going support services for people with severe disabilities signi…cantly increases their rates for employment retention. However, despite the advantages that supported employment a¤ords, evidence suggests that the majority of individuals with intellectual disabilities remain unemployed, underemployed, or employed in segregated workshops (Jones and Bell, 2003; Yamaki and Fujiura, 2002; Rusch and Braddock, 2004). Because the literature to date does not control for the endogeneity of the participation decision, we cannot rule out the possibility that the apparent e¤ectiveness of job coaching results from cream-skimming those individuals who would be likely be employed with or without the program. We take advantage of a unique panel data set of all clients served by the South Carolina Department of Disabilities and Special Needs between 1999 and 2005 to investigate the extent job coaching causes increases in employment. Supportive employment

2

services in South Carolina are provided to individuals with mental retardation by 38 boards that serve county or multi-county areas. Hence, we have variation in the availability of job coaches over time and across boards. The data also contain information on individual characteristics, such as IQ and the presence of secondary autism or emotional and behavioral problems that are likely to a¤ect both employment propensity and likelihood of receiving job coaching. We examine the employment outcomes of the SC program to determine the average e¤ects of the training, who bene…ts most from the training, and the extent to which observed and unobserved characteristics of the applicants a¤ect their selection into the program and the extent to which they bene…t from job coaching. Our econometric approach follows Aakvik, Heckman, and Vytlacil (2005) in allowing the bene…ts from participation in the program to di¤er among individuals who are observationally equivalent and the decision to participate in the program to depend in part on the anticipated di¤erential bene…t. We report both marginal and average treatment e¤ects using Maximum Likelihood estimates, and then analyse the extent to which creamskimming— selecting applicants who are likely to have better outcomes even without job coaches— a¤ects our conclusions about the e¤ectiveness of the program since as in Aakvik, Heckman, and Vytlacil(2005) it is possible to …nd that the program participants tended to have observed and unobserved characteristics that would lead them to gain least from the training program. While controlling for any cream-skimming is important for correctly evaluating the program’s e¤ectiveness, we are most interested in …nding out whether there is a way to more e¤ectively target the program at the individuals who will most bene…t. As the availability of job coaches in South Carolina is limited, making the most e¢ cient use of the job coaches that remain becomes increasingly critical. Our correlation coe¢ cients are very imprecisely estimated that we cannot reject the null hypothesis of no selection on unobservables. However if the possibility of correlation of unoservables is ignored our estimates detoriates greatly. Contribution of this paper is the identi…cation and "imposition" of creamskimming on observables and unobservables as perceived by the observing economist. We use several di¤erent de…nitions of creamskimming (variety of de…nitons exist in the literature: see Heckman et al., 2002). For example, in some studies cream-skimming is said to occur when program administrators systematically admit persons who would likely have high employment rates and earnings 3

even in the absence of the program (see, e.g, Bassi, 1983; Anderson et al., 1993). We …nd evidence for this kind of cream skimming. People who participate in the supported employment have both observable and unobservable factors that produce the lowest gains in employment compared to what they would experience without treatment. Since we do not …nd any evidence of "good" cream skimming (assigning the individuals with highest gains from job coaching to the program) we proceed our analysis with imposition of various schemes of targeting in order to see if we can improve the program. We describe the Supported Employment Program in South Carolina is Section 2 and then discuss our data and variables in the following section. Section 4 describes the latent variable models of program participation and employment and our estimation strategy. Section 5 gives an overview of the preliminary empirical results. Section 6 gives the results for the cream skimming simulations and policy analysis and Section 7 concludes.

2

Supported Employment in SC

3

Supported Employment in South Carolina

The South Carolina supported employment system is a centerpiece of the day services o¤ered to individuals with mental retardation. Every county Disability and Special Needs (DSN) Board o¤ers the program to the individuals they serve. Supported employment programs have four components: 1) assessing skills and developing a plan for achieving competitive employment; 2) identifying a job suitable for the individual; 3) placement and job-site training; 4) follow-up. Job coaching begins when a DDSN bene…ciary is referred to a coach by her case manager. Following a referral, there may be period of instruction and assessment aimed at improving the client’s general job skills and awareness of community-based employment opportunities. Once a speci…c job has been identi…ed and a job coach assigned, the process is expected to last at least a year beginning with six months of on-site training followed by at least six months of followup in which the coach maintains monthly contact with the client. While independence and job stability are the goal, retraining and "follow along" may last for a year or more. Finding a good match, according to our discussion with o¢ cials in the program, is a big part of the coaching process. Bad matches result in rapid turnover. Our measure of 4

employment success, de…ned below, will be based on employment in the year following any receipt of job coaching services and will exclude employment for low pay or short duration. Jobs for low pay do not satisfy the policy objective of competitive employment in integrated settings (rather than sheltered workshops), and therefore we do not count them as successful program outcomes. Using a lag of job coaching status allows us to look at employment outcomes of stable nature and also controls for the possible endogeneity of job coaching status. In South Carolina, 38 local boards provide supported employment services to individuals with mental retardation, and the programs may di¤er by board, particularly before 2003, when statewide standards for supported employment were put into place. Job coaches must have a high school degree or equivalent and pass state law enforcement checks, but are often inexperienced and lack formal training. Larger boards may have a job coach supervisor, while smaller boards may be supervised by a day services director at the board who has many other non-employment related responsibilities. Larger boards may also have developed a network of employer contacts that enables good placements, while smaller boards are more dependent on the job development skills of the individual job coach and the community ties of board members. While boards may try to make job coaches available for everyone who would like one, only a fraction of working age adults served by the board receive job coaching in any year. Some families and individuals served by DDSN opt for non-vocational day services (including recreation and leisure activities) rather than job coaching. These options might be selected because the individual does not want to work, has had a unsatisfactory work experience, or the family is concerned about the logistics of employment which include planning for reliable transportation, a regular sleep schedule, and potential for unpleasant social experiences. The demand for supported employment services at each DSN Board is a function of the number of adults served by the Board, the reputation for success or failure that has developed, and the sta¤ support of the program. Some DSN Boards have a waiting list of 10-20 individuals at any given time and other Boards have a di¢ cult time recruiting participants. We do not have data on waiting lists, but o¢ cials at the DDSN tell us that waits between the referral and onset of supportive employment services have generally been declining over the period of our data. We do not directly observe the process by which individuals are allocated to job coaches. Selection into job coaching may be based on observable characteristics recorded 5

in the DDSN record available to us such as the DSN board identi…er or individual characteristics such as IQ, age and emotional or behavioral problems. Our empirical strategy must also allow for the possibility that there are unobservable individual characteristics that a¤ect both coaching and employment. We discuss this is more detail in Section 4. One individual factor we do not observe (but job coaches and individuals do) is whether employment will a¤ect disability bene…ts. Most adults with mental retardation are eligible and do receive SSI. Earnings from employment can result in lower SSI bene…ts if the individual’s adjusted earnings are su¢ ciently large. Most working individuals with mental retardation do not reach the substantial gainful activity (SGA) standard, which translates to full-time work (37.5 hours per week) at $6.53 per hour. Individuals with mental retardation who work competitively, with or without supported employment are usually eligible to maintain their Medicaid bene…ts which include health insurance and disability related services. Although the SSA has policies and procedures to encourage employment of people who receive SSI, the SSA is a complex system which requires some knowledge of the procedures and a substantial level of persistence to navigate. South Carolina Service Coordinators are assigned to every individual who is eligible to receive services for mental retardation and they assist individuals and families to understand their entitlements and navigate the system. In most cases when supported employment services are o¤ered to an individual, the …rst discussion focuses on the implications for their SSI bene…ts.

4

Data and Variables

The data consist of individuals in South Carolina who have mental retardation and are clients of one of the 38 disability boards in South Carolina at any time between years 1999 and 2005.1 To be included, an individual must be between 21 and 65 years of age (inclusive) during the year and have an IQ score above 26 and below 75. Individuals whose primary diagnosis is autism are excluded. Because there are very few individuals whose race is not identi…ed as African American or white in the data, these individuals are also excluded. Over all seven years, there are 62,826 person-year observations. Descriptive statistics for the sample are shown in Table 1. About half 1

The data are stripped of personal identi…ers and are part of an ongoing system of surveillance of employment. The employment surveillance system has university IRB approval.

6

(51 percent ) of the sample is African American, and just under half (46 percent ) of the sample is female. The average age and IQ are, respectively, 37.7 and 50.4. About 24 percent of the sample has some emotional or behavioral problems reported, and about the same percent live in a supervised setting. Table 1 also provides descriptive statistics separately for individuals who receive some job coaching and those who do not. On average, the job coached group consists of individuals who have higher IQ’s (54.6 versus 49.7) and who are younger (35.82 versus 38.09). Job coached individuals are also more likely to be African American (55 percent versus 51 percent ), male (56 percent versus 53 percent ), and have no emotional or behavioral problems (25 percent versus 19 percent ). Job coaching typically consists of 6 months of on-site training and at least 6 months of follow-up. Our goal is to see whether coaching enables the individual to continue working after the coach has left the job site (but may still be o¤ering continued support via monthly phone calls or visits). Hence we measure the e¤ect of job coaching in year t 1 on the probability of employment in the subsequent year t. Because this requires 2 years of observation, we can model employment outcomes for 6 years (20002005). We construct an (unbalanced) panel of employment outcomes that includes an individual in year t whenever his history is observed in the previous year. The one exception is that individuals who were not observed in the data in t-2 were included and classi…ed as unemployed in t-1. This group includes individuals who just turned 21 and were by de…nition excluded from our data. These individuals were excluded because they are eligible to stay in special education programs up to age 21; and they may be receiving job coaching in school and/or working but we do not observe this. Individuals who are not receiving any services from DSSN (including but not limited to job coaching) are also missing in year t-2, and while it is possible that some are misclassi…ed as being unemployed in t-2, the employed in this group would be unlikely to be receiving job coaching in t-1. Our data is measured in calendar years (January to January), so there is necessarily some imprecision in how we measure the timing of events. We construct additional samples to check the robustness of our results. The pooled cross section will include multiple observations from some individuals if they are unemployed in more than one year between 1999-2003. This means our pooled cross section may over-represent the substantial fraction of the sample who are never employed in our observation period. Hence, we construct a pooled cross section with 7

no duplicates in which individuals are only counted in the …rst year for which they are observed. This means that the employment status of this long-term unemployed group will be measured in the …rst eligible year (2001). To see if this has any e¤ect, we are planning to delete observations form later years and construct a true-cross section for employment outcomes for 2001. The descriptive statistics for the pooled cross sections with and without duplicates are given in Table 2. Table 1 : Descriptive Statistics by Job Coaching Status Pooled Cross-section Pooled Cross-section No Duplicates With Duplicates Got no Job Coach Got Job Coach Got no Job Coach Got Job Coach (N=9126) (N=772) (N=32,967) (N=2520) Variable Mean StdDev Mean StdDev Mean StdDev Mean StdDev Employment* 0.078 0.269 0.43 0.495 0.051 0.221 0.308 0.462 IQ score 50.6 12.8 53.8 12.0 50.0 13.1 53.9 11.5 EmBehPr* 0.227 0.419 0.202 0.402 0.235 0.424 0.193 0.395 Female* 0.473 0.499 0.452 0.498 0.482 0.500 0.481 0.500 Black* 0.512 0.500 0.557 0.497 0.506 0.500 0.574 0.495 Age 35.4 12.2 36.7 9.9 37.9 11.9 35.3 10.1 Age-squared 1400 949 1445 789 1578 960 1350 779 Supervised* 0.199 0.400 0.328 0.47 0.217 0.412 0.270 0.444 Unemp 6.150 1.952 6.138 2.001 6.893 2.003 6.715 2.094 Large board* 0.489 0.500 0.421 0.494 0.485 0.500 0.440 0.497 FTE ratio 0.014 0.007 0.017 0.009 0.013 0.007 0.016 0.009 Job coac.freq 0.151 0.078 0.193 0.079 0.135 0.074 0.175 0.075 * =Dummy variable. Takes values of 0 or 1.

Since supported employment is intended to facilitate stable employment in integrated settings (rather than sheltered workshops), we screen for employment in jobs with very low pay or very short duration. For the purposes of this study, employment is de…ned as earning at least $50 per week for 23 weeks or more (see, for example, Howarth et al., 2006; Pierce et al., 2003; Moran et al., 2002). Because our data does not di¤erentiate between on-going on-site coaching, follow-up contact, and any re-training that occurs if there are job changes, we utilize a bivariate measure of job coaching (some or none) in year t 1. About 15.5 percent of the sample is employed in any given year, but as shown in Table 2, this varies from a high of 20 percent in 2000 to a low of 11 percent in 2004. The overall labor market conditions worsen during the sample period with the average county unemployment rate rising from the lowest point of 3.82 percent in 2000 to 7.3 percent in the 2005. Mirroring these employment trends, the probability of receiving 8

job coaching also falls during the period, from over 16 percent receiving job coaching at the beginning of the sample to only 10 percent by the end. This decrease in job coaching may be attributed to tightening state budgetary constraints, but may also re‡ect better accounting of job coaching hours due to an increase in auditing e¤orts. The reduction in job coaching at the individual level is also seen when aggregated to the disability board level. Of those receiving any services from a given board, the percent receiving job coaching services has declined from 18 percent to 11 percent over the sample period.

Table 2. Means of Employment and Job Coaching Variables by Year 1999 2000 2001 2002 Job coached 0.16 0.16 0.17 0.15 (0.37) (0.36) (0.37) (0.35) Employed 0.15 0.18 0.20 0.16 (0.36) (0.39) (0.40) (0.37) Wages (if employed) 122.71 116.17 117.97 120.61 (64.97) (63.95) (63.72) (65.93) County unemployment rate 4.65 3.82 5.64 6.34 (2.55) (1.15) (1.75) (1.75) Percent job coached by the board 0.18 0.17 0.17 0.16 (0.07) (0.07) (0.08) (0.09) Sample size 8356 8691 7840 8812 *Standard deviations shown in parentheses

5 5.1

2003 0.14 (0.35) 0.18 (0.38) 121.51 (72.44) 7.17 (2.03) 0.15 (0.08) 9156

2004 0.12 (0.32) 0.11 (0.32) 112.52 (64.31) 7.29 (1.91) 0.13 (0.06) 9783

2005 0.10 (0.30) 0.12 (0.32) 115.84 (66.45) 7.32 (1.88) 0.11 (0.05) 10188

Model Latent Index Model

In this study we are trying to address several important policy questions. First, "What is the impact of job coaching policy on the employment outcomes of individuals with mental retardation in South Carolina?". Second, "Is there cream skimming or sorting into the job coaching by gain" and if we …nd that no such pattern exists, that there is no evidence for cream skimming, or even if there is evidence of cream skimming, we ask "Can we improve the program by imposing certain cream skimming schemes or by changing the "targeted" population?".

9

We can answer the …rst question by

estimating a simpler version of the latent index model described below. However, in order to answer the remanining questions we will need the knowledge of distributional treatment parameters which we will identify later by using assumptions of the structural factor model. We are closely following the model developed in Aakvik, et al.(2005). There are two latent indices in our model, one underlying the state of job coaching, the other the state of employment.

The assignment of job coaching or the decision

rule regarding job coaching (we can choose either of these interpretations in this case, it is as likely that job coaches are choosing whom to coach as the individuals choosing themselves whether or not they want to be coached) is as follows:

Ji

= Zi

Uti

Ji = 1 if Ji

0; Ji = 0 otherwise

The outcome variable is dichotomous, an individual is either employed or unemployed. We assume that a linear latent index generates the outcome for the job coached state

E1i = Xi

1

E1i = 1 if E1i

U1i 0; E1i = 0 otherwise,

and for the non job-coached state

E0i = Xi

0

E0i = 1 if E0i

U0i 0; E0i = 0 otherwise.

In above equations Zi and Xi are vectors of observed (by us, the econometricians) 10

random variables and Uti , U1i , and U0i are unobserved (again, by econometricians) random variables. Ji is the net utility gain from getting job coached. We de…ne E0i be the employment outcome with no job coaching and E1i be the employment outcome with job coaching. In this model E1i

E0i is the e¤ect of job coaching on employment.

We know several characteristics of the individuals like their gender, race, and age, some characteristics of the disability board they are registered with and the county they live in, such as unemployment rates, that can a¤ect their employment outcomes. Detailed information regarding the data is provided above in the data section. If the di¤erence E1i

E0i is constant across individuals after controlling for X’s, an

OLS estimation of a switching regression model is su¢ cient to identify the treatment e¤ect.

However, it is more likely that the e¤ect varies across treatment status and

maybe across individuals; individuals who use job coaching may have same unobserved characteristics that make them also more likely to be employed.

IV method is de-

veloped to handle selection bias. However, if individuals know more about the e¤ect of treatment on their employment probabilities and are selecting into job coaching not randomly but proportional to their "rewards" (the change in likelihood of employment resulting from job coaching) or if the job coaches are selecting individuals for treatment based on their expectations regarding their employment probabilities, we are facing a problem called essential heterogeneity and in this case IV method fails to identify the mean treatment e¤ect. In order to be able to do our policy analysis we need to …rst estimate the treatment e¤ect of job coaching on employment outcomes correctly, thus we will adress these econometric concerns …rst in our analysis. Throughout most of this paper we assume that there exists a variable that determines the treatment decision but does not directly a¤ect the outcome (exclusion restriction), unobserved random variable are independent of the observed variables (in11

strumental variable assumption), and both treatment and non-treatment is observed for all X’s(required for identi…cation). Aakvik et al, assume that the outcomes are de…ned for everyone and are independent across individuals so that there are no interactions among agents. In our setting this may not be true since the population we study is very close-knit. However, we stick with this assumption for now. Moreover, we assume that the program being evaluated is small, and quite limited in our case, that we do not need to worry about general equilibrium e¤ects. Following section gives a detailed explanation of the empirical strategy we follow.

5.2

6

Estimation Strategy

Using a one-factor structural model and an instrument for identi…cation(model is identi…ed even without the instrument due to the nonlinearities. However, inclusion of the instrument substantially increases the …t of the model), we estimate the distributional treatment parameters from above latent indices and generate gains from job coaching in terms employment probabilities. Comparing relative gains for job coached and not coached individuals we cannot reject the case of no sorting in terms of gain statistically. Even though, this is convenient for us as econometricians when evaluating social programs, we believe "target"ing or cream skimming can improve a program of this kind. We run several counterfactual simulations with di¤erent cream skimming strategies and analyse resulting employment outcomes for treated and non-treated individuals and tried to generate a "real" measure of success for this program compared to the one that is "perceived" through raw employment outcomes.

12

6.1

Structural Factor Model

We will assume the following structure for the error terms of the above latent index model:

where

i

Uti =

i

+ "ti

U1i =

1 i

+ "1i

U0i =

0 i

+ "0i

is the common unobserved factor. We assume following joint distribution for

the unobserved terms in this factor structure assuming access to an iid data2 as follows: 0

1

B "t C B C @ "1 A v N (0; I) "0

Given this, the latent index model can be rewritten as Ji = Z i Ji = 1 if Ji E1i = Xi 1 E1i = 1 if E1i E0i = Xi 0 E0i = 1 if E0i

i

+ "ti 0; Ji = 0 otherwise + "1i 0; E1i = 0 otherwise

1 i

+ "0i 0; E0i = 0 otherwise

0 i

2

Aarvik et al(1999) shows that normalization of the variances is not restrictive in a normal factor model like described above.

13

The likelihood function has the following form L=

N Z Y

Pr(JijZi; ) Pr(EijXi; Ji; ) ( )d ;

i=1

where Pr(Ji; EijXi; Zi; ) = Pr(JijZi; ) Pr(EijXi; Ji; ): Since is not observed, we need to integrate it out. We numerically approximate this integration over 100 draws from a standard normal distribution.

6.2

Treatment Parameters

Let denote the standard normal cdf and denote the standard normal pdf. Then the following is the general expressions for the mean treatment parameters produced by a [normal] factor structure model(Aakvik et al, 2005)

AT E

TT

TT

Z

(x) = E [ jX = x] = E [ jX = x; U t = u] df (u) Z = [ (x 1 + 1 ) (x 0 + 0 )] ( )d ;

(x; z; J = 1) = E [ jX = x; J = 1] = E [ jX = x; U t Zi ] Z 1 = E [ jX = x; U t = u] df (u) z ( p2 j ) Z [ (x 1 + 1 ) (x 0 + 0 )] (z j + ) ( )d ;

(x; J = 1) = E

and MT E

h

1 Z

( p2j )jX = x

i Ez

(x; u) = E [ jX = x; U t = u] =

R

Z

[ (x

14

[ (x

1

+

1

1

+

(Z

j

)

) (x 0 + 0 )] + ) ( )d jX = x 1

(x 0 + ( pu2 )

0

;

)] ( + u) ( )d

:

Since our model has a dichatomous outcome variable, these will reduce to MT E

(x; u) = Pr(E1 = 1jX = x; U t = u) = F1jt (x 1 ju) F0jt (x 0 ju); AT E

TT

Pr(E0 = 1jX = x; U t = u)

(x) = Pr(E1 = 1jX = x) Pr(E0 = 1jX = x) = FU1 (x 1 ) FU0 (x 0 );

(x; J = 1) = Pr(E1 = 1jX = x; J = 1) 1 = E FJ;1 (Z E FU j (Z j )jX = x

Pr(E0 = 1jX = x; J = 1) j; X

1)

FJ;0 (Z

j; X

0 )jX

=x ;

where Fjjt (ed jet ) = Pr(Uj ej jUt = et ) for j = 0; 1., FUt is the distribution of Ut ; and Ft;j is the joint distrbutions for the unobservables; Ft;1 = FU 1U t ; Ft;0 = FU 0U t . Note that AT E TT (x) (x; J = 1) = 0 if 0 = 1 = 0:

7

Estimation results

We have the data for all people registered with the board. We do not distinguish between applying for job coaching and receiving job coaching. Our conversations with the director of the program made it clear that anyone that apply (or recomended) for job coaching will receive it. Thus, our estimated treatment e¤cets are de…ned for the all registered MR individuals in the data.

15

Table 3:Employment and Job Coaching Probability Estimates Employment Job Coaching Not Job Coached Job Coached Constant -2.7692*** -0.7217 -7.3647*** (0.6103) (1.1838) (0.3723) female -0.2208*** -0.0872 -0.1048** (0.0576) (0.0941) (0.0571) black 0.1945*** 0.1107 0.1870*** (0.0600) (0.1025) (0.0585) Age 0.6643*** 0.4226 1.8915*** (0.2257) (0.4234) (0.1797) Age-squared -0.0894*** -0.0602 -0.2429*** (0.0297) (0.0535) (0.0223) IQ 0.0112*** 0.0017 0.0188*** (0.0032) (0.0045) (0.0023) Emotional Problems -0.2948*** -0.1064 -0.3401*** (0.0804) (0.1247) (0.0717) Supervised 0.2058*** 0.3543*** 0.6314*** (0.0930) (0.1327) (0.0700) Unemployment rate -0.0675*** -0.0694*** (0.0166) (0.0257) JobCoaching Frequency 5.0373*** (0.3892) 0.534* 0 (0.347) 0.060 1 (0.323) Number of Observations 9898 Log likelihood -5428.75 Standard errors in parentheses: * signi…cant at 10%; ** signi…cant at 5%; *** signi…cant at 1% Note that we cannot reject 1 = 0 = 0; that is, we caqnnot reject that there is no sorting on gains (ATT=TT) in this analysis. This shows that we have no evidence that those who bene…t the most from it are those most likely to participate in it. We also estimate the same model with the restriction that all factor loadings 1 and Fixing 0 equal to zero, since we …nd them to be insigni…catly di¤erent from zero. the factor loadings to zero imposes the restriction that the error terms are independent across equations, and thus does not allow for selection on unobservables related to the employment equations. The resulting estimates are similar to those reported in the 16

tables for the more general models with factor loadings estimated. However, even though estimated coe¢ cients do not change by muhc estimated treatment parameter values are altered signi…cantly. Moreover, the …t of the model to the data is slightly worse, thus we conclude that a model with unobservables on which agents select is more consistent with the data and even though we cannot reject that 1 and 0 are equal to zero, we assume they are not.

8

Estimated mean treatment parameters

Figure 1 gives the the MTE pro…le. The MTE parameter positive for everyone and is increasing in Ut.. Recall that higher values of Ut imply lower probabilities of program participation since they imply higher costs. Thus, in terms of unobservables, those most likely to participate bene…t the least from the program.

.4 .2 0 -.2

Marginal Increase in Probability of Employment

Marginal Treatment Effect Profile

0

.2

.4

.6

.8

1

Indifference value

meanmte hi

lo

This evidence is consistent with our estimates for ATE and for the e¤ect of treatment on the treated: AT E = 0:215

17

and T T = 0:207 Using the strategy introduced in Heckman and Vytlacil (2000, 2001, 2005), and Aakvik et al(2005) the e¤ect of treatment on the treated is an integrated version of MTE with most of the weight being placed on MTE values with small U t values who are more likely to participate in the program. ATE weights MTE more uniformly, by the generated weights from the data and accordingly is larger.. Figure 1 also tells us the estimated treatment e¤ect vary substantially with observed characteristics. The degree to which the treatment e¤ect varies with observable characteristics can also be seen by studying the marginal e¤ect of each observable characteristic on the expected treatment e¤ect. Table 4 reports the marginal e¤ects of observable characteristics on Marginal Treatment Parameters. Table 4: Marginal E¤ects of Observables h i on MTE @E(4jX=x) Ex @xK female -0.007 black 0.015 age 0.064 age2 -0.01 iqscore 0.0004 emotbehprobx -0.007 supervised 0.079 unemploy -0.013

8.1

Cream-skimming: the relationship between selection into the program and outcomes

Our main question is whether those who bene…t the most from it are those most likely to participate in it. We have already noted that ATE is greater than TT; i.e., that randomly selected persons bene…t more from the program than those who participate in it. This suggests that the combinations of Ut and Z 0 values that promote program participation are perversely associated with the observed and unobserved factors associated with gains from the program. In order to determine the extent of cream-skimming on both observables and unobservables, it is necessary to relate the treatment status observed and unobserved characteristics that determine treatment probability. Given the factor structure model, we can easily determine how variation in Ut a¤ects U1 and U0 . Using our normalizations 18

Cov(U1 ; U0 ) = Cov(Ut ; U1 ) = Cov(Ut ; U0 ) =

0 1 0

= 0:032 = 0:06 = 0:53 1

Given this model on unobserved random variables we are interested in following correlations 0

Corr(U1 ; U0 ) = p

1+

2 1

p1

1+

1 Corr(Ut ; U1 ) = p p 2 1+

0 Corr(Ut ; U0 ) = p p 2 1+

2 1 2 0

2 0

= 0:028

= 0:042 = 0:33

(We should talk about these) From the …rst correlation, we have that the unobservables determining employment status in the no-training and training states are are very weakly correlated. The point estimate is positive but quite small; this is in line with our failire to …nd sorting in gains. From the latter two correlations, the unobservables that promote participation are also positively correlated with the unobservables that promote employment in training and nontraining status .Thus, higher Ut is associated with both higher U0 and U1 so that persons with low values of Ut (who are more likely to participate in the program) are more likely to have lower values of " employment gains"; holding constant X and Z: Hence, selection is perverse on unobservables: treatment e¤ects are the lowest for those most likely to participate.

9

Policy Simulations

Now we determine what is going on in our data, using our structural model we can analyze if we can improve the program by changing the way the supported employment participants are determined. We want to simulate several di¤erent allocations of job coaching to individuals to see whether a more targeted approach would increase the e¤ectiveness of the program and if so by how much. We consider four alternative 19

schemes for allocating job coaches to individuals: 1) Random assignment 2) Coach the ones most employable if coached 3) Coach the ones most employable if not coached 4) Coach the ones with highest gains. We as econometricians can only observe the observed characteristics but individuals within the system of supported employment, oth providers and clients may also be aware of the unobservables. Thus we consider selection both on observables and unobservables for the last 3 countercatual allocations. We also analyse how an increase in the number number of job coached individuals will e¤ect the employment probabilites in the data. The random lottery assignment serves as our baseline for measuring the gains attributable to the current regime. How much better is the current allocation than a hypothetical random re-assignment of all existing job coaches across DDSN clients served in the state? Note that we allow for reassignment within a board and across boards, allowing us to capture the employment gains produced by the allocation of coaches by boards and (via funding choices) to boards. We next consider allocating coaches based on an individual’s potential for employment if coached. Case workers and job coaches have had opportunity to observe the outcomes of many individual-level coaching “experiments’and may use their experience to guide coaches to those who are most likely to attach to the labor force following coaching. Whether these individuals are also the ones who have the most to gain from the program is an empirical question. For completeness, we also consider targeting individuals who are the most likely to be employed without a coach. Caseworkers could potentially select these “easy” cases as the ones to target. Finally, we consider perfect targeting on gains. There are likely to be factors that predict coaching gains that caseworkers can observe that we, the econometricians, cannot. By assuming perfect knowledge of potential gains, including gains due to “unobservables”that we can estimate in our model but do not observe, we obtain the limiting case for the possible gains from improving targeting e¢ ciency. To compare the results of these alternative job coaching allocations, we simulate the employment outcomes under each and compare the aggregate employment. Reallocating job coaches may cause aggregate employment to rise or fall depending on the relative e¢ ciency of the current allocation system relative to the alternative. To better understand where the relative gains and losses are, we also disaggregate gains 20

from re-assignment to the actual participants and non-participants. How many of those job coached would be re-assigned? How would employment of those who are currently coached change? An example helps to explain why this matters. Suppose that (credit to the Roy model) Bob’s marginal bene…t from coaching is high but his employment probability if coached is still low. Joe is likely to be employed with or without the program. Now compare the possible outcomes of sorting on employability to sorting on gains. Sorting on employability makes job coaching look very successful: Joe is coached and likely to be employed while Bob is not coached and likely to be unemployed. Sorting on gains will allocate the coach to Bob rather than Joe, and the job coached will not appear to be very successful compared to those who are not coached but the overall employment level will be higher. The third metric we consider is the marginal gain in employment from increasing the number of job coaches by 1%. Our procedures for the simulations are as follows. For the lottery we assign a random draw from a uniform distribution to each person and assign job coaching to the ones with the highest 772 draws. For assignments that are based on treated and untreated employment probabilities and the gains, we …rst construct a simulated sample with our estimates with 100 random draws for each individual, then we calculate the marginal treatment e¤ect for each unique X for each preset value of Z :averaging the changes in employment probabilities over s: Once the MTEs are generated we weigh them according to our simulation schemes in order to generate the desired summary stats. To calculate the random lottery assignment, we assign a draw from the uniform to each individual, rank by the lottery number, and then re-assign the job coaches to those with the highest lottery numbers.

21

Table 5: Policy Simulations Employment Rate Employment Rate Employment Rate for Untreated for Treated Data Estimated Simulation 1 Simulation 2 Simulation 3 Simulation 4 Simulation 5 Simulation 6 Simulation 7 Simulation 8

Supported Employment(SE) Assignment rule for Simulations: S1: Lottery S2: highest gain in probability of employment in observables S3: highest gain in probability of employment observables and unobservables S4: highest probability of employment after SE in observables S5: highest probability of employment after SE in observables and unobservables S6: highest probability of employment prior to SE in observables S7: highest probability of employment prior to SE in observables and unobservables S8: 1% Increase in number of job coaches

10

Conclusion

References [1] Aakvik, A., Heckman, J., Vytlacil, E., (2005). Estimating treatment e¤ects for discrete outcomes when responses to treatment vary: an application to Norwegian vocational rehabilitation programs, Journal of Econometrics V.125 pp.15–51 [2] Anderson, K.H., Burkhauser, R.V., Raymond, J.E., 1993. The e¤ect of creaming on placement rates under the jobtraining partnership act. Industrial and Labor Relations Review 46 (4), 613–624.

22

[3] Basu, A. J. Heckman, S.Navarro-Lozano, Sergio Urzua(2007), "Use of instrumental variables in the presence of heterogeneity and self-selection: An application in breast cancer patients", unpublished manuscript [4] Bassi, L.J., 1983. The e¤ect of CETA on the postprogram earnings of participants. Journal of Human Resources 18 (4), 539–556. [5] Baum, C.F., Scha¤er, M.E., Stillman, S. 2007. ivreg2: Stata module for extended instrumental variables/2SLS, GMM and AC/HAC, LIML and k-class regression. http://ideas.repec.org/c/boc/bocode/s425401.html [6] Bjorklund, A., Mo¢ t, R., 1987. The estimation of wage gains and welfare gains in self-selection models.Review of Economics and Statistics 69, 42–49. [7] Blanck, P.D., Americans with Disabilities Act and the Emerging Workforce, Washington, DC: American Association on Mental Retardation, 1998, pp. 3-10. [8] Heckman, J., 1997. Instrumental variables: a study of implicit behavioral assumptions used in making program evaluations. Journal of Human Resources 32, 441– 462. [9] Heckman, J., Honore, B., 1990. The empirical content of the Roy model. Econometrica 58, 1121–1149. [10] Heckman, J., Hotz, J., 1989. Choosing among alternative methods of estimating the impact of social programs: the case of manpower training. Journal of the American Statistical Association 84, 862–874. [11] Heckman, J., Navarro, S., 2004. Using matching, instrumental variables and control functions to estimate Heckman, J., Vytlacil, E., 1999. Local instrumental variables and latent variable models for identifying and bounding treatment e¤ects. Proceedings of the National Academy of Sciences 96, 4730–4734. [12] Heckman, J., Vytlacil, E., 2000. The relationship between treatment parameters within a latent variable framework. Economic Letters 66, 33–39. [13] Heckman, J., Vytlacil, E., 2005. Econometric evaluations of social programs. In: Heckman, J., Leamer, E. (Eds.), Handbook of Econometrics, Vol. 5. North-Holland, Amsterdam. 23

[14] Heckman, J., Lochner, L., Taber, C., 1998c. General equilibrium treatment e¤ects: a study of tuition policy. American Economic Review 88 (2), 381–386. [15] Heckman, J., LaLonde, R., Smith, J., 1999. The economics and econometrics of training programs. In: Ashenfelter, O., Card, D. (Eds.), Handbook of Labor Economics, Vol. III. North-Holland, Amsterdam. [16] Mank, D., Cio¢ , A. and Yovano¤, P., Employment outcomes for people with severe disabilities: Opportunities for improvement, Mental Retardation 36 (1998), 205216. [17] McGaughey M., W.E. Kiernan, L.C. McNally and D.S. Gilmore, A peaceful coexistence?State MR/DD agency trends in integrated employment and facility based services, Mental Retardation 33 (1995), 170-180. [18] Mo¢ t, R., 2001. Estimation of Limited Dependent Variable Models with Dummy Endogenous Regressors: Simple Strategies for Empirical Practice: Comment. Journal of Business and Economic Statistics 19, 20–23. [19] Rusch, F. R., & Braddock, D. (2004). Adult day programs versus supported employment (1988–2002): Spending and service practices of mental retardation and developmental disabilities state agencies. Research and Practice for Persons with Severe Disabilities, 29, 237–242. [20] STATA [21] Test, D. W., Carver, T., Ewers, L., Haddad, J., & Person, J. (2000). Longitudinal job satisfaction of persons in supported employment. Education and Training in Mental Retardation and Developmental Disabilities, 4, 365–373. [22] West, M. D., Wehman, P. B., Wehman, P. (in press). Competitive employment outcomes for persons with intellectual and developmental disabilities: The national impact of the best buddies job program. Journal of Vocational Rehabilitation. [23] Wehman, P., Revell, W. G., & Brooke, V. (2003). Competitive employment: Has it become the "…rst choice" yet? Journal of Disability Policy Studies, 14 (3), pp. 163-173. 24

[24] Yamaki, K., & Fujiura, G. T. (2002). Employment and income status of adults with developmental disabilities living in the community. Mental Retardation, 40, 132–141.

25