Journal of Social Research & Policy, Vol. 5, Issue 2, December 2014

Validation and Predictive Ability of the North Carolina Family Assessment Scale for the Intensive In-Home Visitation Program in Kentucky RAMONA F. STONE1 University of Kentucky Lexington, KY, USA

GERARD M. BARBER University of Louisville Louisville, KY, USA

SARAH HENDRIX Union College Barbourville, KY, USA

Abstract The Community Collaboration for Children (CCC) program is a complex initiative implemented in Kentucky by a network of state and non-governmental agencies that provide intensive in-home services to families at risk for child abuse and neglect. The primary focus of this program is to maintain children who are at risk of being removed from their family, in their own homes while supporting and building family strengths in areas such as safety, stability, and interaction skills. The program’s evaluation research design is longitudinal, data was collected quarterly by the services providers, social and child workers. This paper focuses on the validation of the North Carolina Family Assessment Scale (NCFAS), using the intake data collected from 1959 families who participated in the CCC intensive in-home services during July 1, 2006 through December 31, 2009. NCFAS is a practice tool utilized by service providers to assess families on five domains: environment, child wellbeing, family interaction, family safety and family capability as related to child wellbeing. The factors extracted using two approaches - general factor analysis and a congeneric, single-factor analysis- were used to test the predictive ability of each subscale using logic regression analyses, while controlling for the intensity of in-home visitation services. Both factor analysis approaches yielded valid and reliable results. Of the five NCFAS domains, the family interaction was the strongest predictor of case outcome, assuming that families are provided with 11 to 20 hours of services.

Keywords: Family functioning; Family safety; Child wellbeing; Family preservation; Family permanency; In-Home visitation; Scale validation; Reliability.

Introduction In the United States, state government agencies have the mission to deliver quality services that enhance the health, safety, and wellbeing of their children. To serve this mission, family preservation programs are implemented in all states and include primary, secondary and tertiary prevention programs funded with federal, state and local support. Family preservation programs are based on the Homebuilders model developed in Tacoma, Washington during the 1970s. 1

Postal Address: Department of Health Behavior, College of Public Health, University of Kentucky, Bowman Hall Suite 333, 152 Washington Ave, Lexington, KY 40536. E-mail Address: [email protected]

6 | JSRP

Ramona F. Stone, Gerard M. Barber, Sarah Hendrix

In Kentucky, the first family preservation program was implemented in 1985 as a pilot, with private foundation funding; in 1990 the state legislature passed the Family Preservation Act (FPA) and allocated funding for 47 out of the 120 counties to establish family preservation programs. Services offered under the FPA were intended to be a short-term, intensive, and acute intervention services to prevent removal of children from their homes (Westat, 2002). Over time the family preservation program services grew, as other sources of funding became available. One such program is the federally funded community-based initiative to prevent and reduce the child abuse and neglect in Kentucky, called the Community Collaboration for Children (CCC). CCC is primarily focused on secondary prevention services that were provided by contracted agencies across the state. Secondary prevention programs are designed for the families identified to be at risk for child abuse and neglect, but not yet substantiated, while primary preventions focus on the population at large and tertiary ones on the families with substantiated abuse and neglect. This article reports on data collected for the evaluation of the CCC’s intensive In-Home Services herein referred as IHS, a secondary prevention program, utilized when children are at risk to be removed from their homes and placed with relatives, in kinship care or in out-ofhome care. Thus, the permanency goal for the families enrolled in this program is to maintain the children with their parents. The IHS program is open to families who self-refer or families may be referred by a variety of community agencies including churches; however, about 80% were referred by the Kentucky Department of Community Based Services (DCBS) within the Cabinet for Health and Family Services (CHFS). Families referred by the DCBS are expected to participate in order to avoid removal of their children from the home. Once a referral was made, regardless of the source, a home visit is scheduled within five working days. The focus is on a comprehensive treatment plan intended to address the family’s practical and material difficulties, together with their behavioral problems or mental health needs. The IHS program itself entails a series of home visits during which the family and the local service providers work together toward identifying and addressing the family needs and reach the goal to maintain the children in the home. The timing of these home visits is based upon the needs of the families and at the discretion of the program provider. The IHS services seek to develop, support, and empower the family by teaching positive child development practices and problem solving skills, as well as by assisting parents and coordinating available community resources. Services combine skill-based intervention with maximum flexibility so that they were available to families according to their unique needs. Service providers teach the families how to live together safely, while addressing their immediate basic needs, refer families for community resources and counseling programs as needed, and may also include parent education intervention among their services. Parent education, when available, follows an established, nationally recognized, research-based curriculum. The qualifications of the service providers include 30 or more college hours in a human services area or a completed high school education with one year of experience providing similar services. They are expected to advocate for the best interest of the child, to develop specific goals with the family, and to guide them in locating more specialized resources as needed. The primary focus of home visits is to provide individualized parent education through the use of mentoring and coaching techniques. Topics include child development, age appropriate behavior, communication skills building, mutual trust, and increasing self-esteem and confidence in parenting. In working with the family over time, the service provider has the opportunity to observe parents’ and children’s behavior in the home, to understand the underlying issues and to help them to successfully maintain the children in the home. Background From 1992 through 2010, the Cabinet for Health and Family Services (CHFS) contracted with the University of Louisville to develop and implement an evaluation plan tailored to address each component of the Community Collaboration for Children (CCC) program. It was funded by the federal Community-Based Child Abuse Prevention Program (CBCAP) and authorized

Validation and Predictive Ability of the North Carolina Family Assessment Scale …

7 | JSRP

by the Title II of the Child Abuse Prevention and Treatment Act Amendments of 1996 and reauthorized in 2010. The CCC program in Kentucky went through four phases: a startup period (Phase I) when decisions regarding the types of programs to be funded with CCC funds were made by the local Area Development Districts (ADD). The program evaluation design during this phase was qualitative (Stone and Barber, 2008). After the Adoption and Safe Families Act (ASFA) was signed into law in 1997, the CHFS stipulated what types of programs or services would be provided to families to help the state achieve the federally mandated outcomes required by the new policy (Phase II) and to ensure that the agency made reasonable efforts to prevent unnecessary out of home child placement. As a result, although the programs funded within the CCC network during Phase II continued to vary greatly across the state, the better defined outcomes and the more rigorous program expectations lead to the redesign of the CCC program evaluation. Prior to 2004, all CCC programs reported client information in an aggregate format and the service definitions varied greatly from one agency to another. These variations made it difficult to compare programs across the state and to measure changes in individual family outcomes (Stone & Barber, 2008). During the contract years of 2004-2006 (Phase III), the CHFS and the Service Regional Administrators (SRA) made even stronger recommendations about the services needed to meet the ASFA mandatory goals. More emphasis was placed on family team meetings, intensive inhome services (secondary prevention), and supervised visitations (tertiary prevention). The intent was to work with fewer families and bring about specific changes that were consistent with CHFS’ outcome priorities as mandated by ASFA. By funding a limited, but more focused number of services, and by stressing the importance of comparing program performance measures and of child and family outcomes, the CCC evaluation developed comparative performance measures and implemented systematic digital data collection processes. The evaluation continued to rely on qualitative data, but it added a quantitative component. Standardized reporting measures for monitoring program outputs, such as service provision (e.g., number of home visits, number of hours of home visits completed), family and child outcomes (e.g., safety, wellbeing, environment, parental capabilities, family interactions), and children’s placement or permanency status were also collected. The data collection instruments were further improved during Phase IV for the 2006-2010 contracts. Specifically, the data collection burden was reduced by eliminating fields that were confusing or yielded a large number of cases with missing information. During Phase IV, more focused and more accurate information on CCC program performance and child and family outcomes were collected. Identifying information was collected so that the CCC data could be matched with the child welfare administrative records from The Worker Information SysTem (TWIST); new confidentiality and security procedures for data storage and reporting were set and approved by the Institutional Review Boards (IRB) at the CHFS and at the University of Louisville. The purpose of the IHS research evaluation was to determine whether the services were provided to the intended population, in the intended quantity (number of home visits) and with the intended intensity (number of hours per visit), and to evaluate the outcomes of the program. The evaluation plan was also developed in conjunction with the CHFS technical assistance and with direct feedback from service providers. This paper stems from the quantitative research evaluation of the intensive in-home services provided in Kentucky by a network of state and private non-profit agencies. It focuses primarily on the validation of the North Carolina Family Assessment Scale (NCFAS) as a family functioning assessment practice tool Methods and data Methods A quasi-experimental design, with quarterly data collection points over a 42-month time period was undertaken; data were collected quarterly from the service providers, for 1959 families who enrolled in the program between July 1, 2006 and December 31, 2009. At a minimum,

8 | JSRP

Ramona F. Stone, Gerard M. Barber, Sarah Hendrix

providers submitted a baseline assessment for each family completed at the time of intake into the program and a closure assessment for all cases that were closed by the end of the study period. A case was noted as closed when the case worker’s assessment indicated that the child or children were safe of abuse and neglect (successful closure), or when the assessment indicated that the child needed to be removed and placed in out-of-home care (unsuccessful closure). Each service provider collected data with a standardized assessment of family functioning for program evaluation. The standardized instrument utilized is known as the North Carolina Family Assessment Scale (NCFAS). The NCFAS validation was tested with the intake data using two factor analysis methods: the general factor model and, its special case, the single-factor model known as the congeneric factor analysis. The factors extracted with the two approaches were saved as regression variables and were used in the bivariate and multivariate analyses. Data collection The data collection forms included identifiers, such as names and social security numbers, necessary to link the CCC data overtime and to match it with the TWIST CHFS administrative child welfare data. However, once the data was matched and unduplicated, the identifiers were deleted from the dataset that was used for the validation and predictive analyses. The data was collected over 14 quarters from the service providers who completed the forms for each family. The intake form was completed immediately after the first home visit. At the end of each quarter, for as long as the family was active in the program, the service provider continued to submit an assessment form, until the case was closed and thus, a closure form was submitted instead. In rare cases, families were referred to the program for more services. On average, the time span between case intake and case closure was about six-weeks. The service providers reported that the time spent on completing a quarterly reporting family assessment form was about 30 minutes. Data Items The assessment forms had three main sections: 1) Socio-demographic characteristics; 2) Services provided, including the number of visits and number of hours provided while the family was active in the program, and 3) Family functioning assessment measured by the North Carolina Family Assessment Scales (NCFAS). The demographic section collected data about each family member, and included questions about physical and mental disability. The services provided to families over multiple quarters were summated with a focus on changes in the family functioning assessment measures. The outcome of primary interest was the family’s permanency status at case closure which was measured with a dichotomous variable. Code “1” was assigned to closed cases successful in maintaining their children in their home and a “0” code was assigned to cases that were either closed due to the children being removed from the family, or due to the inability to provide services (e.g. parents’ lack of cooperation, lost contact, moved out of the area) due to the family dropping out of the program. In some cases, the children had different permanency statuses; for instance, children ages 16 or older could have requested consideration for emancipation, which does not assume forceful removal from the family. Thus, if any of the children were maintained in the family, the case was coded as successful. The family functioning assessment was measured with the NCFAS scale, well known and widely used in the United States. Developed by two researchers from North Carolina, Drs. Raymond S. Kirk (University of North Carolina at Chapel Hill) and Kellie Reed-Ashcraft (Appalachian State University) (Kirk & Reed, 2000) as a case practice tool for effective service planning and goal setting (Kirk, 2012; Kirk & Reed-Ashcraft, 2004; Kirk & Griffith, 2007), the NCFAS is a multidimensional scale with five domains: Environment (E), Parental Capabilities (PC), Family Interactions (FI), Family Safety (FS), and Child Well-being (CW), which allow for a comprehensive assessment of family functioning. For each domain a series of items were developed to form a sub-scale. The NCFAS scale was validated for use by family preservation programs, for clients with different levels of risk - including high-risk families, which generally struggle with multiple inter-related problems.

Validation and Predictive Ability of the North Carolina Family Assessment Scale …

9 | JSRP

One of the reasons this scale was chosen, besides its proven validity and reliability with this population, is that it does not require that the person who administers it has a minimum degree or license. The original NCFAS has 39 items; four items, irrelevant to IHS or impossible to collect, were eliminated. Each of the sub-scales has 5 to 10 items, measured on a six-point ordinal scale ranging from -3 (serious problem) to the 0 point (baseline or adequate) to +2 (clear strength). Note that the “overall” item of each sub-scale is not an average of the domain items, but rather an overall worker assessment of the focus of the respective domain (e.g., overall environment, overall family interactions, etc.); workers were asked to assess the overall score after they assessed all other items on the scale. This article utilizes the NCFAS data collected at the time of program enrollment (intake), the case status at closure successful (when children maintained with parents) vs. non-successful (when children were placed with relatives, kinship care, or in out-of-home care), and the total number of hours and home visits provided during the time the family was active in the program (summated hours or visits recorded in the quarterly forms available for each family). Data Analyses Data analyses plan included descriptive statistics of socio-demographic and service measures for the entire caseload and for the cases that completed the program successfully. The intent was to identify any variables that could help set apart the successful cases within the caseload from the unsuccessful ones, at the time of intake. Then, NCFAS scale validation was conducted using a general factor model and a congeneric factor model; the extracted factors were saved as regression variables. Next, t-tests were used to identify statistically significant differences in the regression factors between the successful and the unsuccessful cases. The factors that were significantly different between the successful and the unsuccessful cases were used as predictors of the case outcome using logistic regression models. Finally, a Pearson’s correlation analysis of the extracted factors provided insights regarding the scale’s concurrent validity. The data management and the data analyses were conducted with IBM SPSS 22. Results Descriptive Analyses During the 42-month study period 1959 families were enrolled in and provided with CCC intensive in-home visitation services and 1741 of the 1959 cases, or 88.9% of all cases were closed during the study period. As shown in Table 1, the 1959 CCC families included a total of 7220 clients of which 3006 were adults and 4214 were children. The majority of the children (77.5%) were age 12 or younger with 38.9% age 5 or younger, 38.6% were ages 6 to 12 and 22.4% were teenagers. Thirty five and a half percent (35.5%) of the families had three or more children. Table 1 displays additional information regarding the family structure and sociodemographic characteristics of the families served by the CCC intensive in-home visitation programs. Overall, the majority of the caseload (82.6%) had a female head of household (HOH), and almost half of the families (49.6%) were single-adult households. The average age of the head of household was 34, with 70% ranging between ages 20 and 45. About a fifth of the caseload (19.6%) were minority (non-Caucasian) families. About a quarter of the caseload had at least one adult with a diagnosed physical or mental disability and 23.9% of the families had at least one child with one or more disabilities. Overall, in the CCC programs there were 762 families (38.9%) with at least one member (adult and/or child) with a physical or mental disability.

10 | JSRP

Ramona F. Stone, Gerard M. Barber, Sarah Hendrix

Table 1: Caseload Characteristics

Cases Closed Demographics Adults Children Ages 0 to 1 Ages 2 to 5 Ages 6 to 12 Ages 13 to 18 Households Single adult Minority Female HOH Age HOH (𝑥̅ , σ) Adult with Disability Child with Disability Anyone with Disability With 1-2 children 3-4 children 5+ children Services Home visits scheduled Home visits completed Hours completed <1 hour 1-10 hours 11-20 hours 21-30 hours 31+ hours Services (𝑥̅ , σ) Home visits completed Hours completed Family Assessment (adequate or better) Environment Parental Capabilities Family Interactions Family Safety Child Well-Being

Intake /All Caseload N % 1959 100.0 1741 88.9

Successful Cases N % 1202 61.4 1202 100.0

3006 4214 480 1159 1627 948

100.0 100.0 11.4 27.5 38.6 22.5

1869 2568 286 705 1009 568

100.0 100.0 11.1 27.5 39.3 22.1

971 383 1618 34 478 469 762 1263 590 106

49.6 19.6 82.6 14.4 24.4 23.9 38.9 64.5 30.1 5.4

564 219 975 34 293 309 479 783 359 60

46.9 18.2 81.1 10.8 24.4 25.7 39.9 65.1 29.9 5.0

21103 17710 29518 68 750 724 239 178

100.0 83.9 100.0 3.5 38.3 37.0 12.2 9.1

14285 12474 21161 11 325 556 183 127

100.0 87.3 100.0 0.9 27.0 46.3 15.2 10.6

9.19 15.46

7.51 13.54

10.43 17.75

7.73 13.38

943 780 1072 505 724

48.1 39.8 54.7 25.8 37.0

625 522 704 324 485

52.0 43.4 58.6 27.0 40.3

Note: 𝑥̅ = mean, σ=standard deviation; †%successfully closed of all cases; closed of all closed cases

‡

% successfully

During the study period, the IHS programs provided 29518 hours or about 15 hours/case (standard deviation of 13.5 hours) and 17710 in-home visits to 1959 cases, or an average of about 9 visits per family (standard deviation of 7.5 visits). There were 68 families or 2.6% who received less than one hour of services and were referred for other community services; 38.3% or 750 cases received 1 to 10 hours of in-home services; 37% (724 cases) received 11 to 20 hours; and, 21.3% (417 cases) received 21+ hours of in-home services. The bottom of Table 1 shows the number and the proportion of cases that scored adequate or better on the NCFAS tool at intake. Recall that each NCFAS item ranges from -3 (serious problem) to the 0 point (baseline or adequate) to +2 (clear strength). Although the adequate or better scores were intended to identify the issues on which the family did not need an intervention (Kirk, Kim and Griffith, 2005), in reality service providers used the “0” baseline/adequate score as a neutral point, for situations in which they could not assess the

Validation and Predictive Ability of the North Carolina Family Assessment Scale …

11 | JSRP

family functioning on a specific item. It was noted that family safety items were especially difficult to assess at intake any other way than with a “0”. Majority of families were assessed to have adequate (0) or better (1, 2) score on the family interaction domain, and 48.1% on the environment domain. These percentages were comparable with NCFAS literature (Kirk, Kim & Griffith, 2005) for the environment and child wellbeing domains, but the IHS program had a higher percentage of adequate/better scores on parental capability and family interactions, and a lower percentage on the family safety domain. Further, at right, Table 1 shows the patterns of service provision for the successfully closed cases, the families that successfully maintained their children in the home. Of the 1741 closed cases, 1202 or 69% were successful in maintaining their children in the home; this means that 61.4% of the original IHS caseload maintained the children with parents. A majority of the CCC children that removed from their home were placed with relatives or in kinship care and of the 531 closed but unsuccessful cases, only 25 cases were cases of children removed from the family and placed in out-of-home or foster care. Successfully closed cases were very similar in their demographics and family structure with the overall caseload, but there were differences in the family functioning intake scores and in service provision. The proportion of families with adequate or better family functioning scores at intake were slightly higher for successfully closed cases than for the overall caseload. These differences in percentage points seem small, but a series of χ2 tests show that all but family safety sub-scale (p=.134), were statistically significant (p<.001). The proportion of completed visits of all scheduled visits was significantly higher for the successful cases than for the unsuccessful cases. Families with multiple and complex problems are very hard to serve because they have a greater tendency to reschedule, cancel or miss appointments, which makes it even more difficult to provide them with social services. Overall, about 13% of the scheduled visits were missed or cancelled by parents in the unsuccessful cases, 5% of the visits were rescheduled, and 20% of the scheduled and/or rescheduled visits were attempted by the CCC staff could not be completed due to parents missing the appointment. The two measures of service provision, number of home visits and number of hours of visits completed, were highly correlated with each other (R=.850, p<.001); but, the total hours of visitation completed was more strongly associated with the outcome (χ2 (2)=206.5, p<.001) and thus, a recoded version of it was used in the logistic regression analyses. NCFAS Validation The NCFAS scale validation with the Kentucky’s CCC IHS population was conducted using both a general factor analysis (Reed-Ashcraft, Kirk & Frasier, 2001; Hattie, 1985; DeVellis, 2003) and a congeneric factor analysis (Hattie, 1985; DeVellis, 2003). Principal Axis Factoring (PFA) extraction and Varimax rotation (Reed-Ashcraft, Kirk & Frasier, 2001; Kirk & Reed, 2000; Kirk & Reed-Ashcraft, 2004; Kirk & Griffith, 2007; Kirk, 2012; Kirk, Kim & Griffith, 2005) seeks to explain the common variance in the set of items (Hair et al., 1998; Velicer, Eaton & Fava, 2000; Tabachnick & Fidell, 2007; Ledesma & Valero-Mora, 2007; DeVellis, 2003) measuring family functioning. Varimax rotation maximizes moderate and high correlations between items, and minimizes the low correlations (Tabachnick & Fidell, 2007), to achieve the best orthogonal solution. Table 2 shows the number of cases included in the analyses, the number of items loading on each factor, the reliability coefficient, Cronbach’s α, and the proportion of variance explained for the published model, general factor analysis model and for the congeneric factor analysis model. The published model data originates from several sources, and it allowed a comparison of the CCC IHS models to other NCFAS validation results; the cutoff point for the published models was |.50| (Reed-Ashcraft, Kirk & Frasier, 2001). The published model had four factors (Table 3) extracted with PAF and Varimax rotation; the reliability coefficients varied from 0.767 for family safety sub-scale to 0.922 for the environment sub-scale (Kirk & Griffith, 2007) indicating “good” to “very good” levels of reliability (DeVellis, 2003). The overall proportion of variance explained was 61.3% with the environment explaining the largest proportion of variance (20.1%), followed by the child wellbeing (16.5%), family interaction (14.2%) and ending with family safety (4.2%); none of the parenting capabilities items loaded on any factors.

Ramona F. Stone, Gerard M. Barber, Sarah Hendrix

12 | JSRP

The average Corrected Item-Total Correlation (CITC), a coefficient used to estimate a scale’s convergent validity, varied between α=.577 for the environment sub-scale to α=.776 for family safety sub-scale; it is desired to be 0.6 or above. The overall proportion of variance explained by the general factor analysis model was 69.1%. The largest proportion of variance was explained by the environment sub-scale (40.5%), followed by the family interaction (15.8%), and by child wellbeing (6.5%) and family safety (6.3%). Table 2: Comparisons of Scale Statistics Model

Domain

Published models Overall Environment Child Well-Being Family Interactions Family Safety Parental Capabilities General Factor Analysis Overall Environment Family Interaction Child Wellbeing Family Safety Parental Capability Congeneric Factor Analysis Environment Family Interaction Parental Capability Family Safety Child Wellbeing

N

Items

α

% Variance Explained

n/a 1188 979 881 1036 1187

n/a 10 8 5 6 7

n/a .922 .801 .771 .767 .814

61.31 20.13 16.48 14.24 4.70 n/l

1616 1616 1616 1616 1616 n/l

18 8 4 3 3 n/l

.911 .899 .880 .807 .884 n/l

69.089 40.471 15.780 6.550 6.288 n/l

1877 1374 1881 1430 1311

9 5 5 6 6

.903 .871 .847 .859 .903

57.098 66.862 62.374 59.046 67.391

(n/a= not available; n/l = no loadings; α from Kirk and Griffith, 2007; %Variance explained from Reed-Ashcraft, Kirk, and Frasier, 2001, with N=288) General Factor Analysis: PAF with Varimax rotation was conducted with the 35 items collected at the time of IHS intake, using a cutoff point of |0.40| (Lance, Butts & Michaels, 2006; Petkov, Harvey & Battersby, 2010). The validation process included an iteration of factor and reliability analyses until the best solution for the data was found. The overall reliability coefficient was .911 (Table 2); the sub-scale reliability coefficients varied between .807 for the child wellbeing sub-scale and .899 for the environment scale, indicating “very good” levels of reliability (DeVellis, 2003). The environment (E) subscale, the first extracted factor, had 8 out of the 10 original items had loadings greater than |.40|, explained 40.47% of the variance (Table 3), and had excellent reliability (α=.899, CITC =.577). Family interaction (FI), the second extracted factor, also had high reliability (α=.880, CITC = .686) with 4 (out of the 5) item loadings greater than |.40|, and it explained 15.78% of the variance. Child Wellbeing (CW), the third extracted factor had very good reliability (α=.807, CITC= .654) and, it explained 6.55% of the variance; only 3 out of the 7 original items had loadings greater than |.40|. The three items were: child behavior, child mental health, and overall child wellbeing. Family safety (S) is the fourth and last factor extracted and it explained about the same proportion of variance as child wellbeing (6.29%); it included the overall family safety, the child physical abuse, and the child psychological abuse items. The items with loadings smaller than the |.40| cutoff point (Table 3) were: transportation and income (environment domain); relationship with caregivers, with siblings, with peers, and cooperation or motivation to maintain the family (child wellbeing domain); relationship between caregivers (family interaction); domestic violence, child physical and child sexual abuse (family safety). None of the parental capabilities items loaded on any of the extracted factors.

Validation and Predictive Ability of the North Carolina Family Assessment Scale …

13 | JSRP

Table 3: Factor Loadings by Model Item Environment Overall Housing Habitability Housing Stability Financial Management Learning Hygiene Nutrition Community Safety Transportation Income Family Interaction Overall Expectations of Children Bonding Mutual Support Caregivers’ Relationship Child Wellbeing Overall Child Behavior Child Mental Health Caregiver Relationship Siblings Relationships Peer Relationships Cooperation/motiva tion Family Safety Overall Child Psychological Abuse Child Neglect Child Physical Abuse Child Sexual Abuse Domestic Violence Parental Capability Overall Discipline Enrichment Opportunities Supervision Parent Mental Health Parental Physical Health Parent Substance Abuse

Published Model† E CW FI FS

E

General Model FI CW

FS

E

.854

.830

.831

.774

.770

.761

.774

.768

.766

.732

.646

.679

.707 .780 .758 .718 .700 --

.641 .640 .635 .626 <.40 <.40

.707 .693 .709 .675 .642 <.40

Congeneric Model FI CW FS

.799

.829

.910

.593

.727

.788

.685 .670

.669 .638

.758 .733

--

<.40

.632

.727

.778

.849

.838

.818

.850

.775

.781

.812

.805

<.40

.744

.592

<.40

.732

.706

<.40

.689

.614

<.40

<.40

--

.657

.739

.683

.677

.740

--

.650

.711

--

<.40

.740

.712

<.40

.701

.438

<.40

.606

PC

.841 .749 .716 .697 .634 <.40 <.40

† (Reed-Ashcraft et al., 2001); E= environment, FI= family interaction; FS= family safety, PC= parental capability, CW=child wellbeing)

14 | JSRP

Ramona F. Stone, Gerard M. Barber, Sarah Hendrix

The items loading on the environment and family interaction factors, extracted with the general factor analysis, were similar to the published model results, but the amount of variance explained was different. The environment factor in the general model explained 40.5% of the variance as compared to 20.1%; the family interaction factor explained 15.8% as compared to 14.2%. Significant departure of the general model from the literature was also noted on the child wellbeing and family safety factors. Specifically, in the published model, the child wellbeing subscale explained the second largest proportion of variance (16.5%) and included all of the original seven sub-scale items. In the general model, the wellbeing factor was the third extracted factor, had three item loadings (overall, behavior, and mental health) and explained only 6.6% of the variance. The items with loadings below the cutoff point were measures of relationships with siblings, peers, caregivers, and child cooperation. Family safety was the fourth extracted factor in both models, each with three items loadings; they had similar proportion of variance explained but their item composition was different. The psychological abuse item loaded in both models; the other two items were child neglect and the overall family safety (general model) and respectively, sexual abuse and domestic violence (published model) items. Congeneric Factor Analysis: The second approach was to conduct PAF with a Varimax rotation with a cutoff point of |0.40| separately for each domain or sub-scale; the extracted factors (Table 3) were again saved as regression scores. As with the general factor model, reliability analyses (Table 2) were conducted for each domain. The environment sub-scale was computed using 1877 valid cases, included 9 out of 10 items, explained 57.1% of the variance, and had an excellent Cronbach’s reliability coefficient (α=.903). The child wellbeing sub-scale had 6 out of the 7 items with loadings above the cutoff point, with an excellent reliability coefficient (α=0.903), and explained a large proportion of variance (67.4%). Family interaction and family safety retained all of their items, had very good reliability coefficients (α>.85), and explained 66.9% and respectively 59% of the variance. The congeneric approach allowed for the examination of the parental capability subscale, which was not extracted with the general model, or with the published model; 5 of its 7 items had loadings of |.40| or above, had very good reliability (α=.847) and explained 62.4% of the variance in the data. Overall, with the congeneric approach, 31 of the 35 items had factor loadings greater than |.40| as compared to the general model where only 18 of the 35 items were retained in the factor analysis. The four items with loadings below the cutoff point were: income (E domain), child cooperation/motivation to maintain the family (CW domain), and parental physical health and substance abuse (PC domain). Differences in NCFAS Means Successful Vs. Unsuccessful Cases All factors extracted with the two methods, general and congeneric factor analyses were saved as regression scores in the dataset. The congeneric model yielded five factors and the general model yielded four factors, for a total of nine new variables in the dataset. The differences in factor scores between the two outcome groups (successful and not successful in maintaining the children in the home) were tested with a series of independent t-tests (Table 4). The left side of the table displays the results of the Levene’s test of equality of variances, indicating significant differences between the two groups in the dispersion of several factor scores (environment, family interaction, child wellbeing, and parental capability); the standard deviations showed that the unsuccessful group was a more heterogeneous distribution of factor scores than the successful group. The factor averages were significantly different between the groups on three of the four subscales extracted with the general PAF model: environment (p<.001), family interaction (p=.001), and family safety (p=.051); there was no difference between the successful and the unsuccessful cases in child wellbeing (p=.859).

Validation and Predictive Ability of the North Carolina Family Assessment Scale …

15 | JSRP

Table 4: Successful Vs. Unsuccessful Cases on Key Indicators Levene's Test F General

Congeneric

IHS

Environment Family Interaction Family Safety Child Well-Being Environment Family Interaction Parental Capabilities Family Safety Child Well-Being Home visits completed Hours of home visits

13.116 4.206 2.500 5.782 11.621 4.472 5.645 3.464 9.011 1.245 .102

T-test

Sig.

T

Df

Sig.

.000*** .040* .114 .016* .001** .035* .018* .063† .003** .265 .749

-3.467 -3.364 1.951 -.178 -3.801 -3.338 -2.792 .884 -1.698 -9.508 -9.757

1017.53 1114.18 1614 1095.09 1310.65 971.23 1401.54 1428 853.15 1926 1907

.001** .001** .051† .859 .000*** .001** .005** .377 .090† .000*** .000***

*** p<.001, ** p<.01, * p<.05, †p<.10 The congeneric model approach yielded similar results, although in this case there were no significant differences in family safety (p=.377) and there were marginally significant differences in child wellbeing (p=.090). The remaining three congeneric factors were significantly different between the groups: environment (p<.001), family interaction (p=.001), and parental capability (p=.005). Note that the family safety factor extracted with the general model indicated that unsuccessful cases scored better at intake than their counterparts, but the congeneric model results showed no statistical significant differences between the two groups. The groups were also significantly different in the amount of services received; the successful cases received on average an additional 3.27 home visits or 6.1 hours of visitation as compared to their counterparts.

Figure 1: NCFAS Intake Factor Scores by Case Outcome These findings suggest that on average, the child wellbeing scores were fairly similar for the two groups (Table 4, Figure 1), but the successful cases had significantly better off start-up scores on environment, family interaction, and parental capability than their counterparts. The unsuccessful cases started at a disadvantage, as measured by their intake family functioning scores, and also had a lower program participation as measured by the number visits and of hours of home visits.

16 | JSRP

Ramona F. Stone, Gerard M. Barber, Sarah Hendrix

Logistic Regression To determine the validity of the NCFAS scale in relation to the program’s goal to prevent children’s removal from the home, the extracted intake factor scores were tested as predictors of the outcome at closure (Kirk, Kim & Griffith, 2005) with logistic regression. While crossvalidation analyses using randomly split files were conducted during the preliminary analyses, the analysis presented here used the entire dataset. NCFAS is a practice tool intended to measure family functioning on multiple domains, and it should logically be able to detect changes over time, if they occur. Therefore, closure scores should be strongly related to the case outcome, to likelihood that the children are maintained in or removed from their home; moreover, they should be correlated to the number of hours of services provided during the program. This paper attempts to identify whether the intake scores are related to the outcome, just as the closure scores are. This is important because NCFAS intake assessments assist practitioners in their decisions regarding the type and the intensity of services the family needs in order to successfully maintain their children in the home. The t-tests (Table 4) identified five intake sub-scale scores that were statistically different (at a p<.05 level) between successful cases as compared to the unsuccessful cases, and two that were marginally significant (p<.10). These seven variables were further used as predictors of case success (1=successful, 0=unsuccessful) using logistic regression. NCFAS are standardized variables, with a mean of 0 and a standard deviation of 1, thus one unit on this scale equals one standard deviation. The unadjusted odds ratios showed that all seven sub-scales were significantly related to the case closure outcome; the minimum increase in the likelihood of success was 11.4% (child wellbeing) and the maximum was 24% (environment) for every 1standard unit increase in the factor score. The unadjusted model for the safety factor seems to indicate that the likelihood to maintain the children in the home decreases with family safety improvement; certainly, this does not make any sense. At intake, child abuse and neglect or domestic violence are generally not substantiated, and thus providers used the scale’s “0” point, which unfortunately had a double meaning of baseline or adequate. This is a limitation of the scale that should and could be addressed by adding an “unknown” or “unable to assess” category. Nevertheless, family safety factor became non-significant (p=.965) when adjusting for the number of hours of services provided. All other sub-scales continued to explain a significant amount of variation in the data. Next, logistic analyses were adjusted for the number of hours of intensive in-home visitation received, to account for the impact of service intensity on the likelihood of success. The number of hours was recoded into three categories: up to 10 hours of home visitations, 11 to 20 hours, and 21 or more hours of intensive home visits; the last category (21 or more hours) was used as a reference group. Table 5 displays the logistic regression coefficients, the odds ratios and the p-values for all seven adjusted logistic models. The adjusted odds-ratios showed that for every 1-standard unit increase on the environment and family interaction scores, the likelihood that the family will maintain the children in the home increased by about 30%. Note that the odds ratios for environment and family interaction were similar (1.305 vs. 1.29, and respectively 1.305 vs. 1.330) in the general and congeneric models, indicating that no matter the extraction method the results were valid and reliable. Thus, families who successfully maintained their children in the home were significantly better off at intake on their environment and family interaction domains. Families with a positive outcome at closure also had better off child wellbeing and parental capability factors (extracted with the congeneric factor analyses); every 1-standard unit difference in the child wellbeing score was associated with a 14.9% increase in the likelihood of success. Similarly, every 1-standard unit improvement in the parental capability score increased the odds of a successful outcome by 27.3%. The adjusted models provided an insight with regards to the intensity of services necessary for families to succeed in maintaining their children in the home. All logistic models indicated that families who received at least 11 to 20 hours of services were not significantly different in their likelihood to reach the permanency goal when compared to families who received 21 or more services. Therefore, the optimal number of hours of services is between 11 and 20 hours of visitation; additional services should be provided only after a careful case assessment.

Validation and Predictive Ability of the North Carolina Family Assessment Scale …

17 | JSRP

Table 5: Unadjusted and Adjusted Logistic Regression Models General Factors Environment Hours:21+ Hours: 1_10 Hours:11_20 Constant Family Interaction Hours:21+ Hours: 1_10 Hours:11_20 Constant Family Safety Hours:21+ Hours: 1_10 Hours:11_20 Constant Congeneric Factors Environment Hours:21+ Hours: 1_10 Hours:11_20 Constant Family Interaction Hours:21+ Hours: 1_10 Hours:11_20 Constant Child Wellbeing Hours:21+ Hours: 1_10 Hours:11_20 Constant Parental Capability Hours:21+ Hours: 1_10 Hours:11_20 Constant

N 1580

1580

B .215

.664 .192

.663 1580 -.114

1826

1343

1283

1831

Unadjusted S.E. Sig. .057 .000

.053 .058

.053 .063

.000 .001

.000 .069

OR 1.240

B .266

S.E. .060

1.943 1.211

-1.404 .149 1.212 .255

.152 .160 .127 .063

1.941 .892

-1.431 .111 1.237 -.036

.153 .161 .127 .066 .150 .160 .126

.660

.053

.000

1.934

-1.348 .169 1.174

.202

.052

.000

1.224

.266

.054

1.762 1.222

-1.372 .096 1.119 .285

.139 .145 .115 .065

1.937 1.114

-1.592 -.044 1.344 .139

.171 .176 .143 .064

1.920 1.161

-1.387 .146 1.184 .241

.164 .174 .135 .056

1.771

-1.408 .072 1.153

.139 .146 .115

.567 .200

.661 .108

.652 .149

.571

.049 .061

.058 .061

.059 .052

.049

.000 .001

.000 .078

.000 .004

.000

Adjusted Sig. .000 .000 .000 .354 .000 .000 .000 .000 .491 .000 .589 .000 .000 .291 .000 .000 .000 .000 .507 .000 .000 .000 .000 .802 .000 .031 .000 .000 .401 .000 .000 .000 .000 .623 .000

OR 1.305 .246 1.160 3.359 1.290 .239 1.117 3.445 .965

R2 .117 .161

.115 .160

.106 .147

.260 1.184 3.236 1.305 .254 1.101 3.061 1.330 .204 .957 3.833 1.149 .250 1.158 3.267 1.273

.110 .151

.123 .170

.109 .151

.109 .149

.245 1.074 3.167

Note: The top R2 is Cox and Snell, the second one is Nagelkerke’s R2 estimate. The adjusted logistic regression models had similar coefficients of determination (R 2); the Nagelkerke R2 showed varied between 14.7% and 17%, while Cox& Snell varied between 10.6% and 12.3%. The goodness-of-fit, Hosmer-Lemeshow χ2 test was marginally significant for the environment congeneric factor (χ2 (8) =14.654, p=.066), and for the environment general factor (χ2 (8) =13.436, p=.098); all other χ2 tests had non-significant p-values (p>.25) indicating a good model fit for the data. Further insight in the sub-scales ability to predict case status at closure was provided by the adjusted regression models’ sensitivity (ability to predict correctly the successful cases) and specificity (ability to predict correctly the unsuccessful cases). All of the seven tested subscales classified correctly between 68% and 71% of the cases; but there were notable differences in their specificity and sensitivity. The lowest specificity was recorded on the environment factors, while the highest one was recorded by the family safety (60.5%); family safety had the lowest sensitivity (73.8%) while family interaction had the highest (82%). The environment subscales classified correctly 48-49% of the unsuccessful cases, and had similar sensitivity (80.4% vs. 79.3%); both family interaction scales classified correctly 70-71% of the cases, with a sensitivity of about 50% and specificity of 81-82%. To summarize, the subscales extracted with the two approaches yielded similar results when used as predictors of case outcome at the time of closure, but the two family interaction

18 | JSRP

Ramona F. Stone, Gerard M. Barber, Sarah Hendrix

sub-scales appeared to explain more of the variation in the data while adjusting for the number of hours of home visits provided, have better sensitivity and specificity, and fit the data the best. Convergent Validity Finally, the NCFAS scale was tested for its convergent validity using Pearson’s correlation (Table 6). While all factors extracted with the congeneric model were significantly correlated with each other (bottom quadrant to the right), the general model factors were not. Specifically, the environment factor was not correlated with the family interaction of with child wellbeing; also, even when correlations were significantly different from zero, they were weak (R<.30). In contrast, the congeneric factors were all significant and many were moderately (.30.70) correlated with each other. Table 6: Pearson’s R Correlation Coefficients for Concurrent and Predictive Validity

FI CW FS E FI CW FS PC

General Model E FI .042 1 .008 .118** ** .079 .074** ** .187** .969 ** .231 .932** .127** .401** .326** .218** .386** .520**

CW

FS

Congeneric Model E FI

CW

FS

1 .055* .094** .321** .905** .200** .406**

1 .194** .261** .153** .872** .334**

1 .407** .261** .454** .519**

1 .351** .598**

1 .489**

1 .603** .468** .692**

*p<.05, **p<.01 The family interaction scales, identified with logistic regressions as the strongest predictors of outcome, were highly correlated with each other (R=.932); in addition, they were significantly correlated with child wellbeing and family safety, and with parental capability, suggesting that improvement on one was associated with improvement on the others. Taken all together, the family interaction factor, regardless of the extraction method used, appears to be the best indicator to predict whether the children will be maintained in the family or not, assuming that the family receives between 11 and 20 hours of home visitation. Discussion The primary focus of the CCC program was to provide supports to parents and families allowing them to avoid removal and subsequent placement of their children in out of home care. Families were provided with intensive in-home services on an individualized basis and also referred to outside supportive services as needed. There were commonalities across a statewide service area in terms of program objectives, but there were also significant differences in the actual program implementation and processes. Agencies varied greatly in the number of home visits and the number of hours of visits provided per family; some agencies had fewer referrals and hence served a smaller number of families, but they provided a larger number of home and hours of visits. Other agencies had a much larger number of referrals, and thus provided fewer one-hour home visits to a larger number of families. Individual agencies also differed in the proportion of successful cases, although all met their goal of 60% or more of the successfully closed cases. The CCC program utilized the NCFAS tool to measure family functioning on five domains: environment, family interaction, parental capabilities, child wellbeing, and family safety. Data were collected at intake, at the end of each quarter the family was active in the program, and at case closure. The NCFAS intake scores were considered more appropriate than the closure scores for validation and for predictive analyses by informing the providers about the family needs that must be addressed in order to avoid the removal of children from the home

Validation and Predictive Ability of the North Carolina Family Assessment Scale …

19 | JSRP

Furthermore, it has been suggested before (Tungate, 2006) that the intake scores were more reliable and were better predictors of permanency outcomes than the closure scores. NCFAS is a family assessment practice tool widely used by the family prevention programs across the United States; it is a valid measure of family functioning, and has five domains (Kirk, Kim & Griffith, 2005; Kirk, 2012; Reed-Ashcraft, Kirk & Frasier, 2001). To validate the tool with the CCC population, two approaches were employed. First, the general validation model was fit. This model is commonly used because it allows for multiple latent concepts to underlie the scale items, closer to how the real-world data is (DeVellis, 2003). With this method all items belonging to all sub-scales were included in the same factor analyses and through an iterative process a smaller number of reliable and valid factors were extracted. Second, the congeneric model presumes that factor and reliability analyses are conducted by scale domain. The factors extracted with both methods were saved as regression scores and used as predictors of program outcome in subsequent analyses; the main difference between the two sets of factors, was the parental capability factor which was not extracted with the general factor analysis. Descriptive statistics were conducted to identify differences in the intake scores, between the successful and the unsuccessful cases, with the intent to use them as predictors of case outcome. The tests of differences in factor means between successful and unsuccessful cases yielded a list of seven potential outcome predictors. The t-tests showed that the cases with children who were maintained in the home had significantly better start-up scores on environment, family interaction, and parental capability. Improvements on the family interaction domain were strongly correlated with improvements in child wellbeing, parenting capabilities, family safety and environment. The regression findings suggest a tipping point in the number of hours of visitation (11 to 20 hours), critical for the success of the intervention. This was also a point of diminishing returns, because the family gains were minimal after 20 hours of services and the need for continued family support should be reassessed when reaching this point. The data support the recommendation that at intake a special attention should be given to the family interaction domain, as this factor was the best predictor of case success while adjusting for the number of hours of home visitation.

Conclusions The NCFAS scale validation yielded slightly different results depending on the method of analysis. These differences in the results of the two methods demonstrate that measurement tools, even when they are validated and have high reliability, may still have limitations that need to be considered, especially when used as practice tools, for decision making. The identification of one or more NCFAS domains that set apart the successful cases from the unsuccessful ones at intake could enable family preservation workers to provide the worse off families a more intensive and a more focused case management, to increase their chances to succeed. It was apparent that stabilization of the interaction among family members was key to avoid removal of children from the family, and that six additional hours or three home visits could make a significant difference in the case outcome. Because these differences were often due to family’s lack of ongoing participation, additional research should consider not only how service intensity is related to families’ motivation to change, but also how other factors- such as, family characteristics, type of abuse or neglect, family history of abuse and neglect, provider characteristics, etc. might be related to agency’s ability to provide services to families. Additional research on family maintenance of permanency outcomes over 6 months, 1 year, and 3 years might further inform on the predictive ability of this instrument across all of its domains. Finally, because the data was collected directly by the workers in the field and parents were not asked directly for the information, future research might consider better ways to integrate workers’ and families’ perception of the family functioning, which might have valuable implications for future in-home family preservation programs.

20 | JSRP

Ramona F. Stone, Gerard M. Barber, Sarah Hendrix

References 1.

DeVellis, R.F. (2003). Scale Development: Theory and Applications. Thousand Oaks, CA: Sage Publications.

2.

Hair, J. E., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate data analysis. Upper Saddle River, USA: Prentice Hall.

3.

Hattie, J. (1985). Methodology review: assessing unidimensionality of tests and items. Applied Psychological Measurement, 9(2), pp. 139-164. http://dx.doi.org/10.1177/014662168500900204

4.

Kirk, R.S., & Reed, K.B. (2000). Designing and developing a new assessment and evaluation tool for family preservation service programs. F. Jacobs, P. Hrusu Williams, J. Kapuscik, & E. Kates (Eds.). Evaluating family preservation services: A guide for state administrators. Medford, MA: Family Preservation Services Project, Tufts University.

5.

Kirk, R.S., & Reed-Ashcraft, K.B. (2004). NCFAS Research Report. Guide submitted to the National Family Preservation Network. Retrieved September 5, 2014, http://nfpn.org/Portals/0/Documents/ncfas_scale_development.pdf.

6.

Kirk, R.S., Kim, M.M., & Griffith, D.P. (2005). Advances in the reliability and validity of the North Carolina Family Assessment Scale. Journal of Human Behavior in the Social Environment, 11(3/4), pp. 157-176. http://dx.doi.org/10.1300/J137v11n03_08

7.

Kirk, R.S., & Griffith, D.P. (2007). An Examination of Intensive Family Preservation Services. In cooperation with the National Family Preservation Network. Technical report submitted to the Annie E. Casey Foundation, Retrieved October 1, 2014 from http://tinyurl.com/khc9muo.

8.

Kirk, R.S. (2012) Development, Intent, and Use of the North Carolina Family Assessment Scales and their Relation to Reliability and Validity of Scales. In cooperation with the National Family Preservation Network. Retrieved September 5, 2014 http://www.nfpn.org/Portals/0/Documents/ncfas_scale_development.pdf.

9.

Lance, C. E., Butts, M. M., & Michaels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9(2), pp. 202–220. http://dx.doi.org/10.1177/1094428105284919

10. Ledesma, D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: an easy-to-use computer program for carrying out parallel analysis. Practical Assessment, Research and Evaluation, 12(2), pp. 1-11. 11. Petkov, J., Harvey, P., & Battersby, M. (2010). The internal consistency and construct validity of the partners in health scale: validation of a patient rated chronic condition self-management care. Quality of Life Research, 19(7), pp. 1079-1085. http://dx.doi.org/10.1007/s11136-010-9661-1 12. Reed-Ashcraft, K.B., Kirk, R. & Fraser M. (2001). The reliability and validity of the North Carolina Family Assessment Scale. Research on Social Work Practice, 11(4), pp. 503-520. http://dx.doi.org/10.1177/104973150101100406

Validation and Predictive Ability of the North Carolina Family Assessment Scale …

21 | JSRP

13. Stone, R.F., & Barber, G.M. (2008). Final Evaluation Report: Community Collaboration for Children Programs 2006-2008. Submitted to the Cabinet for Health and Family Services, Frankfort, Kentucky. 14. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics. Fifth Edition. Boston: Allyn & Bacon. 15. Tungate, P. N. (2006). Legacy Children: Whose Legacy Are They? PhD Dissertation, University of Louisville. 16. Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. G. D. Goffin and E. Helmes (Eds.), Problems and Solutions in Human Assessment. (pp 44–71), Boston: Springer. http://dx.doi.org/10.1007/978-1-4615-4397-8_3 17. Westat Chapin Hall Center for Children James Bell Associates (2002). Evaluation of Family Preservation and Reunification Programs: Final Report. Department of Health and Human Services Assistant Secretary for Planning and Evaluation, Washington, DC. Retrieved October 13, 2014 from http://aspe.hhs.gov/hsp/evalfampres94/final/Vol1/chapt3.htm.