CFRP Policy Brief | B.038.0718
In May 2016, the Texas Department of Family and Protective Services (DFPS) enacted a policy to expand the pool of potential caseworkers by changing hiring criteria to allow Child Protective Service (CPS) to hire qualified candidates with relevant experience who have not completed a bachelor’s degree. The agency contracted Dr. Cynthia Osborne and the Child and Family Research Partnership (CFRP) at the LBJ School of Public Affairs at The University of Texas at Austin to conduct an implementation and outcomes evaluation of the change in hiring policy to examine whether variation in caseworker preparedness, retention, and quality is associated with educational background, specifically completion of a bachelor’s degree. This methodology summary accompanies the findings memo CFRP submitted to the agency in July 2018 (“Summary of Findings from the Evaluation of Caseworkers without a Bachelor’s Degree”) to provide the agency with details on CFRP’s methodology for analyzing caseworker retention and casework quality. For these analyses, CFRP used three data sources provided by the agency: CAPPS, IMPACT, and the Caseworker Education Tracking Log for staff hired under the policy.
In October 2017, DFPS provided CFRP with the Caseworker Education Tracking Log, a record of caseworkers hired without a bachelor’s degree (no BA/BS). The Tracking Log provided information on the 225 caseworkers hired between June 2016 and August 2017 including their CAPPS employee ID, start date as a CPS caseworker, and stage of service. CFRP used caseworkers’ employee IDs to match with employment information in CAPPS1 and identify their hiring department, county, and region to use in selecting a comparison sample.
To construct a comparison group of caseworkers with a bachelor’s degree (BA/BS) for the outcomes analyses of retention and casework quality, CFRP used data from CAPPS to identify a pool of caseworkers who were hired for the first time during the same time window as the 225 no BA/BS caseworkers (between June 1, 2016 and August 31, 2017). CFRP used stratified sampling to match caseworkers in the potential comparison sample with the 225 no BA/BS caseworkers using department (unit), hire date, county, stage, and region. The first level of sampling selected the BA/BS caseworker hired closest in time into the same department as the no BA/BS caseworker. Of the 225 no BA/BS caseworkers, 199 had at least one eligible BA/BS caseworker who was hired into the same department.2 For the remaining 26 no BA/BS caseworkers who did not have a hire in the same department during the sampling time period, comparison BA/BS caseworkers were selected based on matching county and stage. Sixteen of the no BA/BS caseworkers had an eligible comparison caseworker from their county and stage. For the remaining 10 no BA/BS caseworkers, comparison BA/BS caseworkers were selected by matching on region and stage.3 This three-level stratification process resulted in a comparison sample of 229 caseworkers with a bachelor’s degree for the outcomes analyses of retention and casework quality.
Caseworker Retention Analyses
To assess whether there are differences in retention between caseworkers without a bachelor’s degree and their counterparts with a bachelor’s degree, the samples described in the previous section were further limited to caseworkers hired between June 2016 and January 2017 to allow CFRP to observe the caseworkers’ retention behavior for up to 12 months by December 31, 2017.4 As a result, CFRP analyzed retention for 263 caseworkers (139 no BA/BS and 124 BA/BS).
CFRP used two analytic strategies to examine retention of caseworkers without a bachelor’s degree. First, CFRP used hiring and termination/demotion data from the Caseworker Education Tracking Log and CAPPS to assess caseworker retention over the first 12 months of employment and examine the overall proportion of caseworkers retained by educational background (no BA/BS compared to BA/BS) using survival curves. In the second approach, CFRP estimated Cox proportional hazard models using the same retention data to predict a caseworker’s likelihood of remaining employed during the first year based on educational background. These models allowed CFRP to incorporate additional control variables for stage of service, region of hire, age at hire, and gender.
Casework Quality Outcomes
To assess whether there are differences in casework quality between caseworkers without a bachelor’s degree and their counterparts with a bachelor’s degree, CFRP began with the 225 no BA/BS caseworkers and 229 BA/BS caseworkers identified in the base samples and applied additional restrictions to ensure that outcome measures could be attributed to the caseworkers. CFRP limited the sample to caseworkers who did not change specialty between their hire date and December 31, 2017 (or their termination date if they were no longer working as a caseworker with DFPS) and had a minimum of five stages for which they were assigned as the primary caseworker (for each respective quality outcome). In addition, the sample was further restricted based on measure-specific eligibility criteria, described in more detail below. For caseworkers with more than 12 months of eligible casework, only the first 12 months of casework quality data were included in the analyses.
For each casework quality measure, CFRP calculated an overall success rate for each caseworker (total completed contacts/timely closures out of all eligible stages) for use in regression analyses.
Timely Initial Contact
To assess timely initial contact, CFRP examined the proportion of investigations for which the caseworker attempted contact with a parent or child on the case within the required time window based on the priority level, as well as the proportion of investigations for which the contact was successful.5 CFRP included investigations in the calculation of timely initial contact if the caseworker was assigned as the primary caseworker on the investigation stage the same day the investigation opened and the caseworker remained assigned to the investigation stage for at least one day for Priority 1 (P1) investigations or at least three days for Priority 2 (P2) investigations. Additionally, eligible investigations were limited to those assigned on or after the caseworker’s hire date through December 30, 2017 (for P1) and through December 28, 2017 (for P2) or through the caseworker’s termination date (if prior to December 28, 2017).
Approximately 10,000 investigations met the criteria for inclusion in the timely initial contact measure. After caseworkers who did not meet the eligibility criteria for timely initial contact were excluded, the final investigations caseworker sample for this measure was 191 (89 no BA/BS and 102 BA/BS).6
Timely initial contact was measured using stage-level documentation in IMPACT that contained information regarding the date the contact occurred, whether the contact was with a principal on the investigation, and the purpose of the contact.7 For analyzing attempted initial contact, CFRP included attempted contacts within the one-day (P1) or three-day (P2) window, regardless of whether the contact was successful. For analyzing successful initial contact, CFRP used the indicator of whether the contact was not successful to limit the contacts only to those that were successful.
Timely Investigation Closure
To assess timely investigation closure, CFRP examined the proportion of investigations closed within 45 days of the date the investigation stage opened.8 CFRP found that the number of investigations that were assigned to the same investigator continuously from the day the investigation opened through 45 days after it opened was too small to limit the sample using this criteria. As a result, CFRP considered an investigation eligible for inclusion in the timely closure analyses if the caseworker was assigned the investigation for 80 percent of the first 45 days the investigation was open. Using this criteria, most caseworkers were assigned within the first week of the investigation (approximately 82 percent of investigations assigned to our sample caseworkers during the time period were assigned for 80 percent of the investigation).9
Approximately 9,500 investigations were eligible for inclusion in the timely closure measure. After caseworkers who did not meet the eligibility criteria for timely stage closure were excluded, the final investigations caseworker sample for this measure was 188 (88, no BA/BS and 100 BA/BS).10
Monthly Face-to-Face Contact
Per DFPS statute, CVS caseworkers are to make face-to-face contact with each child in substitute care at least one time per calendar month for the entire time the child is in CPS conservatorship.11 To assess monthly face-to-face contact with children in CPS care, CFRP examined the proportion of Substitute Care (SUB) stages for which caseworkers recorded a successful, face-to-face contact with the child. For each month during the analysis time frame, CFRP used stage assignment information in IMPACT to limit SUB stages for inclusion in calculations to the stages for which the caseworker was assigned as the primary caseworker for the full calendar month.12 For example, if a caseworker was assigned to a child in a SUB stage on January 1 and unassigned on January 15, the stage was not included in the calculation of their monthly face-to-face contact outcome for January because the caseworker did not work the stage for the full calendar month. Using this approach, CFRP’s measure of timely face-to-face visits only includes stages for which we could be sure the caseworker was responsible for completing the contact in a given month.
Monthly face-to-face contact for children in substitute care is more challenging to capture in IMPACT as caseworkers can document the required face-to-face contact in the child’s substitute care stage (the recommended location), in a family level stage (e.g., family substitute care if the child is still in Temporary Managing Conservatorship or family reunification if the family is working towards reunification while the child is still in CPS care), or in a separate child stage (e.g., an adoption stage if the child is moving towards a potential adoption). Therefore, to try to most accurately capture the number of children who had face-to-face contact with a caseworker in a given month CFRP used contact information from IMPACT from all stages in which a child was indicated as having participated in a face-to-face contact while they also had an active substitute care stage.13 Caseworkers were credited with making the monthly face-to-face visit if the contact was documented in IMPACT as a successful face-to-face visit that included the child in the substitute care stage.
Approximately 2,200 children in substitute care stages were eligible for the monthly face-to-face measure. After caseworkers who did not meet the eligibility criteria for monthly face-to-face contact were excluded, the final conservatorship caseworker sample for this measure was 99 (45 no BA/BS, 54 BA BS).14
Family Based Safety Services (FBSS)
Timely Initial Contact
To assess timely initial contact with families receiving Family-Based Safety Services (FBSS), CFRP examined the proportion of Family Preservation (FPR) stages for which caseworkers recorded a contact within 10 days of the FPR stage opening.15 CFRP used stage assignment information in IMPACT to identify families with FPR stages assigned to FBSS caseworkers at the time the FPR stage started who also remained assigned to the caseworker for 10 calendar days from the date the stage started.16 To be included in the analyses, FPR stages had to start by December 21, 2017.
Approximately 600 families in FPR stages were assigned to sample caseworkers for the first 10 days of the stage. After caseworkers who did not meet the eligibility criteria for timely initial contact were excluded, the final FBSS caseworker sample for this measure was 44 (21 no BA/BS and 23 BA/BS).17
Timely initial contact was measured through stage-level documentation in IMPACT that recorded the date, method, and purpose of the contact. FBSS caseworkers were credited with completing a timely initial contact if a contact was documented in IMPACT as occurring within the first 10 days of the FPR stage opening and was a successful face-to-face contact with a principal on the FPR stage, regardless of the purpose of the contact.
Monthly Face-to-Face Contact
To assess timely face-to-face contact with families receiving FBSS, CFRP examined the proportion of FPR stages for which caseworkers recorded a face-to-face contact with at least one parent or child on the stage in each month the stage was assigned to the caseworker.18 CFRP used the same responsibility criteria here as what was used for CVS monthly face-to-face and considered sample caseworkers responsible for making face-to-face contact only in those months for which they were assigned as primary on the stage for the entirety of the calendar month.19
There were approximately 1,600 families in FPR that were eligible for the monthly face-to-face measure. After caseworkers who did not meet the eligibility criteria for monthly face-to-face contact were excluded, the final FBSS caseworker sample for this measure was 69 (36, no BA/BS and 33 BA/BS).20
Monthly face-to-face contact was measured using stage-level documentation available in IMPACT. Caseworkers were credited with having successfully met with the family face-to-face if the contact was documented in IMPACT as a successful face-to-face contact with a principal on the stage that occurred during the calendar month for which the caseworker was assigned as a primary on the FPR stage.
For all casework quality measures, differences between the two groups (no BA/BS and BA/BS) were assessed using regression analyses. CFRP examined casework quality during their first 12 months of tenure (or up until the date of termination). For each casework quality measure, CFRP ran a series of stepwise regression models, beginning with an indicator for educational background and progressively incorporating explanatory variables in the models to better understand the association between educational background and casework quality.21
Contact/Closure Attribution in IMPACT
IMPACT does not have documentation associated with the specific caseworker who actually attempted/made the contact with a principal on a stage or “closed” the stage. In the absence of this information, CFRP assumed that if a contact was documented in IMPACT during the time a caseworker was identified as the primary caseworker assigned to the stage that she was responsible for making the contact. Similarly, if a caseworker was assigned as primary to a stage on the date that the stage closed, CFRP credited the caseworker with closing the stage. To assess the accuracy of this approach to attribution, CFRP examined the IMPACT stage assignment data to determine if other caseworkers were assigned as primary during the same time as the focal caseworkers.
For the investigations that were eligible for the initial timely contact measure, there were no other overlapping caseworkers who were assigned during the same initial days of the investigation. With respect to timely closure, only three of the eligible stages (approximately .15 percent of eligible stages) had two caseworkers assigned for 80 percent of the first 45 days of the investigation (including on the date the stage should close). For the FBSS timely initial contact measure, there were no other caseworkers (other than the focal caseworker) who was assigned for the first 10 days of the FPR stage. Given the limited overlap in stage assignment, it is likely that even without having definitive information regarding who made contact the appropriate caseworker received “credit” for the successful contact/closure.
It is more challenging to identify overlap for the CVS and FBSS monthly face-to-face casework quality measures because the timeline for these measures are significantly longer (e.g., CVS substitute care stages are frequently one year or longer) and there is a greater likelihood of overlapping periods of primary assignment. Approximately 50 percent of FBSS and CVS stages had some time of overlap of caseworker assignment, however CFRP’s more conservative approach towards defining responsibility for making contact at the monthly level (i.e., caseworker had to have the stage assigned for the full calendar month) decreases the chance of mistakenly attributing contact to caseworkers.