Case-supportive Technology Virginia - Evaluation Overview

Authors

Quality Improvement Center for Workforce Development (QIC-WD)

Document Type

Article

Citation

© The Quality Improvement Center for Workforce Development (QIC-WD)

Abstract

The QIC-WD evaluation was conducted with the support of the Virginia Department of Social Services to determine if a Case-Supportive Technology intervention was effective in improving workforce and child welfare outcomes.

Research Questions

The site-level evaluation for Virginia was designed to understand implementation of and outcomes related to two technology innovations: (1) transcription and (2) a mobility application. See the Intervention Overview for more information.

Relationships among intervention components, outputs, and outcomes were mapped out in Virginia’s logic model. Initial research questions of interest included how well the components of the intervention were received by staff, usage of the technology, and perceived impact on time savings, etc. These variables were measured to assess 1) uptake of the components of the intervention by staff and 2) impact of uptake on outcomes.

The efficacy evaluation addressed a substantial number of research questions about short- and medium-term outcomes. Research questions were posed based on surveys administered at baseline and several follow-up periods. These questions included:

• To what extent was each component of the intervention (transcription and COMPASS|Mobile) associated with increases in the workforce’s: (1) work-life balance, (2) perceived social support, and (3) efficiency pre vs. post intervention?
• To what extent was the intervention associated with decreases in the workforce’s: (1) stress and (2) burnout pre vs. post intervention?
• To what extent was the intervention associated with higher levels of the workforce’s: (1) job satisfaction and (2) intent to stay, and lower levels of: (3) intent to leave and related leaving behaviors, and (4) absenteeism pre vs. post intervention?
• To what extent was the intervention associated with organizational culture and climate, particularly in the targeted area of stress?
• How did uptake of the components of the intervention and changes in the attitudes and behaviors described above vary by demographic and personality characteristics?

The evaluation also addressed longer-term research questions about the associations between the intervention and turnover pre- vs. post-intervention. Furthermore, the evaluation assessed whether any impact on turnover was mediated by changes in job satisfaction, intent to stay, intent to quit, or absenteeism. Finally, the efficacy evaluation assessed the extent to which the intervention was associated with enhanced child and family engagement and safety or permanency outcomes, and whether or not such changes were mediated by reductions in turnover.

Evaluation Design

The process evaluation was conducted by administration of online diaries, typically every two weeks over the course of a few months after the roll-out of each intervention component. These 5-10 minute surveys were descriptive in nature and examined staff perceptions and self-reports about the impact of the technology on documentation tasks, case practice, time use, benefits, and challenges. These assessments allowed the site team to identify and address problems during early implementation in a manner that was responsive to user’s needs and supportive of successful implementation.

A multiple time series design was employed to evaluate the effectiveness of the case-supportive technology intervention. For evaluation purposes, leadership from 18 localities served on the QIC-WD implementation team and staff from those localities completed surveys. (See the Site Overview for more information.) Both survey data and administrative data from child welfare and human resources (HR) systems were collected for these 18 localities as well as the entire state for comparison purposes.

Frontline workers and supervisors in the participating localities completed a staff survey with measures assessed at baseline (i.e., pre-intervention rollout), and in follow-up surveys. Short- and medium-term measures included personality and attitude scales (that might mediate uptake or outcomes), perceived work stress, burnout, perceived social support, and organizational culture and climate. Survey data were collected either in-person or using a secure online platform. Long-term outcomes of workforce turnover, family engagement, child safety, and permanency were measured using child welfare and human resource administrative data extracts. Survey data were linked together and with administrative data for mediation analyses. All data were stored and managed in a secure location, accessible only to the QIC-WD evaluation team and the site data coordinator.

Timeline

Diaries were administered after transcription rolled out every two weeks between February and March 2018. Diaries were administered after mobility rolled out about every two to four weeks between late October 2019 and early March 2020. After Covid-19 caused a shift to remote work, the diaries were administered again between late April and late June 2020, to capture how technology aided the workforce in the pivot to remote work in response to the pandemic. A final diary after the addition of Structured Decision Making (SDM) was administered three times in May 2021.

The first baseline was administered between December 2017 and January 2018 before transcription was rolled out. The second survey was administered between March and August 2019 before mobility was rolled out. The third survey was administered between June 2021 and August 2021 (after SDM tools were added to mobility).

Surveys of staff promotion, transfers and exits from the agencies were conducted monthly between February 2018 and August 2021.

Administrative data regarding usage of transcription and impact on documentation in Safe Measures was gathered between February 2018 and August 2021 (3.5 years). Administrative data regarding usage of mobility was gathered on transactions beginning in October 2019 through August 2021.

HR and child welfare administrative data were transmitted to the evaluation team periodically starting in the summer of 2020. Both HR and child welfare data were extracted from the two years before the QIC-WD engaged the site (2016 and 2017), the three years the site was engaged in the process evaluation (2018, 2019, 2020), and the two years following the intervention (2021 and 2022, planned).