Organizational Change Process Milwaukee - Evaluation Overview

Authors

Quality Improvement Center for Workforce Development

Date of this Version

9-2021

Document Type

Article

Abstract

The QIC-WD evaluation was conducted with the support of the Division of Milwaukee Child Protective Services (DMCPS) to determine if an Organizational Change Process intervention (Availability Responsiveness Continuity or ARC) was effective in improving workforce and child welfare outcomes.

Research Questions

Process Evaluation

1. Did the ARC purveyor follow the training curriculum with fidelity and quality delivery?
2. Did the ARC Team Leaders follow the ARC implementation model with fidelity and quality?
3. Were participants satisfied with the quality of leadership, training, and group dynamics?
4. Did participants learn the required concepts and skills?
5. What were the mediators of fidelity and training outcomes?
6. Did the fidelity evaluation process serve a continuous quality improvement function for the ARC implementation?

Outcome Evaluation

To what extent did the ARC intervention lead to improvement in organizational culture and climate?

1. To what extent did the ARC intervention result in team members having increased knowledge and skills essential to organizational culture and climate?
2. To what extent did the ARC intervention result in changes in frontline staff perceptions of organizational leadership style and involvement in decision making?
3. To what extent did ARC result in improved communication within the workplace?
4. To what extent did the ARC intervention impact organizational commitment, job satisfaction, and intentions to stay?
5. To what extent did changes in organizational culture and climate impact actual retention/turnover?
6. To what extent were changes in organizational culture and climate associated with family engagement and outcomes?
7. What impact did other demographics have on uptake of intervention and changes in attitudes and behaviors in questions above?
8. To what extent did staff retention impact family engagement and outcomes? Were fidelity and readiness to implement workforce interventions enhanced over the course of the study?

Evaluation Design

Process Evaluation:

In the evaluation of the Milwaukee implementation of ARC, a thorough fidelity and implementation evaluation was conducted. The core constructs of the process evaluation included:

• Fidelity of the purveyor to the training component of the ARC intervention and fidelity of the site to implementation of ARC;
• Quality of curriculum delivery using the Observer Quality Rating Tool as an online survey; and
• Participant Engagement through the Participant Survey containing measures focused on their training experience: satisfaction, facilitator alliance and group cohesion.

Outcome Evaluation:

The Milwaukee Evaluation utilized a time series, quantitatively driven mixed methods design. The design was a quasi-experimental time series design as there were repeated measures pre- and post-intervention. There were two baseline measures taken on organizational culture and climate (2018/2019). The intervention was launched in 2019, and multiple mid- and post-test measures were taken, including two more administrations of the Organizational Social Context Measure (OSC) and cross-site survey (2020/2021), as well as quarterly surveys with ARC Team and Organizational Action Team (OAT) members from January 2019 to July 2021. In addition to these repeated quantitative measures comprising the time series design, the study employed a mixed methods approach through the collection of several forms of qualitative data, including focus groups and interviews with workers, supervisors, ARC and OAT team members, as well as content analysis of meeting minutes and ARC/OAT team materials for fidelity/process evaluation.

All frontline workers that engaged in direct practice with families and children and people who supervised those workers who were employed at DMCPS were included in the study. Approximately 120 frontline workers and 22 supervisors met the inclusion criteria and were part of the study. Additionally, there were approximately 10 staff in leadership or other divisional positions that completed specific surveys if they served on the OAT team or attended specific ARC events.

Outcome measures included: organizational culture and climate, ARC principles adoption/adherence, workplace innovation, working alliance, community partner relationships, team cohesion, use of data, leadership/member exchange behaviors, leadership style, procedural justice, overall fairness/justice, and conflict resolution.

All survey data was collected using multiple methods—on paper when the team was in-person and on-line for all other surveys. In addition to surveys, focus groups and interviews were conducted at baseline and endpoint with frontline staff, supervisors, and administrators. These included those directly involved with ARC as well as the general workforce. Administrative data was from human resources and the statewide automated child welfare information system (SACWIS) databases. The historic data from both types of databases draws from January 2016 to December 2018 and was extracted in 2021.

Timeline

For the outcome evaluation, all frontline staff and supervisors completed the OSC on an annual basis from 2018 to 2021 between January and May. These frontline staff and supervisors also completed the cross-site survey and ARC annual survey from 2019 to 2021, using a monthly staggering of survey of administration. The staggering of these baseline data collection efforts was due to the length and response burden of the surveys and intended to enhance the response rate and quality of data. Intervention training fidelity data were collected monthly in 2019; other fidelity data were collected through surveys and content analysis of meeting minutes. The ARC/OAT team members completed surveys on ARC implementation on a quarterly basis from January 2019 to July 2021. Furthermore, focus groups/interviews with frontline workers and supervisors were conducted in spring 2018 and then again in fall 2021. Focus groups and interviews were also conducted with ARC/OAT team members in spring and summer of 2021.