Supportive Supervision and Resiliency Ohio - Final Summary

Authors

Quality Improvement Center for Workforce Development (QIC-WD)

Document Type

Article

Citation

© The Quality Improvement Center for Workforce Development (QIC-WD)

Abstract

Supportive Supervision and a Resilient Workforce

Ohio Department of Job and Family Services (ODJFS) is a state-led, county-administered child welfare system. Ohio’s 83 single-county agencies and two multi-county agencies are responsible for the delivery of child protective services and ongoing case management in Ohio’s 88 counties. In 2017, ODJFS had an annual turnover rate of about 27%. They applied to be a Quality Improvement Center for Workforce Development (QIC-WD) site with the goal of strengthening their child welfare workforce.

When ODJFS started working with the QIC-WD, a Workforce Implementation Team (WIT) was established to participate in a needs assessment process, determine an intervention, and support implementation. The needs assessment included discussions with multiple county leaders, surveys, and a review of Human Resources (HR) data over the period of months. The results pointed to issues related to supervision at every level within the agencies. The QIC-WD also conducted surveys with 588 Ohio child welfare workers across the nine counties involved in the project and found that the organizational culture and climate across all participating counties was above average in rigidity and resistance, and below average in engagement. The survey also revealed that, on average, 53% of respondents had recently excperienced elevated levels of secondary traumatic stress (STS) symptoms. It was noted that supportive supervision could enhance engagement and address rigidity, resistance, and STS. Focus groups with 90 supervisors across the nine counties found that they desired more support from both managers and peers, were willing to support staff more, and believed it was important to engage staff directly in learning resilience skills. A theory of change that addressed supervision and STS was ultimately used to guide program development.

The WIT created Coach Ohio, a multi-level supportive supervision intervention, to help child welfare staff within the original six Ohio implementation counties prevent and mitigate the effects of secondary trauma, employee dissatisfaction, and disengagement from families and children served by the agencies (see the Intervention Background for more information). The team used a logic model to connect the intervention activities and expected outcomes. Coach Ohio includes an adaptation of Resilience Alliance (RA) and the Atlantic Coast Child Welfare Implementation Center (ACCWIC) Coaching Curriculum.
This video highlights the experience of workers, supervisors, and administrators who were involved in Coach Ohio.

The site began to implement Coach Ohio in six counties. Directors, managers, administrators, and frontline supervisors were trained in the ACCWIC coaching model. The coaching model involves being present, listening, reflecting and clarifying, questioning, giving feedback, and holding staff accountable. Coaching was used to reinforce RA concepts once that part of the intervention began. The intervention counties offered 24-weeks of RA and the counties varied in how they managed RA facilitation. In most counties, leaders and frontline staff met separately. For leaders, separate RA groups enhanced peer support. For frontline staff, separate RA groups facilitated a sense of safety to talk openly about their experiences. Participants followed an RA manual, participated in 1-hour weekly groups, and practiced skills in between sessions with support and coaching from their supervisors. Peer-led support groups continued in the intervention counties after the 24-weeks of RA. These groups met in-person until the COVID-19 pandemic caused offices to close, and then meetings were virtual.

The QIC-WD evaluation team carefully monitored the implementation of Coach Ohio, including the fidelity to both the ACCWIC coaching model and RA. Fidelity was measured by checking on adherence, dosage, quality, and staff engagement, as well as the impact of the intervention over time. Evaluation of fidelity to each aspect of the intervention found that the ACCWIC coaching model was strong with high dosage (98%), high adherence to the curriculum (92%), high quality ratings by participants in the coaching training and follow-up sessions (82%), high engagement of participants and high scores on a transfer of learning measure that research has shown is predictive of actual knowledge transfer to the field (79%). Frontline staff reported their supervisors high on usage of the coaching skills (73% rated 4 or higher on 5-point scale) and in each area of being present (78%), listening (69%), reflecting (69%), asking questions (71%), giving feedback (76%) and holding accountable to use the RA skills (72%).

Evaluation of fidelity to implementation of the RA groups found that 81% of participants had very high exposure to the intervention. In other words, they attended at least 20 of the 24 sessions (or 80% of the sessions offered). Adherence by facilitators to the RA curriculum was also high (97%), ratings of quality of the RA sessions by participants was high (90%), as was a sense of engagement by facilitators during sessions (83%). There were also high scores on a transfer of learning measure (78%), indicating that participants gained skills during their RA sessions that they could apply to their job.

For all but one county, a longitudinal quasi-experimental study design was employed to assess the impact of the intervention over time. The results focus on the comparison between four intervention counties and three comparison counties. (Results from the large county that engaged in a wait-list control design are not included here but are similar.) A mix of survey and administrative data were used to evaluate the intervention. There were no differences at baseline between those in intervention vs. comparison counties in the factors assessed. So, even though the design was quasi-experimental, there was equivalence at baseline between the two groups.

Key findings include:

• Coach Ohio participants reported higher levels of active coping, resilience, work-life balance, and supervisor support than those in the comparison counties, although they did not show more optimism which may be a more stable trait.
• STS symptoms significantly increased over time in staff in the comparison counties, particularly after pandemic-related lockdowns. Although participants in the intervention condition (Coach Ohio) also reported that their STS symptoms persisted over time (6-months after the intervention began), symptoms did not increase.
• Coach Ohio participants reported higher levels of work engagement and job embeddedness post-intervention, than did comparison participants.
• Coach Ohio participants reported more job satisfaction, intent to stay, and lower intent to leave (overall, thinking about quitting and looking for a job) than did those in the comparison counties.
• Resignation rates were comparable in the intervention and comparison counties prior to the implementation of Coach Ohio. Once the intervention began, resignations were significantly lower in the intervention counties than in the comparison counties. In fact, caseworkers in the intervention were 2.73 times more likely to remain with the agency than those in the comparison counties. The exit rates did not change for those in the intervention counties but declined in the comparison counties during the pandemic, wiping out the differences. The follow-up finding was confounded by the pandemic effect.
• The study examined all components of organizational culture and climate (OCC) from before the intervention began and annually for several years. Three of the four targeted OCC areas showed improvements in the intervention counties but not in the comparison counties. Those in the intervention perceived reductions in the rigidity of the agency but not resistance (culture measures), as well as reductions in stress, and increases in engagement (climate measures).

Fidelity to all components of the Coach Ohio implementation was high and buy-in and uptake by county agencies involved in the intervention was strong. Therefore, using a quasi-experimental evaluation design, comparisons of attitudes, perceptions, emotions, and behaviors can be attributed to the intervention. The QIC-WD evaluators were able to conclude that the multi-level Coach Ohio intervention had the desired impact.

County and state child welfare leaders and staff saw the value of the Coach Ohio intervention as it was being rolled out, including during the COVID 19 pandemic. Higher resourced counties immediately invested in continuing to train new child welfare supervisors in the ACCWIC Coaching Model and facilitating RA groups for new child welfare employees (once QIC-WD support ended). They also expanded these Coach Ohio components with other social service staff in their agencies. Lower resourced counties requested, and will be receiving, state funds to continue Coach Ohio in their counties. Through the Ohio Workforce Initiative, counties that did not implement Coach Ohio as part of the QIC-WD project will have the opportunity to do so in the future. A Coach Ohio Implementation Manual is available so that other counties can deliver the model as tested by the QIC-WD with the goal of strengthening their child welfare workforce.