Supportive Supervision and Resiliency Ohio - Key Findings

Authors

Quality Improvement Center for Workforce Development (QIC-WD)

Document Type

Article

Citation

© The Quality Improvement Center for Workforce Development (QIC-WD)

Abstract

Background

The Ohio Department of Job and Family Services (ODJFS) started working with the QIC-WD in October 2017 to better understand its turnover problem. Eight counties volunteered to be part of the entire study. They participated in a thorough needs assessment process which led to creation of Coach Ohio, a multi-level blend of two interventions that addressed key needs across the counties regarding work-related traumatic stress and supportive supervision. Four counties and half of the largest county participated in the intervention while three counties and the other half of the largest county participated as comparison counties. Among these counties the baseline average turnover rate was 20%, typical of child welfare agencies nationally at the time according to a study by Edwards and Wildeman.

The Organizational Social Context (OSC) and the Secondary Traumatic Stress (STS) scales were administered in February 2018. The OSC measures organizational culture (proficiency, rigidity, and resistance) and climate (engagement, functionality, and stress) and scores for each county were normed in comparison to 81 other child welfare jurisdictions across the nation. The profiles across the participating Ohio counties were remarkably similar indicating very high levels of rigidity and resistance and low engagement. This coupled with the high rates of secondary trauma symptoms of arousal (50%), avoidance (47%) and intrusive thoughts (48%) and the presence of elevated post-traumatic stress disorder symptoms by 53% of staff, based on meeting cut-off scores, led the team to develop an intervention to address both sets of challenges through the adoption of a supportive supervisory approach to reinforce gains in resilience skills to directly address secondary trauma through use of an adapted version of a structured group process developed in New York called Resilience Alliance (RA).

The site logic model outlined the intervention-specific activities and the expected outcomes based on the overarching theory of change. The evaluation included a study of the implementation across all intervention counties. A quasi-experimental design was utilized with seven counties and a wait-list control design was utilized with the largest county to evaluate the effectiveness of the intervention.

Workforce Demographics

Most participating supervisors (N = 106) were white females (female = 83%, White = 74%, Hispanic = 2%, and Black = 21%, non-Hispanic, multi-race = 3%, Asian < 1%). The average age of supervisors was 42 years old, and the average supervisor had been in the child welfare field for 14 years. Most supervisors (70%) had a bachelor’s degree (28% had a Bachelor of Social Work degree) and an additional 62% had a master’s degree (more than half of which were in social work). Mean tenure as a supervisor was 8.2 years.

Most participating workers were white females (female = 85%, White = 70%, Hispanic = 2%, and Black = 25%, non-Hispanic, multi-race = 3%, Asian, Indigenous/ Pacific Islander, and Other = 2%). The average age of workers was 38 years old, and the average worker had been in the child welfare field for 10 years. Most workers (74%) had a bachelor’s degree, (25% had a Bachelor of Social Work degree) and an additional 31% had a master’s degree (half of which were in social work). Mean worker tenure was 9.5 years.

Top administrators and managers were first trained in the ACCWIC Coaching Model adapted to support resilience skill building to alleviate stress and trauma with subsequent coaching to ensure uptake in work with supervisors and proficiency in the model’s use. Then supervisors received the same training and coaching before the roll-out of RA groups. Resilience Alliance groups were held for affinity groups (e.g., supervisors met together, caseworkers met together) weekly for six months (June-December 2019) while supervisors of each level of staff reinforced each week’s learning through coaching and support.

Evaluation Findings

Out of the 150 managers (N = 26) and supervisors (N = 124) who attended one of the six ACCWIC Coaching Model trainings, the dose of activities they were exposed to (i.e., the number of activities each participant was present for during each of the trainings) ranged from 92% to 100% across six trainings with an average of 98%, far exceeding the goal of full exposure to the model. The goal was for 80% of participants to rate training execution (82%), relevance (86%), effectiveness (80%) and engagement (80%) items at a 4.0 or higher on the 5-point scale. There was also significant knowledge gain from pre to post training and participant scores on a validated transfer of learning scale approached 80% as well. Seventy-five percent of managers and supervisors engaged actively in follow-up coaching sessions to master the model and 72% of caseworkers reported that their supervisors utilized the six core behaviors of the coaching model. Thus, there is evidence to suggest that implementation of the coaching model was solid and likely contributed to outcomes along with participation by both supervisors and frontline workers in the Resilience Alliance aspect of the Coach Ohio intervention.

The goal was for the majority of RA participants to have exposure to at least 75% of RA material by attending 18 sessions or more. This goal was exceeded for the members of caseworker groups (N = 306) in four of the intervention counties in the larger quasi-experiment with attendance rates in the four counties of 80% (attendance at an average of 19 sessions, see image below). But participants in the RA groups that ran in a large county (N = 107) only attended between 10 (42%) and 15 (63%) RA sessions (55% or attendance at an average of 13 session) and only 29% of participants attended 18 or more sessions. So less than one-third received the full dosage in that county. Despite low attendance rates, including the large county results in the overall sample did not lower adherence rates which were 96% across all sessions. Ratings of quality of RA sessions approached 80% and engagement of participants by facilitators was 83%. Transfer of learning, however, was a bit lower at 70%. Thus, the low dosage rates in the largest county raised enough concerns that in subsequent analyses of outcome data (for this overview) the focus was on the remaining seven counties.

Overall, these high levels of fidelity to the Coach Ohio components and no significant differences between groups in baseline measures indicates that any differences between the intervention and comparison groups can be attributed to the intervention. After participating in Coach Ohio for six months, many of the expected impacts outlined in the project’s theory of change were found. First it was expected that Coach Ohio would impact seven attitudes and skills targeted in the intervention. It was hypothesized that those in the intervention group would experience more optimism, resilience, active coping, emotional regulation, and perceived support, and less stress and elevated STS. While the two outcomes of optimism and emotional regulation were not different between groups, the five variables of resilience, active coping, and perceived supervisor support were all greater in the intervention group than the control group while work stress and elevated levels of STS were lower.

Second, it was expected that Coach Ohio would impact eight short-term outcomes. It was hypothesized that those in the intervention group would experience more work-life balance, job satisfaction, work engagement, job embeddedness in the organization, and intent to stay with the agency and less thinking about quitting, looking for a job, and intent to leave. All eight short-term outcomes were significantly different between the two groups, that is, the intervention group had positive outcomes compared to the comparison group.

The study also examined the change in elevated levels of STS over time in the two groups to see if Coach Ohio served as a resilience factor upon onset of the pandemic. At baseline (February 2019, before the intervention) there were no differences between the levels of elevated STS in the two groups. Immediately after the intervention (December 2019) there were significant differences between levels of STS in the two groups, with the comparison group demonstrating an increase in STS. By the summer of 2020, (three to five months into the pandemic) a significant difference remained between levels of STS in the two groups, indicating that RA helped participants manage their STS.

Third, it was expected that Coach Ohio would impact long-term outcomes. It was hypothesized that those in the intervention group would experience reduced turnover from pre-intervention to during the intervention and post-intervention. It was also hypothesized that the organizational culture and climate would improve over time.

The hypothesis was supported for the staff present in the agency between pre-intervention (from January 2017 through May 2018) and during the intervention (from June 2018 through December 2019) periods. At baseline for the period before the intervention there was no significant difference in turnover rate between intervention counties (15%) and comparison counties (18%). But during the intervention period, there was a significant difference in turnover rate between intervention counties (10%) and comparison counties (24%). In fact, caseworkers in the intervention counties were 2.73 times more likely to remain with the agency than those in the comparison counties. However, because of the lockdowns and uncertainty wrought by the pandemic post-intervention (January 2020 through May 2021 when people were just beginning to get vaccinated before the surge in turnover that occurred after September 2021), turnover remained flat in both intervention counties (11%) and comparison counties (10%).

Fourth, hypotheses were mostly supported for the OSC analyses comparing standardized scores.

• Between 2018 (61.5) and 2021 (41.2) there was a significant decrease in seeing the organizational climate as stressed in intervention counties, but no significant difference between 2018 (58.9) and 2021 (41.0) was found in comparison counties.
• Between 2018 (60.6) and 2021 (54.0) there was a significant decrease in seeing the organizational culture as rigid, but in comparison counties no significant difference between 2018 (67.1) and 2021 (62.4).
• Between 2018 and 2021 there was a significant increase in both aspects of engagement (personalization 24.7 to 35.5, respectively, and personal accomplishment 28.9 to 36.7, respectively) in intervention counties, but no significant difference in the same time period in comparison counties (personalization 32.8 to 43.8, respectively, and personal accomplishment 33.7 to 43.7, respectively).
• No differences in resistance to change were found between groups.

In summary, the implementation of Coach Ohio was strong. Eighty-two percent of expected outcomes (14/18) were stronger in intervention than comparison counties or improved over time. Thus, the intervention shows promise through this quasi-experiment. A replication of the Coach Ohio model integrating both management/supervisor supportive supervision and coaching, directly engaging staff at all levels in learning resilience skills, and learning from one another in a peer support context is needed using a randomized control trial to overcome threats to validity in the quasi-experiment.

The Team

This project would not have been possible without the support of ODJFS and partnership in participating counties: Champaign County Department of Job and Family Services (DJFS) (Stacy Cox, Angie Robeson, Sara Wright), Crawford County DJFS (Melinda Crall, Andy Nigh), Hamilton County DJFS (Bev Kroeger, Amy Story), Huron County DJFS (Anna Wyss-Zilles, Brenda Milks, Lara Hozalski), Knox County Children and Family Services (Scott Boon, Courtney Lower, Danielle Wendal), Montgomery County Children Services (Amy Amburn, Jewell Good, Stephanie Marshall), Summit County Children Services (Dawn Boudrie, Amy Davidson, Sara Grubb, Michelle Matthews, Valerie Nash), Trumbull County Children Services (Richard Tvaroch, Lori Yedlicka), and Wayne County Children Services (Lisa Cygan, John Dillon, Michael Smith, Deanna Stika-Nichols). Representatives noted above from the counties and central office noted below served on the Workforce Implementation Team (WIT). This broad team identified an intervention to respond to shared workforce challenges and met regularly to plan for implementation and evaluation activities. Some counties participated in the intervention whereas others were in the comparison group. Counties and the state provided essential data and valuable insight throughout the project.

The QIC-WD would like to acknowledge the work of additional team members who contributed to the development, implementation and evaluation of Coach Ohio. QIC-WD team members included Anita Barbee, Cathy Fisher, Michelle Graef, Penny Putnam Collins who formed the WIE team, Michael Cunningham, Katy Henry, Lisa Purdy, Jenny Taylor and Andrew Winters who provided data collection, entry, checking, cleaning and analysis support, and Ashley Long, Jonathan Litt, and Robert Blagg who provided data download, organization and visualization support. The Data Coordinator at ODJFS was Kristine Monroe, and the Site Implementation Manager position was held by Tequila Washington, Vince Ciola, and Melissa Palmer all of ODJFS. Carla Carpenter, Lakeisha Hilton, and Lindsay Williams, of ODJFS, also served on the WIT in the first year. Finally, Sally Fitch and Laura Hughes from the Institute for Human Services, at the time, provided invaluable support on the WIT and facilitating participant training credits.