Telework Washington - Evaluation Overview

Authors

Quality Improvement Center for Workforce Development (QIC-WD)

Document Type

Article

Citation

© The Quality Improvement Center for Workforce Development (QIC-WD)

Abstract

The QIC-WD evaluation was conducted with the support of the Washington State Department of Children, Youth & Families (DCYF) to determine if a telework program was effective in improving workforce and child welfare outcomes. The evaluation, as outlined in the Logic Model, was approved by the Washington State Institutional Review Board (IRB).

Research Questions

The evaluation included two components, a process evaluation and an outcome evaluation. The following research questions were used to guide the evaluations.

Process Evaluation Questions

• Were telework-eligible staff aware of the telework program and its timelines, and do they understand how to obtain information about the program?
• Were telework-eligible staff satisfied with the telework program and the information resources regarding the program?
• Were telework-eligible staff satisfied with the process for applying to telework and its requirements?
• What was the degree of uptake of telework by eligible staff, and what were the facilitators and barriers?
• Did telework-participating staff implement their plan to telework
• Did staff with telework agreements continue to telework according to their plan?
• Were teleworkers satisfied with their experience while teleworking, including telework supports?

Outcome Evaluation Questions

• Did teleworkers perceive or experience better work conditions?
• Did teleworkers perceive improved work-life balance?
• Were teleworkers as productive or more productive with respect to work processes?
• Were teleworkers transferring internally at the same rates, or less frequently?
• Were teleworkers less likely to leave the agency altogether?
• Did teleworkers have greater organizational commitment?
• Did teleworkers experience less job-related stress, traumatic stress, and burnout?
• Did teleworkers use less leave and overtime?
• Did teleworkers achieve improved child welfare case outcomes?

Evaluation Design

The evaluation was organized as a cluster randomized control trial where clusters of telework-eligible staff were assigned to intervention or control groups at the office level. Eligible DCYF Child Welfare Field Operations (CWFO) staff in the intervention offices were invited to telework if they desired. The implementation of telework was divided into two components: Region 6 control and intervention offices as an initial implementation, and Regions 1-5 control and intervention offices were put into operation later as a full implementation (see the Site Overview for a map and implementation timelines). Implementation evaluation occurred among the intervention offices. Other data collection activities were conducted with both intervention and control groups. All eligible field staff were invited to participate in the following data collection activities:

1. Surveys—Two surveys were developed and administered at different points in time. Measures were included to capture perceptions of job stress and work environment, intentions to quit, perceptions of peer and supervisor support, workload, resilience, home life and dependent child care, work engagement, telework attitudes, reasons for teleworking, satisfaction with telework policy, and appropriateness of technology and telework supports.
2. Focus groups (converted to virtual interviews)—Included a set of open-ended questions about telework and Covid-19 experiences.
3. Fidelity survey—Intervention-office staff with telework agreements were invited to participate.
4. Administrative data—Two general types of administrative data were included: Human Resources (HR) and child welfare data (FamLink). HR personnel data included demographics, organizational affiliation, work history, staff development, leave use, salary history, and other personnel data. The FamLink data include child and family level characteristics and events (e.g., placements) used to construct child welfare related outcomes and productivity. Under IRB auspices all employee and child identifiers were encrypted to enforce confidentiality while retaining the capacity to link data across sources.
5. Autocene records—Telework-specific data from the application and agreement system for telework application tracking developed specifically for the project (Autocene), as described in the Intervention Overview. These data included telework start dates, planned telework days, etc. and were used to help assess the telework implementation. Autocene records were also linked through encrypted IDs to other data sources including surveys and HR data.

The outbreak of Covid-19 changed the nature of what could be evaluated in some key ways:

• Almost all DCYF CWFO staff statewide were required to telework within the first few weeks of the outbreak. The original program called for eligibility to telework only for certain classifications of staff who had worked for DCYF for 18 months and been in their current positions for at least six months.
• Staff who were originally eligible to telework, but elected not to, were also required to telework.
• Instead of a maximum of two days of telework per week, almost all staff began teleworking at least four days a week, and most of these were teleworking full time.
• The Autocene information system set up to track and process telework agreements became incidental, although it may be modified and utilized in the future as in-person work becomes possible.

Timeline

The initial implementation (in Region 6) began in June of 2019 with eligible staff in the intervention offices, and the control offices implemented in March of 2020. The remaining five regions implemented the intervention offices in September 2019, and the control offices were scheduled to implement in June 2020. The experiment was effectively ended as of mid-March 2020 as the Covid-19 pandemic occurred. The first survey was administered in March 2020 and the second survey was administered in summer 2020. The fidelity survey was administered in spring 2020. Interviews were conducted between March 2020 and January 2021. Construction of the administrative data and the analysis are ongoing.