Competency-based Personnel Selection Oklahoma - Evaluation Overview

Authors

Quality Improvement Center for Workforce Development (QIC-WD)

Document Type

Article

Citation

© The Quality Improvement Center for Workforce Development (QIC-WD)

Abstract

The QIC-WD evaluation was conducted with the support of the Oklahoma Human Services (OKDHS) to determine if a Competency-based Personnel Selection process was effective in improving workforce and child welfare outcomes.

Research Questions

The primary research questions concerned the ability of the structured hiring tools to predict job performance, tenure, and turnover. It was hypothesized that the use of the standardized hiring process would result in the selection of candidates who performed better and had lower rates of turnover and longer tenure than candidates hired using existing selection processes, see Logic Model. Intermediate outcomes examined included employee feelings of self-efficacy, perceptions of fit with the organization and job demands, organizational commitment, engagement, and job satisfaction. Improved performance and reduced turnover were hypothesized to lead to reductions in caseworker stress and burnout and fewer changes of caseworker for children and families. This increased continuity was expected to increase child safety and reduce time to permanency.

Evaluation Design

The evaluation encompassed an implementation evaluation, a validation study, and a randomized control trial (RCT). The implementation evaluation gathered participant reactions, training, and fidelity data. A variety of measures were used. Focus groups were conducted with hiring panel members in early implementation districts to gain in-depth feedback for program improvement. Training data were analyzed for accuracy and consistency in scoring of mock interview excerpts and participant perceptions of readiness to implement the standardized hiring process. Survey questions were used to assess applicant reactions and hiring panel buy-in and satisfaction with the hiring process. Administrative data and observations by OKDHS Programs Analysts of a sample of hiring interviews were used to assess adherence to the standardized hiring protocol and identify areas where further coaching was needed to increase fidelity.

The validation study examined the extent to which a set of commercial and academic measures of personality, cognitive abilities, and attitudes predicted job performance in a sample of current Child Welfare Specialist (CWS) Is and IIs with the goal of identifying measures to supplement the standardized hiring process.

The RCT component evaluated performance, tenure, turnover and child and family outcomes, as well as attitudinal variables linked to turnover in the literature. The unit of assignment was the district or geographic area covered by a hiring authority, stratified by rural/urban composition and region of the state. Foster Care/Adoptions units, which have a different organizational structure, were randomized at the Field Manager level rather than at the district level. (See the Site Overview for more information.)

Measures for this component included:

• a culture and climate assessment administered to all current CWS staff,
• a survey of supervisor attitudes linked to turnover in previous research,
• a new-hire survey administered in two parts; at one month after hire and at 6 months after hire as they assume a caseload,
• a performance assessment completed by the supervisor for each new hire at 6 months from the hire date, and
• administrative data from human resources and child welfare systems.

Timeline

Evaluation activities began in three early implementation sites in March 2020 with virtual focus groups with hiring panel members and an analysis of training data. Use of the standardized hiring process was suspended in March 2020 with Covid-19 shutdowns, and during this time period the process was adapted to a virtual and paperless format. Evaluation activities resumed with the validation study, which began statewide in August 2020 and continued throughout the study period. Use of the standardized hiring process for the experimental group restarted following virtual refresher training for the early implementation counties in December 2020 and continued through the end of March 2021 with virtual training for hiring panel members in the full implementation districts and foster care/adoptions units. Full implementation began on a rolling basis as hiring panel members completed training and were authorized to begin using the standardized hiring process. The first RCT surveys were administered in April 2021. New-hire surveys and associated supervisor performance assessments continued on a rolling basis throughout the study period.