Competency-Based Personnel Selection Oklahoma - Key Findings

Authors

Quality Improvement Center for Workforce Development

Date of this Version

7-2023

Document Type

Article

Abstract

Background

The QIC-WD began working with Oklahoma Human Services (OKDHS) in 2017. At that time OKDHS had an average turnover rate of 25% among Child Welfare Specialist (CWS) I and IIs. Agency leaders expressed a desire to better understand turnover rates with an eye toward increasing the retention of more competent performers. A thorough needs assessment led to the decision to design and implement a competency-based personnel selection intervention to address inconsistencies in hiring and better identify candidates with the desired competencies to be hired as a CWS. The evaluation was designed to include an implementation evaluation; a validation study on selected personality, cognitive and attitudinal measures; and a cluster randomized control trial (cRCT) comparing the use of the standardized hiring process structured interview, writing sample and typing test in the experimental group with “business as usual” hiring in the control group. The goal of the evaluation was to determine whether the use of the standardized hiring process developed for this intervention resulted in the selection of candidates who performed better, and had lower rates of turnover and longer tenure, than candidates hired using existing selection processes. Specific intervention components and expected outcomes were outlined in the logic model and theory of change. Full implementation began in September of 2020 on a rolling basis as hiring panel members completed training and were authorized to begin using the standardized hiring process.

Workforce Demographics

Data were collected from 399 newly hired CWS I and IIs across the state as part of the cRCT. Of these survey respondents 83% were female, 69% were White, 11% identified as Indigenous or Pacific Islander, 8% were African American, 8% were Hispanic, and 4% identified as multi-racial or other, and their average age was 35 years. The majority (85%) had bachelor’s degrees. Another 12% had master’s degrees. A little over 20% had social work degrees. Of the incumbents in the validation study 83% of survey respondents were female, 61% were White, 10% identified as Indigenous or Pacific Islander, 12% were African American, 7% were Hispanic, 8% identified as multi-racial or other, and 2% identified as Asian. The average age was 36 years.

Evaluation Findings

Hiring panel members and administrative assistants tasked with implementing the standardized hiring process showed high organizational readiness for change, favorable attitudes toward participating in a research study, and the knowledge and skills needed to implement with fidelity. Participants in focus groups, as well as survey respondents, indicated that they felt a standardized hiring process was needed, commenting “We wanted some more consistency in our hiring practices and decision making and this is a good way to do that.” Over 70% of administrative assistants rated their level of knowledge of the standardized hiring process following training from “moderate” to “very high.” An average of 84% of hiring panel members rated themselves as “confident” or “very confident” that they would be able to conduct the structured interviews and provide accurate ratings of interview and writing sample responses following training. They were able to score simulated candidate responses correctly in training evaluations with an average accuracy of 66%. Hiring panels were able to conduct and score 65% of selection protocol elements adequately when observed in a sample of candidate interviews, offering evidence of transfer of learning and adherence to the standardized hiring protocol.

During full implementation 716 interviews were conducted using the standardized hiring process, mostly by interviewer teams of two or three people. A total of 227 individual interviewers participated. Interviewers conducted as few as one interview and as many as 22 interviews, but, on average an interviewer conducted six interviews.

Candidates in the experimental group selected via the competency-based hiring process were significantly more likely to be satisfied with the hiring process than candidates in the control group and to agree that the process provided an opportunity to show their capabilities, provided valuable information about the job, and was conducted fairly. Candidates who were hired or offered a position scored significantly higher on a composite hiring score that included the interview, writing sample, and typing test than those who were not hired or offered a position. This composite score did not differ significantly by race (defined as a contrast between white and a combined group of other races/ethnicities), gender, and age (defined as less than 40 and 40 and above).

Those hired using the competency-based selection process were significantly more likely to have higher scores on a measure of work engagement and greater job satisfaction at six months than control hires, although hypothesized differences in self-efficacy, fit, and organizational commitment did not materialize. No significant differences were noted on additional measures of employee well-being: burnout, stress, secondary traumatic stress, or work-life balance. Based on supervisor ratings of behaviors associated with risk of turnover, the experimental group was less likely than control hires to have begun to express dissatisfaction with the job or show less interest in clients. However, there were no significant differences in supervisor ratings of other high-risk behaviors such as doing minimal work, leaving early, or displaying a negative attitude. There were also no group differences in intent to leave measured at six months.

Contrary to expectations, there was no statistically significant difference between the experimental and control groups in supervisor assessments of performance at six months. However, higher scores on the standardized hiring process structured interview and the composite hiring tool score were associated with higher scores on the supervisor assessment. Higher performance assessment scores were significantly associated with supervisor ratings of pre-quitting behaviors regardless of group.

CWSs who were hired using the standardized hiring process had a lower risk of leaving during the study period (by 22%) compared to the control group hired using the “business as usual” hiring process. Although fewer people in the experimental group (i.e., those hired using the standardized hiring process) left during the study period, they tended to leave more quickly than those in the control group. During the study period, the average tenure in the job for those who left was 4.3 months for the experimental group and 5.7 months in the control group.

The study found that higher scores on the EAS Verbal Reasoning Test and the global factors of Self-Control and Tough-Mindedness from the 16-PF personality assessment significantly predicted higher scores on the performance assessment completed by supervisors. Higher scores on these factors were also negatively related to turnover. For every 1-point increase in Tough-Mindedness score the odds of leaving decreased by 11% and for every 1-point increase in Self-Control score odds of leaving decreased 10%. Self-Control and Tough-Mindedness relate to how someone approaches decision-making and the activities of the job, according to the 16-PF manual. Individuals high on Tough-Mindedness lean toward familiar practices and tend to be objective and practical. Individuals high on Self-Control approach work in a structured and methodical way, are rule conscious, grounded, and perfectionistic. The remaining three global personality factors of extraversion, independence, and anxiety were not significantly related to either supervisor’s assessment of performance or staff turnover. It is possible that the personality traits captured by the Self-Control and Tough-Mindedness factors are well suited to the bureaucratic environment of a child welfare agency. (NOTE: A variety of cognitive and personality measures are available that capture these and other traits. Agencies wishing to undertake a similar selection process should select measures relevant to their organizational context and needs and validate any test under consideration to ensure that its use is legally defensible.)

In sum, although the hypothesis that supervisor ratings of performance would be higher in the experimental group than the control group was not supported, higher scores on the competency-based hiring tools were predictive of better performance. Moreover, the finding that the experimental group was characterized by lower turnover during the study period suggests that competency-based selection processes merit further investigation in child welfare.

The Team

This project would not have been possible without the partnership of OKDHS. They identified a statewide team that participated in monthly meetings over multiple years to plan and support the initiative and its evaluation. They were led by Site Implementation Managers, Constanzia Nizza and CeCe Anthony, and project sponsor Kevin Haddock. The state also provided essential data led by their Data Coordinator, Jessica Winscher, and implementation support from their training partners at the University of Oklahoma.

The QIC-WD would like to acknowledge the work of the team members who contributed to the development, implementation, and evaluation of the Competency-based Selection Process. QIC-WD Team members included Megan Paul, Cindy Parry, Penny Putnam Collins, Michelle Graef, Ida Drury, Stephanie Weddington, Ashley Long, Jonathan Litt, and Robert Blagg.