Risks to the Promise

Despite this promise, AI-generated work analyses still face risks. Importantly, if the underlying job postings or job descriptions themselves reflect biases or omissions, subsequent outputs will reproduce those errors. Raisch and Krakowski (2021) describe this as the automation–augmentation paradox: full automation or full human judgment alone each brings risks. Unchecked, AI-generated job criteria could inadvertently identify qualifications that are not job-related or otherwise skew job requirements, leading to potential error in making talent management decisions.

Emerging evidence shows that human review improves AI output.  Several researchers have found that human review and expert input are beneficial in adapting AI output to local organizational contexts, aligning with cultural norms across global regions, and ensuring tools are applied ethically, legally, and strategically (e.g., Budwar et al., 2022; Oswald et al., 2020; Putka et al., 2023). While speed and reduced costs are benefits provided by AI solutions, it appears that human involvement plays a critical role in ensuring content accuracy and user acceptance.

Importantly, human review did not involve recreating the work analysis from the ground up.  Instead, supervisors and managers reviewed AI-generated content to verify its accuracy, remove information that was not reflective of the job, add missing responsibilities or skills, clarify ambiguous language, and tailor the content to the organization’s specific context. In most cases, reviewers refined and contextualized rather than replaced the AI-generated output, improving on the original AI-generated solution.  

Although this additional review required some time, it represented a small fraction of the effort traditionally required to develop comparable work analysis information from scratch. The resulting improvements in accuracy, completeness, and downstream usefulness suggest that this investment of human expertise provides substantial returns in quality. 

The APTMetrics study also examined the downstream impact of AI + human-review on a variety of HR processes.  A panel of highly experienced work analysts (i.e., Industrial-Organizational psychologists with extensive experience gathering and using work analysis data) were asked to rate the usefulness of the job content from the two AI-generated work analysis tools and the AI + human-review solution. Raters were blind to the source of the content provided. Raters evaluated the usefulness of the content from each source for a variety of different talent management applications such as selection, performance management, and pay equity analyses.

The AI + human-reviewed job content was rated as significantly more useful (p<.001) than the job content generated by AI alone across all 12 talent management use cases (see Figure 1).