A Commentary from Previsor on the Risks of Profiling in the use of Pre-Employment Psychometric Assessment Tools in building a Conscious Hiring Process.
Overview
This commentary describes the profiling approach to using pre-employment psychometric assessments, compares it to the predictive approach to using assessments, and
then contrasts these two very different approaches. It then describes
the dramatically different results derived from an analysis in which
the two methods were compared side-by-side in the selection process for
a given organization. At KeenHire, we use both approaches in tandem to deliver the best hiring process to our customers. The Key is in the Role Analysis (referred to as job analysis in this article).
What is Profiling and How Does it Compare to the Predictive Model?
The profiling approach involves assessing a small group of employees
that have been identified by the organization as high-performers, and
using the resulting test scores to determine how to evaluate job
candidates. Candidates whose assessment scores more closely resemble
the scores obtained by the high-performers are judged to be better
“fits” for the role, and are forecasted to be better hires. A
predictive approach, by contrast, involves identifying job-relevant
traits based on an in-depth job analysis, and selecting those
candidates who earn higher scores on measures of those traits.
Seven Problems with the Profiling Approach to Assessment
Most
assessment experts do not advocate using a profiling approach, even
though it would be easy for them to implement, due to seven major
problems with profiling. These are summarized below.
#1: The "Ceiling Effect" Problem
The scores obtained by
the high-performing employees asked to complete the assessment define
the ceiling—the maximum possible score that can be obtained on the
assessment. This means that it is actually impossible to identify a
candidate who is a better fit for the position than current
high-performers. The high performers used to develop the model don’t
just represent “desirable,” they represent “the best imaginable. ”The
profiling model has no answer for the question, “If a candidate came
along who was an even better fit for the position than your current
high performers, how would you know?”
#2: The Not Enough Data Problem
The profiling approach
relies on a large number of conclusions and decisions made on the basis
of a very small amount of data. Profiling assumes that the test scores
obtained from a handful of employees produces information that will be
consistent with and generalizable to job candidates. But the
unfortunate reality is that small data samples are very rarely
conclusive. Models built on the basis of test scores from five people,
or even fifteen people, are very unlikely to be accurate. They may not
even be close, and can even lead to false conclusions. At KeenHire, we measure the actual on the job performance of every participant in the job benchmark study. If the on the job results are not exceptional, we encourage our customers to exclude that participant from the study.
#3: The (Non)Essential Trait Problem
Profiling is more
likely to incorrectly identify unimportant traits as job-relevant, or
overlook traits that are essential to job success. This is because the
profiling approach is based on a few individuals and their traits, not
on how relevant those traits are to the job demands or requirements.
Maybe the handful of assessed employees just happen to share traits
that are not actually job relevant. Maybe they differ on traits that
are job relevant because one or two of them have learned to compensate
for a limitation in one area with particular strengths in others. Maybe
the traits common to the top-performers who were tested are the same
traits possessed by weaker performers—but since weaker performers were
not assessed as part of the approach, there is no way to know. Any of
these possibilities would result in a profiling model that has no value
in identifying future high performers.
#4: The “Over-Cloning” Problem
Profiling emphasizes
similarities without any regard to whether or not they are important
similarities. For example, if the profile is created based on a
ten-scale personality instrument, but only four of those scales are
actually relevant for success in the position, the organization is
needlessly forcing consistency and uniformity in six areas. Good,
high-potential candidates may be passed over because they are not
sufficiently similar to current employees in areas that have no bearing
on their likelihood of success. At KeenHire, our process begins with an Organizational Development approach. We work diligently to build a balanced organization by first digging deep into who the organization is, we seek to understand the current corporate culture, the desired culture, the strategic objectives and goals as well as the barriers to achievement. We then assess and evaluate the Executive team,the mission critical and key contributors. We look for what is working, what is not and what is missing, the presence of which would make a difference. Only after we have a full picture of the current state and the desired future state do we make recommendations. No one with any level of discernment or sanity really wants the "Stepford" culture. Most companies simply do not understand how to build a high performance culture. That is where we come in and make the difference.
#5: The Missing Job Analysis Problem
Profiling does not
include a job analysis—which makes the assessment process harder to
defend if legally challenged. Federal guidelines pertaining to testing
specifically mention a job analysis that examines and documents the
actual work behaviors that are important for a given job. Profiling
focuses exclusively on the traits of some employees performing the job,
not on the work behaviors that are necessary for performing that job.
Profiling therefore lacks an essential set of documentation that is an
established requirement in federal guidelines. At KeenHire, we all too often see this crucial step missing. Some companies do not have the patience, the time, or the where with all to sit through the process. Actually, in our search business, we do not accept an Executive Search assignment unless this discipline is followed.
#6: The Measurement Precision Problem
Profiling methods,
even if they are conducted using well-designed, reliable tests,
inevitably reduce the precision of test scores, yielding a less
accurate evaluation of job candidates. Because profiling relies on
subtracting job applicants’ scores from high-performing employees’
scores, any measurement inaccuracy encountered when testing the high
performers is applied to each and every evaluation of each and every
candidate who later applies to the organization. This means that
measurement errors are potentially doubled when using a profiling
approach.
#7: The Re-inventing the Wheel Problem
Profiling
approaches do not use or benefit from relevant existing data and past
findings, but rather treat each situation as a unique one. Each
situation, each application, each position requires development of a
completely new framework and each and every organization is required to
discover anew exactly what is relevant for success. The profiling
approach assumes that the assessment scores obtained by a handful of
individuals should outweigh everything else that is known about a given
job role, about what is relevant for success in that type of role, and
any past findings with a given test for that type of role.
Most assessment professionals would probably use the profiling approach if it were as effective as the predictive approach. This would actually be easier for them than the method they do use. But investigations into the approach have emphatically demonstrated that profiling just does not work very well. Consider the following example, in which assessment scores from high-performing employees were used to model the profiling approach and compare it to the predictive approach.
The bottom line is effective Performance Management, Recruitment and Employee Retention begin with a clearly defined game plan. You can not reduce unwanted employee turnover if you do not understand the reasons it is happening. Every customer of KeenHire learns why they get the results they have been getting when they participate in our comprehensive workforce & role analysis process. If you want to make the difference you have been hired to make you need to do your homework, investigate what your current state it, define what you need for a future state and make it a priority to do the tough work to figure out how to get there.
