Summary of the ProfileXT by Scott G. Howard

Scott G. Howard
SGHoward and Associates
Denver, CO 80111

I was asked to provide an independent evaluation and summary of the ProfileXT for Profiles International, Inc of Waco, TX. Below is the description of that evaluation. My background includes twenty years of test development and validation work, doctorate in psychology with specialization in employment testing and development, and continuous experience in test construction and validation with public and private-sector organizations. 

Introduction:  

The ProfileXT is an instrument that brings together three of the key groups of information required for employee selection and development. These groups are: Behavioral Traits, Occupational Interests and Thinking Style. Representative results from each of these groups are compared to success/failure patterns for real employees (termed job Match patterns), and outcome comparisons are reported. The ProfileXT Report, a graphic and narrative summary of the testing results will yield one of several summaries depending upon the application, (i.e. selection, development, coaching, career guidance, etc.). In combination with traditional selection and development methods like structured interviews and work-sampling, the ProfileXT provides data which independently quantifies, verifies and identifies traits and characteristics relevant to future performance. As a reference point, the ProfileXT allows comparison to other information gathered, and provides important information for follow-up (such as in questions during a second or third interview). By taking this approach, the ProfileXT adds what is sometimes called incremental validity to the process of selection, development planning and coaching.

Specific reliability and validity information is described in the manual for the test, titled "The ProfileXT Technical Manual, September 2005." This summary includes a review of these areas of test utility as expressed in statistical findings from the manual. A sampling of the statistical presentations was made, and all formulae and table inclusions from the ProfileXT Technical Manual are correct and conforming to standards for such presentation (American Psychological Association Manual, 2005).

While many will apply the results of the ProfileXT in the aggregate (such as using Job Match Patterns), it is also important to consider the foundation of the measures, and the constructs being measured and reported. This summary also reviews the basic scales measured, the items comprising those scales, and the general input that yields the PXT Report and its underpinnings.

Finally, the ProfileXT is reviewed in relation to practical application, impact on protected groups, disparity of result by group, and predicting performance as a composite means of identifying utility (in this case, economic value of the measure as intended for use).

 The ProfileXT :

 Behavioral Traits

The nine Behavioral Traits (hereinafter, the "nine BT indicators") and one Distortion Scale are described in the PXT Manual by name and by constructs purported to measure. Each of the nine BT indicators is classically derived as to traits under scrutiny for more than 50 years in the applied psychology literature (Gough, Harrison 1956). Sometimes recombined for brevity (the "Big Five" Personality characteristics) and sometimes placed in new, more modern categories, these classically derived traits in the ProfileXT have been tried-and-true measurements and will likely hold up so long as human behavior exists. Energy Level, Sociability, Independence, etc, some of the nine BT indicators within PXT, will serve to describe vital information in selection and coaching, among other applications. Each of the PXT scales shows thorough construction and efficiency of administration so as not to "over-measure" with redundant questions. Most important; however, the nine BT indicators have industry-leading predictive validity for overall job performance (as expressed in Job Match and validity data).

Separately, the Distortion scale has been derived to express a brief answer to a commonly asked question of all psychological testing: can this be faked? While the ProfileXT Behavioral Traits are drawn from well-designed questions and producing constructs not readily apparent to the test-taker, the Distortion scale allows for an extra measure of certainty. It displays a simple indicator and description of the test-taker's approach to the measure, and whether or not the results should be viewed as an accurate portrayal. As a point on a continuum of confidence, the Distortion Scale has high internal consistency and allows modification of the result interpretation so that test-taking behavior is not an issue.

Construct validity is particularly strong for the PXT nine BT indicators, with some of the most respected tests in the industry used to assess congruence, such as the California Psychological Inventory, the G-ZT Survey and Gordon Personal Profile-Inventory. The ProfileXT shares with sound instruments like these, some of the development approaches key to strong psychological measures. Correlative results were viewed by using standardized regression weights taken from the PXT Manual, and suggest that PXT behavior traits are at or above required validity coefficient measures for tests of this kind. Varying sample sizes and consistent outcome measures by job type allow for a comparison across studies. When reviewing the personnel measures cited in the many real-world studies in the manual, validity, multiple regression R and gain in validity from using the PXT show strong predictive validity for the ProfileXT's nine BT indicators.

Occupational Interests

The ProfileXT Occupational Interest Section is again an application of lengthy, well-researched elements of fitness-for-jobs and careers. In this section, the interests draw upon the decades' long work begun with J.L. Holland's Theory of Vocational Choice (Holland, J.L., 1959. Journal of Counseling Psychology). The theory that spawned most credible interest measures (including the Strong Vocational, Strong-Campbell, Kuder-Preference, etc.) is behind the ProfileXT Occupational Interests section. The six major categories are placed into occupational themes, rank-ordered, plus secondary and discussion section of the PXT Report, and provide insight into the candidate's interest at or above a level measured against the classic interest measures.

 Of primary importance in reviewing the support data for this section is the fact that the ProfileXT interest measures have distinct internal consistency. This means that the scales separately identify areas of interest, in rank order, and keep the rankings clear of distortion from other scales. Patterns for those with strong interest correlate strongly with job satisfaction indicators of people on their specific job. Interest groupings (where research has shown similarity) also correlate at significant levels, for groupings within the ProfileXT measures.

Thinking Style

Of the principal measures available to psychologists for predicting job success, coaching career clientele and identifying areas for development, learning measures rank among the highest. In a meta-analysis of these measures completed for the University of Iowa and Michigan State University (1998, Psychological Bulletin), general mental ability was found, when combined with behavioral traits and work sample data, to have the greatest applied validity and predictive ability compared to all other types of assessments.

The ProfileXT Thinking Section combines the most accepted types of general learning measures from concrete/static-type learning measures, with abstract/dynamic-type measures. In doing so, five scores (four original and one combined) are produced. The data comparing the results of the four-plus-one scale results to job-criteria further support the idea that the PXT is a strong measure of job success. In terms of variance of scores to performance, the reliability measures and coefficients of validity are at levels that meet or exceed accepted standards put forth as testing guidelines by the U.S. Department of Labor's publication: Testing and Assessment: An Employer's Guide to Good Practices, 1999.

Any organization using measures of general ability, in whole or in part, should monitor adverse impact potential. Of particular interest to test users should be the impact data found in the ProfileXT Technical Manual, Appendix C. These data support the contention that the Thinking Section of the ProfileXT does not have a discernible impact on groups identified as protected. Further, the results appear to demonstrate no adverse impact concerns with regard to pre-employment testing when using standard errors of measure, for age, sex or race. Data indicate uniformity of results within accepted ranges of deviation both within and across groups. Samples of results from protected groups do not show adverse impact based on group inclusion, and separately, job success (outcome) measures are consistent between majority and minority populations reported.

Conclusions and Recommendations

The ProfileXT has been designed with the purpose of identifying key behavioral, interest and learning traits. When applied to hiring, coaching and development of people, these results stand on their own as valid, reliable and useful. Useful in this case means specific business utility, where the predictive validity coefficient of the aggregate test result is directly proportional to economic justification. ProfileXT has strong utility, and its technical data reviewed for this summary, suggest that it is fair and unbiased in its recommendations when applied for the purposes designed.

The foundational measures of the ProfileXT include the major components of useful tests gleaned from more than 85 years of research in psychology. These include behavioral construct traits, interests, knowledge and quickness of learning. When combined with behavioral interview findings and work-sampling data (resumes, education, work history, etc.), decisions based using the ProfileXT should have a positive impact on retention, development and incremental value-added utility for individuals who benefit from the PXT results.

To fully realize its potential, it is strongly recommended that the results be used to develop and train people according to strengths and weaknesses identified in the PXT Report. It is also recommended that the individual organization keep statistics on its test job match patterns, successful and unsuccessful people, their traits, and how the demands of the job change in time. By doing so, more custom and specific job match patterns may be developed, and the patterns can be modified to reflect changes in the particular organization's needs.