In a study published in the Journal of Applied Psychology, researchers found evidence that artificial intelligence (AI) chatbots, employing sophisticated machine learning algorithms, can effectively infer personality traits from text interactions. This innovative approach, tested on over one thousand undergraduate students, demonstrates a potential new frontier in personality assessment — and may offer an alternative to traditional methods, such as questionnaires.
The study builds on existing personality assessment methods. In the past, personality traits have been measured through self-report questionnaires in which participants answer a series of questions about their feelings, thoughts, and behaviors. However, these methods are considered to have drawbacks, such as participants’ honesty and the practical challenges of long surveys. The new study explores AI’s potential in this field, using chatbots to analyze language use during online interactions and draw correlations with known personality traits.
The motivation behind this research was to investigate whether AI could offer a more efficient, reliable, and potentially unbiased alternative to traditional personality assessments. In an era where AI is increasingly becoming a part of our daily lives, the researchers aimed to harness its potential for understanding complex human characteristics like personality.
The methodology involved 1,444 undergraduate students recruited from a large southeastern public university in the United States. Participants first completed a well-established Big-Five personality measure (IPIP-300), and then interacted with an AI chatbot for about 20 to 30 minutes. The chatbot analyzed their text responses, using machine learning algorithms to infer personality scores. This method allowed for a direct comparison between the established self-reported personality scores and the new, machine-inferred scores.
The findings of the study were multi-faceted, yet telling. The machine-inferred personality scores showed acceptable reliability — meaning the AI method could consistently and dependably measure personality traits. The factor structure of these scores was found to be comparable to that of self-reported scores, suggesting that the AI method can capture the complexity of human personality similarly to traditional questionnaires.
However, the study noted some challenges in differentiating between certain traits — a factor known as “discriminant validity.” Despite these challenges, the AI method demonstrated potential incremental validity, particularly in predicting academic performance and social adjustment among college students. This suggests that AI-inferred scores could add unique predictive value beyond traditional self-report measures in certain contexts.
It is important to note that the participant pool was mainly comprised of young, female, college students, may not represent the broader population, potentially limiting the generalizability of the findings. This, and the default interview questions used by the AI chatbot were not specifically designed to probe personality traits, which might have influenced the outcomes. The study’s approach to analyzing language also raises questions about content validity, as the deep learning models used are not always interpretable or directly linked to theoretical concepts of personality.
As AI continues to evolve, its role in psychological assessment and other areas of human understanding is likely to become increasingly significant — and the findings in this study may serve as the first steps towards new insights into the complexities of human nature.
The study, “How Well Can an AI Chatbot Infer Personality? Examining Psychometric Properties of Machine-inferred Personality Scores”, was authored by Jingyan Fan, Tianjun Sun, Jiayi Liu, Teng Zhao, Bo Zhang, Zheng Chen, Melissa Glorioso, and Elissa Hack.
