Test personality/depression/cognitive models with large models! Measuring psychological traits through game plot development

Psychometrics play an important role in mental health, self-understanding, and personal development.

Traditional psychometric methods rely mainly on participants filling out self-report questionnaires, which are measured by recalling behaviors and emotions in daily life.

Although such a measurement method is efficient and convenient, it may induce resistance among participants and reduce their willingness to be measured.

With the development of large language models (LLM), many studies have found that LLM can show stable personality traits, imitate human subtle emotions and cognitive patterns, and can also assist various Various social science simulation experiments provide new research ideas for many psychological research fields such as educational psychology, social psychology, cultural psychology, clinical psychology, and psychological counseling.

Recently, a research team from Tsinghua University proposed an innovative psychometric paradigm based on a multi-agent system based on a large language model.

Different from traditional self-report questionnaires, this study generates an interactive narrative-type game customized for each participant , users can customize the type and theme of the game.

As the game plot develops, participants need to make different choices based on various plots from a first-person perspective, thereby affecting the progress of the plot. By studying participants' choices at key moments in the game, their psychological characteristics can be assessed.

△Comparison of the psychometric paradigm of self-report questionnaires (left) and the psychometric paradigm of interactive narrative games (right)

Contributions of this study Mainly reflected in three aspects:

• proposes a new psychological measurement paradigm, transforming traditional questionnaires into game-based interactive measurement; on the basis of ensuring the reliability and validity of psychological measurement, Enhance participants’ sense of immersion and improve the testing experience.
• In order to achieve the measurement of gamification, this study proposes a multi-agent interaction framework based on a large language model, named PsychoGAT (Psychological Game AgenTs), ensuring the generalization of psychological test scenarios and the robustness of measurement under different game settings.
• Through automated simulation assessment and real-person assessment, this study achieved both psychometric statistical indicators and user experience indicators on tasks such as MBTI personality test, PHQ-9 depression measurement, and cognitive thinking trap test. Demonstrated significant superiority.

Next, let’s take a look at the details of the study.

What does PsychoGAT look like?

△PsychoGAT framework diagram

Agent interaction process:

Given a traditional psychological test questionnaire, participants customize Game type and theme, and then the Game Designer agent will give an overall game design outline.

Then, the Game Controller

(Game Controller) agent generates a specific game plot. In the process, the Critic (Critic) agent will comment on the administrator. The generated content undergoes multiple rounds of review and optimization; the optimized game plot will be shown to the participants. After the participants make corresponding choices, the administrator promotes plot development based on this choice, following this interactive process cycle.

Detailed description of the functions of each agent:

• Game Designer(Game Designer): Use CoT technology to generate first-person narrative The outline of the game and ensuring that the storyline contains situations that enable participants to exhibit the psychological traits currently being measured.

At the same time, the standard psychological self-report questionnaire is adapted according to the current game storyline, making the integration of the two more natural and smooth.

• Game Controller（Game Controller）: The adapted questionnaire will be instantiated in sequence according to the story line of the game and become the part of the story. Plot nodes and provide possible options for participants to choose from.

At the same time, the game administrator returns the participants' choices to the game environment, and controls the plot trend of the game based on the participants' choices. In order to achieve the continuity of the game plot, the administrator agent adopts a "memory update" mechanism.

• Commentator（Critic）: Aims to review and optimize the content generated by game administrators.

Mainly targets the following three issues:

1) Optimize consistency: As the game plot progresses, the long text problem will become more serious, making The "memory update" mechanism also cannot fully guarantee plot consistency.

2)Ensure unbiasedness: The choices of participants will affect the development of the game plot, but before the participants make a choice, the administrator should not preset the plot direction, even if Participants showed clear preferences in their previous choices.

3)Correction of missing items: Conduct a detailed review of the game plot generated by the administrator to check whether it has basic game immersion.

Experiments and results

△Comparison of three common psychological measurement paradigms: traditional questionnaires, psychologist interviews, and the gamification proposed in this study Evaluation.

What is mentioned here are all automated measurements based on AI. In particular, psychologist interviews refer to the current interview paradigm that is combined with large language models and in which large language models play the role of psychologists. .

During the experimental phase, the researchers chose three common psychological measurement tasks: extraversion in the MBTI personality test, PHQ-9 depression detection, and cognitive distortion detection in the early stage of CBT therapy. .

First, the researchers compared it with mature traditional psychological questionnaires to test the psychometric reliability and validity of the study. Furthermore, it is compared with three other automated measurement methods to examine the user experience of different measurement methods.

The researchers first used GPT-4 to simulate the subject and recorded the measurement process and measurement results using different measurement methods. These measurement records were used to calculate subsequent psychometric reliability and validity indicators, as well as user experience indicators.

There are two evaluation indicators: reliability and validity indicators and user experience indicators.

• Reliability and validity index: In psychometrics, to evaluate whether a measurement tool is scientific, it is generally based on reliability(reliability) and validity (validity)Verify in two dimensions.

In this study, two statistical quantities were selected to measure internal consistency as indicators of reliability: Cronbach's Alpha and Guttman's Lambda 6; Pearson's coefficient was used as an indicator of validity to measure aggregation respectively. Validity (convergent validity) and discriminant validity (discriminant validity) .

• User experience indicators, manually evaluated indicators include:

1) Consistency (Coherence, CH) : Whether the content logic is coherent;
2) Interactivity (Interactivity, IA) : Whether there is an appropriate and unbiased response to the user's choice;
3) Interest (Interest , INT) : Whether the measurement process is interesting;
4) Immersion (Immersion, IM) : Whether the measurement process allows participants to be immersed;
5) Satisfaction (Satisfaction, ST) : Overall measurement of satisfaction with the process.

The following are the experimental results.

First of all, the researchers tested whether the PsychoGAT proposed in this study can be used as a qualified psychological measurement tool. The results are shown in the table below.

##△PsychoGAT’s reliability and validity test results (passed, good, excellent)

Further, the researchers compared different psychological Measuring the user experience of the paradigm, the gamified assessment proposed in this study is significantly better than other methods in terms of interactivity, fun and immersion:

△Users of PsychoGAT Experience results, as well as the corresponding results of other comparison methods

In order to ensure the effectiveness of manual evaluation, the researchers calculated the manual evaluation results, and the evaluation consistency in each PsychoGAT index is better than other methods:

△PsychoGAT’s user experience index is due to the consistency of the comparison method in manual evaluation

In order to further analyze PsychoGAT, the researchers first examined different game scenarios Under the circumstances, the reliability and validity of gamification measurement are very robust:

△The robustness of PsychoGAT in measuring reliability and validity in different game scenarios

Then, the role of each agent in PsychoGAT was explored:

△The role of different agents in PsychoGAT

Finally, in order to visually present the game-generated content of PsychoGAT, the researchers used word clouds to visualize the extraversion test and Depression test:

#△PsychoGAT generates visualization of game scenarios for extraversion measurement and depression measurement.

The content of the extraversion test mainly focuses on social situations, while the depression test focuses on personal thinking and emotions.

For more research details, please refer to the original paper.

Paper link:https://www.php.cn/link/4bcd537b6c034e297f0030cf08887426

The above is the detailed content of Test personality/depression/cognitive models with large models! Measuring psychological traits through game plot development. For more information, please follow other related articles on the PHP Chinese website!