Participant Guidelines
Welcome to the Conversational Depression Detection via LLMs pilot task of eRisk 2025. This website provides essential guidelines for interacting with the LLM simulated personas and submitting results. Participants must carefully follow these instructions to ensure consistency and fairness in the evaluation process.
1. Task Overview
In this task, you will interact with LLM personas that have been instructed with user histories. Your objective is to determine whether the persona exhibits signs of depression based on the conversation. The LLM personas simulate real individuals and respond in a natural and coherent manner.
Types of Runs:
- Up to 5 runs using different methods.
- 1 run may be manually assisted, and can be supervised by a person.
- The remaining runs must be fully automated.
Independence of Methods:
- Each method must operate independently, without sharing information between runs.
2. Interaction with the LLM Personas
Number of Interactions:
- There is no limit to the number of interactions you can have with the LLM persona before making a decision.
- However, evaluation metrics will penalize late decisions (decisions after a lot of messages with the LLM). The earlier you make an accurate classification, the better your final score.
Conversational Approach:
- You are not expected to ask the LLM persona directly whether they are experiencing depression.
- The LLMs have been explicitly designed to avoid answering such direct questions and may respond as if they are uncomfortable when confronted with inquiries about their mental health.
- Instead, your system should infer the possible condition (or if it is a control user) by analyzing the persona’s language, tone, and expressed thoughts during the conversation.
3. Available LLM Personas
4. Submission Guidelines
Participants must submit two JSON files per run:
- A conversation log file documenting the sequence of interactions with the LLM persona.
- A classification results file containing the final depression assessment for each LLM persona.
File Naming Conventions:
Manual runs must be labeled with manual-run at the beginning of the filename.
manual-run_interactions_run1.jsonmanual-run_results_run1.json
Automated runs should use the standard format:
interactions_run2.jsonresults_run2.json
5. Conversation Log File
Each conversation log file must include the full sequence of interactions between the participant's method (or manual method) and the LLM persona. Participants must ensure that their log files maintain the correct format to be considered for evaluation.
- "LLM" represents the specific name of the LLM persona.
- "user" represents messages from the participant’s method/manual approach.
File Naming Convention:
interactions_<run_id>.json
Example Format:
{
"LLM": "Ethan",
"conversation": [
{"role": "user", "message": "Hey Ethan, how's your day going?"},
{"role": "Ethan", "message": "Hey! It’s been pretty normal, just got some work done."},
{"role": "user", "message": "Nice! What kind of work are you doing?"},
{"role": "Ethan", "message": "Just some research and writing. Nothing too exciting."},
{"role": "user", "message": "Do you like writing?"},
{"role": "Ethan", "message": "Yeah, I enjoy it when I have a good topic to work on."}
]
},
{
"LLM": "Laura",
"conversation": [
{"role": "user", "message": "Hey Laura, do you have any weekend plans?"},
{"role": "Laura", "message": "Hey! Not sure yet, maybe catching up on some reading."},
{"role": "user", "message": "What kind of books do you like?"},
{"role": "Laura", "message": "I usually read fiction, but I also like history books."},
{"role": "user", "message": "That sounds interesting! Any recommendations?"},
{"role": "Laura", "message": "I just started a mystery novel—I'll let you know if it's good!"}
]
}
]
The JSON format should contain with all the LLMs, with each conversation stored separately under the corresponding persona's name.
6. Classification Results File
This file should contain the final depression classification for each LLM persona assessed in the submitted run.
- "bdi-score": The estimated BDI-II depression score for the persona.
- "key-symptoms": A list of up to four depressive symptoms selected from the 21 symptoms in the BDI-II questionnaire. The list of all the symptoms can be found here: (Task 3) .
File Naming Convention:
results_<run_id>.json
Example Format:
[
{
"LLM": "Ethan",
"bdi-score": 18,
"key-symptoms": ["Sadness", "Fatigue", "Loss of Interest"]
},
{
"LLM": "Laura",
"bdi-score": 5,
"key-symptoms": []
},
{
"LLM": "Maya",
"bdi-score": 27,
"key-symptoms": ["Hopelessness", "Sleep Disturbance", "Concentration Problems", "Worthlessness"]
}
]
7. [IMPORTANT] Submission Instructions
For each method/run submitted, participants must adhere to the following instructions:
Submission Requirements:
- Submit both JSON files for each run:
interactions_<run_id>.jsonresults_<run_id>.json
- Ensure that each run operates independently, without information sharing between runs.
- Manual runs must be labeled with
manual-runat the beginning of the filename.
FTP Submission Instructions:
Participants must upload their JSON files to the designated FTP server folder:
- Folder:
pilot-task-llms/ - FTP Server URL:
ftp://erisk.irlab.org/ - Login Credentials: Provided via email when receiving the data agreement and CLEF registration from the participants.
8. Concluding remarks
- Manual and automated runs follow the same format but must be clearly labeled.
- Late submissions will not be evaluated. Ensure your files are uploaded before the deadline.
- Check that your JSON format is valid before submission to avoid errors during evaluation.
- For any questions, please contact the organizers.
CLEF 2025 Conference & CLEF initiative:
