Task 1: Early Detection of Signs of Depression
This is a continuation of the eRisk 2017 pilot task.
The challenge consists in performing a
task on early risk detection of depression. The
challenge consists of sequentially processing pieces of evidence
and detect early traces of depression as soon as possible. The
task is mainly concerned about evaluating Text Mining solutions
and, thus, it concentrates on texts written in Social Media.
Texts should be processed in the order they were created. In
this way, systems that effectively perform this task could be
applied to sequentially monitor user interactions in blogs,
social networks, or other types of online media.
The test collection for this task has the same format as the collection
described in [Losada
& Crestani 2016]. The source of data is also the same used for eRisk 2017.
It is a collection of writings (posts or
comments) from a set of Social Media users. There are two
categories of users, depressed and non- depressed, and, for each
user, the collection contains a sequence of writings (in
chronological order). For each user, his collection of writings
has been divided into 10 chunks. The first chunk contains the
oldest 10% of the messages, the second chunk contains the second
oldest 10%, and so forth.
The
task is organized into two different stages:
- Training
stage. Initially, the teams that participate in this task will
have access to a training stage where we will release the whole
history of writings for a set of training users (we will
provide all chunks of all training users), and we will indicate
what users have explicitly mentioned that they have been
diagnosed with depression. The participants can therefore tune
their systems with the training data. In 2018, the training data for Task 1 is composed of all 2017 users (2017 training users + 2017 test users)
- Test
stage. The test stage will consist of 10 sequential releases
of data (i.e. done at different dates). The first release will
consist of the 1st chunk of data (oldest writings of all test
users), the second release will consist of the 2nd chunk of
data (second oldest writings of all test users), and so forth.
After each release, the participants will have a few days to
process the data and, before the next release, each
participating system has to choose between two options: a)
emitting a decision on the user (i.e. depressed or
non-depressed), or b) seeing more chunks.
This choice has to be made for each user in the collection. If the system emits a
decision, its decision will be final and it will be evaluated
based on the correctness of the system's decision and the
number of chunks required to make the decision (using a metric
for which the fewer writings required to make the alert, the
better). If the system does not emit a decision then it will
have access to the next chunk of data, that is it will have
access to more writings, but it will have a penalty for a
“later emission”).
Evaluation: The evaluation will take into account not only the
correctness of the system's output (i.e. whether or not the user
is depressed) but also the delay taken to emit its decision. To
meet this aim, we will consider the ERDE metric proposed in [Losada
& Crestani 2016].
The proceedings of the lab will be published in the online CEUR-WS Proceedings and on the conference website.
To have access to the collection all participants have to fill, sign and send a user agreement form
(follow the instructions provided here). Once you have submitted the signed copyright form, you can proceed to register for the lab at CLEF 2018 Labs Registration site
Important
Dates
Task 2: Early Detection of Signs of Anorexia
This is a new task in 2018. The format, source of data and overall organization of the task are equivalent to those used for Task 1.
The challenge consists in performing a
task on early risk detection of anorexia. The
challenge consists of sequentially processing pieces of evidence
and detect early traces of anorexia as soon as possible. The
task is mainly concerned about evaluating Text Mining solutions
and, thus, it concentrates on texts written in Social Media.
Texts should be processed in the order they were created. In
this way, systems that effectively perform this task could be
applied to sequentially monitor user interactions in blogs,
social networks, or other types of online media.
The test collection for this task has the same format as the collection
described in [Losada
& Crestani 2016]. The source of data is also the same.
It is a collection of writings (posts or
comments) from a set of Social Media users. There are two
categories of users: those that have been diagnosed with anorexia, and a control group (non-anorexia). For each
user, the collection contains a sequence of writings (in
chronological order). For each user, his collection of writings
has been divided into 10 chunks. The first chunk contains the
oldest 10% of the messages, the second chunk contains the second
oldest 10%, and so forth.
The
task is organized into two different stages:
- Training
stage. Initially, the teams that participate in this task will
have access to a training stage where we will release the whole
history of writings for a set of training users (we will
provide all chunks of all training users), and we will indicate
what users have explicitly mentioned that they have been
diagnosed with anorexia. The participants can therefore tune
their systems with the training data.
- Test
stage. The test stage will consist of 10 sequential releases
of data (i.e. done at different dates). The first release will
consist of the 1st chunk of data (oldest writings of all test
users), the second release will consist of the 2nd chunk of
data (second oldest writings of all test users), and so forth.
After each release, the participants will have a few days to
process the data and, before the next release, each
participating system has to choose between two options: a)
emitting a decision on the user (i.e. anorexia or
non-anorexia), or b) seeing more chunks.
This choice has to be made for each user in the collection. If the system emits a
decision, its decision will be final and it will be evaluated
based on the correctness of the system's decision and the
number of chunks required to make the decision (using a metric
for which the fewer writings required to make the alert, the
better). If the system does not emit a decision then it will
have access to the next chunk of data, that is it will have
access to more writings, but it will have a penalty for a
“later emission”).
Evaluation: The evaluation will take into account not only the
correctness of the system's output (i.e. whether or not the user
has anorexia) but also the delay taken to emit its decision. To
meet this aim, we will consider the ERDE metric proposed in [Losada
& Crestani 2016].
The proceedings of the lab will be published in the online CEUR-WS Proceedings and on the conference website.
To have access to the collection all participants have to fill, sign and send a user agreement form
(follow the instructions provided here).
Once you have submitted the signed copyright form, you can proceed to register for the lab at CLEF 2018 Labs Registration site
Important
Dates
