eRisk 2017:

Early risk prediction on the Internet: experimental foundations


CLEF 2017 Workshop

Dublin, 11-14 September 2017

Find Out More

CLEF eRisk 2017:

Early risk prediction on the Internet: experimental foundations


The main purpose of this workshop is to explore issues of evaluation methodology, effectiveness metrics and other processes related to early risk detection. Early detection technologies can be employed in different areas, particularly those related to health and safety. For instance, early alerts could be sent when a predator starts interacting with a child for sexual purposes, or when a potential offender starts publishing antisocial threats on a blog, forum or social network. Our main goal is to pioneer a new interdisciplinary research area that would be potentially applicable to a wide variety of situations and to many different personal profiles. Examples include potential paedophiles, stalkers, individuals that could fall into the hands of criminal organisations, people with suicidal inclinations, or people susceptible to depression.

Participate


This is the first year that this workshop runs and there are two possible ways to participate:

Research Papers Submission

The workshop is open to the submission of papers describing test collections or data sets suitable for early risk prediction, early risk prediction challenges, tasks and evaluation metrics or specific early risk detection solutions. We understand that there are two main classes of early risk prediction:

  • With multiple actors. We include in this category cases where there is an external actor or intervening factor that explicitly causes or stimulates the problem. For instance, sexual offenders use deliberate tactics to contact vulnerable children and engage them in sexual abuse. In such cases, early warning systems need to analyse the interactions between the offender and the victim and, in particular, the language of both. Another example of risk provoked by external actions is terrorist recruitment; there is currently massive online activity aiming at recruiting young people (particularly, teenagers) for joining criminal networks
  • With a single actors. We include in this second category cases where there is no explicit external actor or intervening factor that causes or stimulates the problem, but instead the risk comes “exclusively” from the individual. For instance, a teenager developing depression whose process is not caused or stimulated by any intervention or action made by another individual. Of course, there might be multiple personal or contextual factors that affect (or even cause) this process and, as a matter of fact, this is usually the case. However, as it is notfeasible to have access to sources of data associated to all these external conditions, the only element that can be analysed is the language of the individual.

The two classes of risks described above might interact one to each other. For instance, individuals suffering from major depression might be more inclined to fall prey to criminal networks. From a technological perspective, different types of tools are likely needed to develop early warning systems for these two types of risks.

Essentially, we look at early risk prediction as a process of sequential evidence accumulation where alerts are made when there is enough evidence about a certain type of risk. For the single actor type of risk, the pieces of evidence could come from the chronological sequence of entries written by a tormented subject in the Social Media. For the multiple actor type of risk, the pieces of evidence could come from a series of messages interchanged by an offender and a victim in a chatroom or online forum.

Notice that we refer to early risk in a general way and the workshop is open to contributions along these lines in any possible application domain.

Important dates (Research Papers):

  • Submission deadline: May 26, 2017
  • Notification of acceptance:June 16, 2017
  • Camera-ready: July 3, 2017

Submission instructions: We solicit papers up to 12 pages in length. The papers must be written in English and should follow the Springer-Verlag LNCS style. For details see the Springer LNCS Author Instructions. Papers (PDF) must be submitted through EasyChair:(eRisk Workshop Papers). The proceedings of this workshop will be published in the online CEUR-WS Proceedings and on the conference website.

Programme Committee:

  • Leif Azzopardi, University of Glasgow, UK
  • Alvaro Barreiro, University of A Coruña, Spain
  • Meeyoung Cha, KAIST, South Korea
  • Glen Coppersmith, Johns Hopkins University, USA
  • Silvia Gabrielli, CREATE-NET, Italy
  • Manuel Montes, National Institute of Astrophysics, Optics and Electronics, México
  • Josianne Mothe, IRIT, France
  • Raffaele Perego, CNR, Italy
  • Philip Resnik, University of Maryland, USA
  • Paolo Rosso, Universitat Politècnica de València, Spain

Important Dates

Early Detection of Depression

The second way of participation consists in performing a pilot task on early risk detection of depression. This is an exploratory task on early risk detection of depression. The challenge consists of sequentially processing pieces of evidence and detect early traces of depression as soon as possible. The task is mainly concerned about evaluating Text Mining solutions and, thus, it concentrates on texts written in Social Media. Texts should be processed in the order they were created. In this way, systems that effectively perform this task could be applied to sequentially monitor user interactions in blogs, social networks, or other types of online media.

The test collection for this pilot task is the collection described in [Losada & Crestani 2016]. It is a collection of writings (posts or comments) from a set of Social Media users. There are two categories of users, depressed and non- depressed, and, for each user, the collection contains a sequence of writings (in chronological order). For each user, his collection of writings has been divided into 10 chunks. The first chunk contains the oldest 10% of the messages, the second chunk contains the second oldest 10%, and so forth.

The task is organized into two different stages:

  • Training stage. Initially, the teams that participate in this task will have access to a training stage where we will release the whole history of writings for a set of training users (we will provide all chunks of all training users), and we will indicate what users have explicitly mentioned that they have been diagnosed with depression. The participants can therefore tune their systems with the training data.
  • Test stage. The test stage will consist of 10 sequential releases of data (i.e. done at different dates). The first release will consist of the 1st chunk of data (oldest writings of all test users), the second release will consist of the 2nd chunk of data (second oldest writings of all test users), and so forth. After each release, the participants will have a few days to process the data and, before the next release, each participating system has to choose between two options: a) emitting a decision on the user (i.e. depressed or non-depressed), or b) seeing more chunks. This choice has to be made for each user in the collection. If the system emits a decision, its decision will be final and it will be evaluated based on the correctness of the system's decision and the number of chunks required to make the decision (using a metric for which the fewer writings required to make the alert, the better). If the system does not emit a decision then it will have access to the next chunk of data, that is it will have access to more writings, but it will have a penalty for a “later emission”).

Evaluation: The evaluation will take into account not only the correctness of the system's output (i.e. whether or not the user is depressed) but also the delay taken to emit its decision. To meet this aim, we will consider the ERDE metric proposed in [Losada & Crestani 2016].

Being a pilot task, we expect it to be useful for instigating discussion on how to create evaluation laboratories for early risk prediction: proper size of the data, adequate early risk evaluation metrics, alternative ways to formulate early detection tasks, other possible application domains, etc.

Nov 30th, 2016: The training data has been released!. More info: here

You can still join us!:

CLEF 2017 Labs Registration (checkbox ERisk-Task1)
04 NOV
  • Registration for the pilot task opens
  • 4/11/2016

30 NOV
  • Release of the training data
  • 30/11/2016

06 FEB
  • Release of 1st chunk of test data
  • 06/02/2017

  • Release of 2nd chunk of test data
  • 13/02/2017

  • Release of 3rd chunk of test data
  • 20/02/2017

  • Release of 4th chunk of test data
  • 27/02/2017

  • Release of 5th chunk of test data
  • 06/03/2017

  • Release of 6th chunk of test data
  • 13/03/2017

  • Release of 7th chunk of test data
  • 20/03/2017

  • Release of 8th chunk of test data
  • 27/03/2017

  • Release of 9th chunk of test data
  • 03/04/2017

  • Release of 10th chunk of test data
  • 10/04/2017

  • Release of evaluation results to all participants
  • 24/04/2017

26 MAY
  • Deadline for submission of papers (research papers)
  • 26/05/2017

  • Pilot task participant papers due
  • 26/05/2017

16 JUN
  • Notification of acceptance (research papers)
  • 16/06/2017

  • Notification of acceptance (pilot task participants)
  • 16/06/2017

03 JUL
  • Camera ready. Research papers
  • 3/07/2017

  • Camera ready. Pilot task participant papers
  • 3/07/2017

Organizers


Steering Committtee


Ryan Boyd

University of Texas at Austin, USA

Ewout H Meijer

Maastricht University, The Netherlands

Anthony Beech

University of Birmingham, UK

More information


+34 881 816 451

CLEF 2017 Conference & CLEF initiative: