Pre-annotation instructions
Before any judging takes place, assessors should be given the following instructions.
Assume you are given a BDI item, e.g.:
15. Loss of Energy
-I have as much energy as ever.
-I have less energy than I used to have.
-I don't have enough energy to do very much.
-I don't have enough energy to do anything.
The task consists of searching for sentences in the collection that are topically-relevant to the item (related to the question and/or to the answers).
Note: A relevant sentence should provide some information about the state of the individual related to the topic of the BDI item. But it is not necessary that the exact same words are used. Assessors should label the relevance of the sentence taking into account its context (preceding and following sentence).
Examples (assessment of sentences ranked for BDI-item number 15):
I cannot control my energy these days: Relevant
My sister has no energy at all: Non-relevant sentence (because it does not refer to the writer who posted this sentence)
The book was about a highly energetic man: Non-relevant sentence
(because it does not refer to the writer who posted this sentence)
I feel more tired than usual: Relevant
The football team is named Top Energy: Non-relevant
I am totally lonely: Non-relevant (it does not mention energy)
“I have just recharged my batteries: Relevant
I am lost: Non-relevant
Your job is to assess sentences on how topically-relevant they are for a concrete BDI item:
Relevance: judging a sentence as relevant or not relevant (binary).
The relevance grades are:
1.
Relevant: A relevant sentence should be topically-related to the BDI-item (regardless of the wording) and, additionally, it should refer to the state of the individual about the BDI-item.
0.
Non-relevant: A non-relevant sentence does not address any topic related to the question and/or the answers of the BDI-item (or it is related to the topic but does not represent the individual’s state about the BDI-item). For example, for BDI-item 15, a sentence that does not talk about the individual’s level of energy (regardless of the wording), then is a non-relevant sentence.
We advise you to not stop the assessment session in the middle of one BDI-item (this helps to maintain consistency in the judgments). Assessors should label the relevance of the sentence taking into account its context (preceding and following sentence). To measure the assessment effort, we ask you to record the time spent on fully evaluating the sentences presented for each BDI-item.