Argumentation mining evaluation

Github repository

Competition on CodaLab with a leaderboard

Telegram group chat

Key dates

• End of December: training and validation data ready
• 7 February: unlabeled test data ready.
• 20 February: final submission deadline.
• 25 February: evaluation ready.
• 25 March 11:59 am (GTM+3): paper submission deadline

Shared task description

Argument mining (or argumentation mining) is a field of computational linguistics that explores methods for extracting from texts and classifying arguments and relationships between them, as well as constructing an argumentative structure. An argument must include a claim containing a stance towards some topic or object, and at least one premise (“favor” or “against”) of this stance. Often a “premise” is called an “argument” when it is clear from the context which claim is being referred to.

There is a large number of works devoted to argument mining. There are also some shared tasks, but mainly for the English language. In the RuArg-2022 evaluation, for the first time, it is proposed to test the argument mining systems on the Russian language texts. There are many tasks of argument mining. We have selected two of them: stance detection and premise classification.

• Determine the point of view (stance) of the text’s author in relation to the given claim
• Recognize whether the text contains premises “for” or “against” to a given claim.

We have formulated three claims regarding the COVID-19 pandemic (and anti-epidemic measures in general):

1. «Vaccination is beneficial for society».
2. «The introduction and observance of quarantine is beneficial for society».
3. «Wearing masks is beneficial for society».

A collection of sentences was gathered from social networks – comments on posts from social media. These sentences can contain both statements defining the author's stance towards the given claims, and statements with premises “for” / “against” these claims.

Each sentence was annotated by stance and premise for all three claims. Thus, each sentence has six labels.

The following classes (labels) were used:

• «for»;
• «against»;
• «other» (for stance, this class merges classes “neutral”, “unclear” or “for and against”) / “no argument” (for a premise);
• «irrelevant» (for this claim)