SemEval-2017 Task 3: Community Question Answering (2017-08-01T00:00:00.000000Z)

TL;DR

SemEval–2017 Task 3 on Community Question Answering reran the four subtasks from SemEval-2016, providing all the data from 2015 and 2016 for training, and fresh data for testing, and added a new subtask E in order to enable experimentation with Multi-domain Question Duplicate Detection in a larger-scale scenario, using StackExchange subforums.

Abstract

We describe SemEval–2017 Task 3 on Community Question Answering. This year, we reran the four subtasks from SemEval-2016: (A) Question–Comment Similarity, (B) Question–Question Similarity, (C) Question–External Comment Similarity, and (D) Rerank the correct answers for a new question in Arabic, providing all the data from 2015 and 2016 for training, and fresh data for testing. Additionally, we added a new subtask E in order to enable experimentation with Multi-domain Question Duplicate Detection in a larger-scale scenario, using StackExchange subforums. A total of 23 teams participated in the task, and submitted a total of 85 runs (36 primary and 49 contrastive) for subtasks A–D. Unfortunately, no teams participated in subtask E. A variety of approaches and features were used by the participating systems to address the different subtasks. The best systems achieved an official score (MAP) of 88.43, 47.22, 15.46, and 61.16 in subtasks A, B, C, and D, respectively. These scores are better than the baselines, especially for subtasks A–C.

Authors

Hamdy Mubarak

7 papers

Preslav Nakov

14 papers

Karin M. Verspoor

4 papers

TL;DR

Abstract

Authors

References89 items

Representation

SemEval-2016 Task 3: Community Question Answering

The dark side of news community forums: opinion manipulation trolls

Cross-Language Question Re-Ranking

GW_QA at SemEval-2017 Task 3: Question Answer Re-ranking on Arabic Fora

SwissAlps at SemEval-2017 Task 3: Attention-based Convolutional Neural Network for Community Question Answering

ECNU at SemEval-2017 Task 3: Using Traditional and Deep Learning Methods to Address Community Question Answering Task

SCIR-QA at SemEval-2017 Task 3: CNN Model Based on Similar and Dissimilar Information between Keywords for Question Similarity

QU-BIGIR at SemEval 2017 Task 3: Using Similarity Features for Arabic Community Question Answering Forums

NLM_NIH at SemEval-2017 Task 3: from Question Entailment to Question Similarity for Community Question Answering

MoRS at SemEval-2017 Task 3: Easy to use SVM in Ranking Tasks

UINSUSKA-TiTech at SemEval-2017 Task 3: Exploiting Word Importance Levels for Similarity Features for CQA

TakeLab-QA at SemEval-2017 Task 3: Classification Experiments for Answer Retrieval in Community QA

UPC-USMBA at SemEval-2017 Task 3: Combining multiple approaches for CQA for Arabic

bunji at SemEval-2017 Task 3: Combination of Neural Similarity Features and Comment Plausibility Features

FuRongWang at SemEval-2017 Task 3: Deep Neural Networks for Selecting Relevant Answers in Community Question Answering

LearningToQuestion at SemEval 2017 Task 3: Ranking Similar Questions by Learning to Rank Using Rich Features

FA3L at SemEval-2017 Task 3: A ThRee Embeddings Recurrent Neural Network for Question Answering

SimBow at SemEval-2017 Task 3: Soft-Cosine Semantic Similarity between Questions for Community Question Answering

KeLP at SemEval-2017 Task 3: Learning Pairwise Patterns in Community Question Answering

IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Question Answering and Implicit Dialogue Identification

Beihang-MSRA at SemEval-2017 Task 3: A Ranking System with Neural Matching Features for Community Question Answering

Large-Scale Goodness Polarity Lexicons for Community Question Answering

It Takes Three to Tango: Triangulation Approach to Answer Ranking in Community Question Answering

Learning to Rank Non-Factoid Answers: Comment Selection in Web Forums

Learning to Re-Rank Questions in Community Question Answering Using Advanced Features

Hunting for Troll Comments in News Community Forums

Machine Translation Evaluation Meets Community Question Answering

Quit While Ahead: Evaluating Truncated Rankings

SIGIR 2016 Workshop WebQA II: Web Question Answering Beyond Factoids

A Decomposable Attention Model for Natural Language Inference

SLS at SemEval-2016 Task 3: Neural-based Approaches for Ranking in Community Question Answering

SemanticZ at SemEval-2016 Task 3: Ranking Relevant Answers in Community Question Answering Using Semantic Similarity Based on Fine-tuned Word Embeddings

UniMelb at SemEval-2016 Task 3: Identifying Similar Questions by combining a CNN with String Similarity Measures

Joint Learning with Global Inference for Comment Classification in Community Question Answering

SUper Team at SemEval-2016 Task 3: Building a Feature-Rich System for Community Question Answering

Semi-supervised Question Retrieval with Gated Convolutions

CQADupStack: A Benchmark Data Set for Community Question-Answering Research

LSTM-based Deep Learning Models for non-factoid answer selection

Global Thread-level Inference for Comment Classification in Community Question Answering

Exposing Paid Opinion Manipulation Trolls

Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks

Applying deep learning to answer selection: A study and an open task

Thread-Level Information for Comment Classification in Community Question Answering

Finding Opinion Manipulation Trolls in News Community Forums

Pairwise Neural Machine Translation Evaluation

Learning Hybrid Representations to Retrieve Semantically Equivalent Questions

A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering

Answer Sequence Learning with Neural Networks for Answer Selection in Community Question Answering

ICRC-HIT: A Deep Learning based Comment Sequence Labeling System for Answer Selection Challenge

HITSZ-ICRC: Exploiting Classification Approach for Answer Selection in Community Question Answering

JAIST: Combining multiple features for Answer Selection in Community Question Answering

QCRI: Answer Selection for Community Question Answering - Experiments for Arabic and English

SemEval-2015 Task 3: Answer Selection in Community Question Answering

Question Retrieval with High Quality Answers in Community Question Answering

GloVe: Global Vectors for Word Representation

Answer Extraction as Sequence Tagging with Tree Edit Distance

Phrase-Based Translation Model for Question Retrieval in Community Question Answer Archives

Improving Question Recommendation by Exploiting Information Need

Learning to Rank Answers to Non-Factoid Questions from Web Collections

Probabilistic Tree-Edit Models with Structured Latent Variables for Textual Entailment and Question Answering

Tree Edit Models for Recognizing Textual Entailments, Paraphrases, and Answers to Questions

The use of categorization information in language models for question retrieval

Reciprocal rank fusion outperforms condorcet and individual rank learning methods

A syntactic tree matching approach to finding similar questions in community-based qa services

Searching Questions by Identifying Question Topic and Question Focus

Recommending questions using the mdl-based tree cut model

What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA

Statistical Machine Translation for Query Expansion in Answer Retrieval

Automatic question answering using the web: Beyond the Factoid

Finding similar questions in large question and answer archives

A Noisy-Channel Approach to Question Answering

Bridging the lexical chasm: statistical approaches to answer-finding

The Mathematics of Statistical Machine Translation: Parameter Estimation

Alberto

Effective shared representations with Multitask Learning for Community Question Answering