research
∙
09/03/2023
How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets
Crowdsourced annotation is vital to both collecting labelled data to tra...
share
research
∙
08/14/2023
Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI
A key challenge in professional fact-checking is its limited scalability...
research
∙
05/31/2023
Designing Closed-Loop Models for Task Allocation
Automatically assigning tasks to people is challenging because human per...
research
∙
02/14/2023
Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection
Algorithmic bias often arises as a result of differential subgroup valid...
research
∙
02/06/2023
Learning Complementary Policies for Human-AI Teams
Human-AI complementarity is important when neither the algorithm nor the...
research
∙
01/19/2023
New Metrics to Encourage Innovation and Diversity in Information Retrieval Approaches
In evaluation campaigns, participants often explore variations of popula...
research
∙
01/08/2023
The State of Human-centered NLP Technology for Fact-checking
Misinformation threatens modern society by promoting distrust in science...
research
∙
12/29/2022
Voices of Workers: Why a Worker-Centered Approach to Crowd Work Is Challenging
How can we better understand the broad, diverse, shifting, and invisible...
research
∙
12/15/2022
Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks
When annotators label data, a key metric for quality assurance is inter-...
research
∙
06/29/2022
longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks
Developing methods to adversarially challenge NLP systems is a promising...
research
∙
04/15/2022
Fairly Accurate: Learning Optimal Accuracy vs. Fairness Tradeoffs for Hate Speech Detection
Recent work has emphasized the importance of balancing competing objecti...
research
∙
04/11/2022
ProtoTEx: Explaining Model Decisions with Prototype Tensors
We present ProtoTEx, a novel white-box NLP classification architecture b...
research
∙
02/17/2022
The Effects of Interactive AI Design on User Behavior: An Eye-tracking Study of Fact-checking COVID-19 Claims
We conducted a lab-based eye-tracking study to investigate how the inter...
research
∙
02/09/2022
Designing Closed Human-in-the-loop Deferral Pipelines
In hybrid human-machine deferral frameworks, a classifier can defer unce...
research
∙
12/04/2021
In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers
We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsour...
research
∙
11/29/2021
Proceedings of the CSCW 2021 Workshop – Investigating and Mitigating Biases in Crowdsourced Data
This volume contains the position papers presented at CSCW 2021 Workshop...
research
∙
11/19/2021
Data Excellence for AI: Why Should You Care
The efficacy of machine learning (ML) models depends on both algorithms ...
research
∙
09/20/2021
The Case for Claim Difficulty Assessment in Automatic Fact Checking
Fact-checking is the process (human, automated, or hybrid) by which clai...
research
∙
06/17/2021
An Information Retrieval Approach to Building Datasets for Hate Speech Detection
Building a benchmark dataset for hate speech detection presents several ...
research
∙
05/22/2021
Human-AI Collaboration with Bandit Feedback
Human-machine complementarity is important when neither the algorithm no...
research
∙
05/20/2021
The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help
We consider a class of variable effort human annotation tasks in which t...
research
∙
02/25/2021
Towards Unbiased and Accurate Deferral to Multiple Experts
Machine learning models are often implemented in cohort with humans in t...
research
∙
12/24/2020
Understanding and Predicting the Characteristics of Test Collections
Shared-task campaigns such as NIST TREC select documents to judge by poo...
research
∙
12/16/2020
You Are What You Tweet: Profiling Users by Past Tweets to Improve Hate Speech Detection
Hate speech detection research has predominantly focused on purely conte...
research
∙
07/22/2019
A Conceptual Framework for Evaluating Fairness in Search
While search efficacy has been evaluated traditionally on the basis of r...
research
∙
07/08/2019
CobWeb: A Research Prototype for Exploring User Bias in Political Fact-Checking
The effect of user bias in fact-checking has not been explored extensive...
research
∙
06/03/2018
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably
Crowdsourcing offers an affordable and scalable means to collect relevan...
research
∙
04/29/2018
But Who Protects the Moderators? The Case of Crowdsourced Image Moderation
Though detection systems have been developed to identify obscene content...
research
∙
04/29/2018
Toward Safer Crowdsourced Content Moderation
Though detection systems have been developed to identify obscene content...
research
∙
02/01/2018
Correlation and Prediction of Evaluation Metrics in Information Retrieval
Because researchers typically do not have the time or space to present m...
research
∙
01/17/2018
Efficient Test Collection Construction via Active Learning
To create a new IR test collection at minimal cost, we must carefully se...
research
∙
02/08/2017
Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization
A fundamental advantage of neural models for NLP is their ability to lea...
research
∙
06/14/2016
Active Discriminative Text Representation Learning
We propose a new active learning (AL) method for text classification wit...
research
∙
04/20/2014
TurKPF: TurKontrol as a Particle Filter
TurKontrol, and algorithm presented in (Dai et al. 2010), uses a POMDP t...
research
∙
12/05/2012
Evaluating Classifiers Without Expert Labels
This paper considers the challenge of evaluating a set of classifiers, a...
research
∙
11/10/2012