Bayesian Ensembles of Crowds and Deep Learners for Sequence Tagging
share
Current methods for sequence tagging, a core task in NLP, are data hungry. Crowdsourcing is a relatively cheap way to obtain labeled data, but the annotators are unreliable, so redundant labeling and aggregation techniques are required. We evaluate multiple models of annotator reliability and develop a Bayesian method for aggregating sequence labels from multiple annotators. Typically, the process of data collection, aggregation and training a sequence tagger is a pipeline of discrete steps. We integrate these steps by training black-box sequence taggers as components in the aggregation model and accounting for their unreliability. We evaluate our model on named entity recognition and information extraction tasks, showing that our method outperforms previous methods, particularly in small data scenarios that are encountered at the beginning of a crowdsourcing process. Our code is published to encourage adaptation and reuse.
READ FULL TEXT