Active Learning to Maximize Area Under the ROC Curve

Authors

Matt Culver, University of Nebraska-LincolnFollow
Deng Kun, University of Nebraska-LincolnFollow
Stephen Scott, University of Nebraska-LincolnFollow

Date of this Version

2006

Comments

Published in Sixth International Conference on Data Mining, 2006. ICDM '06. Copyright 2006 IEEE. Used by permission.

Abstract

In active learning, a machine learning algorithmis given an unlabeled set of examples U, and is allowed to request labels for a relatively small subset of U to use for training. The goal is then to judiciously choose which examples in U to have labeled in order to optimize some performance criterion, e.g. classification accuracy. We study how active learning affects AUC. We examine two existing algorithms from the literature and present our own active learning algorithms designed to maximize the AUC of the hypothesis. One of our algorithms was consistently the top performer, and Closest Sampling from the literature often came in second behind it. When good posterior probability estimates were available, our heuristics were by far the best.