NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Paper ID:1905
Title:The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the XAUC Metric


		
There was a lot of discussion over this paper. The central concern was this: * did the authors provide sufficient justification for the value of xAUC over other fairness measures * Is a good case made for the measure having normative value. After reading the reviews, the feedback and the discussion, it seems like 1. The authors are very careful to guardrail claims about the applicability of the measure - they in fact do NOT claim that this measure should be viewed as superior to other measures, but merely that it provides a context that can't otherwise be easily seen. That argument is compelling. 2. Reviewers correctly called out the measure as having issues when base rates differ (as many measures do), and to be fair the authors do acknowledge this in the paper, although they don't directly address it. I think this is a potential weakness of the measure, but I don't view it as fatal. Overall, the xAUC Measure does add an extra dimension to the discussions, and separately, I think the idea of adapting AUC to group fairness measures to be quite elegant.