Publications
Also on Google Scholar.
2026
2025
- Characterizing Religious Rhetoric in the U.S. Congressional Record
- TIDEs: A Transgender and Nonbinary Community-Labeled Dataset and Model for Transphobia Identification in Digital Environments
2024
2023
2022
-
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand
Abstract
Natural language processing researchers have identified limitations of evaluation methodology for generation tasks, with new questions raised about the validity of automatic metrics and of crowdworker judgments. Meanwhile, efforts to improve generation models tend to depend on simple n-gram overlap metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics should each more directly benefit and inform the other. We therefore propose a generalization of leaderboards, bidimensional leaderboards (BILLBOARDs), that simultaneously tracks progress in language generation models and metrics for their evaluation. Unlike conventional unidimensional leaderboards that sort submitted systems by predetermined metrics, a BILLBOARD accepts both generators and evaluation metrics as competing entries. A BILLBOARD automatically creates an ensemble metric that selects and linearly combines a few metrics based on a global analysis across generators. Further, metrics are ranked based on their correlation with human judgments. We release four BILLBOARDs for machine translation, summarization, and image captioning.1 We demonstrate that a linear ensemble of a few diverse metrics sometimes substantially outperforms existing metrics in isolation. Our mixed-effects model analysis shows that most automatic metrics, especially the reference-based ones, overrate machine over human generation, demonstrating the importance of updating metrics as generation models become stronger (and perhaps more similar to humans) in the future.
-
Transparent Human Evaluation for Image Captioning
Abstract
We establish THUMB, a rubric-based human evaluation protocol for image captioning models. Our scoring rubrics and their definitions are carefully developed based on machine- and human-generated captions on the MSCOCO dataset. Each caption is evaluated along two main dimensions in a tradeoff (precision and recall) as well as other aspects that measure the text quality (fluency, conciseness, and inclusive language). Our evaluations demonstrate several critical problems of the current evaluation practice. Human-generated captions show substantially higher quality than machine-generated ones, especially in coverage of salient information (i.e., recall), while most automatic metrics say the opposite. Our rubric-based results reveal that CLIPScore, a recent metric that uses image features, better correlates with human judgments than conventional text-only metrics because it is more sensitive to recall. We hope that this work will promote a more transparent evaluation protocol for image captioning and its automatic metrics.
2021
-
Conducting Browser-Based User Studies in Software Engineering
Abstract
User studies of software engineers are an invaluable part of the evaluation of new tools in academic software engineering (SE). Unfortunately, significant barriers to the efficient execution of these user studies still exist. Researchers in software engineering and related fields continue to struggle with recruiting, orchestration, experimental design, and IRB approval (Buse 2011). In this paper we address these challenges by comparing them to problem domains already identified and addressed in the behavioral sciences. We present (1) a survey of the current frameworks for administering online experiments in behavioral science and (2) a pair of software engineering-specific tools extending jsPsych, a front-end framework from behavioral science.