#OPENBOX - Eric Smith - Human Evaluation of Open-domain Conversations - Part 1

OPENBOX aims to bring an easier understanding of open problems that helps in finding solutions for such problems. For the said purpose, I interview researchers and practitioners who have published works on open problems in various areas of Artificial Intelligence and Machine Learning to collect a simplified understanding of these open problems. These are published as podcast series Ideas emerge when curiosity meets clarity. Here we go with OPENBOX to bring clarity to those curious minds looking to solve real-world problems. This project is done in collaboration with ForHumanity. ForHumanity is a 501(c)(3) nonprofit organization that minimizes the downside risks of AI and autonomous systems. ForHumanity develops criteria for an independent audit of AI systems. To know more, visit https://forhumanity.center/. Today, we have Eric with us. Eric is a Research Engineer at Facebook AI Research (FAIR). He is interested in questions around (a) conversational ai-how to make it better, and how to evaluate and (b) questions of bias in language models. He is interested in understanding languages and their underlying constructs. We will cover a paper titled “Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents,” published in May 2022, which he co-authored. This is part 1 of the podcast. In this podcast, Eric discusses human evaluation in open-domain conversational contexts, Likert scales, and subjective outcomes.  --- Send in a voice message: https://podcasters.spotify.com/pod/show/ryan-carrier3/message