Escaping the McNamara Fallacy: Toward More Impactful Recommender Systems Research


Recommender systems are among today’s most successful application areas of artificial intelligence. However, in the recommender systems research community, we have fallen prey to a McNamara fallacy to a worrying extent: In the majority of our research efforts, we rely almost exclusively on computational measures such as pre- diction accuracy, which are easier to make than applying other evaluation methods. However, it remains unclear whether small improvements in terms of such computational measures mat- ter greatly and whether they lead us to better systems in practice. A par- adigm shift in terms of our research culture and goals is therefore needed. We can no longer focus exclusively on abstract computational measures but must direct our attention to research questions that are more relevant and have more impact in the real world. In this work, we review the various ways of how recommender systems may cre- ate value; how they, positively or neg- atively, impact consumers, businesses, and the society; and how we can measure the resulting effects. Through our analyses, we identify a number of research gaps and propose ways of broadening and improving our meth- odology in a way that leads us to more impactful research in our field.

AI Magazine