The comprehensive evaluation of the performance of a system is a complex endeavor: many facets need to be considered in configuring an adequate and effective evaluation setting. A single evaluation method or measure is not able to evaluate all relevant aspects in a complex setting where a multitude of stakeholders are involved. Employing a multi-method evaluation, where multiple evaluation methods or measures are combined and integrated, allows for getting a richer picture and prevents blind spots in the evaluation outcome.
Together with Eva Zangerle, I maintain a website on multi-method evaluation where we collect resources that help to apply multi-method evaluation in research and development. Naturally, this collection is meant to grow over time and we are happy to share your resources and discuss best practices.
- Special Issue 'Perspectives on Recommender Systems Evaluation' @ TORS
- Co-Chair of PERSPECTIVES 2022
- Report on the 1st Workshop on the Perspectives on the Evaluation of Recommender Systems (PERSPECTIVES 2021) at RecSys 2021
- Multi-Method Evaluation: Leveraging Multiple Methods to Answer What You Were Looking For
- Perspectives on the Evaluation of Recommender Systems (PERSPECTIVES)