Evaluation of Personalized Summarization

Abstract

This research aims to address the limitations in evaluating the personalization ofa summarizer model solely based on its accuracy. Current accuracy-based measures,such as ROUGE, fail to consider subjectivity when evaluating personalizedsummarization. To overcome this, we introduce a novel metric called EGISES,which evaluates the degree of personalization by taking into account both theuser profile and the model generated summary. Additionally, we propose PROUGE,a novel metric that combines accuracy and the degree of personalization.We conduct a comprehensive analysis to establish the consistency and reliabilityof EGISES and P-ROUGE. Through this research, we provide a more effectiveand comprehensive approach to evaluating personalized summarizer models, accountingfor both, the accuracy and the personalized nature of the summaries.