(Dhirubhai Ambani Institute of Information and Communication Technology, 2022) Ranpara, Tarang J.; Majumder, Prasenjit
Engaging human annotators to evaluate every summary in a content summarization system is not feasible. Automatic evaluation metrics act as a proxy for human evaluation. A high correlation with human evaluation determines the effectiveness of a given metric. This thesis compares 40 different evaluation metrics with human judgments in terms of correlation and investigates whether the contextual similarity based metrics are better than lexical overlap based metrics, i.e., ROUGE score. The comparison shows that contextual similarity based metrics have a high correlation with human judgments than lexical overlap based metrics. Thus, such metrics can act as a good proxy for human judgment.
