Multi-dimensional evaluation of large language models for text summarization

Abstract

The rapid growth of digital textual data across various domains necessitates effective methods for automatic text summarization. Large language models have demonstrated strong capabilities in generating coherent and informative summaries; however, comprehensive evaluation of their performance remains a challenging task. Existing approaches often rely on a limited number of evaluation measures and fail to capture multiple dimensions of summary quality and practical deployment factors. This study aims to conduct a comparative analysis of leading families of large language models in text summarization tasks using a multi-dimensional evaluation framework. The methodology includes experimental evaluation across diverse datasets, with summaries generated at multiple lengths. The models are assessed using a combination of evaluation measures, including lexical overlap, semantic similarity, factual consistency, and evaluation by large language models for human-like quality. Additionally, efficiency measures such as processing time and computational cost are incorporated into the analysis. The results reveal significant differences in model performance across evaluation dimensions. A key finding is the trade-off between factual consistency and perceived quality: shorter summaries tend to be more factually accurate, whereas longer summaries are rated higher in terms of completeness and readability. The proposed multi-dimensional evaluation approach provides a more comprehensive understanding of the capabilities of large language models and supports informed model selection for specific applications. The findings can be applied in real-world scenarios requiring automated processing of large-scale textual data and highlight directions for future research in improving evaluation methodologies and domain-specific adaptation of large language models.