A Wrong Segment and a Failed Answer Have Different Denominators
Abstract
Factuality risk changes when the unit of evaluation moves from an annotated segment to a whole response. This article reanalyzes the released FELM annotations using a segment-level linear probability model with response-clustered covariance and descriptive response-level summaries. After excluding two records without usable response text, the analysis contains 845 responses and 4,412 segments. Longer segments have a higher adjusted error association, while aggregate segment and response error shares differ substantially. The evidence supports a methodological account of evaluation units and decision loss. It does not establish the performance of a current model, a causal effect of verbosity, or the success of a new factuality intervention.
References
S. Chen, M. Xiong, J. Liu, Z. Wu, T. Xiao, S. Gao, and J. He, “In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation,” Proceedings of the 41st International Conference on Machine Learning, 2024.
S. Chen, Y. Zhao, J. Zhang, I.-C. Chern, S. Gao, P. Liu, and J. He, “FELM: Benchmarking Factuality Evaluation of Large Language Models,” Advances in Neural Information Processing Systems, pp. 44502–44523, 2023.
Tong, H., Xu, G., & Chen, Y. (2026, May). Hadamard Tensor Ring for Efficient Low-Rank Fine-Tuning. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Zhou, Yongxi, Wenbo Ye, Yuanzhe Liu, Zihan Dong, and Junwei Yao. “Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts.” arXiv, 2026, arXiv:2609.08236. https://arxiv.org/abs/2609.08236.
White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
Lewis, Patrick, et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474. https://arxiv.org/abs/2005.11401.
Hu, Edward J., et al. “LoRA: Low-Rank Adaptation of Large Language Models.” International Conference on Learning Representations, 2022. https://openreview.net/forum?id=nZeVKeeFYf9.
Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed., MIT Press, 2018. http://incompleteideas.net/book/the-book-2nd.html.
Published
Versions
- 2026-10-15 (2)
- 2025-04-02 (1)