An Internal Alert for Errors the Policy Can Exploit
Abstract
Sharpness-based alerts and value-aware learning concern different views of model error. Evaluation should compare alert calibration with errors in state-action regions emphasized by the policy. It should independently measure the consequence of acting on an alert. A correlation with static factuality labels cannot establish that the alert prevents policy exploitation of a learned model. This focused synthesis develops an empirical comparison that preserves the terminal action, observation process, and total verification cost. Its proposed tests are prospective; no new simulation run, model-training result, or intervention effect is reported.
References
Z. Qiao, J. Lyu, B. Lyu, Y. Shu, S. Gao, and S. Qiu, “Model-Based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting,” Proceedings of the 2026 International Conference on Learning Representations, 2026.
S. Chen, M. Xiong, J. Liu, Z. Wu, T. Xiao, S. Gao, and J. He, “In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation,” Proceedings of the 41st International Conference on Machine Learning, 2024.
Zhou, Yongxi, Wenbo Ye, Yuanzhe Liu, Zihan Dong, and Junwei Yao. “Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts.” arXiv, 2026, arXiv:2609.08236. https://arxiv.org/abs/2609.08236.
Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed., MIT Press, 2018. http://incompleteideas.net/book/the-book-2nd.html.
Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv, 2020, arXiv:2005.01643. https://arxiv.org/abs/2005.01643.
Kumar, Aviral, Aurick Zhou, George Tucker, and Sergey Levine. “Conservative Q-Learning for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1179–1191. https://arxiv.org/abs/2006.04779.
Lewis, Patrick, et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474. https://arxiv.org/abs/2005.11401.
Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
Bertsimas, Dimitris, and Melvyn Sim. “The Price of Robustness.” Operations Research, vol. 52, no. 1, 2004, pp. 35–53. https://doi.org/10.1287/opre.1030.0065.
Published
Versions
- 2026-10-15 (2)
- 2026-01-31 (1)