An Incomplete Condition-Prompt Grid in a Generated-Image Benchmark

Authors

  • Patrick Sullivan Author
  • Oliver Douglas Author
  • Patrick Porter Author

Abstract

AGIQA-3K's released score table contains 2,982 of the 3,000 cells in its nominal ten-condition, 300-prompt grid. Eighteen combinations are absent: ten under DALLE2_normal and four under each of the two Midjourney labels. Thirteen prompts lack at least one condition. This secondary audit identifies the denominator behind pooled and paired comparisons. It does not infer why rows are absent or assign outcomes to them. The findings make completeness a visible part of evaluating semantic and perceptual judgments rather than an implicit assumption.

References

  1. Meng, Zijie. "Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment." arXiv preprint arXiv:2606.16799 (2026).
  2. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
  3. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
  4. Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
  5. Benjamini, Yoav and Yosef Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 57, no. 1, 1995, pp. 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  6. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  7. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  8. Wilkinson, Mark D., et al. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data, vol. 3, 2016, article 160018. https://doi.org/10.1038/sdata.2016.18.
  9. Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems, vol. 28, 2015. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  10. Bland, J. Martin, and Douglas G. Altman. "Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement." The Lancet, vol. 327, no. 8476, 1986, pp. 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8.
  11. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  12. Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
  13. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  14. Li, Chunyi, et al.. "AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment" arXiv preprint arXiv:2306.04717, 2023. https://arxiv.org/abs/2306.04717.
  15. Radford, Alec, et al.. "Learning Transferable Visual Models From Natural Language Supervision" arXiv preprint arXiv:2103.00020, 2021. https://arxiv.org/abs/2103.00020.
  16. Ke, Junjie, et al.. "MUSIQ: Multi-scale Image Quality Transformer" arXiv preprint arXiv:2108.05997, 2021. https://arxiv.org/abs/2108.05997.
  17. Hosu, Vlad, et al.. "KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment" arXiv preprint arXiv:1910.06180, 2019. https://arxiv.org/abs/1910.06180.
  18. Lin, Hanhe, et al.. "KADID-10k: A Large-scale Artificially Distorted IQA Database" 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), 2019. https://doi.org/10.1109/qomex.2019.8743252.
  19. Wang, Zhou, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. "Image Quality Assessment: From Error Visibility to Structural Similarity." IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, pp. 600-612. https://doi.org/10.1109/TIP.2003.819861.
  20. Zhang, Richard, et al.. "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" arXiv preprint arXiv:1801.03924, 2018. https://arxiv.org/abs/1801.03924.
  21. Mittal, A., et al.. "Making a “Completely Blind” Image Quality Analyzer" IEEE Signal Processing Letters, 2013. https://doi.org/10.1109/lsp.2012.2227726.
  22. Talebi, Hossein and Peyman Milanfar. "NIMA: Neural Image Assessment" arXiv preprint arXiv:1709.05424, 2017. https://arxiv.org/abs/1709.05424.
  23. Kirstain, Yuval, et al.. "Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation" arXiv preprint arXiv:2305.01569, 2023. https://arxiv.org/abs/2305.01569.
  24. Wu, Xiaoshi, et al.. "Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis" arXiv preprint arXiv:2306.09341, 2023. https://arxiv.org/abs/2306.09341.
  25. Caron, Mathilde, et al.. "Emerging Properties in Self-Supervised Vision Transformers" arXiv preprint arXiv:2104.14294, 2021. https://arxiv.org/abs/2104.14294.
  26. Ruiz, Nataniel, et al.. "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation" arXiv preprint arXiv:2208.12242, 2022. https://arxiv.org/abs/2208.12242.
  27. Ye, Hu, et al.. "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models" arXiv preprint arXiv:2308.06721, 2023. https://arxiv.org/abs/2308.06721.
  28. Heusel, Martin, et al.. "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium" arXiv preprint arXiv:1706.08500, 2017. https://arxiv.org/abs/1706.08500.
  29. Gebru, Timnit, et al.. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  30. Pedregosa, Fabian, et al. "Scikit-learn: Machine Learning in Python." Journal of Machine Learning Research, vol. 12, 2011, pp. 2825-2830. https://jmlr.org/papers/v12/pedregosa11a.html.

Downloads

Published

2026-10-21

Issue

Vol. 2 (2026): Archive issue of 2026-10-21

Section

Articles

License

Copyright (c) 2026 Patrick Sullivan, Oliver Douglas, Patrick Porter

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.