Frontiers in Science and Practice
Original publication date: 2026-10-21
Archive migration: 2026-10-30
Patrick Sullivan — India; Oliver Douglas — Zambia; Patrick Porter — Malaysia
AGIQA-3K's released score table contains 2,982 of the 3,000 cells in its nominal ten-condition, 300-prompt grid. Eighteen combinations are absent: ten under DALLE2_normal and four under each of the two Midjourney labels. Thirteen prompts lack at least one condition. This secondary audit identifies the denominator behind pooled and paired comparisons. It does not infer why rows are absent or assign outcomes to them. The findings make completeness a visible part of evaluating semantic and perceptual judgments rather than an implicit assumption.
A table can contain no missing values in its observed score columns and still omit entire combinations of its design variables. These are different forms of incompleteness. An analysis that checks only whether each existing row has a quality mean may miss the absent prompt-condition cells. The present study reconstructs the nominal grid in the released AGIQA-3K score table and identifies which comparisons receive complete support from the rows actually available.
Decoupling Semantics from Distortions motivates evaluating semantic and perceptual information separately (Meng, 2026). That distinction also requires clarity about which observations contribute to each evaluation. The present study does not reproduce the focal model or calculate its predictions. It audits the public score table's support. Its contribution is a bounded data-quality finding: the location of absent combinations, not a diagnosis of generation failure, a claim about missingness mechanisms, or evidence about the unobserved quality of images not represented in the release.
A public benchmark table is a useful empirical object in its own right. AGIQA-3K records quality and alignment judgments for generated imagery (Li et al., 2023). Studying those released scores differs from generating new images or reproducing an image-quality architecture. The distinction should remain visible in the data description and conclusion. Dataset documentation provides a way to explain the collection, row identities, and intended scientific use (Gebru et al., 2021), while software documentation helps make processing and validation choices recoverable (Pedregosa et al., 2011). In particular, prompt identifiers should remain linked across generation conditions rather than disappearing in a flattened analysis. Missing prompt-condition combinations also need an explicit record. A complete-case contrast and an all-observed-row contrast can describe different populations even when both are calculated correctly. The relevant empirical contribution is therefore a reproducible relationship among the available judgments, with its selection and measurement limits stated. It is not evidence that any newly proposed predictor has been trained on images or validated against independent raters when those steps have not occurred.
The input has 300 distinct prompt strings and ten condition labels derived by removing the final image index from source filenames. Their Cartesian grid has 3,000 possible cells. The observed table has 2,982 rows, with each observed prompt-condition combination unique. Reshaping quality scores into this grid reveals 18 absent cells. The audit uses the published metadata and means; it does not download or inspect visual files to determine whether an omitted image exists elsewhere or why its score is unavailable.
DALLE2_normal contributes ten absent cells. Midjourney_lowstep and midjourney_normal contribute four each. The other seven conditions have complete 300-row coverage in the released table. At the prompt level, 287 prompts have all ten conditions and thirteen have at least one absence. Nine prompts have nine observed rows, three have eight, and one has seven. The row-count distribution and missing-cell total are consistent descriptions of the same incomplete grid under different units of summary.
No outcome is imputed. An absent cell is not labeled a failed generation, a zero-quality result, or a randomly missing observation. Those interpretations would require evidence beyond the grid itself. The audit also distinguishes source-label completeness from a complete account of the generation process. A table with every nominal cell present could still omit repeated attempts or unrecorded settings. The present check establishes only the support of the released prompt-condition arrangement used by subsequent score analyses.
Precision and identification are separate properties of a result. A narrow interval can surround an association whose scientific target is unclear. Robust covariance calculations allow particular departures from simple error assumptions, including unequal variance (White, 1980) and dependence within declared clusters (Cameron and Miller, 2015). Their role is to improve uncertainty calculations under those assumptions. They do not establish a causal assignment, guarantee a correct measurement, or expand coverage to conditions missing from the data. The analyst should therefore explain both the averaging population and the source of replication. An evaluation repeated many times on one history may provide information about stochastic variability while supplying little evidence about transfer to another history. Those two sources of uncertainty should remain distinguishable in the final interpretation.
Visual-language alignment and image quality have related but distinguishable measurement foundations. CLIP connects images and text through language-supervised representation learning (Radford et al., 2021). Its similarities can inform an assessment, but they do not automatically define perceptual quality or an application's complete notion of semantic correctness. MUSIQ develops a multiscale image-quality approach (Ke et al., 2021), NIMA studies learned aesthetic assessment (Talebi and Milanfar, 2017), and NIQE offers a natural-scene-statistics-based quality measure (Mittal et al., 2013). These methods illustrate different targets and information assumptions rather than interchangeable instruments. A generated image can look convincing while missing the instruction, or satisfy a prompt while containing visible artifacts. The evaluation should retain those possibilities instead of treating every score as a noisy observation of one universal quality. In a secondary analysis of released human judgments, relationships between quality and alignment describe those judgments. They do not establish how one of these automated instruments would perform unless its predictions are actually obtained and assessed under a suitable independent protocol.
An evaluation protocol should state what remains unfamiliar at assessment time. Leaving out an observation while retaining its close relatives can test interpolation rather than the broader transfer suggested by the study's wording. Validation methods for structured data emphasize matching the split to temporal, spatial, or hierarchical dependence (Roberts et al., 2017). Separately, parameter selection can bias an evaluation if the same observations guide both choices and reporting (Varma and Simon, 2006). These issues are not solved by calling a split a test set. The record should identify shared sources, the information accessible during development, and the stage at which each design choice was fixed. A result can be informative within a narrow boundary while providing no direct evidence about a more distant deployment population.
A pooled all-row average weights conditions according to their observed row counts. An equal-condition average uses another weighting rule. A prompt average over available rows can likewise combine a different condition mixture for incomplete prompts. These are legitimate descriptive quantities when named clearly, but they should not be presented as interchangeable summaries of a fully balanced experiment. The reconstructed grid allows a reader to determine which denominator and weighting population each calculation actually represents.
Paired comparisons need their own observed intersection. A contrast between complete conditions can retain all 300 prompts, while a contrast involving an incomplete condition may use fewer. Restricting every analysis to the 287 complete prompts would restore a common observed condition grid but change the prompt population. It would not explain the missingness or establish unbiased inference about omitted cells. A useful sensitivity report could present both estimands while keeping their support and interpretation distinct.
The pattern of absence alone cannot determine a missing-data mechanism. It does not establish independence from quality, prompt content, or generation difficulty. Those possibilities can be stated as research questions, but assigning one would exceed the available record. A future source audit or new controlled collection could add the required evidence. Until then, the practical response is explicit coverage and bounded conclusions: compare observed judgments transparently and avoid expanding a balanced-looking table format into an unsupported assumption that every intended outcome was measured.
Established quality datasets and metrics remind the analyst that observation populations differ. KonIQ-10k studies quality in a large in-the-wild image collection (Hosu et al., 2019), while KADID-10k supports assessment of declared image distortions (Lin et al., 2019). These are not automatically the same population as generated imagery. Structural similarity (Wang et al., 2004) and learned perceptual similarity (Zhang et al., 2018) also require an appropriate interpretation of the compared images. FID instead compares distributions in a learned representation (Heusel et al., 2017). None of these objects is interchangeable with an individual human alignment judgment. A study should state which references or populations a metric needs and whether those inputs exist in the experiment. Distributional quality cannot be inferred solely from one image's mean score, and a high pairwise resemblance can reward repetition rather than successful instruction execution. The relevant lesson for a score-only analysis is to keep its outcome boundary precise. It can describe published judgments and propose a future metric comparison without reporting that the comparison has already been performed.
An association can suggest a research direction without identifying an intervention. Causal diagrams make it possible to state which variables precede the proposed change, which are consequences, and which selection processes remain uncontrolled (Greenland et al., 1999). Their value lies in exposing assumptions rather than decorating a regression with a causal label. A second issue arises when many outcomes or subgroups are explored: selecting the most favorable comparison can exaggerate the impression of support. False-discovery procedures address a defined family of inferential tests (Benjamini and Hochberg, 1995). They do not turn an exploratory pattern into a replicated mechanism. A careful report distinguishes prespecified comparisons, descriptive sensitivities, and hypotheses requiring new observations. This distinction is especially important when an evaluation offers many plausible metrics.
Replacing an expensive judgment with an automated score changes the measurement process. A positive correlation is evidence of co-movement, but it does not establish that the two procedures can be used interchangeably (Bland and Altman, 1986). A calibrated forecast has a different requirement: its stated probabilities should match relevant observed frequencies (Guo et al., 2017). Neither property follows merely from a favorable overall ranking. A useful comparison preserves paired measurements, identifies the conditions in which disagreement matters, and keeps scale transformations explicit. If the intended judgment concerns recognizability, playability, or physical compatibility, the proxy should be assessed against that judgment directly. Agreement on easy cases cannot supply evidence about the difficult conditions absent from the comparison.
An image that receives a high quality judgment need not win every user comparison. Pick-a-Pic (Kirstain et al., 2023) and Human Preference Score (Wu et al., 2023) motivate attention to preference as a distinct assessed outcome. DINO (Caron et al., 2021) provides self-supervised visual features that can characterize content under another representation. DreamBooth (Ruiz et al., 2022) and IP-Adapter (Ye et al., 2023) illustrate how subject or image references can influence the generation task before scoring begins. These sources help explain why an evaluation should specify both its input evidence and its target judgment. A method with additional references or adaptation opportunities may solve a different information problem. Likewise, a predictor of aggregate alignment may require human information unavailable in a practical no-reference quality system. The report should make such access explicit. A secondary score-table analysis remains informative when it describes its available outcomes precisely and reserves statements about preference prediction, subject conditioning, or visual-feature performance for studies in which those outputs were actually obtained and independently assessed.
A system's reliable operating region may be narrower than the collection of inputs an interface accepts. Selective prediction separates performance on accepted cases from the fraction of cases accepted (Geifman and El-Yaniv, 2017). Reporting only the former can hide a substantial loss of coverage. Uncertainty estimates may also change under distribution shift (Ovadia et al., 2019), so familiar-condition confidence is not automatically evidence about new subjects, scenes, or control histories. A useful evaluation records rejected, ambiguous, and unsupported cases alongside successful ones. This makes it possible to distinguish limited but honest capability from a system that answers broadly with unreliable certainty. Any proposed deferral rule still requires assessment on its intended input population; it is not validated merely by being described.
Reproducibility begins with preserving which scientific objects are being linked. Stable identifiers and reusable metadata help connect inputs, reference observations, model versions, and reported outcomes (Wilkinson et al., 2016). The scoring pipeline also has dependencies that can change the interpretation of a result: preprocessing, representation checkpoints, thresholds, and aggregation code can all matter (Sculley et al., 2015). A compact release should retain those decisions rather than only the final mean. Where source data cannot be redistributed, an access description and hashes can still identify the analyzed inputs. This record allows later users to distinguish a change in the underlying system from a change in how the system was measured. Reproducibility is therefore part of interpretation, not simply a final packaging step.
The released benchmark has eighteen absent prompt-condition cells concentrated in three condition labels. This is a reproducible support finding, not an explanation of absence or an estimate of missing outcomes. It clarifies the populations behind pooled, prompt-level, and paired analyses. Future semantic and perceptual assessment should preserve this coverage information so that a score difference refers to the observations actually available rather than to an assumed complete experiment.