Advances in Knowledge and Society
Original publication date: 2026-09-09
Archive migration: 2026-10-30
Brandon Perry — Madagascar; Grant Allen — Côte d'Ivoire; Ryan Turner — Senegal
Public availability of a person's images does not explain why a benchmark selected that person or which conclusions its observations support. This article examines documentation for subject-preserving video evaluation using public-figure references. It separates collection provenance, scientific coverage, access conditions, and intended use. ARGUS motivates the reference-based task, while the proposed audit concerns benchmark records rather than generated performance. No new images are collected. The outcome is a practical account of what a public-figure evaluation should disclose before broad preservation claims are made.
A collection of accessible portraits can be convenient for evaluation. Convenience does not establish a sampling frame, representative appearance variation, or suitability for every identity task. A benchmark may emphasize people who have many well-lit photographs and who are already familiar to reviewers. Those properties can affect recognition independently of the generator. The documentation should identify the scientific reason for selection rather than relying on the fact that a search returned usable images.
ARGUS uses multiple identity references to support subject-preserving video generation (Meng et al., 2026b). That task creates questions about how reference subjects and their views are assembled. The present article proposes a documentation audit for a public-figure benchmark, without collecting portraits or measuring generated identity. It does not resolve permissions from the appearance of a web page or offer a legal classification. Its central concern is whether the benchmark explains the evidence population and the uses for which its measurements are intended.
Identity evidence has a sampling structure before any recognition score is calculated. LFW defines a particular pair-verification protocol (Huang et al., 2008), and the documented scikit-learn loader makes its prescribed files accessible through a reproducible software interface (Pedregosa et al., 2011). Neither source turns a pair count into a count of independent subjects or images. Dataset documentation should identify how references, identities, comparisons, and splits were assembled (Gebru et al., 2021). For generated video, the analogous record should distinguish a new person from another reference photograph, another trajectory, or another frame. Each adds evidence along a different dimension. A transparent report should preserve those links rather than flattening every observation into one score table. The resulting metadata can support a useful audit even when no images or generated clips are available. Such an audit describes coverage and organization; it does not itself establish recognition accuracy, demographic representativeness, or identity preservation in an unobserved video.
An audit would begin with a record of source locations, acquisition dates, image identifiers, selection rules, and any processing used to create reference sets. The record should distinguish the original observation from a crop, resized copy, or duplicated publication. Several available files can originate from one photographic session and provide less variation than their count suggests. References should therefore be linked to their source observations where that relationship can be established.
The documentation should also describe the task boundary. Assessing recognizability under a specified transformation differs from building a general-purpose identity system or endorsing uses outside evaluation. Access conditions and retention procedures should be stated as observed operational facts. If the project changes its intended use, its earlier documentation should not be treated as evidence that every new use was already considered. Versions permit those changes to be understood without erasing the history of the collection.
Scientific coverage requires its own description. Counts of subjects, views, acquisition conditions, and repeated observations can be reported from documented metadata. Claims about demographic representation need appropriate annotations and a justified interpretation. They cannot be inferred from fame, names, or the apparent diversity of a contact sheet. Where relevant information is absent, the audit should say that coverage is unknown instead of manufacturing a descriptive category from visual impressions.
Replication has several levels: a second measurement, a second trajectory, and a second independently acquired setting are different additions to evidence. Statistical calculations should retain those distinctions. Heteroskedasticity-robust regression inference concerns unequal conditional variances (White, 1980); cluster-robust inference concerns dependencies shared within groups (Cameron and Miller, 2015). The group definition should be justified through the data collection and evaluation process, and not chosen because it produces a preferred significance result. Unequal group sizes also influence what a pooled average represents. Reporting both the group count and observation count permits a reader to see whether the calculation describes common cases, heavily recorded cases, or an explicitly balanced group population. No variance estimator alone resolves that choice of estimand.
An identity score is produced by a measurement system with its own training and assumptions. FaceNet (Schroff et al., 2015), ArcFace (Deng et al., 2018), and CosFace (Wang et al., 2018) offer influential approaches to discriminative face representation. Their outputs should be interpreted in relation to the images and decisions on which they are assessed. VGGFace2 broadens variation within subject observations (Cao et al., 2017), MegaFace makes large distractor populations important (Kemelmacher-Shlizerman et al., 2015), and FairFace addresses imbalance across annotated demographic categories (Kärkkäinen and Joo, 2019). These are complementary evaluation concerns. A generated-video study should not compress them into an unexplained claim that a face benchmark is difficult. It should specify whether difficulty comes from viewpoint, acquisition, distractors, subject population, or the representation itself. Demographic conclusions require appropriate annotations and cannot be inferred merely from identity frequencies or from a generic face-similarity score. A metric's usefulness for one purpose remains an empirical question for another.
A validation split defines which form of transfer has been observed. Related observations can share subjects, scenes, prompts, or acquisition conditions, so randomly splitting rows may leave important information on both sides of the boundary. Structured cross-validation addresses these dependencies (Roberts et al., 2017). Model selection introduces another boundary: choosing a representation, threshold, or tuning parameter using the final assessment data makes the resulting error estimate optimistic (Varma and Simon, 2006). The appropriate held-out unit follows the intended future use. Generalization to a new frame, a new trajectory, and a new subject are not equivalent claims. A study should record those units before fitting or choosing a scoring rule, and preserve an assessment set that has not become part of iterative development.
Public familiarity can change a human assessment. A reviewer who already knows a person may use memories beyond the supplied references, while another reviewer depends entirely on those references. Both can supply useful observations, but they answer different questions about the information available in the task. A proposed evaluation should record the intended role of prior familiarity and test whether the comparison remains meaningful when familiarity varies among observers.
The same concern extends to possible training overlap. Widespread publication of an image does not establish that a particular generation model encountered it. Conversely, lack of a matching filename does not rule out related material. A benchmark can document known overlap checks and their limitations. It should avoid converting an incomplete search into a claim of contamination-free evaluation. If the underlying training record is unavailable, that uncertainty belongs in the interpretation of preservation performance.
Finally, benchmark maintenance should include a way to correct provenance, revise access, and retire unsuitable records while preserving version identifiers for reproducibility. A correction can change the population represented by a result even when the model itself is unchanged. Comparisons across releases should therefore refer to the exact benchmark version. These procedures make documentation useful to scientific interpretation: they connect a reported score to identifiable observations and an explicit scope rather than to an evolving collection with an unchanged name.
Different reference-conditioning mechanisms can receive different amounts of information. DreamBooth adapts a generation model around a subject (Ruiz et al., 2022), while IP-Adapter offers a compatible image-conditioning route (Ye et al., 2023). A fair comparison should preserve how many views, which resolutions, and which adaptation opportunities are supplied. CLIP's language-supervised visual representation (Radford et al., 2021) and DINO's self-supervised features (Caron et al., 2021) can provide useful descriptions of outputs, but their training objectives do not establish a person-recognition standard. The endpoint should specify whether success means matching reference appearance, retaining a recognizable subject, or satisfying a requested scene. Those properties can disagree. A useful analysis would keep the disagreement visible and ask which source of evidence each method uses. The relevant advantage is a measured improvement under a declared information budget, not an unqualified increase in similarity to whatever reference happened to be available.
Describing a contrast accurately can be more valuable than giving it an unjustified mechanistic name. A causal interpretation needs assumptions about how exposure and outcome arose, including selection into the analyzed sample. Causal diagrams provide a language for those assumptions (Greenland et al., 1999). If several alternative metrics or partitions are examined, the investigator should also account for the freedom to select whichever result appears strongest. False-discovery control is relevant to explicit collections of statistical tests (Benjamini and Hochberg, 1995), while a descriptive sensitivity analysis should be presented as such. A model can fit the observed rows well and still leave the principal explanation unresolved. Reporting that boundary preserves the value of the observation without overstating what adjustment or a significance threshold has established.
A score is useful only relative to the question it is supposed to answer. Correlation measures a relationship between varying quantities, whereas agreement concerns the discrepancies produced when one procedure substitutes for another (Bland and Altman, 1986). Confidence estimates add a further question about calibration (Guo et al., 2017). These distinctions matter when a learned evaluator becomes the principal source of evidence about another learned system. The evaluation should preserve examples with conflicting scores and obtain independent target judgments where feasible. Numerical ranges and normalization choices should also be documented. An apparently small difference may have a different practical meaning near a decision threshold than in an unimportant part of the scale. Reporting the intended consequence gives those comparisons an interpretable purpose.
An identity evaluation becomes more informative when it retains the other properties of a generated clip. VBench's multiple dimensions (Huang et al., 2023) help motivate this decomposition. Structural similarity (Wang et al., 2004) compares a particular aspect of image structure; learned perceptual similarity (Zhang et al., 2018) uses another representation; video-distribution metrics (Unterthiner et al., 2018) incorporate temporal visual features. None alone establishes a stable person through every pose and event. A study should identify the relevant time windows and examine local failures as well as sequence averages. It should also distinguish an identity evaluator's inability to score a frame from the generator's inability to preserve a subject. Those are different observations with different remedies. A measurement profile linked to the original clip permits this distinction to be checked and avoids attributing broad capability to a favorable average obtained only on easily measurable views.
A useful reliability claim contains both a performance statement and a scope statement. Selective prediction shows why a system that answers only some cases must report how many it accepts (Geifman and El-Yaniv, 2017). Confidence estimation under a changed data distribution requires independent checking (Ovadia et al., 2019). These requirements apply to generators and to the automated judges used to assess them. If a judge cannot evaluate a profile, an occluded object, or an unusual interaction, that inability should remain visible rather than removing the case from the generator's score silently. Recording the entire decision path permits an analyst to identify whether an apparent improvement came from better outputs, narrower eligibility, or a more restrictive measurement procedure.
An evaluation remains comparable only if its inputs and transformations can be identified. The FAIR principles emphasize persistent identification and reusable scientific metadata (Wilkinson et al., 2016). Work on machine-learning technical debt highlights how dependencies and hidden couplings accumulate in a pipeline (Sculley et al., 2015). In this setting, a score should be linked to the exact references, generation settings, evaluator version, and aggregation rule that produced it. A replacement checkpoint or a revised eligibility filter may change the scientific question even when the output column keeps the same name. Preserving such changes in a release record permits later comparisons to be interpreted honestly. The objective is not maximum documentation volume, but a recoverable account of the decisions that affect the result.
Public-figure evaluation needs more than accessible images and a recognizable list of names. It needs recoverable provenance, selection logic, task boundaries, and versioned coverage records. This article proposes an audit rather than reporting a completed benchmark assessment. Documentation cannot establish generated performance, but it can identify what that performance would mean and prevent convenience of collection from becoming an unsupported claim about the full population of subjects or uses.