Prompt-order dominance and question-weight sensitivity in published language-model evaluations
Citable version that has not completed AIRR screening. Not peer reviewed. Screening record
All nine comparisons between three program-aided and three chain-of-thought prompt orders have positive accuracy differences on each of eight published tasks. We allow rounding and either common standard-deviation convention. Relative question weights up to 1.06 preserve all comparisons. For ASDiv, compatible triples and question outcomes permit a tie at the exact ratio 29/27. Four rows cannot be reconstructed from three raw full-file binary accuracies with a common deviation convention; rounding each order first reconciles one. These conclusions concern historical summaries and specified scoring rules.
Preview PDF on this page v1
Uses your browser's PDF viewer. Open the PDF directly if a preview is unavailable.
Cite this paper
Lluis Eriksson (2026). Prompt-order dominance and question-weight sensitivity in published language-model evaluations. AIRR.SCIENCE, ARR-2026-1EFJS8JXH08Q5SF8, v1. https://airr.science/papers/ARR-2026-1EFJS8JXH08Q5SF8/versions/v1/ Research preprint; ARR-2026-1EFJS8JXH08Q5SF8 v1. Not peer reviewed by AIRR. Working paper; not admitted to the AIRR accepted collection.
Use this version link to cite the manuscript you read. A Working paper label means the version has not completed AIRR screening. AIRR screening is not journal peer review. Import BibTeX, RIS or CSL JSON into your reference manager to apply a citation style.
Version history
The paper identifier remains stable. Each version has its own immutable release, timestamp and version identifier.
- v1 · source snapshot available · viewing
Verification record
- Frontier-model screening
- Not assessed
- Source integrity
- Pass
- Bibliographic integrity
- Partial
- Reproducibility
- Partial
- Lean 4
- Not applicable
Recorded under ARR-VERIFY-1.0. AIRR verification and screening are not peer review.
AI assistance statement
OpenAI Codex participated in analysis, drafting and assessment in the shared authoring conversation. The first exact-PDF scientific assessment is 4.55/10; its full report has SHA-256 9b2608e90891d8a4780c9e6874115ef974242835ba038b2b2b18714fbbb241cb. The author administers AIRR. External peer review has not been established.
Frontier-model screening
Status: not_assessed. Any listed reports correspond to this exact version under ARR-SCREEN-1.0; no absent assessment is represented as a pass.
Model assessments and review context
- Distinct models
- 1
- Range
- 4.55–4.55
- Method
- bounded capability-weighted mean · exact version only
Limited evidence · 0/1 benchmark matches. This is an evidence description, not a probability of correctness.
Only one distinct model; One provider; shared blind spots possible; Fresh blinded review context not documented for every model; Mathematical benchmark not applied to this subject; Adverse reports or unresolved objections require editorial disposition
OpenAI · OpenAI Codex (GPT-6 family, exact serving revision unverified) · involved in manuscript · review context: shared4.55
Recommendation: Minor Revision · Material objections: 0
I examined the three rendered pages and the version-specific evidence. The task table and rounding model support a stronger statement than separation of mean accuracies: all nine cross-order comparisons are positive on each task. The ASDiv construction identifies a limit to extrapolating this conclusion to a change of scoring weights. Empty integer cells diagnose the stated reconstruction assumptions; they do not by themselves establish an upstream reporting error. This is appropriate for a working paper.
- Correctness confidence
- For three observations the largest absolute population-standardized deviation is at most sqrt(2). The lower gap is conservative under both SD conventions. In the ASDiv witness, 1200+432 and 1200+464 give the stated margins; 464−(29/27)432 is exactly zero.
- Rigor
- The integer feasibility test uses sorted triples and closed rounding intervals. Two implementations agreed for all raw-score cells. Query weights are common to the two methods; the construction preserves the historical marginal summaries and makes no claim to the observed overlap.
- Novelty
- The inequality and marginal-outcome extremization are classical. The eight cross-order guarantees, attainable ASDiv limit and row-by-row granularity diagnostic are a specific reanalysis. Methodological priority is not demonstrated.
- Significance
- The archived comparison can certify a historical ordering while still permitting a reversal under modest item reweighting. The results inform release design. One model family and eight related mathematics tasks limit external validity.
- Reproducibility
- The manuscript identifies the exact table and repository revision. Results retain file hashes, both precision protocols and integer witnesses. Two clean source checkouts reproduced the same three-page PDF. Individual historical outputs and the actual upstream rounding code remain unavailable in this assessment.
Strengths
- States a conservative guarantee independent of whether sample or population deviations were used.
- Provides an explicit tie construction with a known question denominator and six realizable run counts.
- Retains all tested rounding models and the empty feasible sets.
Weaknesses
- The weakest 1.06 certificate and the exact 29/27 limit are obtained under different information assumptions. A side-by-side display of those assumptions would aid readers.
- No independent reconstruction from historical model outputs was possible.
- The externally reported 36-question example does not establish anything empirical about its separate 50-classifier replication.
Potential errors
- Filtering or different historical scoring denominators can reconcile empty count sets.
- Common question values can be adversarially assigned in the witness; such weights need not correspond to a plausible application.
- Extrapolation to new prompt orderings needs a sampling assumption absent here.
Strong novelty candidates
- The attainable 29/27 ASDiv tie threshold tied to explicit integer triples alongside an all-eight-task 1.06 robustness guarantee.
Unresolved material objections
None reported.
arr:assessment:cbde4df5-35c6-46ec-a03d-ec6db13cdc0a · prompt ARR-ASSESS-1.0 · response SHA-256 83197576da5ba26fa20590cc7e7a40dec1db6e92df23a37b3dcd23e5b7c3a21d · canonical PDF SHA-256 4de32d447c3c8ae0c9b6a45c1a3f64449e0d5ecb77dab7dc8b94c625f9ed70e4
A model assessment is not peer review or a correctness certificate. AIRR preserves disagreement, exact-version provenance and later reassessments.
Editorial disclosure
Working-paper deposit under exact-hash permission. Collection screening and independent peer review remain outstanding.