Research paperWorking paperARR-2026-1EFJS8JXH08Q5SF8 · v1 · 2026-11-19

Prompt-order dominance and question-weight sensitivity in published language-model evaluations

Lluis Eriksson

Citable version that has not completed AIRR screening. Not peer reviewed. Screening record

Abstract

All nine comparisons between three program-aided and three chain-of-thought prompt orders have positive accuracy differences on each of eight published tasks. We allow rounding and either common standard-deviation convention. Relative question weights up to 1.06 preserve all comparisons. For ASDiv, compatible triples and question outcomes permit a tie at the exact ratio 29/27. Four rows cannot be reconstructed from three raw full-file binary accuracies with a common deviation convention; rounding each order first reconciles one. These conclusions concern historical summaries and specified scoring rules.

Preview PDF on this page v1

Uses your browser's PDF viewer. Open the PDF directly if a preview is unavailable.

Reference this work

Cite this paper

v1 · 2026-11-19

Lluis Eriksson (2026). Prompt-order dominance and question-weight sensitivity in published language-model evaluations. AIRR.SCIENCE, ARR-2026-1EFJS8JXH08Q5SF8, v1. https://airr.science/papers/ARR-2026-1EFJS8JXH08Q5SF8/versions/v1/ Research preprint; ARR-2026-1EFJS8JXH08Q5SF8 v1. Not peer reviewed by AIRR. Working paper; not admitted to the AIRR accepted collection.

Use this version link to cite the manuscript you read. A Working paper label means the version has not completed AIRR screening. AIRR screening is not journal peer review. Import BibTeX, RIS or CSL JSON into your reference manager to apply a citation style.

Version history

The paper identifier remains stable. Each version has its own immutable release, timestamp and version identifier.

  • v1 · source snapshot available · viewing

Verification record

Frontier-model screening
Not assessed
Source integrity
Pass
Bibliographic integrity
Partial
Reproducibility
Partial
Lean 4
Not applicable

Recorded under ARR-VERIFY-1.0. AIRR verification and screening are not peer review.

AI assistance statement

OpenAI Codex participated in analysis, drafting and assessment in the shared authoring conversation. The first exact-PDF scientific assessment is 4.55/10; its full report has SHA-256 9b2608e90891d8a4780c9e6874115ef974242835ba038b2b2b18714fbbb241cb. The author administers AIRR. External peer review has not been established.

Frontier-model screening

Status: not_assessed. Any listed reports correspond to this exact version under ARR-SCREEN-1.0; no absent assessment is represented as a pass.

    Longitudinal frontier-model record

    Model assessments and review context

    Read the scale and limits
    4.55 / 10.00Very good
    Distinct models
    1
    Range
    4.55–4.55
    Method
    bounded capability-weighted mean · exact version only

    Limited evidence · 0/1 benchmark matches. This is an evidence description, not a probability of correctness.

    Only one distinct model; One provider; shared blind spots possible; Fresh blinded review context not documented for every model; Mathematical benchmark not applied to this subject; Adverse reports or unresolved objections require editorial disposition

    Score calculation and evidence

    Unweighted mean: 4.55. Policy: AIRR-RATING-1.0. Method and limits.

    ModelOriginal scoreWeightBasis
    OpenAI · OpenAI Codex (GPT-6 family, exact serving revision unverified)4.551.000Benchmark outside declared mathematical scope

    Download the calculation snapshot

    OpenAI · OpenAI Codex (GPT-6 family, exact serving revision unverified) · involved in manuscript · review context: shared4.55

    Recommendation: Minor Revision · Material objections: 0

    I examined the three rendered pages and the version-specific evidence. The task table and rounding model support a stronger statement than separation of mean accuracies: all nine cross-order comparisons are positive on each task. The ASDiv construction identifies a limit to extrapolating this conclusion to a change of scoring weights. Empty integer cells diagnose the stated reconstruction assumptions; they do not by themselves establish an upstream reporting error. This is appropriate for a working paper.

    Correctness confidence
    For three observations the largest absolute population-standardized deviation is at most sqrt(2). The lower gap is conservative under both SD conventions. In the ASDiv witness, 1200+432 and 1200+464 give the stated margins; 464−(29/27)432 is exactly zero.
    Rigor
    The integer feasibility test uses sorted triples and closed rounding intervals. Two implementations agreed for all raw-score cells. Query weights are common to the two methods; the construction preserves the historical marginal summaries and makes no claim to the observed overlap.
    Novelty
    The inequality and marginal-outcome extremization are classical. The eight cross-order guarantees, attainable ASDiv limit and row-by-row granularity diagnostic are a specific reanalysis. Methodological priority is not demonstrated.
    Significance
    The archived comparison can certify a historical ordering while still permitting a reversal under modest item reweighting. The results inform release design. One model family and eight related mathematics tasks limit external validity.
    Reproducibility
    The manuscript identifies the exact table and repository revision. Results retain file hashes, both precision protocols and integer witnesses. Two clean source checkouts reproduced the same three-page PDF. Individual historical outputs and the actual upstream rounding code remain unavailable in this assessment.

    Strengths

    • States a conservative guarantee independent of whether sample or population deviations were used.
    • Provides an explicit tie construction with a known question denominator and six realizable run counts.
    • Retains all tested rounding models and the empty feasible sets.

    Weaknesses

    • The weakest 1.06 certificate and the exact 29/27 limit are obtained under different information assumptions. A side-by-side display of those assumptions would aid readers.
    • No independent reconstruction from historical model outputs was possible.
    • The externally reported 36-question example does not establish anything empirical about its separate 50-classifier replication.

    Potential errors

    • Filtering or different historical scoring denominators can reconcile empty count sets.
    • Common question values can be adversarially assigned in the witness; such weights need not correspond to a plausible application.
    • Extrapolation to new prompt orderings needs a sampling assumption absent here.

    Strong novelty candidates

    • The attainable 29/27 ASDiv tie threshold tied to explicit integer triples alongside an all-eight-task 1.06 robustness guarantee.

    Unresolved material objections

    None reported.

    arr:assessment:cbde4df5-35c6-46ec-a03d-ec6db13cdc0a · prompt ARR-ASSESS-1.0 · response SHA-256 83197576da5ba26fa20590cc7e7a40dec1db6e92df23a37b3dcd23e5b7c3a21d · canonical PDF SHA-256 4de32d447c3c8ae0c9b6a45c1a3f64449e0d5ecb77dab7dc8b94c625f9ed70e4

    A model assessment is not peer review or a correctness certificate. AIRR preserves disagreement, exact-version provenance and later reassessments.

    Editorial disclosure

    Working-paper deposit under exact-hash permission. Collection screening and independent peer review remain outstanding.

    Record timestamps & download statistics
    Deposit recordedFirst repository commit · full Git SHA recorded
    PublishedGitHub release · tag ARR-2026-1EFJS8JXH08Q5SF8-v1
    Canonical PDF downloads2GitHub release downloads only; direct AIRR PDF reads are not included
    Page viewsNot availablePage-view counts are not available in this public snapshot. Optional operator statistics are separate.
    Definitions and rankings →