Does the rigor of an early trial paper predict its Phase 3 outcome?

Any model can grade a paper. The harder question is whether that grade means anything about what happens next. So we ran a clean test: score the early publication of 166 drug assets whose Phase 3 result is now known, blind to that result, and check every rigor criterion against later success or failure.

What we did

We assembled a cohort of drug assets that each have two things: a pivotal early-phase publication, and a later Phase 3 read-out that is now public. The assets come from ClinicalTrials.gov Phase 1/2 to Phase 3 trajectories, built through the v2 API. Starting from 290 candidate trajectories, we de-duplicated correlated assets from the same program (so one drug is never counted twice) down to 224, then curated a benchmark cohort of 166.

Of the 166, 60 went on to succeed in Phase 3 and 106 went on to fail. The outcome was adjudicated independently of, and after, the rigor review: 83 are trial terminations read from the registry’s stated reason for stopping, and the remaining outcomes were adjudicated from the completed-trial results. The review engine itself saw only the early paper, frozen at the decision point, with no access to anything downstream.

Every early paper was scored by the real Adcurare rigor engine in full audit mode, nothing skipped, at a single pinned engine version. That is the same eight-dimension taxonomy and 1-to-5-star deduction scale the product ships, so nothing here is a special-purpose model built to win this benchmark.

What the engine saw, before any outcome

Read purely as a grader of evidence packages, the corpus is unremarkable and fairly typical of the clinical literature. The mean rating was 2.6 of 5 stars. Split into bands, 47 papers landed in the top half (Strong or Exemplary is uncommon), most sat in the middle, and 45 drew a critical rating that flags the paper for priority expert review. On the pass / concerns / fail verdict, the large majority carried at least one candidate concern.

The headline: rigor doesn’t predict the outcome

When you line the grade up against what actually happened in Phase 3, the two barely separate. On the deterministic rigor score, the assets that later succeeded averaged 81 out of 100; the ones that later failed averaged 79. That gap is not distinguishable from noise. Expressed as discrimination, where 0.5 is a coin flip and 1.0 is perfect, the area under the ROC curve sits near 0.57 no matter which headline number you use.

The pass / concerns / fail verdict tells the same story. A paper the engine flagged was only 1.2 times as likely to belong to a later failure as a clean one, and that interval comfortably includes no effect (odds ratio 1.24, 95% CI 0.58 to 2.64). A single overall rigor grade, on its own, does not sort the winners from the losers.

Disease area does most of the sorting

The dominant driver of Phase 3 outcome in this cohort is not rigor at all. It is the therapeutic area. Oncology and neuropsychiatry assets failed the large majority of the time; cardiometabolic and renal assets succeeded far more often than they failed. Once you know the indication, you have already explained most of the apparent signal.

The proof is in the adjustment. The marginal discrimination of 0.57 falls to 0.54 once you stratify by indication, which means the small amount of separation the overall score appeared to have was largely indication mix, not rigor.

One criterion-level lead survives correction

The overall score is a blunt instrument, but the criteria underneath it are not all equal. Testing each of the eight dimensions individually against later failure, one stands out and holds up after correcting for the fact that we tested many: biological variables, the dimension covering whether sex, age, strain, and other biological sources of variation are handled properly. A paper the engine faulted on that dimension was about three times as likely to belong to a later failure.

Each row is one rigor criterion, plotted as its odds ratio for a later trial failure with a 95% interval, against the no-effect line at 1. We show the effect sizes but not the row-by-row labels: the mapping is the study’s finding, not a teaser. Four criteria clear 1; two are kept in deliberately because they show no association. After Benjamini-Hochberg correction, only the biological-variables dimension reaches q < 0.05.

A couple of finer-grained findings point the same way without clearing the bar: an overstated claim relative to the evidence (odds ratio ~4.4) and a partially-met biological-variables check (~4.9) each land at q ≈ 0.10. We report them honestly as leads, not confirmed effects.

How robust is the lead?

We re-ran the discrimination inside five different slices of the cohort, to check the signal is not an artifact of one subgroup. Restricting to the highest-confidence marquee tier, to sponsor-matched early papers, to confirmed-full-text papers, and to failures that are outright trial terminations, the area under the curve stays in a narrow 0.53-to-0.62 band. The direction of effect is stable; the intervals stay wide. That is exactly the pattern of a real-but-small lead, not a decisive predictor.

What it means, and what it doesn’t

• Rigor grades the evidence package, not the drug. A clean paper can sit under a drug that fails for biology, and a flawed paper can sit under a drug that works. This study is consistent with both.
• The value is in the specific criteria, not a single score. Collapsing eight dimensions into one number throws away the one place a real association lives.
• These are in-sample associations. There is no held-out test set and no prospective arm. Everything here is hypothesis-generating and needs to be confirmed out of sample before it could inform a decision.

How the study was built

Outcome definition

The Phase 3 result was adjudicated from the registry record and public results, separately from and after the rigor review. Half the failures are trial terminations read from the registry; half of all outcomes were adjudicated by LLM from the completed-trial read-out. Terminations for reasons unrelated to efficacy are examined in a sensitivity analysis rather than scored as failures by default.

Blinding

The review engine scored only the early paper. It had no access to the Phase 3 result, the registry outcome, later corrections or retractions, or any outcome-revealing text.

Criterion selection

The rigor criteria are the engine’s predefined, published taxonomy of eight dimensions, fixed before this cohort. They were not discovered in these 166 assets. Association was then tested across those predefined dimensions and their sub-findings.

Statistical model

Per-criterion association with later failure as a 2×2 odds ratio (Fisher exact), and overall discrimination of the composite score by AUC (Mann–Whitney). Indication area is carried through as a Mantel–Haenszel adjustment.

Multiplicity

95% intervals are shown unadjusted for multiple comparisons. Benjamini–Hochberg false-discovery q-values are computed across the criteria tested; after correction, one dimension reaches q < 0.05, so the individual associations are leads, not confirmed effects.

Train / test separation

None. This is a single retrospective cohort with no held-out test set. The figures are in-sample association, not out-of-sample prediction, and require held-out and prospective validation.