Two open-weight models reach the top six: GLM-5.3 is fourth at 86.1, ahead of Opus 5; GPT-6 Astra leads at 87.9.

We’re adding three tasks from TrialPanorama to evaluate how models find relevant studies, select eligible evidence, and interpret clinical findings. We evaluated 25 models on the complete published test splits across all three tasks.

Open-weight models remain competitive as ClinReg expands to systematic literature review. GLM-5.3 ranks fourth overall at 86.1, just 1.8 points behind the leader and ahead of Opus 5. GLM-5.3 Flash scores 85.1, within 0.2 points of Opus 5. On the new SLR composite, Kimi K3 leads open-weight models at 78.1, compared with 82.2 for the overall SLR leader, Opus 5. The task-level results show where open models are closest to the leaders and where further improvement is needed.

Task and dataset

TrialPanorama brings together 1.6 million clinical trial records from 15 global registries, linking them to biomedical ontologies and associated literature. We use its published task definitions and released test datasets to expand ClinReg. Our contribution is evaluating a broader model lineup and reporting task performance alongside answer coverage.

We use the original published test splits in full, without sampling, filtering, or modifying the released records.

Study search tests query formulation. The harness executes the model’s query against PubMed and compares the retrieved papers with the studies included in the source review. Search results are not returned to the model for refinement.

Study screening tests application of eligibility criteria. Each review-level request supplies that review’s criteria and up to 10 candidate papers. The released candidates carry labels derived from the source reviews: 491 included and 488 excluded occurrences. Repeated candidates are preserved.

Evidence summarization tests interpretation of supplied evidence. Each question already embeds study abstracts, and the model selects a multiple-choice answer. Despite its name, this task does not require writing a free-form summary or a full review.

The tasks are independent: search outputs do not feed the screening test, and screening decisions do not determine the evidence supplied for summarization. They evaluate individual capabilities rather than an agent conducting an end-to-end literature review.

Evaluation

We use the reference evaluator’s task-specific scoring semantics and report coverage alongside performance. Gold labels are withheld from the evaluated models. Final results include recovery attempts where needed; complete coverage does not imply every initial request succeeded.

Study search. We report Recall@100 and Recall@500, measuring the fraction of a review’s gold studies recovered within each retrieval cutoff:

Recall@k = Number of gold studies found in the top k results ÷ Total number of gold studies

Recall is calculated per review and averaged with equal weight across scored reviews. The reference evaluator excludes error-bearing evaluations, so each result must be read alongside its scored-review count.

Study screening. We report accuracy, precision, recall, F1, and specificity, using F1 as the primary summary metric. Precision measures how often inclusion decisions are correct; recall measures how many eligible studies are retained. F1 balances the two.

All 25 evaluated models have scored decisions for all 979 candidate-study occurrences in the final screening results.

Evidence summarization. We use accuracy over all 1,000 questions as the primary metric, counting missing or invalid answers as incorrect. We also report macro-F1, which averages F1 equally across gold answer classes. Our macro-F1 class-set and invalid-answer conventions remain unconfirmed against the authors’ calculation. Explanations are retained for inspection, not independently scored for correctness.

Results and observations by task

Study search: retrieval depth changes the ordering

Open-weight models come closest to the task leader in study search. GLM-5.3 Flash reaches 69.3% Recall@500, just 2.4 percentage points behind Gemini 3.8 Flash at 71.7%. MiniMax M3 leads open-weight models at Recall@100 with 39.2%, compared with the overall leader’s 42.8%.

Gemini 3.8 Flash has the highest observed recall at both cutoffs. Muse Spark ranks second at Recall@100 (41.9%), while Gemini 3.7 Flash ranks second at Recall@500 (71.2%).

Selected results illustrate how differently the two cutoffs can behave:

Retrieval depth changes the ranking: models strongest within the first 100 results aren't always strongest by 500 — Opus 5 climbs from 24.6% to 70.2%.

Performance within the first 100 results does not tell the whole retrieval story. Opus 5 moves from 24.6% recall at 100 to 70.2% at 500. MiniMax M3 retrieves more of the reference evidence within the first 100 results, but less by 500.

These metrics answer different practical questions. Recall@100 describes how much evidence is available within a smaller screening pool. Recall@500 measures how much becomes reachable when a reviewer can examine more results. Neither measures precision or directly quantifies the time required for screening.

Among open-weight models, MiniMax M3 has the highest Recall@100 at 39.2%, while GLM-5.3 Flash has the highest Recall@500 at 69.3%.

Study screening: precision and recall reveal different strengths

GLM-5.3 Flash leads open-weight models on screening F1 at 74.6%, slightly ahead of Gemini 3.8 Flash at 74.0%. Screening also presents the largest gap between the best open-weight model and the task leader: 6.2 percentage points behind Opus 5 at 80.8%. GPT-6 Astra and GPT-5.6 Sol follow Opus 5 at 80.3% and 79.5%, respectively.

Screening is where open-weight trails most — and still not by much. GLM-5.3 Flash tops the open field at 74.6% F1, 6.2 points behind Opus 5.

The search leader is not the screening F1 leader. Gemini 3.8 Flash has the highest screening precision, but its lower recall means it misses more eligible candidates than Opus 5 or GPT-6 Astra.

That distinction matters operationally. Higher precision reduces the proportion of included studies that are irrelevant. Higher recall reduces missed eligible evidence. F1 provides a balanced summary, while the underlying precision and recall help teams choose a model for their workflow.

GLM-5.3 Flash has the highest observed F1 among open-weight models at 74.6%.

Evidence summarization: interpreting supplied evidence produces another ordering

Kimi K3 leads open-weight models on evidence accuracy at 91.5%, demonstrating strong performance on interpreting supplied clinical evidence. Opus 5 leads overall at 95.7%, followed by GPT-6 Astra at 94.4% and Opus 4.8 at 94.0%.

Interpreting the evidence favors closed models, narrowly. Kimi K3 answers 91.5% correctly, within 4.2 points of Opus 5's 95.7%.

Kimi K3 has the highest observed scores among open-weight models on both evidence metrics, with 91.5% accuracy and 89.7% macro-F1. Its accuracy is 4.2 percentage points below the observed leader.

These scores measure selection of a supported answer from supplied options. They do not establish the ability to retrieve missing evidence, conduct a complete review, or produce a verified narrative synthesis.

Bringing the tasks into ClinReg

We report the task metrics separately and use one metric from each task for a supplementary SLR composite:

SLR score = (Recall@500 + Screening F1 + Evidence accuracy) ÷ 3

All three components are expressed on a 0–100 scale and receive equal weight. For study search, we choose Recall@500 to emphasize recovery of relevant evidence within a pool of up to 500 retrieved results per review. Recall@100 remains a companion measure of coverage within a smaller screening pool.

TrialPanorama reports both retrieval depths separately and does not designate either as primary. Selecting Recall@500 for the composite is our reporting choice, made after examining the results, rather than part of its reference evaluation. We do not average the two recalls.

Across all three literature-review tasks combined, open-weight Kimi K3 (78.1) lands within striking distance of the overall leader, Opus 5 (82.2).

This chart shows selected models. Kimi K3 leads the evaluated open-weight models with an SLR score of 78.1, followed by GLM-5.3 Flash at 77.5. Opus 5 leads overall at 82.2. The two Gemini Flash models have equal composites at the precision of the supplied task scores. These results do not establish statistically significant differences between models.

In the expanded ClinReg summary, IND, TLF, Lit screening, and SLR each receive 25% weight:

ClinReg overall = (IND score + TLF score + Lit screening score + SLR score) ÷ 4

The existing Lit screening task evaluates full-text eligibility, while TrialPanorama screening evaluates titles and abstracts across multiple reviews. We retain them separately for continuity with the existing leaderboard, while recognizing their overlap: together, the two screening components receive one-third of the overall weight.

Two open-weight models reach the top six: GLM-5.3 is fourth at 86.1, ahead of Opus 5; GPT-6 Astra leads at 87.9.

The chart shows selected models. Two open-weight models place in the overall top six: GLM-5.3 is fourth at 86.1, and GLM-5.3 Flash is sixth at 85.1. GLM-5.3 scores above Opus 5’s 85.3, while GPT-6 Astra leads the full leaderboard at 87.9.

Adding SLR changes the overall ordering: GPT-5.6 Sol rises from fourth to second and Opus 5 from eighth to fifth, while GLM-5.3 moves from second to fourth and Muse Spark from tenth to twelfth. GPT-6 Astra remains first, with its overall score moving from 89.9 to 87.9; these changes reflect the expanded evaluation and revised weighting, rather than changes in the models’ performance on the original tasks.

The overall score summarizes performance across ClinReg, while task-level scores show the strengths and weaknesses behind each model’s ranking.

Interpretation and limitations

The results support including open-weight models in clinical evidence evaluations and selecting them by task. Their strongest models are 2.4 points behind the search leader at Recall@500, 6.2 points behind on screening F1, and 4.2 points behind on evidence accuracy. Across the broader ClinReg benchmark, GLM-5.3 remains within 1.8 points of the overall leader.

Different open-weight families contribute different strengths: MiniMax M3 leads their Recall@100 results, GLM-5.3 Flash leads their Recall@500 and screening F1 results, and Kimi K3 leads their evidence accuracy and SLR composite. No single open-weight model leads every stage. The same is true of the overall task leaders: Gemini 3.8 Flash leads search, while Opus 5 leads screening F1 and evidence accuracy.

These tasks give ClinReg a broader view of clinical evidence work. They make it possible to compare finding, selecting, and interpreting evidence separately, without assuming that strength in one guarantees strength in the others.

Comparison with TrialPanorama’s published results

We compare the best results in TrialPanorama’s Figure 4 with the best scores in our ClinReg evaluation. We use Recall@500 for search and accuracy for evidence summarization. For screening, we compare the directly reported precision and recall; F1 remains our primary screening metric.

Today's models exceed the original Nature study's best marks — search recall rises from 55.8% to 71.7%.

The screening results identify separate metric leaders; no single model achieves both the highest precision and the highest recall.

The paper’s “-TP” models are fine-tuned on TrialPanorama’s training data. Its Qwen3-8B-TP leads the published Recall@500, screening recall, and evidence accuracy results, highlighting the potential of task-specific adaptation in smaller open-weight models.

The newer general-purpose models achieve higher observed scores without additional TrialPanorama fine-tuning by us. These are historical comparisons: model lineups and execution procedures differ, and search also depends on changes in PubMed’s corpus and ranking. They do not establish that general-purpose models are inherently better than specialized models.

Open-weight models remain competitive on the expanded ClinReg leaderboard, with GLM-5.3 ranking fourth overall, just 1.8 points behind the leader. The SLR tasks identify specific gaps to target, making fine-tuning newer open-weight models on TrialPanorama’s training data a promising next experiment.

________________________________________________________________________________________

Appendix

Illustrative examples

The following examples come from TrialPanorama’s released test sets. They illustrate what models receive, what they must produce, and the judgments involved. Questions are shortened and selected abstract excerpts are shown for readability. Reference answers are included below for explanation but are withheld from the evaluated models.

Study search: finding evidence on rehabilitation after stroke

Review question

Does physical rehabilitation improve function and mobility after stroke, and is one rehabilitation approach more effective than another?

Eligibility criteria

Randomized controlled trials of physical rehabilitation approaches aimed at improving function or mobility in adults with a clinical diagnosis of stroke.

Required output

The model generates a PubMed query. The harness executes it and measures recovery of the review’s reference studies within the first 100 and 500 results. Search results are not returned to the model for refinement.

Illustrative query

(stroke[tiab] OR poststroke[tiab])
AND (rehabilitation[tiab] OR physiotherapy[tiab])
AND (randomized controlled trial[pt] OR random*[tiab])

This query was composed for illustration; it is not an evaluated model output, and its retrieval performance has not been measured. The test record supplies reference study IDs rather than a single correct query.

What makes this difficult?

Relevant papers may name a specific therapy without using “rehabilitation” or “physiotherapy.” Adding alternative terms can recover more studies while also retrieving more irrelevant papers. Strict filters can miss eligible studies whose abstracts omit the corresponding details. The model must translate the review’s scope into a search strategy without assuming that every eligibility criterion should become a query filter.

Source: Study-search test split, review PMID 39932103.

Study screening: deciding which studies meet the criteria

Review question

How safe and effective are electronic cigarettes for helping people achieve long-term smoking abstinence?

Eligibility criteria

The review accepts randomized trials and uncontrolled intervention studies in which participants receive an electronic-cigarette intervention. Studies must report an eligible outcome.

Candidate A

Effect of an electronic nicotine delivery device (e-Cigarette) on smoking reduction and cessation: a prospective 6-month pilot study.

Selected abstract excerpt:

In this prospective proof-of-concept study we monitored possible modifications in smoking habits of 40 regular smokers (unwilling to quit) experimenting the 'Categoria' e-Cigarette with a focus on smoking reduction and smoking abstinence. […] Smoking reduction and abstinence rates were calculated. Adverse events and product preferences were also reviewed.

Candidate B

e-Cigarette awareness, use, and harm perceptions in US adults.

Selected abstract excerpt:

We drew data from 2 surveys conducted in 2010: a national online study (n = 2649) and the Legacy Longitudinal Smoker Cohort (n = 3658). We used multivariable models to examine e-cigarette awareness, use, and harm perceptions.

Consider the decision

Would you include or exclude each candidate? Both discuss e-cigarettes and smokers, but do both meet the review’s study-design and outcome requirements?

________________________________________________________________________________________

Reference labels: Candidate A — Include; Candidate B — Exclude.

Candidate A reports smoking outcomes from an intervention study. The criteria explicitly allow uncontrolled intervention studies, so lack of randomization alone is not grounds for exclusion. Candidate B examines awareness, use, and perceptions through surveys rather than an intervention.

What makes this difficult?

Topical relevance is insufficient. The model must apply the particular review’s criteria, including exceptions such as allowing uncontrolled studies, rather than relying on a generic preference for randomized trials.

Source: Study-screening test split, review PMID 39878158; candidate PMIDs 21989407 and 22813087. Labels are from the dataset; the explanation above is our interpretation of the criteria and abstracts.

Evidence summarization: reconciling conflicting findings

Clinical question

How effective and safe are antiepileptic drugs (AEDs) for preventing seizures after craniotomy?

The full test record supplies 10 study abstracts. Two contrasting excerpts illustrate the evidence the model must interpret.

Study 1: Phenytoin and postoperative epilepsy

A significant reduction in the frequency of epilepsy was observed in the group receiving the active drug up to the 10th postoperative week.

Study 2: A prospective randomized trial of perioperative seizure prophylaxis in patients with intraparenchymal brain tumors

The incidence of all seizures was 18% in the observation group and 24% in the prophylaxis group (p = 0.51). […] The prophylaxis group experienced significantly more adverse events (18% vs 0%, p < 0.01).

Answer options

• A: AED treatment is highly effective in preventing both early and late postcraniotomy seizures, with minimal adverse effects.

• B: AED treatment shows a statistically significant advantage in preventing early seizures but not late seizures, with some adverse effects reported.

• C: There is limited, low-certainty evidence to suggest that AED treatment is either effective or not effective in preventing postcraniotomy seizures, with inconsistencies in the reporting of outcomes.

• D: AED treatment is ineffective in preventing postcraniotomy seizures and is associated with a high incidence of adverse effects.

Consider the decision

Which conclusion is best supported? Does a positive trial establish a general benefit? Does a trial without a statistically significant benefit establish ineffectiveness?

________________________________________________________________________________________

Reference answer: C.

The dataset’s explanation states:

The evidence available is of low certainty and does not conclusively show that AED treatment is effective or ineffective in preventing postcraniotomy seizures. There are also inconsistencies in the reporting of outcomes, including deaths and adverse events.

What makes this difficult?

The model must reconcile differing findings without overstating certainty. A positive finding from one study does not settle the overall question, and failure to demonstrate a benefit does not establish that treatment is ineffective. The reference answer applies to the complete record of 10 abstracts; the two excerpts above illustrate the tension rather than reproduce all the evidence supplied for the task.

Source: Evidence-summarization test split, record 4636; review PMID 32343399.