46
08.07.27
On this page
About this use case: A rare disease foundation holds 320 patient records across three genetic diseases — records that took two ethics cycles and six years to assemble, and that external biomarker teams cannot access under any conventional data sharing arrangement. tracebloc lets them validate their panels on this cohort without a single patient record leaving the foundation's infrastructure. Explore the data, submit your own model, and see how your approach compares.
Published biomarker panels rarely survive first contact with an independent cohort. Biomarker validation is the discipline's persistent failure: a panel trained and reported on one institution's patients classifies a second institution's patients significantly worse — often enough to invalidate the clinical claim. For rare disease research teams working across Cystic Fibrosis, Duchenne Muscular Dystrophy, and Spinal Muscular Atrophy, where patient numbers are small and cohorts are hard to assemble, failed replication does not just delay publication. It delays regulatory submission and, ultimately, patient access to treatment.
Dr. Sophie Hartmann, Head of Translational Biomarker Research at a rare disease foundation, deploys a tracebloc workspace loaded with 320 anonymised patient records spanning three rare genetic diseases. Biomarker research teams — internal and external — submit their candidate panel models to the workspace. Inside tracebloc's containerised training environment, each model trains on the 256-record patient cohort — fine-tuning to the specific multi-omic feature distributions, disease subtypes, and clinical covariate patterns present in this dataset — without any patient data leaving Sophie's infrastructure. tracebloc handles orchestration, scores each adapted model against the holdout cohort, and publishes results to a live leaderboard automatically. This is a federated learning application of reproducible benchmarking: contributors test their panels on real patient data, and the data never moves.
In this example evaluation, the best-performing model held up across all three disease subgroups after fine-tuning — but the gap between the top contributor and the second-best was larger on the Spinal Muscular Atrophy cohort than on Cystic Fibrosis, a finding invisible in any single-institution benchmark. The leaderboard captures those subgroup differences persistently. The workspace stays active so that as new candidate panels are developed, they face the same holdout cohort and the same evaluation conditions.
Sophie's team has spent three years assembling a rare disease cohort — 320 patients across Cystic Fibrosis, Duchenne Muscular Dystrophy, and Spinal Muscular Atrophy, each with longitudinal clinical assessments, CFTR mutation profiling, and a suite of functional measures including six-minute walk distance, FEV1 percent predicted, motor function score, and sweat chloride levels. The dataset took two ethics committee cycles, four clinical site agreements, and significant effort to harmonise across measurement protocols. It is, for the field, genuinely valuable.
The problem is that every biomarker panel her team has evaluated — whether developed internally or submitted by academic collaborators — has been trained and tested on the same patient records it was developed on. When those panels are published, other groups attempt to replicate them. They often cannot. Not because the methods are wrong, but because the panels were never tested on an independent cohort before publication.
This is not a niche critique. The biomarker replication crisis is well documented. In rare disease specifically, where each disease subtype may have fewer than 100 patients available globally, overfitting to the available cohort is almost inevitable without systematic independent validation. A six-minute walk distance that predicts progression in Sophie's Cystic Fibrosis patients may perform differently in another centre's cohort where patient age distribution or treatment history differs. A CFTR mutation combination that separates disease classes in one country's registry may not separate them in another's.
The structural problem Sophie faces: the specialists who could validate her panels are at academic centres across Europe. They have their own patient cohorts, their own biomarker expertise, and every incentive to collaborate. But they cannot access her patient records. Ethics approval does not transfer across institutions. GDPR does not permit unilateral data sharing. And even setting aside the legal constraints, centralising sensitive rare disease patient data into a shared research database is a governance commitment that takes months to establish and creates ongoing risk.
She needs a mechanism that lets external research groups test their models on her cohort — calibrating to her specific disease population — without those researchers ever seeing a single patient record.
The evaluation dataset contains 320 anonymised rare disease patient records spanning three disease groups. Full dataset statistics, feature distributions, and class-level analysis are available in the Exploratory Data Analysis tab.
This dataset is augmented. It was constructed to reflect the statistical structure of real-world rare disease cohorts — the disease class distribution, CFTR mutation prevalence patterns, clinical measurement distributions, and feature correlation structure — without containing any identifiable patient information.
| Property | Value |
|---|---|
| Total records | 320 |
| Training cohort | 256 records |
| Holdout cohort | 64 records |
| Features | 254 (250 numerical, 4 categorical) |
| Disease classes | 3 |
| Missing values | None |
| Duplicate records | None |
| Class imbalance ratio | 2.5× |
| Evaluation metric | Multi-class F1 score |
Disease class distribution (full dataset):
| Disease | Patients | Share |
|---|---|---|
| Cystic Fibrosis | 165 | 51.6% |
| Duchenne Muscular Dystrophy | 89 | 27.8% |
| Spinal Muscular Atrophy | 66 | 20.6% |
A note on the features: The 254 features span three categories. Mutation markers (CFTR_c_1 through CFTR_c_100+) are binary variables encoding specific CFTR gene variants — present or absent in each patient. Clinical measures include six-minute walk distance (metres), FEV1 percent predicted, motor function score, sweat chloride concentration (mmol/L), steroid use, and cardiac ejection fraction. SMN1 variant markers are included for the SMA subgroup. No features require imputation — the dataset is complete. The class imbalance (Spinal Muscular Atrophy at 20.6% versus Cystic Fibrosis at 51.6%) is preserved as observed in real cohort distributions. A model that classifies every patient as Cystic Fibrosis achieves 51.6% accuracy — which is why per-class recall and macro-F1 are the metrics that matter here.
Each contributor submitted their candidate biomarker panel model to the tracebloc workspace. The evaluation ran in two phases.
Phase 1 — Out-of-the-box performance. Each model was scored as submitted, with no adaptation to Sophie's patient cohort. This establishes the true baseline: what the published or internally-validated panel actually delivers when applied to an independent patient population without retraining.
Phase 2 — Fine-tuning. Contributors were given access to the training environment inside the tracebloc workspace. Each contributor transferred their model into tracebloc and ran training on the 256-record cohort. This process fine-tuned the model to the specific covariate distributions, mutation prevalence patterns, and disease subtype characteristics in Sophie's patient population — adapting from a generalised biomarker classifier to one calibrated for this specific rare disease cohort. After training, the adapted model was evaluated automatically against the 64-record holdout. Contributors received only their own results back; no contributor had visibility into another's performance before the leaderboard published.
→ View the full model leaderboard — complete contributor rankings, per-class recall, and biomarker feature importance across all submitted panels.
| Contributor | Claimed F1 | Out-of-the-Box | After Fine-tuning | SMA Recall |
|---|---|---|---|---|
| Contributor A | 0.84 | 0.71 | 0.79 | 0.61 |
| Contributor B ✅ | 0.81 | 0.74 | 0.86 | 0.78 |
| Contributor C ⚠️ | 0.88 | 0.77 | 0.83 | 0.52 |
What the numbers reveal:
Contributor B achieved the strongest macro-F1 after fine-tuning at 0.86, and — more critically — held the highest recall on the Spinal Muscular Atrophy subgroup at 0.78. Starting at 0.74 out-of-the-box, the model improved meaningfully after training on 256 real-distribution patient records inside the tracebloc workspace, with its SMA recall rising from 0.63 to 0.78. This is the replication result: not just that the panel works on a new cohort, but that it can be calibrated to perform robustly on the minority subgroup where clinical decisions are hardest.
Contributor C had the highest claimed F1 at 0.88 and the strongest out-of-the-box baseline at 0.77. After fine-tuning it reached 0.83 — a meaningful result, but with SMA recall at 0.52, it misclassifies roughly one in two SMA patients. In a disease where the patient population is small and each misclassification delays or misdirects clinical decision-making, a 52% recall on the rarest subgroup is not a publishable panel.
Contributor A's claimed F1 of 0.84 dropped to 0.71 on the independent cohort before fine-tuning — the largest out-of-the-box degradation in the evaluation. After adaptation it recovered to 0.79, still below both other contributors. The claimed performance was measured on a training distribution that does not match Sophie's patient population.
Illustrative assumptions: Rare disease pipeline with 3 candidate biomarker panels / Phase II endpoint selection decision worth $80M+ in downstream trial investment / Cost of failed replication at regulatory review: 12–18 months delay, estimated €3–6M in additional study costs per episode
| Strategy | SMA Recall | Replication Risk | Regulatory Readiness | Est. Cost of Late Failure |
|---|---|---|---|---|
| Unvalidated (internal only) | Unknown | High | Low — single-cohort evidence | €3–6M per episode |
| Contributor A | 0.61 | Moderate | Partial — SMA subgroup underperforms | €1.5–3M exposure |
| Contributor B ✅ | 0.78 | Low | Strong — reproducible across subtypes | Minimised |
| Contributor C | 0.52 | Moderate-High | Partial — SMA recall insufficient | €2–4M exposure |
The primary value of this evaluation is not the cost saved on one decision — it is the ability to generate reproducible, multi-cohort biomarker evidence without a data sharing agreement, without centralising patient records, and without the 12–18 month governance cycle those agreements require. Contributor B delivers that evidence. The alternative is publishing a panel that has never been independently validated, and discovering at EMA interaction that the evidence base is insufficient.
Sophie selects Contributor B's panel for advancement, with the validation run against her cohort constituting the first independent replication evidence for the panel. The feature importance output from the fine-tuning run identifies which CFTR variant combinations and clinical measures — particularly sweat chloride and motor function score — are driving performance in each disease subgroup, giving her team a mechanistic basis for the panel's design that supports regulatory submission narratives.
The tracebloc workspace stays active after the initial evaluation. New candidate panels — whether developed by Sophie's team or submitted by academic collaborators — face the same holdout cohort under identical evaluation conditions. The leaderboard becomes a persistent record of which biomarker approaches replicate and which do not, turning a one-off validation exercise into ongoing reproducibility infrastructure.
Explore this use case further:
Related use cases: See how the same federated evaluation approach applies to prognostic transcriptomics in neuromuscular disease and combination multi-omics therapy response prediction. For proteomics-based biomarker discovery, see the liquid biopsy rare disease use case. For a broader view of what federated learning applications look like across life sciences, see our guide.
Deploy your workspace or schedule a call.
Disclaimer: The dataset used in this use case is augmented — constructed to reflect the statistical structure of real-world rare disease cohorts, including disease class distribution, CFTR mutation prevalence, clinical measurement distributions, and feature correlation patterns, without containing any identifiable patient information. The persona, contributor names, claimed performance figures, and scenario are illustrative and based on patterns observed across rare disease biomarker research environments. They do not represent any specific institution, research group, or regulatory submission.
