SCI Analysis Observatory

Evidence scope

Scope, workflow status, and summary measures

Full Feature Grouped Nested CvFull Feature Nested Oof Statistical Analysis

Full-feature nested CVAnimal-grouped nested modelingComplete

Statistical evaluationAnalyze nested-CV resultsComplete

Validation animals: 115 animals
Validation experiments: 5 experiments
Outer folds: 5 folds
Full-panel features: 26 features
Unique out-of-fold rows: 396 rows

Linked filters

Selections combine with AND across dimensions.

FrameworkExperimentMetric

No shared filters active.

Aggregate estimate plot

Nested-CV AUC summary

Full Feature Grouped Nested Cv

Measure

catboost

0.876

lgbm

0.869

xgb

0.866

Uncertainty: Point estimates are shown because this table does not supply matching lower and upper interval columns.

View plotted values in tabular form

Nested-CV AUC summary: tabulated plot data
Estimate	Marker	Early Auc Mean
catboost	⬟	0.876
lgbm	✦	0.869
xgb	◆	0.866

Early Auc Mean from Nested-CV summary. Shared filters apply to both this plot and its source table.

Aggregate estimate plot

Animal-clustered metric intervals

Full Feature Nested Oof Statistical Analysis

catboost · early_auc

0.871

catboost · late_auc

0.852

catboost · mean_auc

0.861

catboost · accuracy

0.644

catboost · balanced_accuracy

0.606

catboost · log_loss

0.968

catboost · ordinal_mae

0.407

catboost · brier_gt_early

0.143

catboost · brier_gt_late

0.138

catboost · mean_cumulative_brier

0.141

lgbm · early_auc

0.859

lgbm · late_auc

0.830

lgbm · mean_auc

0.845

lgbm · accuracy

0.636

lgbm · balanced_accuracy

0.596

lgbm · log_loss

0.816

lgbm · ordinal_mae

0.457

lgbm · brier_gt_early

0.156

lgbm · brier_gt_late

0.142

lgbm · mean_cumulative_brier

0.149

xgb · early_auc

0.852

xgb · late_auc

0.817

xgb · mean_auc

0.835

xgb · accuracy

0.621

Uncertainty: Horizontal intervals use Ci Lower and Ci Upper from the published aggregate table.

View plotted values in tabular form

Animal-clustered metric intervals: tabulated plot data
Estimate	Marker	Estimate	Lower interval	Upper interval
catboost · early_auc	■	0.871	0.831	0.908
catboost · late_auc	▲	0.852	0.818	0.887
catboost · mean_auc	◆	0.861	0.832	0.889
catboost · accuracy	◆	0.644	0.594	0.696
catboost · balanced_accuracy	▲	0.606	0.550	0.661
catboost · log_loss	▲	0.968	0.746	1.247
catboost · ordinal_mae	■	0.407	0.344	0.468
catboost · brier_gt_early	●	0.143	0.120	0.167
catboost · brier_gt_late	■	0.138	0.117	0.159
catboost · mean_cumulative_brier	●	0.141	0.124	0.157
lgbm · early_auc	●	0.859	0.812	0.903
lgbm · late_auc	■	0.830	0.793	0.867
lgbm · mean_auc	▲	0.845	0.811	0.878
lgbm · accuracy	▲	0.636	0.585	0.687
lgbm · balanced_accuracy	■	0.596	0.540	0.651
lgbm · log_loss	■	0.816	0.757	0.875
lgbm · ordinal_mae	●	0.457	0.388	0.532
lgbm · brier_gt_early	⬟	0.156	0.135	0.176
lgbm · brier_gt_late	●	0.142	0.126	0.157
lgbm · mean_cumulative_brier	⬟	0.149	0.135	0.163
xgb · early_auc	⬟	0.852	0.802	0.898
xgb · late_auc	●	0.817	0.782	0.854
xgb · mean_auc	■	0.835	0.801	0.869
xgb · accuracy	■	0.621	0.570	0.674

Estimate from Animal-clustered confidence intervals. Shared filters apply to both this plot and its source table.

Full Feature Grouped Nested Cv

Nested-CV summary

Summary method: approved_rows

Indexed rows: 3
Eligible rows: 3

Showing 1–3 of 3 matching rows

Filters combine with AND. Empty and unreported values sort last.

Nested-CV summary. full_feature_grouped_nested_cv.

catboost	5	0.876	0.015	0.860	0.045	0.868	0.021
lgbm	5	0.869	0.030	0.861	0.033	0.865	0.020
xgb	5	0.866	0.026	0.854	0.045	0.860	0.022

Rows per pagePage 1 of 1

Full Feature Grouped Nested Cv

Outer-fold metrics

Summary method: approved_rows

Indexed rows: 15
Eligible rows: 15

Showing 1–15 of 15 matching rows

Filters combine with AND. Empty and unreported values sort last.

Outer-fold metrics. full_feature_grouped_nested_cv.

catboost	1	317	93	79	22	0.886	0.904
catboost	2	317	92	79	23	0.853	0.875
catboost	3	318	92	78	23	0.894	0.822
catboost	4	316	92	80	23	0.870	0.896
catboost	5	316	91	80	24	0.877	0.804
lgbm	1	317	93	79	22	0.887	0.884
lgbm	2	317	92	79	23	0.904	0.852
lgbm	3	318	92	78	23	0.879	0.826
lgbm	4	316	92	80	23	0.840	0.905
lgbm	5	316	91	80	24	0.837	0.838
xgb	1	317	93	79	22	0.884	0.879
xgb	2	317	92	79	23	0.879	0.874
xgb	3	318	92	78	23	0.881	0.819
xgb	4	316	92	80	23	0.822	0.905
xgb	5	316	91	80	24	0.861	0.795

Rows per pagePage 1 of 1

Full Feature Nested Oof Statistical Analysis

Pooled out-of-fold performance

Summary method: approved_rows

Indexed rows: 3
Eligible rows: 3

Showing 1–3 of 3 matching rows

Filters combine with AND. Empty and unreported values sort last.

Pooled out-of-fold performance. full_feature_nested_oof_statistical_analysis.

catboost	396	115	0.871	0.852	0.861	0.644	0.606
lgbm	396	115	0.859	0.830	0.845	0.636	0.596
xgb	396	115	0.852	0.817	0.835	0.621	0.580

Rows per pagePage 1 of 1

Full Feature Nested Oof Statistical Analysis

Animal-clustered confidence intervals

Summary method: approved_rows

Indexed rows: 30
Eligible rows: 30

Showing 1–25 of 30 matching rows

Filters combine with AND. Empty and unreported values sort last.

Animal-clustered confidence intervals. full_feature_nested_oof_statistical_analysis.

catboost	early_auc	higher	0.871	0.831	0.908	0.950	5,000
catboost	late_auc	higher	0.852	0.818	0.887	0.950	5,000
catboost	mean_auc	higher	0.861	0.832	0.889	0.950	5,000
catboost	accuracy	higher	0.644	0.594	0.696	0.950	5,000
catboost	balanced_accuracy	higher	0.606	0.550	0.661	0.950	5,000
catboost	log_loss	lower	0.968	0.746	1.247	0.950	5,000
catboost	ordinal_mae	lower	0.407	0.344	0.468	0.950	5,000
catboost	brier_gt_early	lower	0.143	0.120	0.167	0.950	5,000
catboost	brier_gt_late	lower	0.138	0.117	0.159	0.950	5,000
catboost	mean_cumulative_brier	lower	0.141	0.124	0.157	0.950	5,000
lgbm	early_auc	higher	0.859	0.812	0.903	0.950	5,000
lgbm	late_auc	higher	0.830	0.793	0.867	0.950	5,000
lgbm	mean_auc	higher	0.845	0.811	0.878	0.950	5,000
lgbm	accuracy	higher	0.636	0.585	0.687	0.950	5,000
lgbm	balanced_accuracy	higher	0.596	0.540	0.651	0.950	5,000
lgbm	log_loss	lower	0.816	0.757	0.875	0.950	5,000
lgbm	ordinal_mae	lower	0.457	0.388	0.532	0.950	5,000
lgbm	brier_gt_early	lower	0.156	0.135	0.176	0.950	5,000
lgbm	brier_gt_late	lower	0.142	0.126	0.157	0.950	5,000
lgbm	mean_cumulative_brier	lower	0.149	0.135	0.163	0.950	5,000
xgb	early_auc	higher	0.852	0.802	0.898	0.950	5,000
xgb	late_auc	higher	0.817	0.782	0.854	0.950	5,000
xgb	mean_auc	higher	0.835	0.801	0.869	0.950	5,000
xgb	accuracy	higher	0.621	0.570	0.674	0.950	5,000
xgb	balanced_accuracy	higher	0.580	0.524	0.637	0.950	5,000

Rows per pagePage 1 of 2

Full Feature Nested Oof Statistical Analysis

Paired framework contrasts

Summary method: approved_rows

Indexed rows: 30
Eligible rows: 30

Showing 1–25 of 30 matching rows

Filters combine with AND. Empty and unreported values sort last.

Paired framework contrasts. full_feature_nested_oof_statistical_analysis.

catboost	lgbm	early_auc	higher	0.871	0.859	0.011	-0.014
catboost	lgbm	late_auc	higher	0.852	0.830	0.021	-0.002
catboost	lgbm	mean_auc	higher	0.861	0.845	0.016	-0.002
catboost	lgbm	accuracy	higher	0.644	0.636	0.008	-0.034
catboost	lgbm	balanced_accuracy	higher	0.606	0.596	0.009	-0.034
catboost	lgbm	log_loss	lower	0.968	0.816	0.153	-0.044
catboost	lgbm	ordinal_mae	lower	0.407	0.457	-0.051	-0.117
catboost	lgbm	brier_gt_early	lower	0.143	0.156	-0.012	-0.025
catboost	lgbm	brier_gt_late	lower	0.138	0.142	-0.004	-0.016
catboost	lgbm	mean_cumulative_brier	lower	0.141	0.149	-0.008	-0.017
catboost	xgb	early_auc	higher	0.871	0.852	0.019	-0.006
catboost	xgb	late_auc	higher	0.852	0.817	0.034	0.017
catboost	xgb	mean_auc	higher	0.861	0.835	0.026	0.009
catboost	xgb	accuracy	higher	0.644	0.621	0.023	-0.017
catboost	xgb	balanced_accuracy	higher	0.606	0.580	0.025	-0.019
catboost	xgb	log_loss	lower	0.968	0.844	0.124	-0.068
catboost	xgb	ordinal_mae	lower	0.407	0.467	-0.061	-0.123
catboost	xgb	brier_gt_early	lower	0.143	0.160	-0.016	-0.030
catboost	xgb	brier_gt_late	lower	0.138	0.149	-0.011	-0.024
catboost	xgb	mean_cumulative_brier	lower	0.141	0.154	-0.014	-0.025
lgbm	xgb	early_auc	higher	0.859	0.852	0.007	-0.012
lgbm	xgb	late_auc	higher	0.830	0.817	0.013	-0.011
lgbm	xgb	mean_auc	higher	0.845	0.835	0.010	-0.006
lgbm	xgb	accuracy	higher	0.636	0.621	0.015	-0.012
lgbm	xgb	balanced_accuracy	higher	0.596	0.580	0.016	-0.013

Rows per pagePage 1 of 2

Full Feature Nested Oof Statistical Analysis

Experiment-specific descriptive metrics

Summary method: approved_rows

Indexed rows: 15
Eligible rows: 15

Showing 1–15 of 15 matching rows

Filters combine with AND. Empty and unreported values sort last.

Experiment-specific descriptive metrics. full_feature_nested_oof_statistical_analysis.

catboost	BMDM	36	12	1.000	—	—	—
catboost	IL4	98	45	0.764	0.780	0.772	0.434
catboost	NK1	33	12	—	—	—	—
catboost	SHH	138	16	—	0.620	—	—
catboost	VEGF	91	30	0.665	0.628	0.647	0.384
lgbm	BMDM	36	12	0.957	—	—	—
lgbm	IL4	98	45	0.782	0.807	0.795	0.536
lgbm	NK1	33	12	—	—	—	—
lgbm	SHH	138	16	—	0.612	—	—
lgbm	VEGF	91	30	0.652	0.595	0.624	0.419
xgb	BMDM	36	12	0.895	—	—	—
xgb	IL4	98	45	0.798	0.721	0.759	0.450
xgb	NK1	33	12	—	—	—	—
xgb	SHH	138	16	—	0.555	—	—
xgb	VEGF	91	30	0.655	0.594	0.624	0.390

Rows per pagePage 1 of 1

Full Feature Nested Oof Statistical Analysis

Full 26-feature descriptive ROC curves

Summary method: framework_median_range_fpr_grid_41

Indexed rows: 82
Eligible rows: 1,206

Showing 1–25 of 82 matching rows

Filters combine with AND. Empty and unreported values sort last.

Full 26-feature descriptive ROC curves. full_feature_nested_oof_statistical_analysis.

gt_early	0.000	0.172	0.027	0.317	3
gt_early	0.025	0.525	0.398	0.597	3
gt_early	0.050	0.552	0.439	0.624	3
gt_early	0.075	0.602	0.538	0.679	3
gt_early	0.100	0.638	0.633	0.697	3
gt_early	0.125	0.683	0.647	0.719	3
gt_early	0.150	0.724	0.683	0.742	3
gt_early	0.175	0.742	0.715	0.769	3
gt_early	0.200	0.778	0.769	0.805	3
gt_early	0.225	0.796	0.787	0.828	3
gt_early	0.250	0.814	0.805	0.833	3
gt_early	0.275	0.828	0.814	0.833	3
gt_early	0.300	0.837	0.833	0.842	3
gt_early	0.325	0.842	0.837	0.860	3
gt_early	0.350	0.851	0.851	0.860	3
gt_early	0.375	0.855	0.851	0.878	3
gt_early	0.400	0.869	0.860	0.878	3
gt_early	0.425	0.878	0.869	0.878	3
gt_early	0.450	0.891	0.878	0.891	3
gt_early	0.475	0.896	0.878	0.900	3
gt_early	0.500	0.896	0.882	0.905	3
gt_early	0.525	0.900	0.887	0.919	3
gt_early	0.550	0.923	0.896	0.937	3
gt_early	0.575	0.937	0.900	0.941	3
gt_early	0.600	0.946	0.910	0.959	3

Rows per pagePage 1 of 4

Full-feature validation

Scope, workflow status, and summary measures

Apply filters to tables and figures

Nested-CV AUC summary

Animal-clustered metric intervals

Nested-CV summary

Outer-fold metrics

Pooled out-of-fold performance

Animal-clustered confidence intervals

Paired framework contrasts

Experiment-specific descriptive metrics

Full 26-feature descriptive ROC curves