SCI Analysis Observatory

Evidence scope

Scope, workflow status, and summary measures

Cross Fitted CalibrationPrediction Sensitivity Analysis

Cross-fitted calibrationCross-fitted probability calibrationComplete

Prediction sensitivityPrediction sensitivity analysesComplete

Selected XGBoost evaluation calibrator: logistic
Selected LightGBM evaluation calibrator: logistic
Selected CatBoost evaluation calibrator: none
Calibration selection rule: Select a method only when the animal-clustered 95% CI for mean cumulative Brier improvement excludes zero and pooled log loss does not worsen.
Rows per framework: 396 rows
Sensitivity scenarios: 2 scenarios
Sensitivity estimands: 3 estimands

Linked filters

Selections combine with AND across dimensions.

FrameworkScenarioMetric

No shared filters active.

Aggregate estimate plot

Calibration log loss

Cross Fitted Calibration

catboost · isotonic

1.730

catboost · logistic

1.030

catboost

0.968

lgbm · isotonic

1.528

lgbm · logistic

0.793

lgbm

0.816

xgb · isotonic

1.448

xgb · logistic

0.815

xgb

0.844

Uncertainty: Point estimates are shown because this table does not supply matching lower and upper interval columns.

View plotted values in tabular form

Calibration log loss: tabulated plot data
Estimate	Marker	Log Loss
catboost · isotonic	▲	1.730
catboost · logistic	✦	1.030
catboost	⬟	0.968
lgbm · isotonic	■	1.528
lgbm · logistic	◆	0.793
lgbm	✦	0.816
xgb · isotonic	●	1.448
xgb · logistic	▲	0.815
xgb	◆	0.844

Log Loss from Pooled calibration metrics. Shared filters apply to both this plot and its source table.

Aggregate estimate plot

Sensitivity confidence intervals

Prediction Sensitivity Analysis

xgb · uncalibrated

0.852

xgb · uncalibrated

0.817

xgb · uncalibrated

0.835

xgb · uncalibrated

0.621

xgb · uncalibrated

0.580

xgb · uncalibrated

0.844

xgb · uncalibrated

0.467

xgb · uncalibrated

0.154

xgb · uncalibrated

0.852

xgb · uncalibrated

0.817

xgb · uncalibrated

0.835

xgb · uncalibrated

0.621

xgb · uncalibrated

0.580

xgb · uncalibrated

0.844

xgb · uncalibrated

0.467

xgb · uncalibrated

0.154

lgbm · uncalibrated

0.859

lgbm · uncalibrated

0.830

lgbm · uncalibrated

0.845

lgbm · uncalibrated

0.636

lgbm · uncalibrated

0.596

lgbm · uncalibrated

0.816

lgbm · uncalibrated

0.457

lgbm · uncalibrated

0.149

Uncertainty: Horizontal intervals use Ci Lower and Ci Upper from the published aggregate table.

View plotted values in tabular form

Sensitivity confidence intervals: tabulated plot data
Estimate	Marker	Estimate	Lower interval	Upper interval
xgb · uncalibrated	●	0.852	0.802	0.898
xgb · uncalibrated	●	0.817	0.782	0.854
xgb · uncalibrated	●	0.835	0.801	0.869
xgb · uncalibrated	●	0.621	0.570	0.674
xgb · uncalibrated	●	0.580	0.524	0.637
xgb · uncalibrated	●	0.844	0.778	0.908
xgb · uncalibrated	●	0.467	0.389	0.545
xgb · uncalibrated	●	0.154	0.139	0.170
xgb · uncalibrated	●	0.852	0.794	0.900
xgb · uncalibrated	●	0.817	0.776	0.857
xgb · uncalibrated	●	0.835	0.795	0.869
xgb · uncalibrated	●	0.621	0.563	0.682
xgb · uncalibrated	●	0.580	0.520	0.638
xgb · uncalibrated	●	0.844	0.766	0.915
xgb · uncalibrated	●	0.467	0.383	0.553
xgb · uncalibrated	●	0.154	0.137	0.172
lgbm · uncalibrated	■	0.859	0.810	0.902
lgbm · uncalibrated	■	0.830	0.793	0.869
lgbm · uncalibrated	■	0.845	0.811	0.878
lgbm · uncalibrated	■	0.636	0.586	0.687
lgbm · uncalibrated	■	0.596	0.541	0.649
lgbm · uncalibrated	■	0.816	0.755	0.875
lgbm · uncalibrated	■	0.457	0.387	0.529
lgbm · uncalibrated	■	0.149	0.134	0.163

Estimate from Sensitivity confidence intervals. Shared filters apply to both this plot and its source table.

Cross Fitted Calibration

Pooled calibration metrics

Summary method: approved_rows

Indexed rows: 9
Eligible rows: 9

Showing 1–9 of 9 matching rows

Filters combine with AND. Empty and unreported values sort last.

Pooled calibration metrics. cross_fitted_calibration.

catboost	isotonic	396	115	0.869	0.845	0.857	0.652
catboost	logistic	396	115	0.873	0.848	0.861	0.634
catboost	—	396	115	0.871	0.852	0.861	0.644
lgbm	isotonic	396	115	0.858	0.837	0.848	0.669
lgbm	logistic	396	115	0.864	0.850	0.857	0.662
lgbm	—	396	115	0.859	0.830	0.845	0.636
xgb	isotonic	396	115	0.851	0.837	0.844	0.634
xgb	logistic	396	115	0.861	0.845	0.853	0.641
xgb	—	396	115	0.852	0.817	0.835	0.621

Rows per pagePage 1 of 1

Cross Fitted Calibration

Paired calibration-method comparisons

Summary method: approved_rows

Indexed rows: 12
Eligible rows: 12

Showing 1–12 of 12 matching rows

Filters combine with AND. Empty and unreported values sort last.

Paired calibration-method comparisons. cross_fitted_calibration.

catboost	logistic	log_loss	uncalibrated_minus_calibrated	-0.061	-0.324	0.187	0.378
catboost	logistic	mean_cumulative_brier	uncalibrated_minus_calibrated	0.001	-0.002	0.004	0.812
catboost	isotonic	log_loss	uncalibrated_minus_calibrated	-0.762	-1.325	-0.265	0.002
catboost	isotonic	mean_cumulative_brier	uncalibrated_minus_calibrated	-2.26e-4	-0.005	0.005	0.463
lgbm	logistic	log_loss	uncalibrated_minus_calibrated	0.023	-0.006	0.052	0.935
lgbm	logistic	mean_cumulative_brier	uncalibrated_minus_calibrated	0.007	6.52e-4	0.013	0.985
lgbm	isotonic	log_loss	uncalibrated_minus_calibrated	-0.713	-1.356	-0.213	6.00e-4
lgbm	isotonic	mean_cumulative_brier	uncalibrated_minus_calibrated	0.003	-0.004	0.010	0.769
xgb	logistic	log_loss	uncalibrated_minus_calibrated	0.030	-0.002	0.063	0.970
xgb	logistic	mean_cumulative_brier	uncalibrated_minus_calibrated	0.009	0.003	0.017	0.999
xgb	isotonic	log_loss	uncalibrated_minus_calibrated	-0.604	-1.060	-0.199	2.00e-4
xgb	isotonic	mean_cumulative_brier	uncalibrated_minus_calibrated	0.007	-1.59e-4	0.014	0.971

Rows per pagePage 1 of 1

Prediction Sensitivity Analysis

Sensitivity estimands

Summary method: approved_rows

Indexed rows: 18
Eligible rows: 18

Showing 1–18 of 18 matching rows

Filters combine with AND. Empty and unreported values sort last.

Sensitivity estimands. prediction_sensitivity_analysis.

selected_calibration	xgb	row_weighted_pooled	0.861	0.845	0.853	0.641	0.609
selected_calibration	xgb	equal_animal_weighted	0.847	0.869	0.858	0.730	0.623
selected_calibration	xgb	equal_outer_fold_weighted	0.865	0.854	0.860	0.641	0.609
selected_calibration	lgbm	row_weighted_pooled	0.864	0.850	0.857	0.662	0.628
selected_calibration	lgbm	equal_animal_weighted	0.847	0.871	0.859	0.757	0.644
selected_calibration	lgbm	equal_outer_fold_weighted	0.869	0.861	0.865	0.662	0.628
selected_calibration	catboost	row_weighted_pooled	0.871	0.852	0.861	0.644	0.606
selected_calibration	catboost	equal_animal_weighted	0.851	0.871	0.861	0.740	0.612
selected_calibration	catboost	equal_outer_fold_weighted	0.876	0.860	0.868	0.644	0.605
uncalibrated	xgb	row_weighted_pooled	0.852	0.817	0.835	0.621	0.580
uncalibrated	xgb	equal_animal_weighted	0.834	0.837	0.835	0.719	0.585
uncalibrated	xgb	equal_outer_fold_weighted	0.865	0.854	0.860	0.621	0.581
uncalibrated	lgbm	row_weighted_pooled	0.859	0.830	0.845	0.636	0.596
uncalibrated	lgbm	equal_animal_weighted	0.841	0.843	0.842	0.734	0.597
uncalibrated	lgbm	equal_outer_fold_weighted	0.869	0.861	0.865	0.637	0.597
uncalibrated	catboost	row_weighted_pooled	0.871	0.852	0.861	0.644	0.606
uncalibrated	catboost	equal_animal_weighted	0.851	0.871	0.861	0.740	0.612
uncalibrated	catboost	equal_outer_fold_weighted	0.876	0.860	0.868	0.644	0.605

Rows per pagePage 1 of 1

Prediction Sensitivity Analysis

Sensitivity confidence intervals

Summary method: approved_rows

Indexed rows: 96
Eligible rows: 96

Showing 1–25 of 96 matching rows

Filters combine with AND. Empty and unreported values sort last.

Sensitivity confidence intervals. prediction_sensitivity_analysis.

selected_calibration	xgb	early_auc	0.861	0.817	0.901	0.021	5,000
selected_calibration	xgb	late_auc	0.845	0.811	0.879	0.017	5,000
selected_calibration	xgb	mean_auc	0.853	0.822	0.882	0.015	5,000
selected_calibration	xgb	accuracy	0.641	0.591	0.692	0.026	5,000
selected_calibration	xgb	balanced_accuracy	0.609	0.555	0.664	0.028	5,000
selected_calibration	xgb	log_loss	0.815	0.739	0.891	0.039	5,000
selected_calibration	xgb	ordinal_mae	0.437	0.368	0.508	0.036	5,000
selected_calibration	xgb	mean_cumulative_brier	0.145	0.129	0.161	0.008	5,000
selected_calibration	xgb	early_auc	0.861	0.809	0.903	0.025	5,000
selected_calibration	xgb	late_auc	0.845	0.807	0.882	0.019	5,000
selected_calibration	xgb	mean_auc	0.853	0.817	0.883	0.017	5,000
selected_calibration	xgb	accuracy	0.641	0.585	0.700	0.029	5,000
selected_calibration	xgb	balanced_accuracy	0.609	0.548	0.668	0.031	5,000
selected_calibration	xgb	log_loss	0.815	0.727	0.897	0.044	5,000
selected_calibration	xgb	ordinal_mae	0.437	0.359	0.514	0.040	5,000
selected_calibration	xgb	mean_cumulative_brier	0.145	0.127	0.162	0.009	5,000
selected_calibration	lgbm	early_auc	0.864	0.817	0.906	0.022	5,000
selected_calibration	lgbm	late_auc	0.850	0.820	0.881	0.016	5,000
selected_calibration	lgbm	mean_auc	0.857	0.827	0.886	0.015	5,000
selected_calibration	lgbm	accuracy	0.662	0.613	0.710	0.025	5,000
selected_calibration	lgbm	balanced_accuracy	0.628	0.575	0.680	0.027	5,000
selected_calibration	lgbm	log_loss	0.793	0.720	0.869	0.038	5,000
selected_calibration	lgbm	ordinal_mae	0.409	0.341	0.480	0.035	5,000
selected_calibration	lgbm	mean_cumulative_brier	0.142	0.126	0.159	0.008	5,000
selected_calibration	lgbm	early_auc	0.864	0.809	0.908	0.025	5,000

Rows per pagePage 1 of 4

Prediction Sensitivity Analysis

Uncalibrated row-weighted pooled framework ranking

Summary method: approved_rows

Indexed rows: 3
Eligible rows: 3

Showing 1–3 of 3 matching rows

Filters combine with AND. Empty and unreported values sort last.

Uncalibrated row-weighted pooled framework ranking. prediction_sensitivity_analysis.

1	catboost	0.861	0.606	0.407
2	lgbm	0.845	0.596	0.457
3	xgb	0.835	0.580	0.467

Rows per pagePage 1 of 1

Calibration and sensitivity

Scope, workflow status, and summary measures

Apply filters to tables and figures

Calibration log loss

Sensitivity confidence intervals

Pooled calibration metrics

Paired calibration-method comparisons

Sensitivity estimands

Sensitivity confidence intervals

Uncalibrated row-weighted pooled framework ranking