Evaluation utilities

The evaluation scripts are kept as repository modules rather than packaged estimators. They are intended for experiments, notebooks, and benchmark runs.

noise_cv_eval.py

The 5-fold evaluation module provides helpers for baseline runs, filtered runs, full experiments, and grid experiments:

• run_5cv_baseline
• run_5cv_filters
• run_5cv_experiment
• run_5cv_grid

The classification output includes fold metadata, preprocessing metadata, timing, validity flags, and common metrics such as accuracy, balanced accuracy, macro-F1, macro-precision, and macro-recall.

The removal output includes the predicted removals, true noisy rows when known, exact/partial ground-truth metadata, and confusion-matrix-derived removal metrics.

noise_score_eval.py

Evaluate a continuous noise score as a ranking of suspicious samples.

ROC-AUC and Average Precision evaluate the full ranking across all possible thresholds. Precision@k / Recall@k / F1@k check whether the k most suspicious instances match the k truly corrupted ones, where k is the number of actual noisy samples.

Evaluate every method in noise_scores with evaluate_noise_score_ranking.

Evaluate how useful a score is as a progressive filtering criterion.

For each removal percentage, the top-scoring samples are removed, the estimator is refit, and the metric is computed.

If cv is provided, the procedure is repeated fold-by-fold using StratifiedKFold. The filtering is always computed only on the training fold to avoid test leakage.

When cv is None, the function fits on the full input and leaves the metric evaluation flexible: metric_fn may be a regular sklearn metric, a scorer, or a closure that only needs the fitted estimator.

Evaluate the filtering curve for every method in noise_scores.

Summarize each method's filtering curve.

The AUC is computed over the normalized removal fraction, so the score is comparable whether the curve was expressed in [0, 1] or [0, 100].