AI-agent skill · Claude Code · Codex · Cursor · Gemini CLI
ML Pipeline Auditor reviews machine-learning code and Jupyter notebooks like a senior data scientist. It finds data leakage, test-set misuse, wrong metrics, irreproducible runs and PyTorch training bugs, then writes a scored report (0–100) with the exact fix for each problem. Local and private: nothing is uploaded, your training is never run.
A typical Titanic notebook vs the same model after the auditor's fixes, on the same 20 random splits of the real data.
± 0.005 · audit score 0/100 · 6 kinds of problem: target encoding and SMOTE before the split, scaler and fill values from test rows, depth tuned on the test set, AUC from labels
± 0.027 · ROC-AUC 0.871 · audit score 100/100 · one Pipeline, tuning by cross-validation, test set used once
Start of the raw report for the leaky notebook (the agent then confirms each finding and writes the full report):
## 1. [HIGH] MLA106 Target-based feature (target/mean encoding) computed on the full dataset
- **Where:** `leaky_titanic.py:22` (related lines [32])
- **Found:** Target encoding (mean of target 'survived' per group) computed before train_test_split (line 32).
- **Code:** `df[col + "_rate"] = df.groupby(col)["survived"].transform("mean")`
- **Why it matters:** Each row's feature contains its own label, so the model reads the answer.
- **Fix:** Compute encodings inside CV on training folds only (sklearn TargetEncoder in a Pipeline uses cross-fitting).
- **Confidence:** high
## 2. [HIGH] MLA104 Resampling (SMOTE/over/under-sampling) before the split or outside cross-validation
- **Where:** `leaky_titanic.py:31` (related lines [32])
- **Found:** SMOTE applied to 'X' (all rows) before train_test_split (line 32): resampled rows appear in both train and test.
- **Code:** `X, y = SMOTE(random_state=SEED).fit_resample(X, y)`
- **Why it matters:** Synthetic or duplicated rows end up in both train and test folds, which inflates recall/F1/AUC.
- **Fix:** Resample only the training part, e.g. imblearn.pipeline.Pipeline([('smote', SMOTE()), ('model', clf)]) inside CV.
- **Confidence:** high
## 3. [HIGH] MLA201 Test set used to choose hyperparameters or models
- **Where:** `leaky_titanic.py:37`
- **Found:** A model/parameter is chosen inside this loop by its score on test data.
- **Code:** `f1 = f1_score(y_test, model.predict(X_test))`
- **Why it matters:** Picking the best setting by its test score makes the test score optimistic; it is no longer an unbiased estimate.
- **Fix:** Tune with cross-validation or a separate validation set; touch the test set once, at the end.
- **Confidence:** high
| Project | Set | Files | Findings checked by hand | Verdict |
|---|---|---|---|---|
| scikit-learn official examples | development | 282 | 6 | 3 correct, 3 intentional |
| Hands-On ML (3rd ed.) notebooks | development | 28 | 1 | 1 correct |
| Microsoft ML-For-Beginners | hold-out | 69 | 9 | 6 correct, 3 debatable |
| Python Data Science Handbook | hold-out | 68 | 7 | 6 correct, 1 false alarm |
Real problems it found in famous code:
plot_digits_denoising.py · MLA102: MinMaxScaler fitted on all digits before train_test_split (small, unsupervised leak).plot_digits_kde_sampling.py · MLA102: PCA fitted on all digits, then GridSearchCV.plot_rbf_parameters.py · MLA102: Scaler fitted on all data before GridSearchCV; the authors' own comment says 'We are cheating a bit in this example in scaling all of the data'.17_autoencoders_gans_and_diffusion_models.ipynb · MLA202: Denoising autoencoder uses validation_data=(X_test, X_test) while the rest of the book uses X_valid.05.08-Random-Forests.ipynb · MLA302: metrics.classification_report(ypred, ytest): arguments swapped, so precision and recall per class are exchanged.Hold-out projects were run once with the analyser frozen. Its one false alarm (a K-Means example) was fixed afterwards in v1.0.1. I wrote the labelled cases myself, so the real-world table is the independent check.
Feature selection, target encoding, SMOTE, scaling or imputation fitted before the split or outside CV · target left in X · random split of time series · same entity on both sides
Test set used for tuning or early stopping · model trained on test data · training-only evaluation · threshold tuned on test · no stratification
AUC / log-loss from hard labels · swapped metric(y_pred, y_true) · accuracy as the only metric
Missing seeds · notebook cells run out of order or skipped · unpinned dependencies
Missing optimizer.zero_grad() · missing model.eval() · softmax/sigmoid before a logits loss
Duplicate and near-duplicate rows across the split · shared IDs or groups · test rows older than training rows · single features that predict the target · drift
Post-outcome features, preprocessing in SQL, wrong evaluation unit, metric vs goal, LLM/RAG evaluation contamination and LLM-as-judge bias
ML_AUDIT_REPORT.md: verdict, score, findings ordered by impact, before/after code fixes.--fail-on high blocks merges that add leakage.
SARIF 2.1.0 output shows findings on the changed lines in GitHub.
Pure Python standard library, tested on Python 3.9 to 3.14.
# mlaudit: ignore[MLA102] reason for accepted cases.
Made by Muhammad Shahmeer · Data Science MSc, University of Naples Federico II · © 2026, all rights reserved.