AI-agent skill · Claude Code · Codex · Cursor · Gemini CLI

Is your model's accuracy real?

ML Pipeline Auditor reviews machine-learning code and Jupyter notebooks like a senior data scientist. It finds data leakage, test-set misuse, wrong metrics, irreproducible runs and PyTorch training bugs, then writes a scored report (0–100) with the exact fix for each problem. Local and private: nothing is uploaded, your training is never run.

Get it on Agensi · $12 ↗See the demoTested on real codeGitHub ↗

A 0.99 that was not real

A typical Titanic notebook vs the same model after the auditor's fixes, on the same 20 random splits of the real data.

Leaky notebook reportsF1 0.991

± 0.005 · audit score 0/100 · 6 kinds of problem: target encoding and SMOTE before the split, scaler and fill values from test rows, depth tuned on the test set, AUC from labels

After the fixes (honest)F1 0.749

± 0.027 · ROC-AUC 0.871 · audit score 100/100 · one Pipeline, tuning by cross-validation, test set used once

Start of the raw report for the leaky notebook (the agent then confirms each finding and writes the full report):

## 1. [HIGH] MLA106 Target-based feature (target/mean encoding) computed on the full dataset

- **Where:** `leaky_titanic.py:22` (related lines [32])
- **Found:** Target encoding (mean of target 'survived' per group) computed before train_test_split (line 32).
- **Code:** `df[col + "_rate"] = df.groupby(col)["survived"].transform("mean")`
- **Why it matters:** Each row's feature contains its own label, so the model reads the answer.
- **Fix:** Compute encodings inside CV on training folds only (sklearn TargetEncoder in a Pipeline uses cross-fitting).
- **Confidence:** high

## 2. [HIGH] MLA104 Resampling (SMOTE/over/under-sampling) before the split or outside cross-validation

- **Where:** `leaky_titanic.py:31` (related lines [32])
- **Found:** SMOTE applied to 'X' (all rows) before train_test_split (line 32): resampled rows appear in both train and test.
- **Code:** `X, y = SMOTE(random_state=SEED).fit_resample(X, y)`
- **Why it matters:** Synthetic or duplicated rows end up in both train and test folds, which inflates recall/F1/AUC.
- **Fix:** Resample only the training part, e.g. imblearn.pipeline.Pipeline([('smote', SMOTE()), ('model', clf)]) inside CV.
- **Confidence:** high

## 3. [HIGH] MLA201 Test set used to choose hyperparameters or models

- **Where:** `leaky_titanic.py:37`
- **Found:** A model/parameter is chosen inside this loop by its score on test data.
- **Code:** `f1 = f1_score(y_test, model.predict(X_test))`
- **Why it matters:** Picking the best setting by its test score makes the test score optimistic; it is no longer an unbiased estimate.
- **Fix:** Tune with cross-validation or a separate validation set; touch the test set once, at the end.
- **Confidence:** high

Tested on real code

447files from 4 well-known open-source ML projects
15 / 16hold-out findings correct or debatable (code never seen before)
31 / 31planted problems caught in 44 labelled pipelines
0 / 15clean pipelines wrongly flagged
ProjectSetFilesFindings checked by handVerdict
scikit-learn official examplesdevelopment28263 correct, 3 intentional
Hands-On ML (3rd ed.) notebooksdevelopment2811 correct
Microsoft ML-For-Beginnershold-out6996 correct, 3 debatable
Python Data Science Handbookhold-out6876 correct, 1 false alarm

Real problems it found in famous code:

Hold-out projects were run once with the analyser frozen. Its one false alarm (a K-Means example) was fixed afterwards in v1.0.1. I wrote the labelled cases myself, so the real-world table is the independent check.

What it checks

Leakage

Feature selection, target encoding, SMOTE, scaling or imputation fitted before the split or outside CV · target left in X · random split of time series · same entity on both sides

Validation

Test set used for tuning or early stopping · model trained on test data · training-only evaluation · threshold tuned on test · no stratification

Metrics

AUC / log-loss from hard labels · swapped metric(y_pred, y_true) · accuracy as the only metric

Reproducibility

Missing seeds · notebook cells run out of order or skipped · unpinned dependencies

Deep learning

Missing optimizer.zero_grad() · missing model.eval() · softmax/sigmoid before a logits loss

Your train/test data

Duplicate and near-duplicate rows across the split · shared IDs or groups · test rows older than training rows · single features that predict the target · drift

Manual checklist

Post-outcome features, preprocessing in SQL, wrong evaluation unit, metric vs goal, LLM/RAG evaluation contamination and LLM-as-judge bias

How your agent uses it

  1. Runs the analyser over your notebooks and scripts (26 rules, cell-aware).
  2. Opens every finding and decides: confirmed, intended or false alarm. No blind copying.
  3. Works through the manual checklist for what scanning cannot see.
  4. Checks your actual train/test files when you have them.
  5. Writes ML_AUDIT_REPORT.md: verdict, score, findings ordered by impact, before/after code fixes.
  6. With your OK, applies the fixes and re-scores.

For teams

Pull-request gate

--fail-on high blocks merges that add leakage.

Code-scanning alerts

SARIF 2.1.0 output shows findings on the changed lines in GitHub.

No install

Pure Python standard library, tested on Python 3.9 to 3.14.

Exceptions

# mlaudit: ignore[MLA102] reason for accepted cases.

Get it on Agensi · $12 ↗GitHub ↗

Made by Muhammad Shahmeer · Data Science MSc, University of Naples Federico II · © 2026, all rights reserved.