Core AI/ML Concepts — Datasets, Preprocessing, and the Full Pipeline from Data to Evaluation (with Real Code)
Core ML concepts for developers: dataset types, cleaning, splitting without leakage, Pipeline + ColumnTransformer, evaluation, full LightGBM pipeline.
- Author
- Randhir Jassal
- Published
- Reading time
- 30 min read
- Views
- 5 views
Core AI/ML Concepts — Datasets, Preprocessing, and the Full Pipeline from Data to Evaluation (with Real Code)
Most "ML basics" articles teach you to call
model.fit(X, y)and pretend that''s the job. The actual job — the part where 80% of your time goes and 100% of the mistakes happen — is everything aroundfit: figuring out what your dataset really is, fixing the things broken in it, splitting it without leaking, picking the right evaluation metric, and noticing when your "great" model is actually memorising garbage.This guide is the missing manual. We cover every concept you need to read an ML codebase, write a real pipeline, and recognise when something is wrong — basic to advanced, with code you can run. By the end you understand not just what to do but why each step exists.
TL;DR
- Data > model. A mediocre model on clean, leak-free, well-split data beats a state-of-the-art model on bad data every time.
- The pipeline: raw data → clean → split → preprocess (fit on train, transform everywhere) → train → evaluate → iterate. Every step has a "wrong way" that silently inflates your score.
- Pick the right metric for the task. Accuracy is wrong for imbalanced classification. RMSE is wrong for skewed regression. ROC-AUC is wrong for ranking. Mismatching metric and task is the most common ML mistake.
- Use
sklearn.pipeline.Pipeline+ColumnTransformer. They prevent data leakage by design. Manual preprocessing is how production ML breaks. - Validate the way you''ll deploy. Time-series prod = time-series cross-validation. Tenant-scoped prod = group-aware CV. Random CV on time-series gives you a fantasy score.
1. The mental model — what ML actually is
Machine learning is a function-approximation game. You have:
- An input
X(features describing each example). - An output
y(the thing you want to predict). - A dataset of (X, y) pairs that you trust to be representative.
You search for a function f such that f(X) ≈ y, and that generalises to new (X, y) you haven''t seen. That "generalises" word is the whole reason ML is hard. Memorising your training set is easy; predicting next month''s data is not.
Three things must be true for ML to work at all:
- The signal exists.
ymust depend onXin a learnable way. - The data is representative. Your training set must look like what you''ll see in production.
- You can measure success honestly. If you can''t define "good," you can''t optimise for it.
Everything in this article is in service of those three properties.
2. Dataset types — recognise what you''re working with
2.1 Tabular — rows and columns
The 80% case. Think CSVs, SQL tables, Parquet files.
import pandas as pd
df = pd.read_csv("loans.csv", parse_dates=["application_date"])
df.dtypes
# loan_id int64
# amount float64
# term_months int64
# applicant_age int64
# employment category
# default bool ← target y
Tools: scikit-learn, XGBoost, LightGBM, CatBoost. For tabular, gradient-boosted trees almost always beat deep learning.
2.2 Text — variable-length sequences
emails = ["Free money click here!!!", "Quarterly board agenda attached"]
labels = [1, 0] # 1 = spam
Tools: transformers (BERT for classification, GPT/Llama for generation), sentence-transformers for embeddings, classic TF-IDF + linear models still competitive.
2.3 Images — pixel grids
from PIL import Image
import numpy as np
img = np.array(Image.open("cat.jpg").convert("RGB"))
img.shape, img.dtype # ((224, 224, 3), uint8)
Tools: PyTorch + torchvision, pretrained ResNet/EfficientNet/ViT, transfer learning is the default.
2.4 Audio — time series of samples
import soundfile as sf
audio, sr = sf.read("clip.wav")
audio.shape, sr # ((48000,), 16000)
Tools: librosa for features (mel-spectrograms), Whisper for ASR, Wav2Vec2 for embeddings.
2.5 Time series — observations indexed by time
ts = pd.read_csv("sales.csv", parse_dates=["date"], index_col="date").sort_index()
Tools: Prophet, statsmodels, sktime, XGBoost on engineered lag features.
2.6 Graphs — entities and relationships
import networkx as nx
G = nx.Graph()
G.add_edges_from([(1, 2), (1, 3), (2, 3)])
Tools: PyTorch Geometric, DGL.
2.7 Quick recognition guide
| If your data is… | Treat as |
|---|---|
| CSV / SQL table | Tabular |
| Strings of natural language | Text |
| JPEGs / PNGs | Image |
| WAV / MP3 | Audio |
| Anything indexed by a timestamp where order matters | Time series |
| Anything where relationships matter as much as nodes | Graph |
Most real projects are mixed: tabular features + a text column + a timestamp. Identify the dominant structure and build around it.
3. The big-picture pipeline — five steps, in order
Raw data → Clean → Split → Preprocess → Train → Evaluate
▲ │
└── fit on train ────┘
transform val/test
The most important arrow is the feedback loop: preprocessing is fit on training data and applied to validation/test, never the other way round. Get that wrong and your score is a fantasy.
4. Data cleaning — the unglamorous 60% of the job
4.1 Missing values
df.isna().sum()
# applicant_age 34
# employment 128
# Three honest options:
df_clean = df.dropna() # drop rows
df_clean = df.drop(columns=["employment"]) # drop columns
# Impute using the training set''s stats only
from sklearn.impute import SimpleImputer
num_imputer = SimpleImputer(strategy="median")
df["applicant_age"] = num_imputer.fit_transform(df[["applicant_age"]])
Crucial: missingness itself can be informative. Add a was_missing indicator before imputing:
df["employment_missing"] = df["employment"].isna()
df["employment"] = df["employment"].fillna("UNKNOWN")
4.2 Duplicates
print(df.duplicated().sum())
df = df.drop_duplicates()
df = df.sort_values("updated_at").drop_duplicates(subset=["loan_id"], keep="last")
4.3 Outliers
df["amount"].describe()
df["amount"].plot.box()
q1, q3 = df["amount"].quantile([0.25, 0.75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
df["amount"] = df["amount"].clip(lo, hi)
Outliers can be the actual signal (fraud detection, rare events). Don''t blindly cap them.
4.4 Type & range sanity
assert (df["applicant_age"] >= 0).all()
assert (df["application_date"] <= pd.Timestamp.utcnow()).all()
df["employment"] = df["employment"].replace("", "UNKNOWN")
These asserts are cheap and catch upstream bugs.
4.5 Class balance — check it now
df["default"].value_counts(normalize=True)
# False 0.97
# True 0.03 ← 3% positives — heavily imbalanced
If positives are <10%, classic metrics (accuracy) and naive resampling will mislead you.
5. Splitting — the place leakage starts
You need three sets:
- Train — model fits parameters here.
- Validation — you tune hyperparameters / pick the best model here.
- Test — touch it once, at the very end, to estimate generalisation.
5.1 Random split — the default for IID data
from sklearn.model_selection import train_test_split
X = df.drop(columns=["default", "loan_id", "application_date"])
y = df["default"]
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.20, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp)
stratify=y keeps the same class ratio across splits.
5.2 Time-aware split — when order matters
cutoff = df["application_date"].quantile(0.80)
train = df[df["application_date"] < cutoff]
test = df[df["application_date"] >= cutoff]
Random split here would let your model peek at the future.
5.3 Group-aware split — when rows are correlated
from sklearn.model_selection import GroupShuffleSplit
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y, groups=df["user_id"]))
Every user_id ends up entirely in train or entirely in test.
5.4 The four leakage failure modes
- Information from validation/test leaking into preprocessing (scaling on full dataset before splitting).
- Time leakage (training on data from after the prediction time).
- Group leakage (same user in both splits).
- Target leakage (a feature only available after the target is known).
Every one inflates your score and tanks production.
6. Preprocessing — fit on train, transform everywhere
6.1 Feature scaling
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
# StandardScaler — z-score: default
# MinMaxScaler — scale to [0, 1]
# RobustScaler — uses median + IQR; use when outliers are present
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train[num_cols])
X_val_scaled = scaler.transform(X_val[num_cols])
X_test_scaled = scaler.transform(X_test[num_cols])
Never scaler.fit(X_full) before splitting. That''s leakage.
6.2 Categorical encoding
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder
# OneHotEncoder — for low-cardinality (<50 unique)
ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
X_train_emp = ohe.fit_transform(X_train[["employment"]])
# OrdinalEncoder — when there''s a real order
# TargetEncoder — for high-cardinality categoricals
te = TargetEncoder()
X_train_zip = te.fit_transform(X_train[["zip_code"]], y_train)
6.3 Text vectorisation
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(
max_features=20000,
ngram_range=(1, 2),
min_df=5,
stop_words="english",
)
X_train_text = tfidf.fit_transform(X_train["description"])
X_val_text = tfidf.transform(X_val["description"])
For modern NLP, prefer sentence embeddings:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(X_train["description"].tolist())
6.4 Feature engineering — the most undervalued lever
# Date features
df["month"] = df["application_date"].dt.month
df["day_of_week"] = df["application_date"].dt.dayofweek
df["is_weekend"] = df["day_of_week"].isin([5, 6])
# Ratio features
df["debt_to_income"] = df["total_debt"] / df["annual_income"].clip(lower=1)
# Aggregations from another table
spend_by_user = transactions.groupby("user_id")["amount"].agg(["mean", "sum", "count"])
df = df.merge(spend_by_user, on="user_id", how="left")
# Lag features (time series)
df["units_sold_lag_7"] = df.groupby("product_id")["units_sold"].shift(7)
Domain knowledge wins here. Talk to the business; ask "which two features, if combined, would tell you a lot?"
6.5 Putting preprocessing in a Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
num_cols = ["amount", "term_months", "applicant_age", "debt_to_income"]
cat_cols = ["employment", "zip_code"]
numeric_pipeline = Pipeline([
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
])
categorical_pipeline = Pipeline([
("imputer", SimpleImputer(strategy="constant", fill_value="UNKNOWN")),
("ohe", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])
preprocessor = ColumnTransformer([
("num", numeric_pipeline, num_cols),
("cat", categorical_pipeline, cat_cols),
])
model = Pipeline([
("preprocess", preprocessor),
("clf", LogisticRegression(max_iter=1000, class_weight="balanced")),
])
model.fit(X_train, y_train)
y_val_pred = model.predict(X_val)
The pipeline object is the model — save it with joblib.dump, reload it for serving, and you cannot accidentally apply different preprocessing in production.
7. Choosing a model — taxonomy first, then experiment
7.1 Supervised learning
- Classification —
yis a category. - Regression —
yis a number.
7.2 Unsupervised learning
- Clustering — k-means, DBSCAN, hierarchical.
- Dimensionality reduction — PCA, UMAP, t-SNE.
- Anomaly detection — Isolation Forest, autoencoder reconstruction.
7.3 Self-supervised learning
Generate labels from the data itself. "Predict the next token" (LLMs). "Predict a masked patch" (MAE). The modern foundation of deep learning.
7.4 Reinforcement learning
The model takes actions, receives rewards. Game-playing, robotics.
7.5 The model menu — start simple, always
| Task | Start with | If you need more |
|---|---|---|
| Tabular classification / regression | Logistic / Linear regression baseline | LightGBM / XGBoost / CatBoost |
| Image classification | Pretrained ResNet-50 (transfer learning) | Vision Transformer |
| Text classification | TF-IDF + Logistic regression | Pretrained BERT/DistilBERT |
| Text generation | Off-the-shelf LLM | Fine-tune with LoRA |
| Time-series forecasting | Naive baseline, ARIMA | LightGBM with lag features, Prophet |
| Clustering | k-means baseline | DBSCAN if cluster sizes vary |
Always ship the baseline first. A 5-line logistic regression at 0.78 AUC tells you the problem is learnable. A 200-line transformer at 0.79 AUC tells you you wasted a week.
7.6 Bias / variance trade-off
- Bias = too simple → underfits → bad on train AND test.
- Variance = memorises noise → overfits → great on train, bad on test.
- Big gap between train and val → variance. Try regularisation, more data, simpler model.
- Bad on both → bias. Try a bigger model, better features, less regularisation.
8. Training — the parts that matter
8.1 Loss function
| Task | Loss |
|---|---|
| Binary classification | Binary cross-entropy (log loss) |
| Multi-class classification | Categorical cross-entropy |
| Regression (normal errors) | MSE |
| Regression (heavy-tailed errors) | MAE or Huber |
| Imbalanced classification | Focal loss, or weighted cross-entropy |
| Ranking | Pairwise / listwise (LambdaRank) |
8.2 Optimisers (deep learning)
- SGD with momentum — slowest but often the highest final accuracy.
- Adam / AdamW — adaptive per parameter. Default for transformers.
- Adafactor / Lion — newer, lower memory.
8.3 Learning rate — the single most important hyperparameter
Too high → loss diverges. Too low → training crawls. Sweep LR with a learning rate finder; for transformers fine-tuning, [1e-5, 5e-5]. Schedule with warmup + cosine decay.
8.4 Regularisation
- L1 (Lasso) — drives weights to zero → feature selection.
- L2 (Ridge) — shrinks weights uniformly.
- Dropout (NNs) — randomly zero activations during training.
- Early stopping — stop when val loss stops improving. Free regularisation.
- Data augmentation — flip / rotate / mixup.
9. Evaluation — the part that decides everything
Pick the metric the task actually requires, not the metric your library reports by default.
9.1 Classification metrics
| Metric | What it tells you | When to use |
|---|---|---|
| Accuracy | correct / total | Balanced classes only |
| Precision | of predicted positives, how many real | False positives are costly |
| Recall | of real positives, how many caught | False negatives are costly |
| F1 | harmonic mean of P and R | A single balanced number |
| ROC-AUC | ranking quality across thresholds | Pick the threshold later |
| PR-AUC | precision-recall area | Heavily imbalanced classification |
| Log loss | calibration-sensitive | Probabilities matter |
| Confusion matrix | the actual TP / FP / TN / FN | Always look at this |
from sklearn.metrics import (classification_report, confusion_matrix,
roc_auc_score, average_precision_score, log_loss)
y_pred = model.predict(X_val)
y_proba = model.predict_proba(X_val)[:, 1]
print(classification_report(y_val, y_pred))
print(confusion_matrix(y_val, y_pred))
print("ROC AUC :", roc_auc_score(y_val, y_proba))
print("PR AUC :", average_precision_score(y_val, y_proba))
print("Log loss:", log_loss(y_val, y_proba))
9.2 The accuracy trap
If 97% of your data is "not default" and your model predicts "not default" for everyone, accuracy is 97%. Useless model, great-looking number. For imbalanced classification, lead with PR-AUC and recall@k.
9.3 Regression metrics
| Metric | What it tells you |
|---|---|
| MAE | mean absolute error — interpretable, robust to outliers |
| MSE / RMSE | mean (root) squared error — penalises big errors |
| MAPE | mean absolute percentage error — breaks on zeros |
| R² | proportion of variance explained |
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
y_pred = model.predict(X_val)
print("MAE :", mean_absolute_error(y_val, y_pred))
print("RMSE:", np.sqrt(mean_squared_error(y_val, y_pred)))
print("R² :", r2_score(y_val, y_pred))
For skewed targets, np.log1p(y) before training often gives better behaviour.
9.4 Calibration — when probabilities matter
from sklearn.calibration import calibration_curve
prob_true, prob_pred = calibration_curve(y_val, y_proba, n_bins=10)
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6, 6))
ax.plot([0, 1], [0, 1], "--", label="Perfect calibration")
ax.plot(prob_pred, prob_true, "o-", label="Model")
ax.set_xlabel("Predicted probability")
ax.set_ylabel("Observed frequency")
ax.legend()
If the curve bows, calibrate with CalibratedClassifierCV.
9.5 Cross-validation
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=cv, scoring="roc_auc")
print(f"CV ROC-AUC: {scores.mean():.3f} ± {scores.std():.3f}")
Variants:
- StratifiedKFold — preserves class ratio (classification).
- KFold — vanilla (regression).
- GroupKFold — keeps groups together.
- TimeSeriesSplit — train always before validate.
9.6 Hyperparameter tuning
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from scipy.stats import loguniform
# Grid — exhaustive. Small spaces (<200 combos).
gs = GridSearchCV(model, param_grid, cv=5, scoring="roc_auc", n_jobs=-1)
# Random — sample N. Medium spaces.
rs = RandomizedSearchCV(
model,
param_distributions={"clf__C": loguniform(1e-4, 1e2)},
n_iter=50, cv=5, scoring="roc_auc", n_jobs=-1,
)
# Bayesian (Optuna) — uses past results. Best ROI for expensive trials.
Rule of thumb: grid for small, random for medium, Optuna for large or expensive.
10. The full end-to-end example
Loan-default prediction. Tabular, imbalanced, mixed numeric + categorical, time-aware split.
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.metrics import (classification_report, roc_auc_score, average_precision_score,
confusion_matrix)
import lightgbm as lgb
# ---- 1. Load ----
df = pd.read_csv("loans.csv", parse_dates=["application_date"])
df = df.sort_values("application_date")
# ---- 2. Engineer features ----
df["debt_to_income"] = df["total_debt"] / df["annual_income"].clip(lower=1)
df["log_amount"] = np.log1p(df["amount"])
df["application_dow"] = df["application_date"].dt.dayofweek
num_cols = ["log_amount", "term_months", "applicant_age",
"debt_to_income", "annual_income", "application_dow"]
cat_cols = ["employment", "purpose"]
target = "default"
# ---- 3. Time-aware split ----
cutoff_train_val = df["application_date"].quantile(0.70)
cutoff_val_test = df["application_date"].quantile(0.85)
train = df[df["application_date"] < cutoff_train_val]
val = df[(df["application_date"] >= cutoff_train_val) & (df["application_date"] < cutoff_val_test)]
test = df[df["application_date"] >= cutoff_val_test]
X_train, y_train = train[num_cols + cat_cols], train[target]
X_val, y_val = val [num_cols + cat_cols], val [target]
X_test, y_test = test [num_cols + cat_cols], test [target]
# ---- 4. Preprocessing pipeline ----
preprocessor = ColumnTransformer([
("num", Pipeline([
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
]), num_cols),
("cat", Pipeline([
("imputer", SimpleImputer(strategy="constant", fill_value="UNKNOWN")),
("ohe", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
]), cat_cols),
])
# ---- 5. Model — LightGBM, class-weighted ----
clf = lgb.LGBMClassifier(
n_estimators=500,
learning_rate=0.05,
num_leaves=31,
is_unbalance=True,
random_state=42,
n_jobs=-1,
)
model = Pipeline([
("preprocess", preprocessor),
("clf", clf),
])
# ---- 6. Fit with early stopping ----
model.named_steps["preprocess"].fit(X_train)
X_train_t = model.named_steps["preprocess"].transform(X_train)
X_val_t = model.named_steps["preprocess"].transform(X_val)
clf.fit(
X_train_t, y_train,
eval_set=[(X_val_t, y_val)],
eval_metric="auc",
callbacks=[lgb.early_stopping(stopping_rounds=20)],
)
# ---- 7. Evaluate ----
def report(name, X, y):
proba = clf.predict_proba(model.named_steps["preprocess"].transform(X))[:, 1]
pred = (proba > 0.5).astype(int)
print(f"\n=== {name} ===")
print(f"ROC-AUC : {roc_auc_score(y, proba):.3f}")
print(f"PR-AUC : {average_precision_score(y, proba):.3f}")
print(classification_report(y, pred, digits=3))
print("Confusion matrix:")
print(confusion_matrix(y, pred))
report("VAL", X_val, y_val)
report("TEST", X_test, y_test) # touch ONCE
Every concept in the article is in there: time-aware split, imputation inside the Pipeline (no leakage), class-weighted training (imbalance), early stopping (free regularisation), PR-AUC alongside ROC-AUC, test set touched once at the end.
11. From notebook to production
11.1 The model object IS the pipeline
import joblib
joblib.dump(model, "model.pkl")
# Serving
loaded = joblib.load("model.pkl")
prediction = loaded.predict(new_X)[0]
Preprocessing lives in the pipeline → you cannot accidentally apply different preprocessing in production.
11.2 Drift
- Feature drift — your inputs shift.
- Concept drift — the relationship between X and y changes.
Monitor distribution of features and predictions over time. Alert on shifts.
11.3 Retraining cadence
| Data type | Cadence |
|---|---|
| Stable industrial sensors | Quarterly or on alert |
| Consumer behavior | Weekly or daily |
| Adversarial (fraud, abuse) | Continuous / hourly |
11.4 A/B test new models
Route 5% of traffic to v2, measure business metrics for two weeks, then decide. Val/test score is necessary but not sufficient.
12. The honest stuff
- Most projects fail in data, not modelling.
- Domain knowledge beats algorithms.
- You''ll spend more time labeling and curating than modelling.
- A baseline is mandatory.
- Reproducibility is harder than it looks. Pin every library version, log every hyperparameter, version every dataset.
- Fairness matters. Test across demographic groups before shipping.
- Explainability matters too. SHAP, feature importances, partial-dependence plots.
13. The mental checklist
Before declaring a model "ready":
- Splits are time-aware / group-aware as appropriate.
- Preprocessing is inside the pipeline.
- Validation metric matches the business problem.
- Baseline reported alongside.
- Class imbalance acknowledged — PR-AUC if positives are rare.
- Calibration checked if probabilities matter.
- Test set has been touched exactly once.
- No feature uses data unavailable at prediction time.
- Confusion matrix or residual plot inspected manually.
- Model artifact contains preprocessing + classifier.
- Drift monitoring + retrain plan exist.
- Fairness audited across segments that matter.
14. Closing — the right mental model
Machine learning is engineering with statistics inside it. The model is the last 10% of the work. Everything before it — understanding your data, splitting it honestly, engineering features, picking the metric that matches the business — is what separates a model that ships from a notebook that doesn''t.
Three habits that make ML actually work:
- Always ship a baseline first.
- Suspect leakage by default. Every great-looking score has leakage until proven otherwise.
- Build the pipeline once, retrain forever. A reproducible pipeline beats ten one-off notebooks.
Internalise those, and the rest is technique. Technique you pick up by writing more code.
Go build the pipeline.
Further reading
- Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow — Aurélien Géron.
- Designing Machine Learning Systems — Chip Huyen.
- scikit-learn user guide — the entire ML toolbox.
- LightGBM docs and XGBoost docs — for serious tabular work.
- Made With ML — modern, project-based, free.
Got a confusion matrix that''s confusing you, or a metric that disagrees with your intuition? Email randhir.jassal@gmail.com with the numbers and I''ll help you read them.
Get the next issue
A short, curated email with the newest posts and questions.