Python for AI/ML — From Pandas Tricks to Production Pipelines (NumPy, Matplotlib, Functional Patterns, APIs, JSON, with Real Code)

Most "Python for ML" articles teach you import numpy as np and stop at toy [1,2,3] examples. Then you sit down with a real 4 GB CSV, a flaky vendor API, and a deadline — and none of those tutorials help.

This guide is the one I wish I''d had: deep enough that you understand why NumPy is fast, why pandas iterates rows badly, why requests.get(url).json() is a production bug waiting to happen, and how the same functional tricks that make LISP programmers smile also make pandas pipelines readable. Every section has runnable code. By the end you can read most ML codebases on autopilot and write data pipelines that don''t fall over at 2 AM.

---

TL;DR

• NumPy is fast because it stores data in contiguous C arrays and runs operations in compiled loops — the cost of one Python for loop is what you''re escaping.
• Pandas is NumPy plus labels plus joins plus time series. Master .loc, .groupby().agg(), and vectorised assignment; never iterrows().
• Matplotlib has two APIs — the pyplot "state machine" (fine for notebooks) and the object-oriented fig, ax (use this in production). Most confusion comes from mixing them.
• Functional patterns (map/filter/reduce, comprehensions, functools, itertools) make ML preprocessing readable and composable. Use them.
• APIs + JSON are how data gets into your model. Always set timeouts, retry on transient errors, validate with Pydantic, and never trust the shape of upstream JSON.

---

1. Why Python is still the AI/ML default in 2026

Three reasons that have only gotten stronger:

1. The libraries. NumPy, pandas, PyTorch, scikit-learn, transformers, LangChain, vLLM — every meaningful AI/ML breakthrough since 2010 ships in Python first. New papers come with Python code; .NET / Go / Rust ports arrive months later if at all.
2. The glue. Python is slow but cheap to write, and the heavy work happens inside C/CUDA. You spend 5% of CPU on Python orchestration and 95% on highly-tuned native code. That ratio works.
3. The notebooks. Jupyter is how ML actually gets done — iterative, visual, exploratory. No other language has anything close.

If you''re a .NET / Java / Go developer learning Python for ML, the syntax is the easy part. The hard part is learning the idioms that make Python performant despite being interpreted — and that''s exactly what we''re going to cover.

---

2. NumPy — the bedrock

Everything in Python ML stands on NumPy. Tensors in PyTorch, DataFrames in pandas, datasets in scikit-learn — they all eventually flatten to a NumPy ndarray.

2.1 Why NumPy is fast (the part no tutorial explains)

import numpy as np
import time

# Python list — each element is a full PyObject (28+ bytes overhead).
py_list = list(range(1_000_000))

# NumPy array — one contiguous C buffer of 8-byte floats.
np_arr = np.arange(1_000_000, dtype=np.float64)

# Add 1 to every element, two ways.
t0 = time.perf_counter()
result_py = [x + 1 for x in py_list]
t1 = time.perf_counter()
print(f"Python list comprehension: {(t1-t0)*1000:.1f} ms")

t0 = time.perf_counter()
result_np = np_arr + 1
t1 = time.perf_counter()
print(f"NumPy vectorised add:     {(t1-t0)*1000:.1f} ms")

# Typical output:
# Python list comprehension: 62.0 ms
# NumPy vectorised add:     1.8 ms

That ~35× speedup isn''t magic. It''s three things:

1. Contiguous memory — np_arr is one block of 8 MB; CPU caches love it.
2. No Python objects in the loop — the +1 runs in compiled C, one machine instruction per element.
3. SIMD — modern CPUs add 4 or 8 floats per cycle.

The corollary: the moment you write a for loop over a NumPy array in Python, you''ve thrown the speed away. The whole NumPy mindset is "express your operation as array math, not as a loop."

2.2 The essential array operations

# Creation
a = np.zeros((3, 4))            # 3×4 matrix of zeros
b = np.ones((3, 4))
c = np.full((3, 4), 7.5)
d = np.arange(0, 10, 0.5)       # [0, 0.5, 1, ..., 9.5]
e = np.linspace(0, 1, 11)       # 11 evenly-spaced from 0 to 1
f = np.random.default_rng(42).standard_normal((1000, 3))

# Shape & dtype — read these first when debugging
print(f.shape, f.dtype)         # (1000, 3) float64

# Slicing — same as Python lists, but multi-dimensional
f[:5, :]                        # first 5 rows, all columns
f[:, 0]                         # column 0
f[f[:, 0] > 0]                  # rows where col 0 > 0 — boolean indexing

# Aggregation along an axis
f.mean(axis=0)                  # mean per column → shape (3,)
f.mean(axis=1)                  # mean per row → shape (1000,)

# In-place operations save memory
f += 1.0                        # vs f = f + 1.0 (allocates new array)

2.3 Broadcasting — the secret sauce

Broadcasting lets NumPy combine arrays of different shapes without you writing explicit loops.

# Standardise (z-score) every column of a (1000, 3) dataset
X = np.random.default_rng(0).standard_normal((1000, 3))

mean = X.mean(axis=0)           # shape (3,)
std  = X.std(axis=0)            # shape (3,)
X_z  = (X - mean) / std         # shape (1000, 3) — broadcast (3,) across all rows

print(X_z.mean(axis=0))         # ≈ [0, 0, 0]
print(X_z.std(axis=0))          # ≈ [1, 1, 1]

The (1000, 3) - (3,) works because NumPy broadcasts the (3,) array across each row. Internally it''s the same compiled loop, no memory allocated for an intermediate (1000, 3) mean matrix. This is the trick behind most efficient ML preprocessing.

2.4 Vectorisation in practice — a real ML example

Compute pairwise Euclidean distances between 1,000 vectors in 100 dimensions. The naive way:

# Naive — 4.2 seconds on my laptop, almost all Python overhead.
import math
def pairwise_naive(X):
    n = len(X)
    D = np.zeros((n, n))
    for i in range(n):
        for j in range(n):
            D[i, j] = math.sqrt(sum((X[i, k] - X[j, k])**2 for k in range(len(X[i]))))
    return D

The NumPy way using broadcasting:

# Vectorised — 8 ms. ~500× faster.
def pairwise_fast(X):
    # X has shape (n, d). We want D[i,j] = ||X[i] - X[j]||
    # Broadcasting: X[:, None, :] has shape (n, 1, d)
    #               X[None, :, :] has shape (1, n, d)
    # Their difference has shape (n, n, d).
    diff = X[:, None, :] - X[None, :, :]
    return np.sqrt((diff ** 2).sum(axis=-1))

X = np.random.default_rng(0).standard_normal((1000, 100))
D = pairwise_fast(X)
print(D.shape)  # (1000, 1000)

That X[:, None, :] - X[None, :, :] trick — using None to add a new axis — is the most idiomatic NumPy you''ll write. Practise reading it; it shows up everywhere.

2.5 When NumPy isn''t enough

• Sparse matrices (95%+ zeros) → scipy.sparse. Saves orders of magnitude in memory.
• GPU acceleration → PyTorch (torch.from_numpy(x).cuda()) or CuPy.
• Out-of-core / bigger-than-RAM → Dask or Polars.

For 99% of pre-GPU work, plain NumPy is your tool.

---

3. Pandas — the data layer of ML

Pandas is NumPy with labels. A Series is a 1-D array with an index; a DataFrame is a dict of Series sharing one index. Everything else is conveniences on top.

3.1 Read your data correctly

import pandas as pd

# Read 100k rows with the right dtypes upfront. Inferring is slow + wrong.
df = pd.read_csv(
    "transactions.csv",
    dtype={
        "user_id":      "int64",
        "amount_cents": "int64",
        "merchant":     "category",
        "country":      "category",
    },
    parse_dates=["created_at"],
    usecols=["user_id", "amount_cents", "merchant", "country", "created_at"],
)
print(df.dtypes)
print(df.memory_usage(deep=True).sum() / 1e6, "MB")

Three habits new pandas users skip and regret:

1. Specify dtypes — type inference is slow and often picks object (Python strings) when you meant category.
2. usecols= — only read the columns you need. CSV parsing is I/O-bound; columns you ignore still cost time.
3. parse_dates= — get real datetime64 columns, not strings. Time-series ops only work on real datetimes.

3.2 The two ways to select — .loc and .iloc

# .loc — label-based
df.loc[42, "amount_cents"]
df.loc[df["country"] == "IN", ["user_id", "amount_cents"]]

# .iloc — integer-position-based
df.iloc[0, 0]
df.iloc[-5:, :3]

# DO NOT mix chained indexing — it triggers SettingWithCopyWarning and bugs.
# df[df["country"] == "IN"]["amount_cents"] = 0     # may not modify df
# df.loc[df["country"] == "IN", "amount_cents"] = 0 # correct

Beginners conflate df["col"] (selection) with df.loc[..., "col"] (label access) and end up with the dreaded SettingWithCopyWarning. Memorise: .loc for everything that writes.

3.3 GroupBy — the operation that pays the pandas tax back

# Daily revenue by country
daily = (
    df.groupby([df["created_at"].dt.date, "country"])
      .agg(
          revenue   = ("amount_cents", "sum"),
          orders    = ("user_id", "count"),
          users     = ("user_id", "nunique"),
          avg_value = ("amount_cents", "mean"),
      )
      .reset_index()
)
print(daily.head())

The pattern: groupby(...).agg(named_output=("source_col", "func")). Multiple aggregations, multiple output columns, in one C loop. The same operation written with a Python for loop over rows is typically 100–500× slower.

3.4 Method chaining — readable pipelines

clean = (
    df
    .query("amount_cents > 0")
    .assign(
        amount     = lambda d: d["amount_cents"] / 100,
        day_of_week = lambda d: d["created_at"].dt.day_name(),
        is_weekend = lambda d: d["created_at"].dt.dayofweek >= 5,
    )
    .drop(columns=["amount_cents"])
    .loc[lambda d: d["country"].isin(["IN", "US", "GB"])]
    .sort_values("created_at")
    .reset_index(drop=True)
)

Five steps, no temporary variables. The lambda d: pattern lets you reference the current intermediate DataFrame, so each step composes cleanly.

3.5 Avoid iterrows() — really

# Slow. ~100× the cost.
for idx, row in df.iterrows():
    df.at[idx, "tax"] = row["amount"] * 0.18 if row["country"] == "IN" else 0

# Vectorised.
df["tax"] = np.where(df["country"] == "IN", df["amount"] * 0.18, 0.0)

If you''re writing for over a DataFrame, you''ve missed a vectorised expression.

3.6 Time-series superpowers

# Resample — bucket by time window
hourly_revenue = (
    df.set_index("created_at")
      .resample("1h")["amount_cents"].sum()
)

# Rolling — moving-window stats
hourly_revenue.rolling("24h").mean()

# Shift — for lag features in ML
df = df.sort_values("created_at")
df["amount_lag_1"] = df.groupby("user_id")["amount_cents"].shift(1)

resample, rolling, shift — three operations that handle 90% of time-series feature engineering.

3.7 Merge / join — the SQL inside pandas

users = pd.read_parquet("users.parquet")
features = (
    df
    .merge(users, on="user_id", how="left")
    .assign(account_age = lambda d: (d["created_at"] - d["signup_date"]).dt.days)
)

how= accepts inner, left, right, outer. Always check the shape after a merge — accidental Cartesian joins are how pandas pipelines silently produce wrong models.

3.8 Should you use Polars or DuckDB instead?

For new projects in 2026: maybe. Polars is faster (multithreaded by default, lazy execution); DuckDB is excellent for SQL-on-Parquet. But pandas is everywhere — every tutorial, every notebook, every Stack Overflow answer. Learn pandas first; reach for Polars when you have a real performance problem pandas can''t solve.

---

4. Matplotlib — visualisation that actually works

Matplotlib has two APIs and they confuse everyone:

• pyplot (state machine) — plt.plot(...), plt.xlabel(...). Convenient in notebooks, breaks in scripts.
• Object-oriented — fig, ax = plt.subplots(), then ax.plot(...). Explicit, composable, the only correct choice in production.

Always use the OO API.

4.1 The skeleton you''ll write a thousand times

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(figsize=(10, 6))
x = np.linspace(0, 4 * np.pi, 200)
ax.plot(x, np.sin(x),  label="sin(x)")
ax.plot(x, np.cos(x),  label="cos(x)", linestyle="--")

ax.set_title("Trig functions")
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.legend()
ax.grid(True, alpha=0.3)

fig.tight_layout()
fig.savefig("trig.png", dpi=150, bbox_inches="tight")
plt.show()

Create fig, ax; call methods on ax; finish with tight_layout + savefig. Done.

4.2 Multi-panel figures

fig, axes = plt.subplots(2, 2, figsize=(12, 8), sharex=True)

axes[0, 0].hist(df["amount"], bins=50)
axes[0, 0].set_title("Distribution of amounts")

axes[0, 1].scatter(df["amount"], df["tax"], alpha=0.3)
axes[0, 1].set_title("Tax vs amount")

axes[1, 0].plot(daily["date"], daily["revenue"])
axes[1, 0].set_title("Daily revenue")

axes[1, 1].boxplot([df.loc[df["country"]==c, "amount"] for c in ["IN", "US", "GB"]],
                    tick_labels=["IN", "US", "GB"])
axes[1, 1].set_title("Amount by country")

fig.suptitle("Transaction overview", fontsize=14)
fig.tight_layout()

Indexing axes[row, col] is the entire mental model. sharex=True ties the x-axes together so zooming one zooms all.

4.3 Seaborn for statistical plots

import seaborn as sns

fig, ax = plt.subplots(figsize=(8, 5))
sns.boxplot(data=df, x="country", y="amount", ax=ax)
ax.set_title("Amount distribution by country")
ax.set_ylabel("Amount (₹)")
fig.tight_layout()

Seaborn is matplotlib with sensible defaults for statistical plots. Pass ax=ax to keep the OO model.

4.4 Saving for slides vs notebooks

fig.savefig("plot.svg", bbox_inches="tight")      # slides / PDFs
fig.savefig("plot.png", dpi=150, bbox_inches="tight")  # READMEs / blogs

Always bbox_inches="tight". Always set dpi explicitly for raster output.

---

5. Functional patterns — the part that makes ML code clean

You don''t write much OOP in ML pipelines. You write transformations. Functional patterns match the data-flow nature of the work and make pipelines readable.

5.1 Comprehensions — the Pythonic map / filter

squares = [x * x for x in range(10)]
unique_lengths = {len(s) for s in ["hi", "hello", "hi"]}
char_count = {c: word.count(c) for c in set(word)}

# Filter inline
even_squares = [x * x for x in range(10) if x % 2 == 0]

# Nested — flatten a list of lists
nested = [[1, 2], [3, 4], [5]]
flat = [n for sub in nested for n in sub]   # [1, 2, 3, 4, 5]

# Generator expressions — lazy, memory-efficient
total = sum(x * x for x in range(1_000_000))   # no list materialised

Comprehensions are how Python expresses map/filter. Use them.

5.2 functools — composition, caching, partial application

from functools import reduce, lru_cache, partial, cache

# reduce — the third functional primitive
from operator import mul
factorial_5 = reduce(mul, range(1, 6))   # 120

# lru_cache — memoize an expensive pure function
@lru_cache(maxsize=1024)
def embed(text: str) -> list[float]:
    return slow_embedding_api(text)

# partial — bind some arguments now, call later
def call_with_timeout(url, timeout, headers):
    return requests.get(url, timeout=timeout, headers=headers)

call_api = partial(call_with_timeout, timeout=5, headers={"X-Trace": "abc"})
call_api("https://api.example.com/v1/users")

partial is underused. Any time you''d write a wrapper to bind some arguments, use partial.

5.3 itertools — the lazy combinators

from itertools import chain, islice, accumulate, product

# Flatten lazily
chain.from_iterable([[1, 2], [3, 4]])

# Take the first N from any iterable
islice(infinite_generator(), 100)

# Running totals
list(accumulate([1, 2, 3, 4]))              # [1, 3, 6, 10]

# Cartesian product — grid search over hyperparameters
for lr, batch_size in product([1e-3, 1e-4], [16, 32, 64]):
    train(lr=lr, batch_size=batch_size)

itertools is the standard library''s lazy / functional combinators. They compose beautifully.

5.4 Pure functions — the ML pipeline foundation

# Mutates the input — surprises future you
def add_features(df):
    df["amount_log"] = np.log1p(df["amount"])
    df["weekend"] = df["created_at"].dt.dayofweek >= 5
    return df

# Returns a new DataFrame
def add_features(df):
    return df.assign(
        amount_log = lambda d: np.log1p(d["amount"]),
        weekend    = lambda d: d["created_at"].dt.dayofweek >= 5,
    )

Pure functions compose, test, parallelise, and cache. Mutating ones don''t.

---

6. APIs — the boring part that breaks production at 2 AM

6.1 The wrong way (and why you see it everywhere)

# Don''t do this — every line is a footgun
import requests
data = requests.get("https://api.example.com/v1/users").json()
for u in data["users"]:
    print(u["email"])

Failures hiding here: no timeout, .json() will throw on HTML responses, data["users"] KeyErrors on shape change, no retry on 502/503, no auth headers.

6.2 The right way with httpx + tenacity + pydantic

import httpx
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
from pydantic import BaseModel

class User(BaseModel):
    id: int
    email: str
    country: str | None = None

class UsersResponse(BaseModel):
    users: list[User]
    next_cursor: str | None = None

@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential(multiplier=0.5, max=8),
    retry=retry_if_exception_type((httpx.HTTPStatusError, httpx.TimeoutException)),
)
def fetch_users(client: httpx.Client, cursor: str | None) -> UsersResponse:
    params = {"limit": 100}
    if cursor:
        params["cursor"] = cursor
    r = client.get("/v1/users", params=params, timeout=10.0)
    r.raise_for_status()
    return UsersResponse.model_validate_json(r.content)

def all_users() -> list[User]:
    headers = {"Authorization": f"Bearer {get_token()}"}
    with httpx.Client(base_url="https://api.example.com", headers=headers) as client:
        users: list[User] = []
        cursor: str | None = None
        while True:
            page = fetch_users(client, cursor)
            users.extend(page.users)
            if not page.next_cursor:
                return users
            cursor = page.next_cursor

What''s good here:

• httpx.Client(...) keeps the connection alive — 3–10× faster than fresh requests.get() per call.
• timeout=10.0 — never trust the upstream.
• raise_for_status() turns 4xx/5xx into exceptions you can retry.
• @retry — exponential backoff via tenacity.
• Pydantic models — parses JSON and validates types.
• Cursor pagination — works at any size.

6.3 Async for high throughput

import asyncio
import httpx

async def embed_one(client: httpx.AsyncClient, text: str) -> list[float]:
    r = await client.post("/v1/embeddings", json={"input": text})
    r.raise_for_status()
    return r.json()["data"][0]["embedding"]

async def embed_many(texts: list[str], concurrency: int = 16) -> list[list[float]]:
    semaphore = asyncio.Semaphore(concurrency)

    async def bounded_embed(client, text):
        async with semaphore:
            return await embed_one(client, text)

    async with httpx.AsyncClient(base_url="https://api.openai.com", timeout=30.0,
                                  headers={"Authorization": f"Bearer {OPENAI_KEY}"}) as client:
        return await asyncio.gather(*[bounded_embed(client, t) for t in texts])

embeddings = asyncio.run(embed_many(my_texts))

The Semaphore(16) caps concurrency so you don''t hammer the upstream past its rate limit.

6.4 Caching responses — a one-line speedup

import hishel
import httpx

storage = hishel.FileStorage(base_path=".cache/http", ttl=3600)
with hishel.CacheClient(storage=storage) as client:
    r = client.get("https://api.example.com/v1/users")

In a notebook where you''re re-running the same cells 50 times, this saves both money and patience.

---

7. JSON — the format you''ll deal with every day

7.1 Don''t trust the shape

import json

raw = '{"orders": [{"id": 1, "items": null}, {"id": 2}]}'
data = json.loads(raw)

# Brittle
for o in data["orders"]:
    for item in o["items"]:       # KeyError if missing; TypeError if null
        process(item)

# Defensive
for o in data["orders"]:
    items = o.get("items") or []  # both missing and null become []
    for item in items:
        process(item)

7.2 Pydantic > raw JSON dict, always

from pydantic import BaseModel, Field
from datetime import datetime

class OrderItem(BaseModel):
    sku: str
    qty: int = Field(gt=0)
    price_cents: int

class Order(BaseModel):
    id: int
    created_at: datetime
    items: list[OrderItem] = []
    customer_email: str | None = None

raw = '''
{
  "id": 42,
  "created_at": "2026-05-27T14:35:00Z",
  "items": [
    {"sku": "WIDGET-1", "qty": 2, "price_cents": 1999}
  ]
}
'''

order = Order.model_validate_json(raw)
print(order.created_at.year)
print(order.items[0].qty)

Pydantic gives you parsing, validation, type safety, and good errors in one decorator.

7.3 Serialising NumPy / pandas to JSON

import numpy as np
import json

def to_jsonable(x):
    if isinstance(x, (np.integer,)):  return int(x)
    if isinstance(x, (np.floating,)): return float(x)
    if isinstance(x, np.ndarray):     return x.tolist()
    raise TypeError

json.dumps({"value": np.int64(42)}, default=to_jsonable)

# For DataFrames
df.to_json(orient="records", date_format="iso")

7.4 Streaming big JSON files

import json

with open("events.jsonl") as f:
    for line in f:
        record = json.loads(line)
        process(record)

JSONL (newline-delimited JSON) is the format of choice for big ML datasets.

---

8. Putting it all together — a real ML data pipeline

from datetime import datetime
from pathlib import Path
import httpx
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pydantic import BaseModel
from tenacity import retry, stop_after_attempt, wait_exponential

# ---- 1. API client ----
class User(BaseModel):
    id: int
    country: str | None = None
    signup_date: datetime

class UsersPage(BaseModel):
    users: list[User]
    next_cursor: str | None = None

@retry(stop=stop_after_attempt(4), wait=wait_exponential(0.5, max=8))
def fetch_page(client: httpx.Client, cursor: str | None) -> UsersPage:
    r = client.get("/v1/users", params={"cursor": cursor, "limit": 200}, timeout=10.0)
    r.raise_for_status()
    return UsersPage.model_validate_json(r.content)

def fetch_all_users(base_url: str, token: str) -> pd.DataFrame:
    headers = {"Authorization": f"Bearer {token}"}
    with httpx.Client(base_url=base_url, headers=headers) as client:
        users: list[User] = []
        cursor = None
        while True:
            page = fetch_page(client, cursor)
            users.extend(page.users)
            if not page.next_cursor:
                break
            cursor = page.next_cursor
    return pd.DataFrame([u.model_dump() for u in users])

# ---- 2. Load transactions ----
def load_transactions(path: Path) -> pd.DataFrame:
    return pd.read_parquet(path, columns=[
        "user_id", "amount_cents", "merchant", "created_at"
    ])

# ---- 3. Feature engineering — pure functions ----
def add_time_features(tx: pd.DataFrame) -> pd.DataFrame:
    return tx.assign(
        day_of_week = lambda d: d["created_at"].dt.dayofweek,
        is_weekend  = lambda d: d["created_at"].dt.dayofweek >= 5,
        hour        = lambda d: d["created_at"].dt.hour,
        amount      = lambda d: d["amount_cents"] / 100,
    )

def aggregate_per_user(tx: pd.DataFrame) -> pd.DataFrame:
    return (
        tx.groupby("user_id")
          .agg(
              total_amount  = ("amount", "sum"),
              avg_amount    = ("amount", "mean"),
              n_transactions = ("amount", "count"),
              n_merchants   = ("merchant", "nunique"),
              weekend_ratio = ("is_weekend", "mean"),
          )
          .reset_index()
    )

# ---- 4. Putting it together ----
def build_training_table(users: pd.DataFrame, tx: pd.DataFrame) -> pd.DataFrame:
    return (
        tx
        .pipe(add_time_features)
        .pipe(aggregate_per_user)
        .merge(users, left_on="user_id", right_on="id", how="left")
        .assign(account_age_days = lambda d: (pd.Timestamp.utcnow() - d["signup_date"]).dt.days)
        .drop(columns=["id", "signup_date"])
    )

# ---- 5. Quick visual check ----
def plot_feature_distributions(df: pd.DataFrame, out: Path) -> None:
    fig, axes = plt.subplots(2, 2, figsize=(12, 8))
    df["total_amount"].plot.hist(bins=50, ax=axes[0, 0], title="Total amount")
    df["n_transactions"].plot.hist(bins=50, ax=axes[0, 1], title="Number of transactions")
    df["weekend_ratio"].plot.hist(bins=20, ax=axes[1, 0], title="Weekend ratio")
    df["account_age_days"].plot.hist(bins=50, ax=axes[1, 1], title="Account age (days)")
    fig.tight_layout()
    fig.savefig(out, dpi=150, bbox_inches="tight")

# ---- 6. Run it ----
if __name__ == "__main__":
    import os
    users = fetch_all_users("https://api.example.com", token=os.environ["API_TOKEN"])
    tx    = load_transactions(Path("data/transactions.parquet"))
    training_table = build_training_table(users, tx)
    training_table.to_parquet("data/training_table.parquet", index=False)
    plot_feature_distributions(training_table, Path("plots/feature_distributions.png"))
    print(f"Wrote {len(training_table)} rows of features")

Every concept from this article is in there:

• Pydantic models + retry decorator for the API.
• pandas.read_parquet with columns= for fast load.
• .assign() with lambdas for chained transformations.
• .groupby().agg(...) for per-user features.
• .pipe() to compose pure functions.
• .merge() to join API users with transactions.
• OO matplotlib for the visual check.

This is what production-shaped Python looks like. Once you can read and write code like this comfortably, you''re ready for the model layer (PyTorch, scikit-learn, transformers) — which is the easy part once your data layer is solid.

---

9. The habits that make your Python ML code last

1. Type your function signatures. def add_features(df: pd.DataFrame) -> pd.DataFrame: — your IDE and future self both win.
2. Pure functions for transforms. No df["new_col"] = ... inside helpers. Use .assign() and return.
3. Pydantic at every API / file boundary. Catch shape errors at parse time, not deep in model code.
4. Vectorise before you optimise. A vectorised pandas/NumPy expression is almost always faster than your hand-rolled for loop.
5. Read your intermediate DataFrames + plots. Bugs hide in what you didn''t bother to look at.
6. One notebook, one experiment. Not "scratch.ipynb" with 800 cells.
7. Cache slow I/O during dev. lru_cache, hishel, joblib''s Memory — your iteration loop should be sub-second.

---

10. Closing — the right mental model

Python in ML is a glue language. You''re not optimising Python; you''re using Python to compose fast C/CUDA/Rust kernels (NumPy, pandas, PyTorch, polars, faiss, vLLM). The skill is knowing the idioms that stay inside the fast kernels and avoiding the ones that fall back to interpreted Python.

NumPy operations on whole arrays — fast. Pandas groupby().agg() — fast. Vectorised boolean masks — fast. for loops over rows — slow. iterrows() — slow. Calling Python functions inside apply() — usually slow.

Internalise that one rule and you''ll write idiomatic, fast Python by default. Everything else in this article — APIs, JSON, functional patterns, matplotlib — just lets you stitch the fast kernels together cleanly.

Go write the pipeline. The model is the easy part.

---

Further reading

• NumPy: the absolute basics for beginners — start here if you''ve never touched NumPy.
• Pandas user guide — the GroupBy and Merge chapters are non-negotiable reading.
• Matplotlib OO tutorial — the OO API, explained by the maintainers.
• HTTPX and tenacity — the two libraries every Python API client should use.
• Pydantic v2 docs — JSON validation done right.

---

Stuck on a pandas operation or making an API call that won''t behave? Email randhir.jassal@gmail.com with the code and the error, and I''ll show you the idiomatic way.