When Classical ML Beats LLM

There is a particular kind of pressure in the industry right now where reaching for an LLM has become the default. New project, new problem, someone in the room says "what if we just use GPT?" and the conversation pivots.

Most of the time that instinct is wrong. Not because LLMs are bad—they are remarkable—but because they are a very specific tool with a very specific cost and risk profile. And for a wide class of problems, a logistic regression trained on clean features will outperform a billion-parameter model while running a hundred times cheaper, faster, and more predictably.

This post is about knowing which problems belong in which bucket.

The core tradeoff

Before getting into specific cases, it helps to be honest about what each approach optimizes for.

That table is not a scorecard. It is a starting point for asking the right question: what does your problem actually look like?

Case 1: Fraud detection

Fraud detection is a structured prediction problem. Every transaction carries numeric and categorical signals: amount, merchant category, geography, time of day, velocity, device fingerprint. The label is binary: fraud or not fraud.

This is gradient boosting territory.

import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
 
# Features engineered from raw transaction data
features = [
    "amount", "hour_of_day", "merchant_category_code",
    "distance_from_home_km", "transactions_last_1h",
    "transactions_last_24h", "is_new_device", "country_match"
]
 
X = df[features]
y = df["is_fraud"]
 
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
 
model = xgb.XGBClassifier(
    n_estimators=500,
    max_depth=6,
    learning_rate=0.05,
    scale_pos_weight=(y_train == 0).sum() / (y_train == 1).sum(),
    eval_metric="aucpr",
    early_stopping_rounds=20
)
 
model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    verbose=False
)
 
print(classification_report(y_test, model.predict(X_test)))

An LLM cannot meaningfully process a JSON blob of transaction metadata and return a calibrated fraud probability. It might produce a plausible-sounding answer but the probability estimate will be poorly calibrated, the latency will be unacceptable for real-time decisioning, and you lose all explainability the moment a regulator asks why a transaction was flagged.

XGBoost gives you feature importance. It gives you SHAP values. It gives you sub-millisecond inference and a model you can retrain weekly on fresh labels.

Case 2: Churn prediction

Predicting whether a user will churn in the next 30 days is fundamentally a tabular classification problem. The signals are behavioral: session frequency, last-login delta, feature adoption, support tickets, billing changes.

flowchart LR
  E[User Events] --> FE[Feature Engineering]
  FE --> M[Logistic Regression / XGBoost]
  M --> S[Churn Score 0–1]
  S --> D{Score > threshold?}
  D -->|Yes| CX[CX Intervention]
  D -->|No| IDLE[No action]

A logistic regression baseline is a good starting point because the output probability is well-calibrated and the coefficients are directly interpretable.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import pandas as pd
 
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(class_weight="balanced", max_iter=1000))
])
 
pipeline.fit(X_train, y_train)
 
# Coefficient interpretation
coef_df = pd.DataFrame({
    "feature": features,
    "coefficient": pipeline.named_steps["clf"].coef_[0]
}).sort_values("coefficient", ascending=False)
 
print(coef_df)

The coefficients tell you directly that "days since last login" is three times more predictive than "number of support tickets." That is a business insight, not just a model output. You can act on it immediately without prompting anything.

Case 3: Time-series anomaly detection

Infrastructure monitoring, financial time series, IoT sensor data. The question is always "is this reading anomalous given its recent history?"

This is not a language problem. Isolation Forest, STL decomposition, or a simple z-score over a rolling window will catch real anomalies with minimal false positives and sub-millisecond latency per point.

from sklearn.ensemble import IsolationForest
import numpy as np
 
# Rolling feature window: mean, std, slope over last N points
def extract_window_features(series, window=60):
    rolling_mean = series.rolling(window).mean()
    rolling_std = series.rolling(window).std()
    rolling_slope = series.diff(window) / window
    return np.column_stack([
        series.values,
        rolling_mean.values,
        rolling_std.values,
        rolling_slope.values
    ])
 
X_features = extract_window_features(df["metric"])
X_clean = X_features[~np.isnan(X_features).any(axis=1)]
 
detector = IsolationForest(contamination=0.01, random_state=42)
detector.fit(X_clean)
 
scores = detector.decision_function(X_clean)
anomalies = detector.predict(X_clean) == -1

An LLM asked to "detect anomalies in this time series" will either hallucinate a threshold or describe the problem in English. It cannot process millions of data points per second and give you a continuous anomaly score with a calibrated contamination rate.

flowchart TD
  RAW[Raw Metric Stream] --> FEAT[Window Feature Extraction]
  FEAT --> ISO[Isolation Forest]
  ISO --> SCORE[Anomaly Score]
  SCORE --> THR{Below threshold?}
  THR -->|Anomaly| ALERT[PagerDuty Alert]
  THR -->|Normal| SINK[Log and Continue]

Case 4: Recommendation ranking

When you have explicit signals—clicks, purchases, ratings, dwell time—collaborative filtering and learning-to-rank methods will almost always outperform an LLM retrieval approach for in-domain recommendation.

The reason is that the signal is behavioral, not linguistic. The model needs to learn the implicit latent space of user-item affinity, not the semantic meaning of item descriptions.

from implicit import als
import scipy.sparse as sparse
 
# Build user-item interaction matrix
user_item_matrix = sparse.csr_matrix(
    (df["interaction_weight"], (df["user_idx"], df["item_idx"])),
    shape=(n_users, n_items)
)
 
# ALS collaborative filter
model = als.AlternatingLeastSquares(
    factors=128,
    regularization=0.01,
    iterations=20,
    calculate_training_loss=True
)
 
model.fit(user_item_matrix)
 
# Recommend for a specific user
user_id = 42
recommended_ids, scores = model.recommend(
    user_id,
    user_item_matrix[user_id],
    N=10,
    filter_already_liked_items=True
)

You can layer an LLM on top for explanation ("we recommended this because you seem to enjoy dense thriller narratives") but the ranking itself should come from behavioral signals.

Case 5: Structured classification at scale

Imagine classifying incoming insurance claims into fifty categories. The data is text but it is domain-specific and formulaic. You have ten million labeled historical claims.

A fine-tuned lightweight encoder—or even a TF-IDF plus logistic regression baseline—will be more cost-effective than GPT-4o by orders of magnitude.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import f1_score
 
text_clf = Pipeline([
    ("tfidf", TfidfVectorizer(
        ngram_range=(1, 2),
        max_features=100_000,
        sublinear_tf=True
    )),
    ("clf", LogisticRegression(
        C=5.0,
        class_weight="balanced",
        max_iter=500,
        solver="saga",
        n_jobs=-1
    ))
])
 
text_clf.fit(X_train_text, y_train)
preds = text_clf.predict(X_test_text)
 
macro_f1 = f1_score(y_test, preds, average="macro")
print(f"Macro F1: {macro_f1:.4f}")

The classifier runs in microseconds per document and costs nothing beyond the initial training compute. If you need to classify ten million documents a day, the cost differential between this and an API-based LLM call is the difference between a rounding error and a budget line item.

Decision framework

The question is not "should I use an LLM?" It is "what does this problem actually require?"

flowchart TD
  START([New Problem]) --> Q1{Is the input primarily unstructured text with high variance?}
  Q1 -->|Yes| Q2{Do you need open-ended generation or reasoning?}
  Q1 -->|No| Q3{Is the data tabular or structured?}

  Q2 -->|Yes| LLM[Consider LLM]
  Q2 -->|No| Q4{Is it a classification or extraction task?}
  Q4 -->|Yes| FT[Fine-tuned small model or classical NLP]
  Q4 -->|No| LLM

  Q3 -->|Yes| Q5{Do you have labeled training data?}
  Q5 -->|Yes| CLASSIC[Classical ML — GBT, LogReg, RF]
  Q5 -->|No| Q6{Can you engineer meaningful features?}
  Q6 -->|Yes| CLASSIC
  Q6 -->|No| Q7{Is the problem solvable with rules or heuristics?}
  Q7 -->|Yes| RULES[Rule-based system]
  Q7 -->|No| LLM

A few heuristics that hold up in practice:

• If your input has a fixed schema, try classical ML first.
• If latency is a hard constraint under 50ms, LLMs are usually out.
• If you need calibrated probabilities with regulatory audit trails, classical ML is significantly easier to certify.
• If your training data is labeled and large, a task-specific model will almost always beat a general-purpose LLM without fine-tuning.
• If you need zero-shot generalization on novel language tasks, LLMs earn their place.

The hybrid is often the right architecture

The cleanest production systems are not all-or-nothing. They route problems to the right tool.

flowchart LR
  INPUT[Incoming Request] --> ROUTER{Problem Router}

  ROUTER -->|Structured signal| ML[Classical ML Pipeline]
  ROUTER -->|Unstructured text and generation| LLM_SVC[LLM Service]
  ROUTER -->|Simple rule| RULES[Rule Engine]

  ML --> AGG[Aggregation Layer]
  LLM_SVC --> AGG
  RULES --> AGG

  AGG --> RESP[Response]

A real example: a customer support system might use a classifier to route tickets to the right queue (fast, cheap, high-accuracy), use an LLM to draft a reply suggestion (where generation quality matters), and use a rule engine to flag SLA breaches (where determinism is non-negotiable).

Each tool does what it is actually good at.

What classical ML cannot do well

This post would be dishonest without acknowledging the limits.

• Ambiguous open-ended language tasks: sentiment that requires cultural nuance, intent that depends on context across a long conversation, summarization of complex documents. Classical NLP can do a version of these things, but LLMs do them much better.
• Zero-shot generalization: if you have no labeled data and no domain-specific rules, LLMs give you a starting point that classical approaches simply cannot match.
• Reasoning under uncertainty: questions that require multi-step inference over text benefit from the architecture of large models in ways that a feature-based approach cannot replicate.
• Novel task prototyping: the iteration speed of prompting versus training is dramatically faster. For early product exploration, an LLM gets you to signal quickly even if the production version ends up being a fine-tuned smaller model.

The takeaway

The industry has overcorrected. LLMs are extraordinary for the problems they are designed to solve, and they are used heavily in places where a clean feature matrix and a tree-based ensemble would be faster, cheaper, more explainable, and more accurate.

The mental model that works in production is:

• Default to the simplest model that could plausibly solve the problem.
• Move toward complexity only when that model fails in a specific, measured way.
• Use LLMs for the genuinely hard language problems they are built for, and classical ML for everything that fits in a feature table.

That is not a conservative stance. It is an engineering one.