Training vs. Inference: What Actually Happens When You Ask an LLM a Question

This is the eighth post in our AI & ML Foundations series, building on tokenization. Start with What Is Machine Learning, Really? if you're joining partway through.

Ben ran platform infrastructure at a startup that had built a customer-facing chatbot on top of a fine-tuned language model, and the meeting that prompted this post started with a genuinely reasonable question from finance: "the chatbot answers in under a second — why would updating it to handle our new product line take three weeks and cost several thousand dollars in compute?"

It was a fair question, and the honest answer required drawing a distinction Ben himself hadn't fully separated in his own head until he had to explain it clearly: training and inference are not the same process running at different speeds. They're two entirely different activities, with different costs, different infrastructure, and different things actually happening inside the model.

Training: the slow, expensive process that only happens occasionally

Every technique covered earlier in this series — gradient descent, backpropagation, adjusting millions of weights one small step at a time — is training. It happens rarely (once, or periodically when retraining on new data), it's genuinely expensive (specialized hardware running for hours, days, or weeks), and its entire output is a single artifact: a file full of learned weights.

# Training: happens rarely, produces a static artifact
model.fit(
    training_data,
    epochs=50,
    batch_size=64,
)
model.save("chatbot_weights_v3.h5")

Think of training the way you'd think about a student's entire years of schooling — years of exposure to material, gradually adjusting their internal understanding, culminating in something (call it "what they now know") that then gets used repeatedly, long after the studying itself has stopped.

Inference: the fast, cheap process that happens constantly

Inference is using an already-trained model to actually answer a question — no weight updates, no gradients, no loss function. The weights are frozen; the model reads them and computes an answer.

# Inference: happens constantly, uses the already-learned weights, doesn't change them
model = keras.models.load_model("chatbot_weights_v3.h5")
response = model.predict(user_question)

This is the exam itself — the student walks in with everything they've already learned, applies it to a new question they've never seen phrased exactly that way before, and writes an answer. No further studying happens during the exam. That's precisely why Ben's chatbot could answer in under a second: it was doing inference, a single forward pass through an already-fixed set of weights, not re-learning anything on the spot.

Why the cost structure of the two is genuinely so different

Training touches every weight, repeatedly, across the entire training dataset, for many passes (epochs) — Ben's fine-tuning run adjusted every one of the model's parameters, using every example in the new product-line documentation, dozens of times over, which is precisely why it needed specialized GPU hardware running for days. Inference, by contrast, is a single forward pass — the input flows through the network's fixed layers exactly once, producing an output, with no repeated passes and no per-example dataset to iterate over at all.

import time
 
# Training: 50 epochs over 10,000 examples — genuinely heavy computation
start = time.time()
model.fit(training_data, epochs=50)
print(f"Training took {time.time() - start:.0f} seconds")  # e.g., 14400 (4 hours)
 
# Inference: one forward pass on one input
start = time.time()
model.predict(single_question)
print(f"Inference took {time.time() - start:.3f} seconds")  # e.g., 0.180

This asymmetry is exactly why an application calling a hosted model API pays per-token for inference, continuously, while a fine-tuning job is billed as a separate, one-time (or periodic) training cost — they're genuinely different operations with genuinely different resource footprints, not the same meter running at different speeds.

Why an LLM can't actually "learn" from a single conversation

This distinction also explains a common misconception Ben had to clear up for his own product team: when a user corrects a chatbot mid-conversation ("no, I meant our enterprise plan"), the model does not update its weights based on that correction. It has no mechanism to learn from a live conversation at all — inference never touches the weights. What actually happens is that the correction becomes part of the context (the conversation history sent along with every subsequent message, covered in our OpenAI series), which the model reads fresh, alongside its already-frozen knowledge, on every single turn. The moment the conversation ends, that context is gone; the model's actual weights are exactly as they were before the conversation started.

# The correction lives in the conversation context, NOT in the model's weights
messages = [
    {"role": "user", "content": "What's included in the standard plan?"},
    {"role": "assistant", "content": "The standard plan includes..."},
    {"role": "user", "content": "No, I meant the enterprise plan."},
]
# The model reads all three messages fresh, every time — it hasn't "learned" anything permanent

Where the two processes actually meet: fine-tuning

Ben's three-week, several-thousand-dollar update was fine-tuning — a real, if narrower, training run: taking an already-trained base model and running additional gradient descent steps against new, product-specific examples, producing an updated set of weights that then gets used for inference going forward. It's genuinely training, just starting from a more useful point than random initialization, which is exactly why it's faster and cheaper than training a model from scratch — and still meaningfully slower and more expensive than a single inference call, because it's still adjusting every weight across many passes over real data, the same fundamental process covered throughout this series.

The infrastructure difference, made concrete

The distinction Ben had to explain to finance shows up directly in what each process actually runs on:

# Training: a dedicated, temporary cluster of GPUs, running for hours or days
training_job = launch_training_job(
    instance_type="8x-a100-gpu",
    duration_hours=96,
    dataset=new_product_docs,
)
 
# Inference: a much smaller, always-on service, answering requests continuously
inference_endpoint = deploy_model(
    instance_type="1x-t4-gpu",
    autoscaling=True,  # scales with request volume, not with dataset size
)

Training infrastructure is provisioned for a bounded job and released when it finishes — the same disposable-compute pattern covered for CI runners and Docker containers elsewhere on this blog. Inference infrastructure stays running continuously, sized to handle request volume, completely independent of how large the original training dataset was — a model trained on ten million examples and one trained on ten thousand can serve inference from identically-sized infrastructure, because inference never touches that training data again at all.

Why "just retrain it more often" isn't a free fix

Once Ben's team understood the real cost structure, the temptation was to solve every accuracy complaint by retraining more frequently. This is worth resisting as a reflexive default: every retraining run costs real GPU time, and a model retrained on a schedule regardless of whether anything meaningful actually changed is spending real money for often-marginal benefit. The better instinct, covered in more depth in the next post in this series, is measuring whether a retrain would actually move a meaningful evaluation metric before committing the compute budget — treating retraining as a deliberate, evaluated decision, not a default maintenance chore run on a fixed calendar regardless of evidence it's needed.

What to actually remember from this post

• Training adjusts a model's weights using gradient descent over many passes of data — rare, slow, and genuinely expensive; its output is a static artifact.
• Inference uses an already-trained model's frozen weights to answer one input at a time — constant, fast, and cheap per call, with no weights changing at all.
• A model cannot learn from a single conversation — a mid-conversation correction lives in the context sent alongside each message, not in the model's permanent weights, and disappears once the conversation ends.
• Fine-tuning is genuine training, just starting from a more useful point than random initialization — which is why it's faster than training from scratch, and still meaningfully more expensive than ordinary inference.

Next in the series: Evaluating Models: Why Accuracy Lies, where a fraud-detection model scores 99.9% accurate and still catches almost nothing.