Managing Context, Tokens, and Cost in Production OpenAI Applications

This is the third post in our OpenAI API fundamentals series, building on your first completion and function calling.

We noted in the first post in this series that the API has no server-side memory — every request sends the complete conversation from scratch. That fact has a direct, compounding cost consequence that's easy to miss until a real bill arrives: a 50-turn conversation's 51st message doesn't cost the same as its 1st. It costs roughly 50 times more, because the entire history rides along with every single request.

Tokens: the actual unit everything is billed in

Neither words nor characters map directly onto cost — the actual unit is a token, roughly ¾ of a word in English on average, though it varies by language and content.

import tiktoken
 
encoding = tiktoken.encoding_for_model("gpt-4o")
tokens = encoding.encode("Explain what a load balancer does.")
print(len(tokens))

Both input tokens (everything you send: system prompt, conversation history, the current message) and output tokens (the model's response) count toward cost — and pricing per token differs between the two, with output tokens typically priced higher. This is why an application that sends a large system prompt on every single request, or accumulates a long conversation history, sees cost scale with total conversation length, not with the length of just the newest message.

Why context grows linearly, and why that's a real problem

messages = [{"role": "system", "content": "You are a helpful assistant."}]
 
def chat(user_message):
    messages.append({"role": "user", "content": user_message})
    response = client.chat.completions.create(model="gpt-4o", messages=messages)
    reply = response.choices[0].message.content
    messages.append({"role": "assistant", "content": reply})
    return reply

This is the naive, and extremely common, first implementation of a multi-turn chat: append every message, forever, and send the whole growing list every time. It works correctly. It also means turn 50 sends roughly 50 times the tokens of turn 1 — a cost curve growing without bound for as long as the conversation continues, entirely invisible until someone actually looks at token usage or the bill.

Fixing it: trimming, summarizing, or both

Sliding window — keep only the most recent N messages:

def trim_history(messages, keep_last=10):
    system = [m for m in messages if m["role"] == "system"]
    recent = [m for m in messages if m["role"] != "system"][-keep_last:]
    return system + recent

Simple, and appropriate when older conversation context genuinely stops being relevant after a while — a support chat about today's specific issue rarely needs message 1 by message 40.

Summarization — periodically compress older turns into a shorter summary, using the model itself:

def summarize(messages_to_compress):
    summary_prompt = "Summarize this conversation in 2-3 sentences, keeping key facts:"
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # a smaller, cheaper model for a simple internal task
        messages=[
            {"role": "system", "content": summary_prompt},
            {"role": "user", "content": str(messages_to_compress)},
        ],
    )
    return response.choices[0].message.content

Preserves the gist of older context at a fraction of the token cost of the original messages, and — notice the model choice — this internal, low-stakes summarization step is exactly the right place to use a smaller, cheaper model rather than defaulting to your main, more expensive one for every single call in the system.

Matching model choice to task, deliberately

Not every call in a real application needs the most capable available model. A classification task ("is this support ticket urgent, yes or no") or the summarization step above genuinely doesn't need the same model handling nuanced customer-facing conversations. Routing internal, well-defined, lower-stakes tasks to a smaller model (gpt-4o-mini rather than gpt-4o, in OpenAI's current lineup) is one of the highest-leverage, lowest-effort cost optimizations available — often a 90%+ per-token cost reduction for tasks that don't need the larger model's additional capability at all.

Prompt caching: paying less for repeated context

For applications sending the same large system prompt or reference document on every request, OpenAI's prompt caching automatically discounts the portion of input tokens that exactly match a previous request within a short time window — a meaningful cost reduction for exactly the "large, mostly-static system prompt sent on every call" pattern that's extremely common in real applications, requiring no code change to benefit from beyond structuring the static portion of a prompt consistently.

What to actually remember from this post

Tokens, not words or characters, are the actual billing unit — and both input and output tokens count, typically at different rates.
An unbounded conversation history means unbounded, linearly-growing per-turn cost — invisible until someone actually checks usage.
Sliding windows and periodic summarization are the two standard fixes — trim what's no longer relevant, or compress it, rather than sending the entire unbounded history forever.
Route internal, low-stakes tasks (classification, summarization) to a smaller, cheaper model — one of the highest-leverage cost optimizations available, and one of the easiest to implement.

Next in the series: Building a Retrieval-Augmented Generation (RAG) Pipeline with OpenAI, where we cover how to ground the model's answers in your own actual documents, rather than relying on what it happened to learn during training.