This is the third post in our Amazon SageMaker series, building on notebooks and training jobs and built-in algorithms.
A trained model sitting in S3 isn't useful to anything until it's deployed somewhere that can actually serve predictions. SageMaker offers several distinct deployment shapes, each suited to a genuinely different traffic pattern — choosing the wrong one for a given workload means either paying for idle capacity or accepting latency a use case can't actually tolerate.
Real-time endpoints: always on, lowest latency
predictor = estimator.deploy(
initial_instance_count=2,
instance_type="ml.m5.large",
endpoint_name="churn-prediction-endpoint",
)
result = predictor.predict(new_customer_data)A real-time endpoint provisions instances that stay running continuously, ready to respond with the lowest possible latency at any moment. This is the right shape for genuinely latency-sensitive, continuous, unpredictable traffic — a live production application where a user is actively waiting on a response. It's also, per the previous post in this series, the shape most likely to accrue real, ongoing cost for capacity that sits idle outside of actual traffic — appropriate specifically when that always-on cost is justified by genuinely continuous demand.
Serverless inference: scale-to-zero, cold-start trade-off
from sagemaker.serverless import ServerlessInferenceConfig
serverless_config = ServerlessInferenceConfig(
memory_size_in_mb=2048,
max_concurrency=5,
)
predictor = estimator.deploy(
serverless_inference_config=serverless_config,
endpoint_name="churn-prediction-serverless",
)A serverless endpoint scales down to zero provisioned capacity during idle periods, and back up automatically when a request arrives — billed only for actual inference compute time, genuinely analogous to the disposable, pay-for-what-you-use model covered for training jobs, Docker containers, and GitHub Actions runners throughout this blog. The real trade-off: a request arriving after idle time incurs a cold start — the time to provision capacity before that first request can actually be served — meaningfully higher latency than an already-warm real-time endpoint. This is the right choice for genuinely intermittent, unpredictable traffic (a handful of calls per hour, batch-adjacent use cases) where that occasional cold-start latency is an acceptable trade for not paying for continuously idle capacity.
Asynchronous inference: for requests that genuinely take a while
from sagemaker.async_inference import AsyncInferenceConfig
async_config = AsyncInferenceConfig(
output_path=f"s3://{bucket}/async-predictions",
max_concurrent_invocations_per_instance=4,
)
predictor = estimator.deploy(
async_inference_config=async_config,
instance_type="ml.m5.xlarge",
initial_instance_count=1,
)
response = predictor.predict_async(input_path=f"s3://{bucket}/large-batch-input.csv")For requests that genuinely take longer than a typical synchronous HTTP call should reasonably block for — processing a large input payload, a computationally heavy model — asynchronous inference queues the request and writes the result to S3 once complete, rather than holding a connection open and waiting. This is a genuinely different pattern than either real-time or serverless: neither of those is well-suited to a request that might legitimately take minutes to process.
Multi-model endpoints: consolidating cost across many models
from sagemaker.multidatamodel import MultiDataModel
mdm = MultiDataModel(
name="multi-churn-models",
model_data_prefix=f"s3://{bucket}/multi-models/",
role=role,
)
predictor = mdm.deploy(initial_instance_count=1, instance_type="ml.m5.xlarge")
result = predictor.predict(data, target_model="customer-segment-a.tar.gz")An organization serving many similar models — one per customer segment, one per region — behind separate real-time endpoints pays for each endpoint's infrastructure independently, even though most of them see comparatively low individual traffic. A multi-model endpoint hosts many models behind one endpoint's infrastructure, loading a specific model's artifact into memory on demand based on the target_model parameter — meaningfully reducing infrastructure cost for exactly this "many similar, individually lower-traffic models" pattern, at the cost of a load-time delay the first time a specific model is requested after being evicted from memory.
Choosing between them
- Real-time: continuous, latency-sensitive, unpredictable traffic — a live production application.
- Serverless: intermittent, low-volume traffic where occasional cold-start latency is an acceptable trade for not paying for idle capacity.
- Asynchronous: requests that genuinely take longer than a synchronous call should reasonably block for.
- Multi-model: many similar models, each individually lower-traffic, where consolidating infrastructure meaningfully reduces total cost.
What to actually remember from this post
- Real-time endpoints stay running continuously — lowest latency, at the cost of paying for capacity even during idle periods.
- Serverless endpoints scale to zero, trading occasional cold-start latency for genuinely pay-per-use cost — the right fit for intermittent traffic specifically.
- Asynchronous inference is for requests too long for a synchronous call — queued, with results delivered to S3 rather than blocking a connection.
- Multi-model endpoints consolidate infrastructure cost across many similar, individually lower-traffic models behind one endpoint.
Next in the series: SageMaker Pipelines: Automating the ML Lifecycle, where training and deployment stop being manual notebook steps and become a repeatable, automated pipeline.