SageMaker Pipelines: Automating the ML Lifecycle

This is the fourth post in our Amazon SageMaker series, building on notebooks and training jobs, built-in algorithms, and deployment endpoints.

Every step covered so far in this series — preprocessing, training, evaluation, deployment — has been a manually-run notebook cell, executed by a person, in an order they have to remember correctly, using the exact right parameters each time. This is the same fragile, undocumented, "hope the person doing it remembers the steps" pattern this blog has argued against repeatedly, for Terraform applied by hand, for a deploy process run manually before we automated it with GitHub Actions in our own real case study. SageMaker Pipelines is the fix for the ML lifecycle specifically.

Defining a pipeline as a sequence of steps

from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep, TrainingStep
from sagemaker.workflow.step_collections import RegisterModel
from sagemaker.processing import ScriptProcessor
 
processor = ScriptProcessor(
    image_uri="...",
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
)
 
preprocessing_step = ProcessingStep(
    name="PreprocessData",
    processor=processor,
    inputs=[...],
    outputs=[...],
    code="preprocessing.py",
)
 
training_step = TrainingStep(
    name="TrainModel",
    estimator=estimator,
    inputs={
        "train": preprocessing_step.properties.ProcessingOutputConfig.Outputs["train"].S3Output.S3Uri,
    },
)
 
register_step = RegisterModel(
    name="RegisterModel",
    estimator=estimator,
    model_data=training_step.properties.ModelArtifacts.S3ModelArtifacts,
    content_types=["text/csv"],
    response_types=["text/csv"],
    model_package_group_name="churn-prediction-models",
)
 
pipeline = Pipeline(
    name="churn-prediction-pipeline",
    steps=[preprocessing_step, training_step, register_step],
)
 
pipeline.upsert(role_arn=role)
pipeline.start()

This is genuinely the same conceptual shape as a CI/CD workflow — a defined sequence of steps, each depending on the previous step's output, executed automatically rather than by a person manually clicking through notebook cells in order. preprocessing_step.properties... referencing the previous step's actual output is the pipeline equivalent of a job's needs: dependency in GitHub Actions, covered directly earlier in this blog: an explicit, code-defined dependency, not an implicit assumption about execution order that only exists in someone's memory.

Conditional steps: only registering a model that's actually good enough

from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
from sagemaker.workflow.functions import JsonGet
 
evaluation_step = ProcessingStep(
    name="EvaluateModel",
    processor=processor,
    inputs=[...],
    outputs=[...],
    code="evaluate.py",
)
 
condition = ConditionGreaterThanOrEqualTo(
    left=JsonGet(step_name=evaluation_step.name, property_file="evaluation.json", json_path="metrics.auc"),
    right=0.75,
)
 
condition_step = ConditionStep(
    name="CheckModelQuality",
    conditions=[condition],
    if_steps=[register_step],
    else_steps=[],
)

This is the ML pipeline equivalent of a quality gate in any other CI/CD process — the same instinct behind requiring tests to pass before a deploy proceeds, or requiring a Molecule test to pass before an Ansible role change ships, covered earlier in this blog. A newly-trained model only gets registered (and becomes eligible for deployment) if its evaluated AUC meets a defined bar — a model that trained "successfully" but performs worse than the bar never reaches production, automatically, with no person needing to remember to manually check a metric before deciding whether to deploy.

Triggering pipelines automatically

from sagemaker.workflow.triggers import PipelineSchedule
 
pipeline.put_triggers(
    triggers=[PipelineSchedule(schedule_expression="cron(0 2 * * ? *)")],
)

A pipeline scheduled to run automatically — nightly here — retrains a model against fresh data on a defined cadence, without a person needing to remember to kick off the process manually. This is exactly the "rebuild and rescan on a schedule, independent of whether anyone remembered to trigger it" principle recommended for Docker base images earlier in this blog, applied here to model retraining specifically — genuinely important for a model whose accuracy can quietly degrade over time as the real-world data distribution it's predicting against shifts.

Model Registry: versioning models the way you'd version any other artifact

client = boto3.client("sagemaker")
response = client.list_model_packages(ModelPackageGroupName="churn-prediction-models")
for pkg in response["ModelPackageSummaryList"]:
    print(pkg["ModelPackageArn"], pkg["ModelApprovalStatus"])

Every model that passes the quality gate gets registered as a versioned model package — approved for deployment, or left pending an explicit human approval step, the same manual-approval-gate pattern covered for GitHub Environments and LangGraph's human-in-the-loop interrupts elsewhere in this blog. This gives an organization an actual audit trail: which specific model version is currently deployed, when it was trained, what data and evaluation metrics produced it — rather than a deployed model with no clear record of how it got there.

What to actually remember from this post

A Pipeline replaces manually-run notebook cells with an explicit, automated, code-defined sequence of steps — the same fix GitHub Actions provides for a manual deploy process, applied to the ML lifecycle specifically.
Conditional steps act as quality gates, registering a model only if it meets a defined performance bar — the ML equivalent of requiring tests to pass before a deploy proceeds.
Scheduled pipeline runs keep a model current against a shifting real-world data distribution, without relying on a person remembering to manually retrain it.
The Model Registry provides a genuine audit trail of every trained model version, its approval status, and what produced it.