Yesterday built and registered models; today they meet traffic. Serving is where ML becomes an availability problem, and Azure ML’s endpoint machinery imports the deployment discipline this series established for applications: traffic splitting, safe rollout, autoscale, and honest SKU economics.

Online Endpoints and Safe Rollout

A managed online endpoint is a stable scoring URL fronting one or more deployments, each a registered model plus environment plus instance configuration, with traffic percentages between them. That structure is the blue green machinery from the App Service and Container Apps posts wearing ML clothes: deploy the challenger alongside the champion at zero percent, validate with mirrored traffic (copies of live requests scored but not returned, the highest fidelity test that exists), shift ten percent, watch the model metrics beside the service metrics, then promote or roll back by editing percentages, never by redeploying. Authentication is Entra tokens with managed identity from callers, the endpoint reachable through private link inside the network like every other internal API, and the scoring container’s dependencies pinned by the environment from yesterday, so the artifact that passed evaluation is bit for bit the artifact serving.

Batch Endpoints: The Unsung Majority

Most inference is not a user waiting: nightly propensity scores for the CRM, weekly demand forecasts, embeddings over the document corpus for the search index. Batch endpoints serve that shape, jobs over data assets running on autoscaling cluster compute that returns to zero, outputs landing in the lake for downstream consumption, triggered by the Data Factory and Event Grid machinery from earlier months. The economics are the point: a real time endpoint sized for a nightly job burns capacity 23 hours for nothing, while the batch endpoint spins up spot instances, chews the parquet, and vanishes. The decision rule is the latency budget: humans or systems blocked on the answer in under a second, online; everything else defaults to batch until proven otherwise, and the proof is a requirement, not a preference.

Capacity, Cost, and the AKS Question

SKU choice follows measurement: CPU instances serve most classical models and small transformers comfortably once the scoring code is profiled (the usual sin is preprocessing in Python loops, not the model), GPUs earn their cost for deep models at throughput, and the concurrency settings per instance, workers, max concurrent requests, matter as much as the SKU. Autoscale on the endpoint tracks utilization or schedule with the asymmetric thresholds this series always recommends, minimum instances two for anything with an SLA, and scale to zero reserved for the dev endpoints where cold starts are tolerable. The AKS question resolves by ownership: managed endpoints for teams that want serving as a product, the Kubernetes online deployments (or your own serving stack on the AKS from October) when the platform team already operates clusters and wants the mesh, the GitOps flow, and the Karpenter economics to apply to inference too, with the honest note that you then own the serving layer’s upgrades and incidents. Either way the observability contract is fixed: request logs and latency percentiles to the monitoring stack, model inputs and outputs sampled to the drift monitors from yesterday, and cost per thousand inferences on the dashboard next to cost per transaction, because a model that is accurate but uneconomic is a prototype that escaped.

Cheers
Osama