Why it matters
The same model can be deployed as a shared API, a dedicated endpoint, a container in a customer's network or a quantized file on a laptop, and each choice changes latency, data handling, upgrade cadence and cost. Teams often pick the mode by convenience and discover later that a contract requires isolation, or that an edge deployment cannot hold the context window the product needs. The decision belongs with the product requirements.
Mental model
It is choosing between a shared office, a leased floor and a home study: the work is the same, the neighbours and the maintenance are not. The analogy breaks because a shared office cannot see your documents, while a shared inference endpoint can, which is why data terms are part of the choice.
How it works
Event · step 1 of 6
One revision, then the shared upgrade
At 4 wk
Four ways to serve the same model started on supported revisions: shared managed, dedicated, private and edge. At week 4 the provider moved the shared endpoint to v44 with no deploy in your repository and no action from your team — the shortest upgrade path and the least visible one.
On screen at this event
- v43
- v44
- v42
- v41
- v41
- supported
Next at 6 wk: Dedicated lags, on purpose
- Shared
- v44
- Dedicated
- v42
- Private
- v41
- Edge
- v41
- Unsupported
- 0
Figure. A synthetic 24-week history. The shared endpoint upgrades without you, the dedicated endpoint follows your rollout, and the private and edge deployments move only when someone schedules the work. v41 support ends at week 12; isolation is a separate axis and is not drawn.
What to notice. The same model reaches customers on four cadences. At week 12 two modes are on an unsupported revision. The private cluster recovers when its owner acts; the edge waits for an app release and installs. Deployment mode and rollout strategy are different decisions: where the model runs determines who can upgrade it, and how a version reaches traffic is a separate choice with its own failure.
Text version
Five lanes over 24 weeks. Shared serves v43 for 0–4, v44 for 4–8 and v45 for 8–24. Dedicated serves v42 for 0–6, v43 for 6–14 and v44 for 14–24. The private cluster serves v41 for 0–12, a red v41 unsupported span 12–20, and v42 for 20–24. The edge serves v41 for 0–12 and a red v41 stranded span 12–24. The support lane is v41 supported 0–12, v41 EOL 12–24, and v42 supported 20–24 on a second row, with a red marker v41 support ends at 12. Counters: shared v44, v44, v45, v45, v45, v45; dedicated v42, v43, v43, v44, v44, v44; private v41 until week 20 then v42; edge v41 throughout; unsupported lanes 0, 0, 2, 2, 1, 1.
- Classify the data and the contractual obligations first: residency, retention, isolation and audit.
- List the model capabilities required and check which modes can serve them at the context length you need.
- Choose the rollout mechanism independently: rolling for stateless replicas, blue-green for schema changes, canary for behaviour changes.
- Decide who patches, scales and gets paged, and put that in the cost comparison.
- Test the mode's failure behaviour: what happens when the shared endpoint throttles, or when the private cluster loses a node.
- Document the upgrade path, because a private deployment can strand you on an old model.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Shared managed endpoint | Fastest to start, no operations | Shared quota and limited isolation | A customer requires dedicated capacity |
| Dedicated endpoint | Isolation and predictable capacity | Paid for whether used or not | Load is intermittent |
| Private network deployment | Data stays in the customer boundary | You own upgrades and support | You cannot staff the operations |
| Edge or on-device | No network dependency, data local | Small models and device constraints | The task needs a large context or model |
Good for
Fastest to start, no operations
Cost
Shared quota and limited isolation
Fails when
A customer requires dedicated capacity
- A customer requires dedicated capacity
- Load is intermittent
- You cannot staff the operations
- The task needs a large context or model
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Shared managed endpoint is good for Fastest to start, no operations, costs Shared quota and limited isolation, and fails when A customer requires dedicated capacity. Dedicated endpoint is good for Isolation and predictable capacity, costs Paid for whether used or not, and fails when Load is intermittent. Private network deployment is good for Data stays in the customer boundary, costs You own upgrades and support, and fails when You cannot staff the operations. Edge or on-device is good for No network dependency, data local, costs Small models and device constraints, and fails when The task needs a large context or model
Failure modes
- Mode chosen after the model. The selected model does not fit the deployment target and the plan restarts. Debug: check the context length and memory requirement against the target's documented limits before design.
- Upgrade stranding. A private deployment stays on an old version because nobody owns the upgrade. Debug: list deployed model revisions per environment and flag any past the provider's support window.
- Shared quota surprise. A dedicated-looking endpoint shares a quota with other workloads and throttles under peak. Debug: load-test at peak concurrency and read the provider's limit documentation for that endpoint type.
In production
Record the deployment mode and model revision per environment, and review them together. Rehearse the failure the mode makes possible, whether that is throttling on a shared endpoint or a node loss in a private cluster.
Common misconceptions
People think: Private deployment is automatically more secure.
Actually: It changes the trust boundary but adds patching, access control and monitoring obligations that a managed service would carry.
People think: Deployment mode and rollout strategy are the same decision.
Actually: Where the model runs and how a new version reaches traffic are independent choices with different failure modes.
Interview angle
Separate isolation from rollout, name the requirement that forces each, and account for the operational burden. Explain why the mode constrains model choice. Picking a hosting option and leaving rollout to nobody is the weak version.
Sources
- Martin Fowler, BlueGreenDeployment — the two-environment release pattern.
- Kubernetes, Deployments — rolling update semantics and replica management.
- AWS Well-Architected, SaaS Lens — tenant isolation models and their operational cost.