ModelService Custom Resource
A ModelService is one model as a caller sees it: a stable name that resolves to whichever ModelEndpoint should serve the next request. The endpoints behind it can be replicas Modelplane runs, models bought from a provider, or both, in more than one region. A caller reaches it by naming it as the model in an ordinary OpenAI or Anthropic request to any InferenceGateway that serves it. There is no per-service address.
Concept guide: Expose a Model →
#Metadata
#Example
Manifest
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
name: qwen-72b
namespace: ml-team
labels:
# Matched by an InferenceGateway's serviceSelector. Your label, under your
# own prefix; Modelplane matches it and never interprets it.
example.org/region: eu
spec:
endpoints:
# Entries at the same priority share traffic by weight, so this pair is a
# 90/10 canary across two deployments.
- priority: 0
weight: 90
selector:
matchLabels:
modelplane.ai/deployment: qwen-72b
- priority: 0
weight: 10
selector:
matchLabels:
modelplane.ai/deployment: qwen-72b-next
# A higher priority is only tried when nothing below it has a healthy
# endpoint, which makes this provider a failover for the capacity above.
- priority: 1
selector:
matchLabels:
modelplane.ai/endpoint: together-qwen-72b
#Spec
Lower is preferred. Entries at the same priority share traffic by weight; a higher number is only tried when nothing below it has a healthy endpoint, which is what makes a provider a failover for capacity you run. A request that fails over is retried against the next endpoint and gets that endpoint’s own model name, credential and path. Retrying is only possible until the first byte reaches the caller, because after that the tokens are already sent, so a backend that dies mid-stream truncates the response instead.
Selects ModelEndpoints in this ModelService’s namespace. Scope a service to a region by selecting only endpoints in it; Modelplane stamps an InferenceCluster’s labels onto every endpoint composed there, so the region is declared once on the cluster.
Share of traffic for this entry relative to the other entries at the same priority, spread as evenly as possible across the endpoints it matches. A pair of entries weighted 90 and 10 is a canary. At least 1. A weight of 0 doesn’t deprioritise a backend, it drops it from the gateway’s load assignment entirely, which is indistinguishable from removing the entry and easy to mistake for parking it. Remove the entry instead.
#Status
Observed endpoint counts, across all priorities.
The name a caller passes as the request’s model. Namespaced, so two services can’t collide and the namespace serving a caller is legible in what it passes.