# ModelService Source: /reference/modelservices/ A ModelService is one model as a caller sees it: a stable name that resolves to whichever ModelEndpoint should serve the next request. The endpoints behind it can be replicas Modelplane runs, models bought from a provider, or both, in more than one region. A caller reaches it by naming it as the model in an ordinary OpenAI or Anthropic request to any InferenceGateway that serves it. There is no per-service address. Apply instances as `apiVersion: modelplane.ai/v1alpha1`, `kind: ModelService`. [Concept guide: Expose a Model](/models/model-service/index.md) ## Example ```yaml apiVersion: modelplane.ai/v1alpha1 kind: ModelService metadata: name: qwen-72b namespace: ml-team labels: # Matched by an InferenceGateway's serviceSelector. Your label, under your # own prefix; Modelplane matches it and never interprets it. example.org/region: eu spec: endpoints: # Entries at the same priority share traffic by weight, so this pair is a # 90/10 canary across two deployments. - priority: 0 weight: 90 selector: matchLabels: modelplane.ai/deployment: qwen-72b - priority: 0 weight: 10 selector: matchLabels: modelplane.ai/deployment: qwen-72b-next # A higher priority is only tried when nothing below it has a healthy # endpoint, which makes this provider a failover for the capacity above. - priority: 1 selector: matchLabels: modelplane.ai/endpoint: together-qwen-72b ``` ## Definition The CompositeResourceDefinition this reference is generated from, with the complete OpenAPI schema, validation rules, and defaults: ```yaml apiVersion: apiextensions.crossplane.io/v2 kind: CompositeResourceDefinition metadata: name: modelservices.modelplane.ai spec: group: modelplane.ai names: categories: [crossplane, modelplane] kind: ModelService plural: modelservices shortNames: [ms] scope: Namespaced versions: - name: v1alpha1 served: true referenceable: true additionalPrinterColumns: - name: MODEL type: string jsonPath: .status.model schema: openAPIV3Schema: description: >- A ModelService is one model as a caller sees it: a stable name that resolves to whichever ModelEndpoint should serve the next request. The endpoints behind it can be replicas Modelplane runs, models bought from a provider, or both, in more than one region. A caller reaches it by naming it as the model in an ordinary OpenAI or Anthropic request to any InferenceGateway that serves it. There is no per-service address. type: object required: [spec] properties: spec: type: object required: [endpoints] properties: endpoints: type: array description: >- A priority order over ModelEndpoints, each entry selecting a set of them by label. The two knobs work on different timescales. priority is failure: a tier is only used once the tiers above it have no healthy endpoints left. weight is everything that isn't failure, and is how you shift traffic deliberately, whether canarying a new deployment or preferring capacity you've already paid for until it stops keeping up. Modelplane never adjusts a weight. It is whatever it was last written to be, by a person or by something watching the fleet's load and cost. minItems: 1 maxItems: 32 items: type: object required: [selector] properties: priority: type: integer description: >- Lower is preferred. Entries at the same priority share traffic by weight; a higher number is only tried when nothing below it has a healthy endpoint, which is what makes a provider a failover for capacity you run. A request that fails over is retried against the next endpoint and gets that endpoint's own model name, credential and path. Retrying is only possible until the first byte reaches the caller, because after that the tokens are already sent, so a backend that dies mid-stream truncates the response instead. minimum: 0 maximum: 63 default: 0 weight: type: integer description: >- Share of traffic for this entry relative to the other entries at the same priority, spread as evenly as possible across the endpoints it matches. A pair of entries weighted 90 and 10 is a canary. At least 1. A weight of 0 doesn't deprioritise a backend, it drops it from the gateway's load assignment entirely, which is indistinguishable from removing the entry and easy to mistake for parking it. Remove the entry instead. minimum: 1 maximum: 1000000 default: 1 selector: type: object description: >- Selects ModelEndpoints in this ModelService's namespace. Scope a service to a region by selecting only endpoints in it; Modelplane stamps an InferenceCluster's labels onto every endpoint composed there, so the region is declared once on the cluster. required: [matchLabels] properties: matchLabels: type: object additionalProperties: type: string maxLength: 63 minProperties: 1 maxProperties: 16 status: type: object properties: model: type: string description: >- The name a caller passes as the request's model. Namespaced, so two services can't collide and the namespace serving a caller is legible in what it passes. gateways: type: array description: >- The InferenceGateways serving this service, which is every gateway whose serviceSelector matches it. Empty means no gateway serves this service and no caller can reach it. items: type: object properties: name: type: string hostname: type: string description: The name that gateway answers on, if it has one. address: type: string endpoints: type: object description: Observed endpoint counts, across all priorities. properties: total: type: integer ready: type: integer conditions: type: array items: type: object ```