Modelplane Modelplane docs
Version
main v0.1 v0.2 v0.3 Latest

InferenceCluster Custom Resource

On this page

A Kubernetes cluster registered with Modelplane for model serving.

Concept guide: Register a Cluster →

#Metadata

API version
modelplane.ai/v1alpha1
Kind
InferenceCluster
Scope
Cluster
Short names
ic

#Example

Manifest
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: west-gke
spec:
  cluster:
    source: GKE
    gke:
      region: us-central1
  nodePools:
    - name: h100-pool
      className: h100-8x-byo
      nodeCount: 2
      maxNodeCount: 10
      zones: [us-central1-a]

#Spec

# cluster required object
# aks optional object

AKS cluster configuration. Required when source is AKS.

# credentials optional object

Azure ProviderConfig or ClusterProviderConfig used to authenticate to the Azure API. Defaults to the ClusterProviderConfig named default.

# name optional string 1–253 chars default: default
# type optional enum: ProviderConfig | ClusterProviderConfig default: ClusterProviderConfig
# kubernetesVersion optional string default: 1.34

AKS cluster Kubernetes version. Defaults to a version where Dynamic Resource Allocation (how GPUs bind to pods) is generally available.

# location required string 1–32 chars

Azure region for the cluster (e.g. westeurope, eastus2).

# eks optional object

EKS cluster configuration. Required when source is EKS.

# credentials optional object

AWS ProviderConfig or ClusterProviderConfig used to authenticate to the AWS API. Defaults to the ClusterProviderConfig named default.

# name optional string 1–253 chars default: default
# type optional enum: ProviderConfig | ClusterProviderConfig default: ClusterProviderConfig
# kubernetesVersion optional string default: 1.36

EKS cluster Kubernetes version. Defaults to a version where Dynamic Resource Allocation (how GPUs bind to pods) is generally available.

# region required string 1–32 chars

AWS region for the cluster (e.g. us-west-2).

# existing optional object

Bring-your-own cluster configuration. Required when source is Existing. Modelplane manages the inference stack on the cluster but does not provision the cluster itself.

# cache optional object

ModelCache configuration for this cluster.

# storageClassName optional string 1–253 chars

Name of an existing ReadWriteMany StorageClass for ModelCache PVCs. Modelplane doesn’t provision storage on an existing cluster, so the admin must create the StorageClass (it must support ReadWriteMany dynamic provisioning).

# identitySecretRef optional object

Optional reference to a Secret containing cloud provider credentials for IAM-based authentication. The type selects which cloud identity the ProviderConfigs authenticate as, and must match the cloud the existing cluster runs on.

# key optional string 1–253 chars default: private_key
# name required string 1–253 chars
# type optional enum: GoogleApplicationCredentials | AWSWebIdentityCredentials | NebiusServiceAccountCredentials default: GoogleApplicationCredentials

Cloud identity type the credential authenticates as. Must match the cloud the existing cluster runs on. Defaults to GoogleApplicationCredentials.

# secretRef required object

Reference to a Secret containing a kubeconfig for the existing cluster. The Secret must exist in the modelplane-system namespace.

# key optional string 1–253 chars default: kubeconfig
# name required string 1–253 chars
# gke optional object

GKE cluster configuration. Required when source is GKE.

# credentials optional object

GCP ProviderConfig or ClusterProviderConfig used to authenticate to the GCP API. Defaults to the ClusterProviderConfig named default.

# name optional string 1–253 chars default: default
# type optional enum: ProviderConfig | ClusterProviderConfig default: ClusterProviderConfig
# kubernetesVersion optional string default: 1.35
# region required string 1–32 chars
# nebius optional object

Nebius mk8s cluster configuration. Required when source is Nebius; may be empty, since every field has a default. The cluster is created in the project the referenced ProviderConfig or ClusterProviderConfig sets as its projectID; Nebius projects are bound to a region, so the project also determines where the cluster runs.

# credentials optional object

Nebius ProviderConfig or ClusterProviderConfig used to authenticate to the Nebius API. Defaults to the ClusterProviderConfig named default.

# name optional string 1–253 chars default: default
# type optional enum: ProviderConfig | ClusterProviderConfig default: ClusterProviderConfig
# kubernetesVersion optional string default: 1.34

mk8s cluster Kubernetes version. Defaults to a version where Dynamic Resource Allocation (DRA) is generally available.

# source required enum: GKE | EKS | AKS | Nebius | Vultr | Existing

Cluster provisioning method.

# vultr optional object

Vultr Kubernetes Engine (VKE) cluster configuration. Required when source is Vultr.

# credentials optional object

Vultr ProviderConfig or ClusterProviderConfig used to authenticate to the Vultr API. Defaults to the ClusterProviderConfig named default.

# name optional string 1–253 chars default: default
# type optional enum: ProviderConfig | ClusterProviderConfig default: ClusterProviderConfig
# kubernetesVersion optional string default: v1.36.1+3

VKE cluster Kubernetes version. VKE requires an exact version string including the build suffix; list current versions with vultr-cli kubernetes versions. Defaults to a version where Dynamic Resource Allocation (how GPUs bind to pods) is generally available.

# region required string 1–32 chars

Vultr region for the cluster (e.g. ewr, fra). GPU plans and Vultr File System availability vary by region.

# gateway optional object

Configuration for this cluster’s inference traffic gateway, which fronts the engine pods and is what an InferenceGateway forwards to.

# hostname optional string 1–253 chars

DNS name that resolves to this cluster’s gateway address, published by you once the cluster reports status.gateway.address. An InferenceGateway addresses this cluster by name rather than by address, so replicas here carry no traffic until the name resolves. A name is required because Envoy AI Gateway only applies per-backend model rewriting, credentials and priority failover when every backend in a route is addressed by hostname; given an address it silently stops applying them.

# nodePools optional object[] 1–8 items
# capacityBlock optional object

Capacity Block reservation backing this node pool. EKS only. Large GPU instances (e.g. p5en.48xlarge) are rarely available on demand; AWS allocates them via Capacity Blocks for ML. Set this to back the pool with a Capacity Block you have purchased. The pool’s zones must match the reservation’s Availability Zone, and nodeCount must not exceed the reserved instance count. Omit for on-demand pools.

# capacityReservationId required string 4–64 chars

The ID of the Capacity Reservation backing the Capacity Block (e.g. cr-0123456789abcdef0). Purchasing a Capacity Block yields this ID.

pattern: ^cr-[0-9a-f]+$

# className required string 1–253 chars

Name of the InferenceClass describing this pool’s hardware.

# fabric optional object

High-performance node-to-node fabric for multi-node engines, so a gang’s tensor-parallel traffic isn’t capped by TCP. Omit for standard VPC networking.

# infiniband optional object

InfiniBand fabric configuration. Required when type is InfiniBand and the cluster source is Nebius; not used on AKS.

# fabric required string 1–63 chars

Identifier of the physical InfiniBand fabric to join (e.g. fabric-2). This selects existing Nebius infrastructure, not a name for a new resource: fabrics are per-region - see https://docs.nebius.com/compute/clusters/gpu#fabrics - and multi-node GPU capacity is allocated on specific fabrics, so use the fabric your capacity lives on.

# type optional enum: None | EFA | InfiniBand default: None

Fabric technology. None uses standard VPC networking (TCP). EFA attaches Elastic Fabric Adapter interfaces to each node for GPUDirect RDMA across nodes; EKS only, and only useful on EFA-capable instance types (e.g. p5en.48xlarge). When any pool sets EFA, Modelplane installs the EFA DRA driver on the cluster and the gang’s pods claim EFA devices alongside their GPUs. InfiniBand places the pool’s nodes on a physical InfiniBand fabric for GPUDirect RDMA across nodes; Nebius and AKS only, and only useful on InfiniBand-capable shapes (e.g. gpu-h100-sxm on Nebius, Standard_ND96isr_H100_v5 on AKS). On Nebius the pool joins the fabric named in infiniband.fabric; on AKS the pool’s VM Scale Set placement group lands its nodes on one fabric, and Modelplane installs the NVIDIA network operator on the cluster.

# maxNodeCount optional integer ≥ 1

Maximum node count for autoscaling. Omit for fixed-size pools.

# minNodeCount optional integer
# name required string ≤ 40 chars
# nodeCount optional integer default: 1
# zones optional string[]
# placement optional object

Facts about where this cluster is, projected onto everything Modelplane composes here.

# metadata optional object

Metadata to project.

# labels optional map[string]string

Labels stamped onto every ModelReplica and ModelEndpoint composed on this cluster, so a fact about the cluster is declared once here rather than repeated on each of them. This is how a self-hosted endpoint gets its region: a ModelService selects endpoints by label, so a service scoped to a region selects only the endpoints in it. These are your labels, under your own prefix. Modelplane carries and matches them, and never interprets them, so “eu” means no more to it than “prod”.

# stack optional enum: Standard | Dynamo default: Standard

Which serving stack the cluster installs and composes. Standard (the default) is the Modelplane-composed serving layer: a Deployment or LeaderWorkerSet, Gateway API, and the endpoint picker. Dynamo swaps in NVIDIA’s components: Grove with the KAI Scheduler gang-schedules multi-node engines, and ModelExpress distributes weights. A single-node (Standalone) engine’s workload kind is unaffected - it stays a Deployment - but if it references a ModelCache on a Dynamo cluster it still gets the ModelExpress P2P env and IPC_LOCK, so it can seed peers and load from them like a gang.

# taints optional object[]
# effect required enum: NoSchedule | NoExecute
# key required string ≥ 1 chars
# value optional string

#Status

# cache optional object

Observed ModelCache RWX storage state.

# storageClassName optional string ≤ 253 chars

Effective ReadWriteMany StorageClass name for ModelCache PVCs on this cluster. ModelCache reads this to target the cache PVC.

# gateway optional object
# address optional string

External address of the inference gateway on the remote cluster. Point spec.gateway.hostname at this.

# caCertificate optional string ≤ 16384 chars

PEM certificate of the CA that signed this gateway’s serving certificate. An InferenceGateway validates against it, so it reaches the cluster it meant to rather than whatever else answers on that address. Written once cert-manager on the cluster has issued.

# hostname optional string

The name an InferenceGateway addresses this cluster’s gateway by, echoed from spec.gateway.hostname once the gateway has an address. ModelDeployment composes a ModelEndpoint origin from it, and withholds the endpoint while it’s unset.

# gpuPools optional object[] ≤ 8 items
# devices optional object[] ≤ 16 items
# attributes optional map[string]object
# capacity optional map[string]object
# claim optional enum: DRA | Synthetic
# count optional integer
# deviceClassName optional string ≤ 253 chars
# driver required string ≤ 253 chars
# name required string ≤ 63 chars
# name required string

Node pool name, matching spec.nodePools[].name. Used to pin a ModelReplica to a specific pool via spec.nodePoolName.

# nodes optional integer

Number of nodes in this pool. Derived from maxNodeCount (if autoscaling) or nodeCount.

# namespace optional string

Namespace where the internal XRs (cluster, backend) were created.

# providerConfigRef optional object
# name optional string

Name of the ProviderConfig targeting the remote cluster. Used by ModelReplica to create resources on the cluster.