Enterprise

HAMi Enterprise Air-Gap Deployment Guide

This document is intended for SRE / platform engineers and describes how to deploy HAMi Enterprise in a Kubernetes cluster using the All-in-One air-gap package, complete license activation, enable GPU nodes, and verify with sample workloads.

This delivery package uses Zarf to perform image import, Helm chart installation, and subsequent upgrades in environments without external network access or with restricted networks, reducing the manual effort of syncing images and maintaining installation order.

Zarf is an application packaging and deployment tool for Kubernetes air-gap / semi-air-gap environments; it can bundle images, Helm charts, scripts, and deployment actions into a portable package.

Installation itself does not require a license. Complete the software deployment first, then apply for and import the license.

Without activation, vGPU partitioning and scheduling are unavailable, and functional verification will fail.

Air-Gap Package Contents

The outer delivery package is named as follows:

hami-enterprise-v<VERSION>-airgap-<ARCH>.tar.gz
hami-enterprise-v<VERSION>-airgap-<ARCH>.tar.gz.sha256

The current hami-enterprise-v0.0.3-airgap-amd64.tar.gz contains the following key files:

File	Purpose
`zarf-linux-amd64`	Linux amd64 Zarf CLI
`zarf-init-amd64-v0.82.0.tar.zst`	Zarf init air-gap package
`hami-enterprise-v0.0.3-airgap-amd64.tar.zst`	HAMi Enterprise main deployment package
`zarf-package-hami-example-gpu-burn-amd64-v0.0.2.tar.zst`	GPU burn sample verification package
`zarf-package-hami-example-vllm-qwen-amd64-v0.0.4.tar.zst`	vLLM + Qwen sample verification package
`collect-hami-license-info.sh`	License request information collection script
`collect-cluster-info.sh`	Cluster diagnostic information collection script
`hami/README.md`	hami-enterprise complete values documentation

You can use the following command to inspect the contents of the outer package and confirm the delivery is complete:

tar -tvf hami-enterprise-v0.0.3-airgap-amd64.tar.gz

Prerequisite Checks

Kubernetes version 1.24 or later, with working kubectl and Helm on the administration host.
NVIDIA nodes already have the NVIDIA driver and NVIDIA Container Toolkit, or NVIDIA GPU Operator will install them.
Ascend nodes already have the vendor driver and runtime configured and are visible to Kubernetes. This guide installs only the HAMi Ascend device plugin.
When NVIDIA GPU Operator is used, its default device plugin must be disabled to avoid a conflict with HAMi.

kubectl cluster-info
kubectl get nodes -o wide
helm version

Extract, Verify, and Install Zarf

# Download the delivery package and checksum file
curl -L -O <URL>
curl -L -O <SHA256_URL>

# Verify integrity
shasum -a 256 -c hami-enterprise-v0.0.3-airgap-amd64.tar.gz.sha256

# Extract the outer tar.gz
tar -xzf hami-enterprise-v0.0.3-airgap-amd64.tar.gz

# Enter the extracted directory
cd hami-enterprise-v0.0.3-airgap-amd64

The delivery package already includes the Linux amd64 version of the Zarf CLI. After entering the extracted directory, install the bundled Zarf first:

chmod +x ./zarf-linux-amd64
sudo install -m 0755 ./zarf-linux-amd64 /usr/local/bin/zarf
zarf version

Zarf includes the Helm tool. For subsequent troubleshooting of Helm release, values, and chart status, use zarf tools helm to avoid relying on a separately installed Helm in the target environment:

zarf tools helm version
zarf tools helm list -A

Initialize Zarf

Before deploying a Zarf package to the target cluster for the first time, run zarf init. We recommend the labeled policy so that image references are rewritten only for explicitly labeled namespaces or workloads.

Use the Zarf built-in registry:

zarf init zarf-init-amd64-v0.82.0.tar.zst \
  --agent-mutation-policy=labeled \
  --confirm

Use an external registry:

zarf init zarf-init-amd64-v0.82.0.tar.zst \
  --agent-mutation-policy=labeled \
  --registry-url=harbor.example.com/zarf-amd64 \
  --registry-push-username=<username> \
  --registry-push-password=<password> \
  --confirm

💡 Mixed-architecture clusters must use separate registry prefixes. AMD64 and ARM64 clusters may share the same Harbor instance, but they must not share the same Zarf registry project or repository prefix. Images in a Zarf air-gap package are packaged for the target architecture. Deploying AMD64 and ARM64 packages sequentially to the same image path does not merge identical tags into a multi-platform manifest. A later push may replace the earlier tag and cause Pods on the other architecture to pull an incompatible image after a restart or rescheduling.

During cluster initialization, use an architecture-specific --registry-url for each architecture. For example, use registry.example.com/zarf-amd64 for AMD64 and registry.example.com/zarf-arm64 for ARM64. Create the corresponding Harbor projects or repository prefixes in advance, and ensure that the account used by Zarf has push and pull permissions.

External registry parameter descriptions:

Parameter	Description
`--registry-url`	External image registry address
`--registry-push-username`	Username used to push images
`--registry-push-password`	Password used to push images

After initialization, zarf package deploy imports the images bundled in the package and uses an admission webhook to rewrite image references for managed workloads to the Zarf registry. With the labeled policy, only resources labeled zarf.dev/agent: mutate, or resources in a namespace with that label, are processed; a label on the resource takes precedence over the namespace label. The HAMi main package applies this label to the namespaces it manages. For offline workloads created separately, confirm that the namespace is labeled first. Label changes do not affect existing Pods; recreate the Pods to trigger rewriting.

Use --insecure-skip-tls-verify for the relevant command only when the TLS certificate of the target registry genuinely cannot be verified and the man-in-the-middle risk has been assessed. In production, fix the certificate chain or configure a trusted CA for the Zarf Agent instead of treating skipped TLS verification as a default.

Deploy HAMi Enterprise

All components in the HAMi Enterprise main package are set as optional. Select --components based on your deployment scenario. Please keep the component order as documented.

The component list is as follows:

Component Name	Description	Required	Recommended
`tools`	Operations toolkit: jq, nerdctl, etc.	No	As needed
`hami-deploy-scripts`	HAMi deployment and preflight scripts	Yes	Yes
`hami`	hami-enterprise Helm chart	No	Yes
`prometheus-crds`	Prometheus Operator CRDs, installed with Server-Side Apply	No	Install when monitoring is enabled
`prometheus`	kube-prometheus-stack Helm chart	No	As needed
`gpu-operator`	NVIDIA GPU Operator; the bundled values disable its built-in device-plugin	No	As needed

HAMi Enterprise deployment requires a custom values file to override cluster, scheduling, and monitoring configuration. Zarf v0.82.0 supports --values directly, so --features="values=true" is no longer required; a small number of fields can also be overridden with --set-values key.path=value. The merge order is: Chart defaults → bundled values/*.yaml → --values → --set-values. The prometheus and gpu-operator components include bundled values and usually require no repeated configuration.

Prepare Custom Values

The in-package hami/README.md provides complete values documentation for hami-enterprise. Common configuration items at a glance:

Parameter	Description	Default
`dra.enabled`	Whether to enable and deploy DRA	`false`
`scheduler.leaderElect`	Whether to enable leader election for `hami-scheduler` across multiple nodes. Strongly recommended to disable for single-node clusters.	`true`
`scheduler.replicas`	Adjust the number of `hami-scheduler` instances	1
`scheduler.kubeScheduler.image.registry`	Image registry for the `kube-scheduler` image used by `hami-scheduler`	`registry.cn-hangzhou.aliyuncs.com`
`scheduler.kubeScheduler.image.repository`	Image repository name for the `kube-scheduler` image used by `hami-scheduler`	`google_containers/kube-scheduler`
`scheduler.kubeScheduler.image.tag`	Version of the `kube-scheduler` image used by `hami-scheduler`; should match the target cluster	`""`

Minimal configuration example: my-overrides.yaml

dra:
  enabled: false

scheduler:
  leaderElect: true

The HAMi scheduler depends on a kube-scheduler image that matches the target Kubernetes cluster version. Additional configuration is only required when the target cluster's kube-scheduler is not running inside the cluster (and its image cannot be reused directly from kube-system); this is common in cloud-managed Kubernetes control plane clusters. If a kube-scheduler Pod exists in the cluster, you can skip the following configuration:

scheduler:
  kubeScheduler:
    image:
      registry: your-registry.example.com
      repository: google_containers/kube-scheduler
      tag: v1.29.8

The built-in kube-scheduler image version in this air-gap package is v1.36.0. If the target cluster's Kubernetes version differs and you cannot reuse the existing in-cluster kube-scheduler, you must prepare and import a matching kube-scheduler image in the offline environment yourself.

Before deployment, it is recommended to check the merged values result offline:

zarf package inspect values-files hami-enterprise-v0.0.3-airgap-amd64.tar.zst \
  --components=tools,hami-deploy-scripts,hami,prometheus-crds,prometheus,gpu-operator \
  --values=my-overrides.yaml

Execute Deployment

Ascend clusters: If the target cluster uses Ascend accelerators, add ascend-device-plugin immediately after hami in the deployment command --components list. This component uses the values bundled with the package; no additional configuration is required in my-overrides.yaml.

Minimal installation, deploying only the HAMi Enterprise core components:

zarf package deploy hami-enterprise-v0.0.3-airgap-amd64.tar.zst \
  --components=hami-deploy-scripts,hami \
  --values=my-overrides.yaml \
  --confirm

Full installation, including tools, HAMi, PrometheusCRD, Prometheus, and GPU Operator:

zarf package deploy hami-enterprise-v0.0.3-airgap-amd64.tar.zst \
  --components=tools,hami-deploy-scripts,hami,prometheus-crds,prometheus,gpu-operator \
  --values=my-overrides.yaml \
  --confirm

If a component remains stuck for a long time, the installation has encountered an issue. You can use zarf tools helm to diagnose the component; if the failure is caused by incorrect values during Helm rendering or installation, fix my-overrides.yaml and rerun the same zarf package deploy ... command.

After an interrupted deployment, resolve the issue and resume with the same zarf package deploy ... --components=... --values=... command. Zarf will skip re-importing images when the digest has not changed; it will perform a Helm upgrade when Helm charts or values have changed.

If the target resources are already managed by another Helm release and you have confirmed that they should be transferred to the current Zarf package, review the resource scope and then use --take-ownership. Use --force-conflicts only for Server-Side Apply field ownership conflicts. It overwrites fields managed by other field managers, so use it only after confirming that this deployment may take over those fields; do not use it as a general retry option.

Enable GPU Nodes

HAMi device-plugin starts only on nodes labeled gpu=on. Run the following command for every GPU or Ascend node that HAMi should manage:

kubectl label nodes <node-name> gpu=on

Nodes and Monitoring

kubectl describe node <node-name>
kubectl api-resources --api-group=monitoring.coreos.com

When Prometheus is used, the ServiceMonitor labels must match Prometheus.spec.serviceMonitorSelector. When VictoriaMetrics is used, the labels must match VMServiceScrape.spec.serviceScrapeSelector.

Query the following NVIDIA metrics in the monitoring system and confirm that the results are not empty:

DCGM_FI_DEV_GPU_UTIL
HostCoreUtilization
GPUDeviceCoreAllocated

Obtain License Request Information

Run this step after all selected components have started. The environment requires kubectl and jq. Use either method below.

Download and Run the Script

curl -fsSLO https://public.hami.run/collect-hami-license-info.sh
bash collect-hami-license-info.sh

Run a Local Copy

bash collect-hami-license-info.sh

The script outputs JSON in the following format:

{
  "esn": "96565d61-986a-4918-aafb-448ff6e3746b",
  "deviceInstances": [
    {
      "uuid": "GPU-ceee905d-48ac-93de-a81b-17c00e1e5e02",
      "deviceType": "NVIDIA A10"
    }
  ]
}

Send the JSON output to Dynamia.ai sales or technical support to obtain the license.

Post-Activation Verification

kubectl -n hami-system get pods
kubectl describe node <gpu-node>
kubectl get events --field-selector involvedObject.name=hami-license -n hami-system
kubectl get nodes -o custom-columns='NODE:.metadata.name,LICENSE:.metadata.annotations.hami\.io/nvidia-license'

A LicenseValid event indicates that license validation succeeded. Confirm that the selected component Pods are Running or Completed and that accelerator resources are registered on the managed nodes.

Sample Workload Verification

The outer air-gap package already contains two independent Zarf sample packages, so there is no need to manually kubectl apply local YAML.

GPU Burn Verification

zarf package deploy zarf-package-hami-example-gpu-burn-amd64-v0.0.2.tar.zst --confirm

After deployment, check the Deployment / Pod status:

kubectl -n hami-example get deploy turbo-gpu-burn
kubectl -n hami-example get pods -l app=turbo-gpu-burn
kubectl -n hami-example logs -l app=turbo-gpu-burn --tail=50

This sample creates hami-example/turbo-gpu-burn Deployment; clean it up as needed after verification:

kubectl -n hami-example delete deploy turbo-gpu-burn

vLLM + Qwen Verification

zarf package deploy zarf-package-hami-example-vllm-qwen-amd64-v0.0.4.tar.zst --confirm

After deployment, check the inference service status:

kubectl -n hami-example get deploy vllm-qwen3
kubectl -n hami-example get pods -l app=vllm-qwen3
kubectl -n hami-example get svc vllm-qwen3-webui

Once the Pod is ready, access Open WebUI via any node IP + NodePort (30081):

# Get cluster node IP (any reachable node is fine)
kubectl get nodes -o wide

# Access via browser
# http://<node-ip>:30081

Open WebUI is automatically connected to the vLLM sidecar in the same Pod; open the page to start interacting.

If the Pod remains Pending, first check whether the license is activated, whether GPU nodes are labeled with gpu=on, and whether the node GPU drivers are normal.

Troubleshooting

Common inspection commands:

kubectl get pods -A | grep -E 'hami|gpu-operator|prometheus|vllm|gpu-burn'
kubectl get events -A --sort-by=.lastTimestamp | tail -50
zarf package list

Collect diagnostic information:

bash collect-cluster-info.sh

Common Issues

Symptom	Checks and Resolution
`hami-device-plugin` is not running	Confirm that the node has the `gpu=on` label, then check `kubectl -n hami-system get pods`.
`hami-device-plugin` repeatedly restarts	Check whether NVIDIA GPU Operator still enables its default device plugin. Set `devicePlugin.enabled=false`.
Image pull fails	Check network access from the target cluster to the image registry, the image repository and tag, and any required `imagePullSecrets`.
HAMi metrics are unavailable	Check whether the Prometheus or VictoriaMetrics selector matches the labels on the monitoring object.
A workload remains `Pending`	Check license activation, the `gpu=on` node label, available accelerator resources, and events from `kubectl describe pod`.

Limitations of Installing Components with Zarf

Control the Image Rewrite Scope Explicitly with Labels

We recommend initializing Zarf with zarf init --agent-mutation-policy=labeled. This policy rewrites only resources labeled zarf.dev/agent: mutate, or resources in a namespace with that label. An individual workload that must retain its original image reference can be labeled zarf.dev/agent: ignore, and a resource label takes precedence over the namespace label. This allows image rewriting to be controlled per resource, even within the same cluster or namespace, without having the Agent manage every application namespace by default. Only images included in the Zarf package can be rewritten and pulled in an offline environment. After changing labels, recreate existing Pods because they are not updated automatically.

Manage Each Set of Resources Through a Single Delivery Pipeline

Zarf manages the chart releases in a package through Helm. Managing the same Kubernetes resources in parallel through another native Helm workflow can still cause release ownership, field ownership, and upgrade ordering conflicts. Continue routine upgrades with a newer Zarf package. If existing resources must be transferred to Zarf, review the release and resource scope before using --take-ownership. Use --force-conflicts only for Server-Side Apply field conflicts that you have confirmed may be overwritten; it should not be used as a routine installation option.

Support

Email: info@dynamia.ai
Sales / technical support: 400-026-7800
Customers with active commercial agreements should submit support requests through their dedicated support channel.

Air-Gap Package Contents​

Prerequisite Checks​

Extract, Verify, and Install Zarf​

Initialize Zarf​

Deploy HAMi Enterprise​

Prepare Custom Values​

Execute Deployment​

Enable GPU Nodes​

Nodes and Monitoring​

Obtain License Request Information​

Download and Run the Script​

Run a Local Copy​

Post-Activation Verification​

Sample Workload Verification​

GPU Burn Verification​

vLLM + Qwen Verification​

Troubleshooting​