Zarf Migration Guide

This document guides the migration of a cluster already running HAMi Enterprise/AI Platform to a Zarf offline-package deployment. The intended audience is customer-side SREs, platform engineers, or Kubernetes administrators responsible for cluster delivery, upgrades, and operations.

Package names, component names, and values file names should be replaced with those in the actual delivery package. The examples below use:

<PACKAGE_FILE>       # HAMi Enterprise Zarf main deployment package, e.g. hami-enterprise-*.tar.zst
<VALUES_FILE>        # Values file used for the migration, e.g. my-overrides.yaml

Recommended Migration Path

First determine how HAMi Enterprise is currently installed:

--adopt-existing-resources and --force-conflicts are two important zarf package deploy parameters.

Current state	Recommended approach	Use --adopt-existing-resources by default?	Use --force-conflicts by default?
No HAMi Enterprise in the cluster	Deploy directly with Zarf	No	No
Helm release with the same name already exists	Let the Zarf package take over subsequent deployments and upgrades	No	No
Helm release with a different name already exists	Migrate during a maintenance window; do not let two releases manage the same resources	Case by case	No
Resources created by Kustomize or kubectl apply	Use --adopt-existing-resources to take over the resources	Yes	Only if field conflicts occur
Only some fields were changed by HPA or manual kubectl	Deploy normally and handle SSA field conflicts if they occur	No	As needed

Do not leave the old Helm release and the Zarf-managed release managing the same objects for a long time. Subsequent upgrades, rollbacks, or uninstalls on either side may affect resources already taken over by the other side.

Limitations of Installing Components with Zarf

Cannot mix internal and external image registries in the same namespace (using multiple registries in the same cluster has limitations)

Zarf uses an admission webhook to silently replace the images of all containers in a managed namespace with versions from the private registry. In a cluster, some namespaces can bypass Zarf and access other registries normally, but you cannot use images from different registries within the same namespace.You can ignore this behavior by labeling the namespace with zarf.dev/agent=ignore, allowing you to use other registries in that namespace instead of the registry specified by zarf init --registry-url ....

Native Helm cannot be used to manage the same set of resources

Once Zarf has taken over a set of components, attempting to manage the same resources with another Helm workflow can easily lead to ownership conflicts, upgrade collisions, or ambiguous state. This is not a Zarf-specific issue; it is an almost inevitable result when delivery boundaries are not clearly defined. Zarf makes it convenient to spin up a set of components, but you cannot install them and then kick Zarf aside and return to Helm for daily iteration.Therefore, unless necessary, please rely on the Migua AI team for regular version upgrades and do not iterate HAMi Enterprise on your own.

Difficult to iterate flexibly

Zarf's design means that every transfer carries the full software stack. This inherently precludes "flexibility."

Pre-Migration Preparation

1. Verify Tool Availability

kubectl version
kubectl get nodes
zarf version

If Helm is not installed separately in the target environment, you can use the Helm bundled with Zarf:

zarf tools helm version
zarf tools helm list -A

2. Confirm the Current Installation Method

Check the current Helm release:

zarf tools helm list -A

If a release corresponding to HAMi Enterprise is found, record its namespace and release name. A common scenario is:

namespace: hami-system
release:   hami

Check HAMi-related resources:

kubectl get ns hami-system
kubectl -n hami-system get pods,deploy,ds,svc,cm,secret

Check Helm ownership information on resources:

kubectl -n hami-system get deploy,ds,svc,cm,secret,sa,role,rolebinding \
  -o custom-columns='KIND:.kind,NAME:.metadata.name,MANAGED_BY:.metadata.labels.app\.kubernetes\.io/managed-by,RELEASE:.metadata.annotations.meta\.helm\.sh/release-name,RELEASE_NS:.metadata.annotations.meta\.helm\.sh/release-namespace'

Check HAMi-related cluster-scoped resources:

kubectl get clusterrole,clusterrolebinding,mutatingwebhookconfiguration,validatingwebhookconfiguration \
  -o name | grep -i hami

3. Back Up the Current Configuration and State

If the current deployment is a Helm release:

zarf tools helm -n hami-system get values <OLD_RELEASE> -o yaml > hami-current-values.yaml
zarf tools helm -n hami-system get manifest <OLD_RELEASE> > hami-current-manifest.yaml
zarf tools helm -n hami-system status <OLD_RELEASE> > hami-current-status.txt

If the current deployment is not managed by Helm, it is still recommended to export the existing YAML:

kubectl -n hami-system get deploy,ds,svc,cm,secret,sa,role,rolebinding -o yaml > hami-current-resources.yaml

Save the current runtime state:

kubectl -n hami-system get pods -o wide
kubectl get nodes --show-labels
kubectl get events -A --sort-by=.lastTimestamp | tail -100

If the cluster already has HAMi license Secrets or customer-specific custom configurations, back those up as well.

4. Prepare Migration Values

Usually, start from the current Helm values:

cp hami-current-values.yaml <VALUES_FILE>

Please verify the following key items:

• Whether DRA is enabled.
• Whether the hami-scheduler replica count is suitable for the cluster scale.
• Whether scheduler leader election is enabled.
• Whether the kube-scheduler image version matches the target Kubernetes version.
• Whether GPU nodes use the gpu=on label.
• Whether to keep the names of existing webhooks, RBAC, Services, and ConfigMaps.

Offline Pre-Deployment Checks

Before deployment, it is recommended to inspect the merged values result:

zarf package inspect values-files <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true"

Then inspect the manifests that the Zarf package will deploy:

zarf package inspect manifests <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true" \
  > hami-zarf-rendered.yaml

Key items to check:

• Whether the target namespace is correct.
• Whether the target release name matches the migration plan.
• Whether resource names correspond to existing resources.
• Whether images all come from the offline package or a registry accessible to the target environment.
• Whether values correctly cover the configurations required by the current cluster.

Scenario 1: Existing Helm Release with the Same Name

Applicable conditions:

• HAMi Enterprise is currently installed by Helm.
• The old release name matches the HAMi release name in the Zarf package.
• The namespace matches.
• Going forward, deployments and upgrades will be managed uniformly through the Zarf package.

In this case, migration is usually performed as a Helm upgrade. Do not add --adopt-existing-resources or --force-conflicts by default.

Perform the migration:

zarf package deploy <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true" \
  --confirm

Post-migration checks:

zarf package list
zarf tools helm -n hami-system status hami
kubectl -n hami-system get pods
kubectl -n hami-system rollout status deploy/hami-scheduler

After the migration, switch the routine upgrade entry point to the Zarf package. Unless you are executing a clearly defined rollback plan, do not use the old Helm workflow to directly upgrade the same release again.

Scenario 2: Existing Helm Release with a Different Name

If the old release name differs from the release name in the Zarf package, this is a high-risk migration. Do not directly deploy a new Zarf release over the same set of resources.

It is recommended to choose one of the following approaches during a maintenance window:

Approach	Applicable situation	Description
Use a Zarf package whose release name matches the old one	Prefer in-place migration	Confirm that the release name in the delivery package matches the old environment
Uninstall the old release before deploying Zarf	Short downtime or rebuild is acceptable	Cleanest lifecycle
Use --adopt-existing-resources to take over	Resources must be retained and cannot be rebuilt	Requires item-by-item verification of resource ownership; higher risk

If you choose to uninstall the old release and then deploy Zarf, first confirm that values, manifests, license Secrets, and necessary configurations have been backed up. Then execute during the maintenance window:

zarf tools helm -n hami-system uninstall <OLD_RELEASE>

zarf package deploy <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true" \
  --confirm

If you choose to take over existing resources, confirm at least the following:

• The old Helm release will not be upgraded, rolled back, or uninstalled afterward.
• The resource names rendered by Zarf correspond to the existing resources.
• The resource namespace does not contain resources with the same or similar names belonging to other systems.
• A full backup exists and a maintenance window is available.

Example takeover command:

zarf package deploy <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true" \
  --adopt-existing-resources \
  --confirm

Only consider adding --force-conflicts when the deployment log clearly shows Server-Side Apply field conflicts and you have confirmed that those fields should be managed by Zarf/Helm:

zarf package deploy <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true" \
  --adopt-existing-resources \
  --force-conflicts \
  --confirm

This is not the default migration command; use it only after the takeover scope and the source of field conflicts have been confirmed.

Scenario 3: Existing Resources Managed by Kustomize or kubectl apply

If the existing HAMi Enterprise resources are not a Helm release but were created by Kustomize or kubectl apply, the migration focus is to bring those resources under the Zarf-managed Helm chart.

Recommended process:

1. Export the existing YAML.
2. Render the target manifests with Zarf.
3. Compare resource names, namespaces, labels, annotations, and key spec fields.
4. Use --adopt-existing-resources to take over.
5. Use --force-conflicts only when Server-Side Apply field conflicts are encountered.

Takeover command:

zarf package deploy <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true" \
  --adopt-existing-resources \
  --confirm

If the failure message clearly indicates a field ownership conflict, execute:

zarf package deploy <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true" \
  --adopt-existing-resources \
  --force-conflicts \
  --confirm

Prometheus and GPU Operator

Migrating HAMi Enterprise does not mean you must also take over Prometheus or NVIDIA GPU Operator at the same time.

If Prometheus or GPU Operator already exists in the cluster, it is recommended to migrate only HAMi first:

zarf package deploy <PACKAGE_FILE> \
  --components=hami-deploy-scripts,hami \
  --values=<VALUES_FILE> \
  --features="values=true" \
  --confirm

When Prometheus or GPU Operator also needs to be migrated, evaluate them separately based on their own releases, namespaces, values, and resource ownership.

Special notes:

• If GPU Operator already exists in the cluster, usually do not deploy the gpu-operator component from the Zarf package again.
• If the default NVIDIA device-plugin from GPU Operator conflicts with the HAMi device-plugin, disable the default device-plugin via GPU Operator values.
• --force-conflicts cannot resolve runtime conflicts caused by two device-plugins running simultaneously.

Post-Migration Acceptance

Check the Zarf package status:

zarf package list

Check the Helm release:

zarf tools helm -n hami-system status hami
zarf tools helm -n hami-system get values hami

Check core components:

kubectl -n hami-system get pods -o wide
kubectl -n hami-system rollout status deploy/hami-scheduler
kubectl get nodes --show-labels | grep gpu=on

Check certificates and the GPU scheduling chain:

bash collect-hami-license-info.sh
kubectl describe node <GPU_NODE_NAME>

If the delivery package includes sample workloads, you can continue using the GPU burn or vLLM examples to verify the scheduling chain.

Rollback Recommendations

The rollback approach depends on the migration method:

Migration method	Rollback approach
Same-name Helm release migration	Use the values and manifest saved before migration and follow the established Helm/Zarf rollback procedure
Uninstalled old release and redeployed with Zarf	Reinstall using the old release values and the original Helm chart
--adopt-existing-resources takeover	Before rolling back, confirm resource ownership to avoid the old release's uninstall deleting resources in use
Kustomize / kubectl apply takeover	Restore using the exported YAML and change records from before migration

Any rollback should be performed in a maintenance window. Do not delete a release or namespace without confirming resource ownership.

Common Issue Handling

Symptom	Possible cause	Resolution
invalid ownership metadata	The resource already exists but Helm release ownership does not match or is missing	Decide whether --adopt-existing-resources is needed, or clean up conflicting resources during a maintenance window
field manager conflict / SSA conflict	Some fields are managed by another manager	After confirming that the fields should be managed by Zarf/Helm, use --force-conflicts
Resources are deleted after uninstalling the old Helm release	The old release still considers itself the owner of those resources	Do not let two releases manage the same resources; confirm ownership before rollback
hami-device-pluginCrashLoopBackOff	May conflict with the NVIDIA default device-plugin	Disable the built-in device-plugin of GPU Operator and check the GPU driver
Workload stays Pending	License not activated, GPU nodes not labeled, or insufficient GPUs	Check license, gpu=on labels, and kubectl describe pod events
Scheduler startup abnormal	Values, image version, or leader election configuration mismatch	Verify <VALUES_FILE> and the target Kubernetes version

References

• Zarf package deploy options: https://docs.zarf.dev/commands/zarf_package_deploy
• Zarf resource adoption: https://docs.zarf.dev/tutorials/8-resource-adoption
• Zarf Helm chart configuration: https://docs.zarf.dev/ref/components