ishenwei/nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fc0dde291fa7c19f90564e88e6a04ed4c26b5ba6
nexus/knowledgebase/csd-wiki/ICSD/Transform-the-suite-to-a-Helm-deployment-on-24.2.FP1_688996419.md
weishen 3f2e1765d8 Auto-sync: 2026-04-18 17:09
2026-04-18 17:09:43 +08:00

19 KiB
Raw Blame History

Transform-the-suite-to-a-Helm-deployment-on-24.2.FP1_688996419

Helm upgrade procedure

SEQStepsDurationDowntime

Prerequisites

1

Make sure current suite is upgraded to version 24.2.fp1

1 minNA
2

Run the following command to make sure that all the OMT and suite pods are ready

1 min

kubectl get pod --all-namespaces|grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v 6/6|grep -v Completed

Preparation

1

Download the ESM helm chart to bastion

2 minsNA

Download helm chart (24.2.FP1) & Unzip & Verify the signature

s3://deploy-packages/artifacts/24.2/24.2.FP1-04-29-2024-Build-185/ESM_Helm_Chart-24.2.1.zip

2Get environment variables1 min
NAMESPACE=`kubectl get namespace|grep itsma | cut -f1 -d " "`
SYSTEM_USER_ID=$(kubectl get configmap -o jsonpath='{.data.system_user_id}' itsma-common-configmap -n $NAMESPACE)
SYSTEM_GROUP_ID=$(kubectl get configmap -o jsonpath='{.data.system_group_id}' itsma-common-configmap -n $NAMESPACE)
SIZE=$(kubectl get configmap -o jsonpath='{.data.itom_suite_size}' itsma-common-configmap -n $NAMESPACE)
echo NAMESPACE: $NAMESPACE SYSTEM_USER_ID: ${SYSTEM_USER_ID}, SYSTEM_GROUP_ID: ${SYSTEM_GROUP_ID}, SIZE: ${SIZE}

NOTE: If your bastion session is expired, run this get ENV variables again. If it expires after you delete the ns, replace the variables manually when executing the command

3(Optional) Backup the suite external ingress and suite integration ingress2 mins

For ESK ENV, you will create the ingresses (suite and integration) after helm installation.

If you don't have the backup files, please do the backup in advance. For example：

NOTE: The ingress names may vary across environments

kubectl get ingress sma-ingress -n $NAMESPACE -o yaml > suite-ingress-backup.yaml

kubectl get ingress sma-int-ingress -n $NAMESPACE -o yaml > sma-int-ingress-backup.yaml

Note: After helm installation you have to re-create related ingress and alb.

4

Create additional volumes

1 min

For EKS:

Run the following commands on the bastion node.

NOTE: Please replace the mount point with the actual value:

sudo mkdir -p /mnt/efs/var/vols/itom/itsma/config-volume

sudo mkdir -p /mnt/efs/var/vols/itom/itsma/logging-volume

sudo chown -R $SYSTEM_USER_ID:$SYSTEM_GROUP_ID /mnt/efs/var/vols/itom/itsma/config-volume
sudo chown -R $SYSTEM_USER_ID:$SYSTEM_GROUP_ID /mnt/efs/var/vols/itom/itsma/logging-volume
sudo chmod g+w /mnt/efs/var/vols/itom/itsma/config-volume
sudo chmod g+w /mnt/efs/var/vols/itom/itsma/logging-volume
sudo chmod g+s /mnt/efs/var/vols/itom/itsma/config-volume
sudo chmod g+s /mnt/efs/var/vols/itom/itsma/logging-volume

5

Check folder permissions

5 mins

Using the following command to check and change folder permission (Please replace the mount point with actual value:):

Check folder permission:

sudo find /mnt/efs/var/vols/itom -type d -exec stat --format='%u:%g %A %n' '{}' \;| grep -v $SYSTEM_USER_ID:$SYSTEM_GROUP_ID

If the result doesn't include :, change the ownership of each directory or file that you created by using the chown -R : command.

For example (the command find and update the folders' permission exclude the log folder):

sudo find /mnt/efs/var/vols/itom -type d -not -path "/mnt/efs/var/vols/itom/itsma/global-volume/logs/*" -exec chown $SYSTEM_USER_ID:$SYSTEM_GROUP_ID {} +

6

Sync data to new Helm persistent volumes

6 mins

NOTE: You may want to clean up the tenant-import and tenant-export folders if the size of these 2 volumes are too large before sync.

Sync data to new PV:

cd ESM_Helm_Chart-2x.x/scripts/transformation
chmod u+x syncData.sh

Run sycnData command with your own path, for example:

sudo./syncData.sh --globalVolumePath /mnt/efs/var/vols/itom/itsma/global-volume --smartanalyticsVolumePath /mnt/efs/var/vols/itom/itsma/smartanalytics-volume --configVolumePath /mnt/efs/var/vols/itom/itsma/config-volume

NOTE: you need to press 'y' twice to complete the script

Also you could use the following command to avoid bastion connection interruption during data sync

NOTE: Please replace the mount point to actual value you use:

nohup sh -c "printf 'y\ny\ny\ny\n' | sudo./syncData.sh --globalVolumePath /mnt/efs/var/vols/itom/itsma/global-volume --smartanalyticsVolumePath /mnt/efs/var/vols/itom/itsma/smartanalytics-volume --configVolumePath /mnt/efs/var/vols/itom/itsma/config-volume" </dev/null >nohup.out 2>&1 &

Watch the progress by below command:

tail -f nohup.out

Gov Cloud ENV took 6 mins with below data size:

The 'global-volume' requires an additional 77 MB of free disk space.
The 'config-volume' requires an additional 70 MB of free disk space.

7

Retrieve system configurations and generate values.yaml file

  • Generate basic values.yaml

  • Generate customized values.yaml

5 mins

Before you begin, ensure that the jq and yq tools are installed

Generate basic values.yaml file based on the suite environment

cd ESM_Helm_Chart-2x.x/scripts/transformation
chmod u+x generateBasicValuesYaml.sh
./generateBasicValuesYaml.sh

NOTE: you need to press 'y' twice to complete the script

Copy the values.yaml file to the ESM_Helm_Chart-2x.x/charts/ directory.

Save a copy of the values.yaml file in a secure location as backup.

Generate customized values.yaml

Go to the ESM_Helm_Chart-2x.x/scripts/custom_settings directory.Change the script file permission and run script:

cd ESM_Helm_Chart-2x.x/esm-1.0.0+2x.x-xxx/scripts/ custom_settings

chmod u+x generateCustomSettings.sh

./generateCustomSettings.sh

The script generates a customized_values.yaml file in the current directory.

Copy the customized_values.yaml file to the ESM_Helm_Chart-2x.x/charts/ directory.

Save the customized_values.yaml file to a secure location as a backup copy.

7

Back up OMT and SMA

30 mins

Backup the whole SMAX (RDS/EFS/K8S)

Maintain Window

1

Stop SMA and OMT

5 mins45 mins

1.Stop OMT & SMA:

$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n $NAMESPACE
$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n core

2.Check the pods of OMT & SMA are all stopped.

kubectl get pod -n $NAMESPACE|grep -v -E 'throttling|opentelemetry|toolkit|Completed'
kubectl get pod -n core |grep -v Completed

NOTE: If your promethues pods are under core namespace, it takes time to shut down these pods

2

Clean up classic SMA resources

5 mins

Use the below command to clean the SMA resource:

kubectl delete ns $NAMESPACE

NOTE: Deleting the ns will re-created the ingress and ALB. You will create new ingress and ALB for EKS in helm installation step. Then bound the newly created the ingress alb to FedRAMP domain

Verify the ns is deleted:

kubectl get ns

Use the following command to check what resources are being used:

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n $NAMESPACE

Patch the ingress use the following command:

kubectl patch -n $NAMESPACE --type=json --patch='[{"op": "remove", "path": "/metadata/finalizers/0"}]'

3

Sync incremental data

2 mins

Sync incremental data:

Go to the ESM_Helm_Chart-2x.x/scripts/transformation directory and run below command again:

sudo./syncData.sh --globalVolumePath /mnt/efs/var/vols/itom/itsma/global-volume --smartanalyticsVolumePath /mnt/efs/var/vols/itom/itsma/smartanalytics-volume --configVolumePath /mnt/efs/var/vols/itom/itsma/config-volume

NOTE: to sync incremental data you only need to press y for one time

The incremental sync should not cost long time as the first sync, while still you can use the below command to avoid bastion connection interruption during sync data

NOTE: Please replace the mount point with your actual environment:

nohup sh -c "printf 'y\ny\ny\ny\n' | sudo./syncData.sh --globalVolumePath /mnt/efs/var/vols/itom/itsma/global-volume --smartanalyticsVolumePath /mnt/efs/var/vols/itom/itsma/smartanalytics-volume --configVolumePath /mnt/efs/var/vols/itom/itsma/config-volume" </dev/null >nohup.out 2>&1 &

4

Update the label for the core namespace

1 min

Update the label for core:

kubectl patch ns core -p '{"metadata":{"labels":{" deployments.microfocus.com/deployment-name":"cdf "}}}'

5Create a deployment for the suite1 min

Create a deployment for suite:

$CDF_HOME/bin/cdfctl deployment create -d $NAMESPACE
NOTE: This command creates a deployment with your original suite namespace as the deployment name.

6Refine existing PVs1 min
  1. Go to the ESM_Helm_Chart-2x.x/scripts/transformation directory.cd ESM_Helm_Chart-2x.x/esm-1.0.0+2x.x-xxx/scripts/transformation
  2. Run the following commands:

    chmod u+x refinePV.sh
    ./refinePV.sh $SIZE
    NOTE: Press "y" after you see this message "Do you want to create pvs based on the same nfs server and nfs root path with the global-volume:? [y/n]"

  3. Run the following command to verify the PV creation.
    kubectl get pv|grep -E "config-volume|logging-volume|data-volume"|grep itsma
  4. Run the following command to verify the PV status is Available (Below command is supposed to return no result)
    kubectl get pv|grep itsma|grep -v -E "db-volume|global-volume|smartanalytics"|awk '{if ($5!="Available") print $0}'
7Restore vault data
  • Copy vault data from core-volume to global-volume
  • Copy vault secrets to the suite namespace
5 mins

Copy vault data from core-volume to global-volume
On bastion run below command:

sudo cp -R /mnt/efs/var/vols/itom/itsma/core/vault /mnt/efs/var/vols/itom/itsma/global-volume/
sudo chown -R $SYSTEM_USER_ID:$SYSTEM_GROUP_ID /mnt/efs/var/vols/itom/itsma/global-volume/vault

NOTE: Please replace the mount point with your actual environment

Copy vault secrets to the suite namespace
  1. Create script "copyVaultCredentials.sh" > Restore_vault_data
  2. Run below command
    chmod u+x copyVaultCredentials.sh
    ./copyVaultCredentials.sh $NAMESPACE
NOTE: Replace with a unique release name that you want to specify for the suite deployment.
8Start OMT5 mins
  1. Start OMT:
    $CDF_HOME/bin/cdfctl runlevel set -l UP -n core
  2. Check that all the OMT pods are ready:
    kubectl get pods -n core|grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed
9Install the ESM helm chartCreate ingress for the suite (EKS only)15 mins

Login Bastion and change folder to the ESM_Helm_Chart-2x.x/charts/ directory.
Run the helm install command:

helm install esm-1.0.0+2x.x-xxx.tgz -n $NAMESPACE -f values.yaml -f customized_values.yaml --set global.nodeSelector.Worker=label

N OTE: This release name is the one you gave in step 7

During the helm installation, monitor the status of the itom-nginx-ingress service by running the below command:

kubectl get svc -n $NAMESPACE|grep itom-nginx-ingress-svc

[EKS Only] Once the service is available, create the ingress for the suite & ingress for the SMAX integration.

For example:

kubectl create -f sma-ingress.yaml
kubectl create -f sma-ingress-integration.yaml

NOTE: The ingress yaml files are the ones you exported in Preparation - Step 3

Bound the newly created the ingress alb to FedRAMP domain

Add the newly created service port to the EKS worker's inbound security group

Reference:

10Enable helm autopass3 mins
  1. Log in to the control plane node or bastion node.
  2. Change to the ESM_Helm_Chart-2x.x/scripts/transformation directory.
  3. Change the permission of the updateAutopassKey.sh file:
    chmod u+x updateAutopassKey.sh
  4. Run the script to sync the autopass key.
    ./updateAutopassKey.sh -n $NAMESPACE
    It will restart the itom-bo-license-deployment and autopass-lm-v2 pods.
  5. After the above pods are up and running, enter the autopass page with the new URL: https:///autopass
11Ensure all suite pods & job are ready2 mins

Check helm install pod status

kubectl get pod -n $NAMESPACE|grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed

kubectl get job -n $NAMESPACE

Note: please run the following command t o delete the xruntime-upgrade-tenants job if it is 0/1:

kubectl delete job -n $ns `kubectl get job -n $ns | grep xruntime-upgrade-tenants | awk '{print $1}'`

Post-tasks

1Clean up unused pods in the OMT namespace1 minsNA

Clean up unused pods in the OMT namespace via below command:

helm upgrade apphub $CDF_HOME/charts/apphub-1.2*.tgz --reuse-values --set global.services.suiteDeployment

NOTE: If you are going to restore a single namespace after running above command, please run below workaround to fix this issue

Fix OMT clusterrolebinding and clusterrole

After helm transformation, OMT 5443 portal will no longer be in use.

2Delete unused PVs1 mins
kubectl delete pv $NAMESPACE-db-volume
kubectl delete pv $NAMESPACE-smartanalytics-volume
kubectl delete pv $NAMESPACE-global-volume
3Delete unused nfs folders15 minsRun the following command to clean unused nfs folder:

dbVolume=

globalVolume=

smartanalyticsVolume=

Where: and are the NFS paths of db-volume, global-volume, and smartanalytics-volume, respectively.

For example:

dbVolume=/mnt/efs/var/vols/itom/itsma/db-volume

globalVolume=/mnt/efs/var/vols/itom/itsma/global-volume

smartanalyticsVolume=/mnt/efs/var/vols/itom/itsma/smartanalytics-volume


sudo rm -rf $dbVolume
sudo rm -rf $smartanalyticsVolume
sudo rm -rf $globalVolume/data
sudo rm -rf $globalVolume/image
sudo rm -rf $globalVolume/resources
sudo rm -rf $globalVolume/config
sudo rm -rf $globalVolume/certificate
sudo rm -rf $globalVolume/tenant-export
sudo rm -rf $globalVolume/tenant-import
sudo rm -rf $globalVolume/jdbc
sudo rm -rf $globalVolume/l10n
sudo rm -rf $globalVolume/va/configs
sudo rm -rf $globalVolume/va/stopwords
sudo rm -rf $globalVolume/di
sudo rm -rf $globalVolume/logs

Verification

1SMAX30 minsNA
2NSACM Sanity
3Audit Sanity
4Audit-Collector Sanity

Build Info:

Product Build/Chart
OMT 24.2-159
SMAX/HCMX Metadata 24.2.FP1-b11
SMAX/HCMX Helm chart esm-1.0.1+24.2.1-38

Doc Link:

Transform SMAX To Helm

Related pages

Reference in New Issue View Git Blame Copy Permalink