ishenwei/nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
059fffb9c2f5e8c9ffbb76799637ce8bfe4e7bdb
nexus/knowledgebase/csd-wiki/ICSD/Transform-the-suite-to-a-Helm-deployment-on-24.3.2_688996421.md
weishen 3f2e1765d8 Auto-sync: 2026-04-18 17:09
2026-04-18 17:09:43 +08:00

22 KiB
Raw Blame History

Transform-the-suite-to-a-Helm-deployment-on-24.3.2_688996421

Helm upgrade procedure

SEQStepsDurationDowntime

Prerequisites

1

Make sure current suite is upgraded to version 24.3.2

1 minNA
2

Run the following command to make sure that all the OMT and suite pods are ready

1 min
kubectl get pod --all-namespaces|grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed

Preparation

1

Download the ESM helm chart to bastion

2 minsNA

Download & Unzip & Verify the signature

2Get environment variables1 min

Run below commands to get ENV variables:

NAMESPACE=`kubectl get namespace|grep itsma | cut -f1 -d " "`
SYSTEM_USER_ID=$(kubectl get configmap -o jsonpath='{.data.system_user_id}' itsma-common-configmap -n $NAMESPACE)
SYSTEM_GROUP_ID=$(kubectl get configmap -o jsonpath='{.data.system_group_id}' itsma-common-configmap -n $NAMESPACE)
SIZE=$(kubectl get configmap -o jsonpath='{.data.itom_suite_size}' itsma-common-configmap -n $NAMESPACE)
echo NAMESPACE: $NAMESPACE SYSTEM_USER_ID: ${SYSTEM_USER_ID}, SYSTEM_GROUP_ID: ${SYSTEM_GROUP_ID}, SIZE: ${SIZE}

NOTE: If your bastion session is expired, run this get ENV variables again. If it expires after you delete the ns, replace the variables manually when executing the command

3Backup the suite external ingress and suite integration ingress.

If you don't have Backup the ingress yaml files which will be used in helm install step to the tmp folder:

NOTE: The ingress names may differ between farms.

kubectl get ingress sma-ingress -n $NAMESPACE -o yaml > suite-ingress-backup.yaml

kubectl get ingress sma-int-ingress -n $NAMESPACE -o yaml > sma-int-ingress-backup.yaml

4

Create additional volumes

Configure NFS volume

1 min

For EKS:

Run the following commands on the bastion node.

NOTE: Please change the mount point with the actual value:

sudo mkdir -p /mnt/efs/var/vols/itom/itsma/config-volume
sudo mkdir -p /mnt/efs/var/vols/itom/itsma/logging-volume
sudo chown -R $SYSTEM_USER_ID:$SYSTEM_GROUP_ID /mnt/efs/var/vols/itom/itsma/config-volume
sudo chown -R $SYSTEM_USER_ID:$SYSTEM_GROUP_ID /mnt/efs/var/vols/itom/itsma/logging-volume
sudo chmod g+w /mnt/efs/var/vols/itom/itsma/config-volume
sudo chmod g+w /mnt/efs/var/vols/itom/itsma/logging-volume
sudo chmod g+s /mnt/efs/var/vols/itom/itsma/config-volume
sudo chmod g+s /mnt/efs/var/vols/itom/itsma/logging-volume

5

Check folder permissions

10 mins

Using the following command to check and change folder permission (Please replace the mount point with actual value:):

Check folder permission:

sudo find /mnt/efs/var/vols/itom -type d -exec stat --format='%u:%g %A %n' '{}' \;| grep -v $SYSTEM_USER_ID:$SYSTEM_GROUP_ID

If the result doesn't include :, change the ownership of each directory or file that you created by using the chown -R : command.

For example (the command find and update the folders' permission exclude the log folder):

sudo find /mnt/efs/var/vols/itom -type d -not -path "/mnt/efs/var/vols/itom/itsma/global-volume/logs/*" -exec chown $SYSTEM_USER_ID:$SYSTEM_GROUP_ID {} +

6

Sync data to new Helm persistent volumes

35 mins

NOTE: Please use the syncData.sh script which in 24.4 GA release package.

NOTE: You may want to clean up the tenant-import and tenant-export folders before sync.

cd ESM_Helm_Chart-2x.x/scripts/transformation
chmod u+x syncData.sh

Use the following command to avoid bastion connection interruption during sync data (Please replace the mount point to actual value you use):

nohup sh -c "printf 'y\ny\ny\ny\n' | sudo./syncData.sh --globalVolumePath /mnt/efs/var/vols/itom/itsma/global-volume --smartanalyticsVolumePath /mnt/efs/var/vols/itom/itsma/smartanalytics-volume --configVolumePath /mnt/efs/var/vols/itom/itsma/config-volume" </dev/null >nohup.out 2>&1 &

Watch the progress by below command:

tail -f nohup.out

For the information, 35 mins for below data size:

The 'global-volume' requires an additional 33 G of free disk space.
The 'config-volume' requires an additional 1.5 G of free disk space.

7

Retrieve system configurations and generate values.yaml file

  • Generate basic values.yaml

  • Generate customized values.yaml

5 mins

Before you begin, ensure that the jq and yq tools are installed

Create a values.yaml file based on the suite environment

cd ESM_Helm_Chart-2x.x/scripts/transformation
chmod u+x generateBasicValuesYaml.sh
./generateBasicValuesYaml.sh

Copy the values.yaml file to the ESM_Helm_Chart-2x.x/charts/ directory.

Save a copy of the values.yaml file in a secure location as backup.

Generate customized values.yaml

Go to the ESM_Helm_Chart-2x.x/scripts/custom_settings directory.Change the script file permission and run script:

cd ESM_Helm_Chart-2x.x/esm-1.0.0+2x.x-xxx/scripts/ custom_settings

chmod u+x generateCustomSettings.sh

./generateCustomSettings.sh

The script generates a customized_values.yaml file in the current directory.

Copy the customized_values.yaml file to the ESM_Helm_Chart-2x.x/charts/ directory.

Copy the customized_values.yaml file to a secure location as a backup copy.

8

Back up OMT and SMA

30 mins

Backup the whole SMAX (RDS/EFS/K8S)

Maintain Window

1

Stop SMA and OMT

10 mins70 mins

1.Stop OMT & SMA:

$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n $NAMESPACE
$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n core

2.Check the pods of OMT & SMA are all stopped.

kubectl get pod -n $NAMESPACE|grep -v -E 'throttling|opentelemetry|toolkit|Completed'
kubectl get pod -n core |grep -v Completed

NOTE: In SaaS simulation ENV, promethues pods are under core namespace and it takes time to shut down these pods

2

Clean up classic SMA resources

5 mins

Use the below command to clean the SMA resource:

kubectl delete ns $NAMESPACE

Verify the ns is deleted:

kubectl get ns

Use the following command to check what resources are being used:

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n $NAMESPACE

Patch the ingress use the following command:

kubectl patch ing sma-ingress -n $NAMESPACE --type=json --patch='[{"op": "remove", "path": "/metadata/finalizers/0"}]'

3

Sync incremental data

2 mins

NOTE: Please use the syncData.sh script which in 24.4 GA release package.

Sync incremental data:

Go to the ESM_Helm_Chart-2x.x/scripts/transformation directory and run below command again:

sudo./syncData.sh --globalVolumePath /mnt/efs/var/vols/itom/itsma/global-volume --smartanalyticsVolumePath /mnt/efs/var/vols/itom/itsma/smartanalytics-volume --configVolumePath /mnt/efs/var/vols/itom/itsma/config-volume

NOTE: The incremental sync should not cost long time as the first sync, while still you can use the below command to avoid bastion connection interruption during sync data(Please change the mount point as your actual environment):

nohup sh -c "printf 'y\ny\ny\ny\n' | sudo./syncData.sh --globalVolumePath /mnt/efs/var/vols/itom/itsma/global-volume --smartanalyticsVolumePath /mnt/efs/var/vols/itom/itsma/smartanalytics-volume --configVolumePath /mnt/efs/var/vols/itom/itsma/config-volume" </dev/null >nohup.out 2>&1 &

Then watch the progress by below command:

tail -f nohup.out

4

Update the label for the core ns

1 min

Update the label for core:

kubectl patch ns core -p '{"metadata":{"labels":{" deployments.microfocus.com/deployment-name":"cdf "}}}'

5

Create a deployment for the suite

1 min

Create a deployment for suite:

$CDF_HOME/bin/cdfctl deployment create -d $NAMESPACE

NOTE: This command creates a deployment with your original suite namespace as the deployment name.

6

Refine existing PVs

1 min

  1. Go to the ESM_Helm_Chart-2x.x/scripts/transformation directory.

    cd ESM_Helm_Chart-2x.x/esm-1.0.0+2x.x-xxx/scripts/transformation

  2. Run the following commands:

    chmod u+x refinePV.sh 
./refinePV.sh $SIZE
NOTE: Enter 2 'y' when execute the shell

  3. Run the following command to verify the PV creation.

    kubectl get pv|grep -E "config-volume|logging-volume|data-volume"|grep itsma

  4. Run the following command to verify the PV status is Available (Below command is supposed to return no result)

    kubectl get pv|grep itsma|grep -v -E "db-volume|global-volume|smartanalytics"|awk '{if ($5!="Available") print $0}'

7Restore vault data
  • Copy vault data from core-volume to global-volume
  • Copy vault secrets to the suite namespace
5 mins

Copy vault data from core-volume to global-volume
On NFS server run:

sudo cp -R /mnt/efs/var/vols/itom/itsma/core/vault /mnt/efs/var/vols/itom/itsma/global-volume/
sudo chown -R $SYSTEM_USER_ID:$SYSTEM_GROUP_ID /mnt/efs/var/vols/itom/itsma/global-volume/vault

Copy vault secrets to the suite namespace

  1. Create script "copyVaultCredentials.sh" > Restore_vault_data
  2. Run below command
    chmod u+x copyVaultCredentials.sh
    ./copyVaultCredentials.sh $NAMESPACE
NOTE: Replace with a unique release name that you want to specify for the suite deployment.
8

Start OMT

5 mins
  1. Start OMT:
    $CDF_HOME/bin/cdfctl runlevel set -l UP -n core
  2. Check that all the OMT pods are ready:
    kubectl get pods -n core|grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed
9

Install the ESM helm chart

Create ingress for the suite (EKS only)

35 mins

Login Bastion and change folder to the ESM_Helm_Chart-2x.x/charts/ directory.
Run the helm install command:

helm install esm-1.0.0+2x.x-xxx.tgz -n $NAMESPACE -f values.yaml --set global.nodeSelector.Worker=label -f customized_values.yaml

NOTE: This release name is the one you gave in step 7

During the helm installation, monitor the status of the itom-nginx-ingress service by running the below command:

kubectl get svc -n $NAMESPACE|grep itom-nginx-ingress-svc

[EKS Only] Once the service is available, create the ingress for the suite & ingress for the SMAX integration.

For example:

kubectl create -f sma-ingress.yaml
kubectl create -f sma-ingress-integration.yaml

NOTE: The ingress yaml files are the ones you exported in Preparation - Step 3

Bound the newly created the ingress alb to Ops domain(**- smax.esm.com)

Add the newly created service port to the EKS worker's inbound security group

Reference:

10

Enable helm autopass

3 mins

  1. Log in to the control plane node or bastion node.

  2. Change to the ESM_Helm_Chart-2x.x/scripts/transformation directory.

  3. Change the permission of the updateAutopassKey.sh file:

    chmod u+x updateAutopassKey.sh

  4. Run the script to sync the autopass key.

    ./updateAutopassKey.sh -n $NAMESPACE

    It will restart the itom-bo-license-deployment and autopass-lm-v2 pods.

  5. After the above pods are up and running, enter the autopass page with the new URL: https:///autopass.

Ensure all suite pods & job are ready

2 mins

Check helm install pod status

kubectl get pod -n $NAMESPACE|grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed

Kubectl get job -n $NAMESPACE

Post-tasks

0

(Optional) Reinstall lost services:

Toolkit,

Monitoring,

Opentelemetry

After install the helm version suite, certain ports in suite will be reset. We have to re-enable them again in suite. Please run following command"

1.Run the following patch command:

kubectl patch svc idm-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/2","value":{"name":"ssl","port":443,"protocol":"TCP","targetPort":8443}},{"op":"add","path":"/spec/ports/3","value":{"name":"metrics","port":444,"protocol":"TCP","targetPort":8444}}]'#expose nginx portkubectl patch svc itom-nginx-ingress-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/1","value":{"name":"http-metrics","port":10254,"protocol":"TCP","targetPort":10254}}]'#expose redis portkubectl patch svc itom-xruntime-redis-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/1","value":{"name":"https-metrics","port":9121,"protocol":"TCP","targetPort":9121}}]'#expose rabbitmq port#kubectl patch svc itom-xruntime-rabbitmq-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/4","value":{"name":"http-metrics","port":9419,"protocol":"TCP","targetPort":15691}}]'#expose port for JMX platformkubectl patch svc itom-xruntime-platform-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/3","value":{"name":"platform-metrics","port":5555,"protocol":"TCP","targetPort":5555}}]'kubectl patch svc itom-xruntime-platform-offline-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/3","value":{"name":"platform-offline-metrics","port":5555,"protocol":"TCP","targetPort":5555}}]'kubectl patch svc itom-xruntime-platform-offline-ng-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/3","value":{"name":"platform-offline-ng-metrics","port":5555,"protocol":"TCP","targetPort":5555}}]'kubectl patch svc itom-xruntime-platform-readonly-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/3","value":{"name":"platform-readonly-metrics","port":5555,"protocol":"TCP","targetPort":5555}}]'#expose port for JMX gatewaykubectl patch svc itom-xruntime-gateway-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/3","value":{"name":"gateway-metrics","port":5555,"protocol":"TCP","targetPort":5555}}]'#expose port for JMX service portalkubectl patch svc itom-xruntime-serviceportal-svc -n $(kubectl get namespace |grep itsma | cut -f1 -d " ") --type=json -p='[{"op":"add","path":"/spec/ports/3","value":{"name":"serviceportal-metrics","port":5555,"protocol":"TCP","targetPort":5555}}]'

2.Go to ITOM Marketplace to download all Service Monitor definitions

3.Unzip the package and navigate to the folder prometheus/servicemonitor and run the following command:

namespace=`kubectl get namespace |grep itsma| cut -f1 -d " "`
sed -i -e 's//'$namespace'/g' *.yaml

sed -i -e 's/insecureSkipVerify: false/insecureSkipVerify: true/g' *.yaml
kubectl delete -f./
kubectl create -f./

1

Clean up unused pods in the OMT namespace

5 minsNA

Run the following command to remove cdf-apiserver, cdfapiserverdb, frontendIngress, itom-frontend-ui, and itom-mng-portal resources:

helm upgrade apphub $CDF_HOME/charts/apphub-1.2*.tgz --reuse-values --set global.services.suiteDeploymentManagement=false -n core

kubectl delete deploy suite-conf-pod-itsma -n core --ignore-not-found=true

kubectl delete svc suite-conf-svc-itsma -n core --ignore-not-found=true

kubectl delete ingress suite-conf-ing-itsma -n core --ignore-not-found=true

If you are restoring a single namespace after running above command, it will fail, please do workaround with below wiki:

OMT clusterrolebinding and clusterrole

After helm transformation, OMT 5443 portal will no longer be in use.


Run the following command to clean up OMT install portal ingress for ALB controller:

kubectl delete ingress -n core -l app=install-ingress
2

Delete unused PVs

1 mins
kubectl delete pv $NAMESPACE-db-volume
kubectl delete pv $NAMESPACE-smartanalytics-volume
kubectl delete pv $NAMESPACE-global-volume
3

Delete unused nfs folders

15 mins

Run the following command to clean unused nfs folder:

dbVolume=

globalVolume=

smartanalyticsVolume=

Where: and are the NFS paths of db-volume, global-volume, and smartanalytics-volume, respectively.

For example:

dbVolume=/mnt/efs/var/vols/itom/itsma/db-volume

globalVolume=/mnt/efs/var/vols/itom/itsma/global-volume

smartanalyticsVolume=/mnt/efs/var/vols/itom/itsma/smartanalytics-volume


sudo rm -rf $dbVolume
sudo rm -rf $smartanalyticsVolume
sudo rm -rf $globalVolume/data
sudo rm -rf $globalVolume/image
sudo rm -rf $globalVolume/resources
sudo rm -rf $globalVolume/config
sudo rm -rf $globalVolume/certificate
sudo rm -rf $globalVolume/tenant-export
sudo rm -rf $globalVolume/tenant-import
sudo rm -rf $globalVolume/jdbc
sudo rm -rf $globalVolume/l10n
sudo rm -rf $globalVolume/va/configs
sudo rm -rf $globalVolume/va/stopwords
sudo rm -rf $globalVolume/di
sudo rm -rf $globalVolume/logs

Verification

1SMAX30 minsNA
2NSACM Sanity
3Audit Sanity
4Audit-Collector Sanity

Doc Link:

Transform SMAX To Helm

Related pages

Reference in New Issue View Git Blame Copy Permalink