ishenwei/nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d2ae5b39488b8e76e70d71a6881ee53caeefa2dc
nexus/knowledgebase/csd-wiki/ICSD/AWS-RDS-certificate-update--Helm-Simulation-env_686088156.md

11 KiB
Raw Blame History

AWS-RDS-certificate-update--Helm-Simulation-env_686088156

TasksProductsDurationSteps

Preparation

Download the new AWS RDS certificate bundle PEM file5 mins

Download the new AWS RDS certificate bundles for specific AWS region from the Certificate bundles for specific AWS Regions.

for example, for region of us-west-2, download the below certificate:

Upload the certificate bundle to the bastion.

Note

  1. There is no dependency on below operations for each product. You may prepare the yaml files (this can be done before the maintain window) and run the steps in parallel to reduce the ESM level downtime.
  2. During the restart the applications (OMT/Suite/CMS/OO/Audit) are not able to access, which is considered as the downtime of this whole process.
OMT5 mins

1. Acquire database info before running the script:

Note: Above are OOB values, if you are not using OOB values, you may get the values with below commands:

kubectl get cm default-database-configmap -n -o yaml

The result is like:

DEFAULT_DB_CDFIDM_PASSWORD_KEY: defaultdb_cdfidm_user_password
DEFAULT_DB_CDFIDM_USERNAME: cdfidm
DEFAULT_DB_HOST:xxxxxyyyyy.us-west-2.rds.amazonaws.com
DEFAULT_DB_NAME: cdfidmdb

You may find the db user, db name and PASSWORD_KEY value from database configmap.

2. Get the cmfidm db password.
kubectl get pod -n $CDF_NAMESPACE | grep "itom-idm" | head -1 | awk '{print $1}'
kubectl exec -n $CDF_NAMESPACE -c idm -- get_secret

For example:
kubectl exec $(kubectl get pod -n $CDF_NAMESPACE | grep "itom-idm" | head -1 | awk '{print $1}') -n $CDF_NAMESPACE -c idm -- get_secret defaultdb_cdfidm_user_password

Take the note of your which you will be using in OMT certificate replacement.

Reference: https://docs.microfocus.com/doc/OMT/24.2/ModifyExternalDatabaseConfiguration

SMAX & HCMX10 mins

  1. [Optional] If you don't remember the path of your custom my-values.yaml file, run the following command to get your yaml file.

    helm get values -n > my-values.yaml

  2. Confirm if tlsEnabled is set to true and replace the caCertificates under "database" with the new PEM content directly or base64 all the content (e.g cat | base 64).

    Example:

    global:
  database:
    tlsEnabled: true
    tlsMode: verify-full    # or use verify-ca
... ...
database:
  caCertificates: 
    pg_ca.crt:  or PEM

    Note:

    If you are using base64 encoded content, make sure the encoded content is formatted in one line, as below:If you are not using base64 encoded content, please refer to below format:

  3. Run the command to check if job exist in ENV. It should return nothing. If job exist, make sure it's in Completed status and delete it.
    kubectl get job -n |grep -v NAME | awk '{print $1}'
    Run below command to get the pod status - if they are "Completed", it means the jobs are done and you may delete the jobs with 'delete job' command:
    kubectl get pods -n |grep job
    If above pods are all completed status, delete related jobs:
    kubectl delete job -n

NOTE: The yaml file with new pem content replaced will be used in RDS certificate replacement.

Reference: https://staging.docs.microfocus.com/doc/SMAX/Main/ChangeCertForPostgreSQL

CMS5 mins

1.Get the CMS values.yaml from current running deployment by running below command

helm get values -n >values.yaml

2.Replace the content of caCertificates.postgresql.crt in values.yaml with the content of AWS RDS certificate bundle got at above step.

Note: every line of certificate content starts with 4 indentation in values.yaml

OO5 mins

1.Get the OO values.yaml

helm get values -n > values.yaml

2.Replace the content of caCertificates.postgresql.crt in values.yaml with the content of AWS RDS certificate bundle got at above step.

Note: every line of certificate content starts with 4 indentation in values.yaml

Audit5 mins
  1. Get the Audit values.yaml
    helm get values -n > values.yaml
  2. Replace the content of caCertificates.RE_ca_dbcrt in values.yaml with the content of AWS RDS certificate bundle got at above step.

    Note: every line of certificate content starts with 4 indentation in values.yaml

Maintain Window

Update the certificate in application side

OMT5 mins

Navigate to the $CDF_HOME/bin directory, run the updateExternalDbInfo.sh script with below parameters:

NOTE: you may find the DB parameters in preparation steps.

./updateExternalDbInfo.sh -H -p -d -u --dbpassword --component itom-idm --cacert
For example:
./updateExternalDbInfo.sh -H xxxxyyyy.us-west-2.rds.amazonaws.com -p 5432 -d cdfidmdb -u cdfidm --dbpassword --component itom-idm --cacert /home/centos/ us-west-2-bundle.pem

SMAX/HCMX

30 mins

  1. Run the following commands to apply DB setting change. The yaml file is the one with new pem content replaced in preparation steps.
    helm upgrade -n -f my-values.yaml
    For example:
    helm upgrade itsma ESM_Helm_Chart-2x.x/charts/esm-1.0.0+2x.x-xxx.tgz -n itsma-xxyy -f values.yaml

  2. Run the following command to make sure that all SMAX pods are ready before next step.
    kubectl get pod -n < ESM_NAMESPACE > |grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed

  3. Restart the suite
    $CDF_HOME/bin/cdfctl runlevel set -l DOWN -n
    Wait till all pods are down, then run below command to bring the application up
    $CDF_HOME/bin/cdfctl runlevel set -l UP -n

CMS

20 mins

1. Update the deployment by running helm upgrade command. The yaml file is the one with new pem content replaced in preparation steps.

helm upgrade -n -f values.yaml

2. Restart CMS

$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n

wait till all pods are shut down

$CDF_HOME/bin/cdfctl runlevel set -l UP -n

3. Monitor pod status:

kubectl get pod -n NAMESPACE > |grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed

NOTE: You may do this in parallel with SMAX restart

OO20 mins

1.Update the deployment by running helm upgrade command. The yaml file is the one with new pem content replaced in preparation steps.

helm upgrade -n -f values.yaml

2. Restart OO

$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n
wait till all pods are shut down
$CDF_HOME/bin/cdfctl runlevel set -l UP -n

3. Monitor pod status:

kubectl get pod -n NAMESPACE > |grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed

NOTE: You may do this in parallel with SMAX restart

Audit5 mins
  1. Update the deployment by running helm upgrade command. The yaml file is the one with new pem content replaced in preparation steps.
    helm upgrade -n -f values.yaml
  2. Restart Audit

$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n
wait till all pods are shut down
$CDF_HOME/bin/cdfctl runlevel set -l UP -n

3. Monitor pod status:

kubectl get pod -n NAMESPACE > |grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed

NOTE: You may do this in parallel with SMAX restart

Update the certificates of AWS RDS DB instances.

Update the certificate on AWS RDS DB instances.10 mins

1.Login AWS console, go to the RDS instances that you want to update the certificates.

2.Select the RDS instance, click modify button

3.Change the Certificate authority.

If your primary certificate CA is rds-ca-2019, it's recommended to select the rds-ca-rsa2048-g1 CA as new value

4.Save the change, and select immediate effect.

Repeat the steps for all your RDS instances

Reference in New Issue View Git Blame Copy Permalink