3 lines
11 KiB
Markdown
3 lines
11 KiB
Markdown
# AWS-RDS-certificate-update--Helm-Fedramp-simulation-ENV_688983269
|
|
<table><colgroup><col> <col> <col> <col> <col> <col></colgroup><tbody><tr><td><p><strong>Tasks</strong></p></td><td><p><strong>Products</strong></p></td><td><p><strong>Steps</strong></p></td><td><p><strong>Duration</strong></p></td><td><p><strong>Downtime</strong></p></td><td></td></tr><tr><td><p><strong>Prepare:</strong></p><p><strong>Certificate File Preparation</strong></p></td><td><p><strong>Download the new AWS RDS certificate bundle PEM file</strong></p></td><td><div><strong>Download the new AWS RDS certificate bundles for specific AWS region from the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.RegionCertificates">Certificate bundles for specific AWS Regions</a>.</strong><p>For region of us-gov-west-1, download the below certificate:</p><p><img height="250" src="attachments/688983269/688983236.png"></p><strong>Upload the certificate bundle to the bastion.</strong></div></td><td></td><td></td><td></td></tr><tr><td rowspan="4"><p><strong>Prepare:</strong></p><p><strong>Update certificate configuration in application side</strong></p></td><td><p><strong>OMT</strong></p></td><td><p>1. Acquire database info before running the script:</p><p>You may get the db user, db name and PASSWORD_KEY values from database configmap with below commands:</p><p>kubectl get cm default-database-configmap -n <CDF NAMESPACE> -o yaml</p><p>The result is like:</p><p><strong>DEFAULT_DB_CDFIDM_PASSWORD_KEY</strong>: defaultdb_cdfidm_user_password</p><p><strong>DEFAULT_DB_CDFIDM_USERNAME</strong>: cdfidm</p><p><strong>DEFAULT_DB_HOST</strong>: xxxxxyyyyy<a href="http://us7-simulation-smax.ctz1ruxhq5vf.us-west-2.rds.amazonaws.com/">.us-west-2.rds.amazonaws.com</a></p><p><strong>DEFAULT_DB_NAME</strong>: itom-cdf-idm</p><p>2. Get the cdfidm db password:</p><p>kubectl get pod -n $CDF_NAMESPACE | grep "itom-idm" | head -1 | awk '{print $1}'</p><p>kubectl exec -it <idm pod> -n $CDF_NAMESPACE -c idm -- get_secret <password key></p><p>For example:</p><p>kubectl exec -it $(kubectl get pod -n $CDF_NAMESPACE | grep "itom-idm" | head -1 | awk '{print $1}') -n $CDF_NAMESPACE -c idm -- get_secret defaultdb_cdfidm_user_password</p><p><strong>Note:</strong> Record the database info and password, they will be used in execute command</p></td><td></td><td></td><td><p><a href="https://docs.microfocus.com/doc/SMAX/24.2/ModifyExternalDBConfig">https://docs.microfocus.com/doc/SMAX/24.2/ModifyExternalDBConfig</a></p></td></tr><tr><td><p><strong>SMAX & HCMX</strong></p></td><td><div><ol><li>[Optional] If you don't remember the path of your custom <strong>my-values.yaml</strong> file, run the following command to get your yaml file.<br>helm get values <ESM_RELEASE_NAME> -n <ESM_NAMESPACE> > <ESM_VALUES_YAML></li><li>Confirm if tlsEnabled is set to true and replace the caCertificates under "database" with the <strong>new PEM content</strong> directly or base64 all the content (e.g cat <certFile> | base 64)<br>Example:<br>global:<br>database:<br>tlsEnabled: true<br>tlsMode: verify-full # or use verify-ca<br>......<br>database:<br>caCertificates:<br>pg_ca.crt: <cert-file-base64-encoded> or PEM<br><p><strong>Note:</strong></p><strong>If you are using base64 encoded content, make sure the encoded content is formatted in one line, as below:</strong><strong><img height="81" src="attachments/688983269/688983243.png"></strong> <strong>If you are not using base64 encoded content, please refer to below format:</strong></li><li><img height="136" src="attachments/688983269/688983248.png"></li><li>Run the command to check if job exist in ENV. It should return nothing. If job exist, make sure it's in Completed status and delete it.<br>kubectl get job -n <ESM_NAMESPACE>|grep -v NAME | awk '{print $1}'<br>Run below command to get the pod status - if they are "Completed", it means the jobs are done and you may delete the jobs with 'delete job' command:<br>kubectl get pods -n <EMS_NAMESPACES> |grep job<br>If above pods are all completed status, delete related jobs:<br>kubectl delete job <the jobs identified in get job command> -n <EMS NS></li></ol><p>NOTE: The yaml file with new pem content replaced will be used in RDS certificate replacement.Reference: <a href="https://staging.docs.microfocus.com/doc/SMAX/Main/ChangeCertForPostgreSQL">https://staging.docs.microfocus.com/doc/SMAX/Main/ChangeCertForPostgreSQL</a></p></div></td><td></td><td></td><td><p><a href="https://docs.microfocus.com/doc/SMAX/24.2/ModifyExternalDBConfig">https://docs.microfocus.com/doc/SMAX/24.2/ModifyExternalDBConfig</a></p></td></tr><tr><td><p><strong>CMS</strong></p></td><td><div><ol><li>Get the CMS values.yaml from current running deployment by running below command</li></ol><p>helm get values <CMS_RELEASENAME> -n <CMS_NAMESPACE> > values.yaml</p><p>2. Replace the content of caCertificates.postgresql.crt in values.yaml with the content of AWS RDS certificate bundle got at above step.</p><p><strong>Note:</strong>every line of certificate content starts with 4 indentations in values.yaml, for example:</p><p><img height="187" src="attachments/688983269/688983253.png"></p></div></td><td></td><td></td><td></td></tr><tr><td><p><strong>Audit</strong></p></td><td><div><ol><li>Get the Audit values.yaml</li></ol><p>helm get values <Audit_RELEASE_NAME> -n <AUDIT_NAMESPACE> > values.yaml</p><p>2. Replace the content of caCertificates.RE_ca_dbcrt in values.yaml with the content of AWS RDS certificate bundle got at above step.</p><p><strong>Note:</strong> every line of certificate content starts with 4 indentations in values.yaml, for example:</p><p><img height="192" src="attachments/688983269/688983256.png"></p></div></td><td></td><td></td><td></td></tr><tr><td rowspan="6"><p><strong>Execute certificate update in application side</strong></p><p><strong>Note:</strong> There is no dependency on each application.</p></td><td><p><strong>OMT</strong></p></td><td><p>Navigate to the $CDF_HOME/bin directory, run the updateExternalDbInfo.sh script with below parameters:</p><p>./updateExternalDbInfo.sh -H <DB connection URL> -p <DB port> -d <DB name> -u <database username> --dbpassword <database password> --component itom-idm --cacert <Database Certificate></p><p>For example:</p><p><em>.</em>/updateExternalDbInfo.sh -H <a href="http://saas-simulation-smax.ctz1ruxhq5vf.us-west-2.rds.amazonaws.com/">xxxxyyyy.us-west-2.rds.amazonaws.com</a> -p 5432 -d cdfidmdb -u cdfidm --dbpassword <password> --component itom-idm --cacert /home/ssm-user/us-gov-west-1-bundle.pem</p><p>Reference: <a href="https://docs.microfocus.com/doc/OMT/24.2/ModifyExternalDatabaseConfiguration">https://docs.microfocus.com/doc/OMT/24.2/ModifyExternalDatabaseConfiguration</a></p></td><td><p>1min</p></td><td><p>0</p></td><td></td></tr><tr><td><p><strong>SMAX & HCMX</strong></p></td><td><ol><li>Run the following commands to apply DB setting change. The yaml file is the one with new pem content replaced in preparation steps.<br>helm upgrade <ESM_RELEASE_NAME> <ESM_CHART_FILE> -n <ESM_NAMESPACE> -f my-values.yaml<br>For example:<br>helm upgrade itsma ESM_Helm_Chart-2x.x/charts/esm-1.0.0+2x.x-xxx.tgz -n itsma-xxyy -f values.yaml<br></li><li><p>Run the following command to make sure that all SMAX pods are ready before next step.<br>kubectl get pod -n < <em>ESM_NAMESPACE</em> > |grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed<br></p></li><li><p>Restart the suite (Another option is to do the restart later via one command along with other products)<br>$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n <ESM_NAMESPACE><br>Wait till all pods are down, then run below command to bring the application up<br>$CDF_HOME/bin/cdfctl runlevel set -l UP -n <ESM_NAMESPACE></p></li></ol></td><td><p>4mins</p></td><td><p>0</p></td><td></td></tr><tr><td><p><strong>CMS</strong></p></td><td><ol><li>Update the deployment by running helm upgrade command. The yaml file is the one with new pem content replaced in preparation steps.<br><em>helm upgrade <cms_releasename> <CMS CHART FILE> -n <cms_namespace> -f values.yaml</em></li><li>Restart CMS (Another option is to do the restart later via one command along with other products)<br>$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n <CMS_NAMESPACE><br>wait till all pods are shut down<br>$CDF_HOME/bin/cdfctl runlevel set -l UP -n <CMS_NAMESPACE></li></ol><p>NOTE: You may do this in parallel with SMAX restart</p></td><td><p>1min</p></td><td><p>0</p></td><td></td></tr><tr><td><p><strong>Audit</strong></p></td><td><ol><li>Update the deployment by running helm upgrade command. The yaml file is the one with new pem content replaced in preparation steps.<br><em>helm upgrade <audit_release_name> -n <audit_namespace> -f values.yaml <audit_CHART_FILE></em></li><li>Restart Audit (Another option is to do the restart later via one command along with other products)</li></ol><p>$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n <Audit_NAMESPACE><br>wait till all pods are shut down<br>$CDF_HOME/bin/cdfctl runlevel set -l UP -n <Audit_NAMESPACE></p><p>NOTE: You may do this in parallel with SMAX restart</p></td><td><p>1min</p></td><td><p>0</p></td><td></td></tr><tr><td><p><strong>Restart pods (Alternative)</strong></p></td><td><p>You may also do the helm upgrade for all products in parallel without restarting.</p><p>Then do the restart against all products whose RDS certificates were changed</p><p>For example:</p><p>$CDF_HOME/bin/cdfctl runlevel set -l DOWN -n <OMT_NAMESPACE>,<ESM_NAMESPACE>,<AUDIT_NAMESPACE>,<CMS_NAMESPACE><br>wait till all pods are shut down and start the pods<br>$CDF_HOME/bin/cdfctl runlevel set -l UP -n <OMT_NAMESPACE>,<ESM_NAMESPACE>,<AUDIT_NAMESPACE>,<CMS_NAMESPACE></p></td><td><p>14mins</p></td><td><p>14mins</p></td><td></td></tr><tr><td><strong>Monitor the restart till all pods are started</strong></td><td><p>kubectl get pod -n < <em>ESM_NAMESPACE</em> > |grep -v 1/1|grep -v 2/2|grep -v 3/3|grep -v 4/4|grep -v Completed</p></td><td></td><td></td><td></td></tr><tr><td><p><strong>Update the certificates of AWS RDS DB instances.</strong></p></td><td><p><strong>Update the certificate on AWS RDS DB instances.</strong></p></td><td><p>1.Login AWS console, go to the RDS instances that you want to update the certificates.</p><p>2.Select the RDS instance, click modify button</p><p>3.Change the <strong>Certificate authority.</strong></p><p><strong>If your primary certificate CA is rds-ca-2019, it's recommended to select the rds-ca-rsa4096-g1 CA as new value</strong></p><p><strong><img height="250" src="attachments/688983269/688983260.png"></strong></p><p>4.Save the change and select <strong>immediate effect</strong>.</p><p>5.Repeat the steps for all your RDS instances</p></td><td><p>2mins</p></td><td><p>0</p></td><td></td></tr></tbody></table>
|