26 lines
2.9 KiB
Markdown
26 lines
2.9 KiB
Markdown
# Workaround-to-avoid-EFS-credits-used-up-after-upgrading-SaaS-UCMDB-to-24.3.2_688996457
|
|
### Background:
|
|
|
|
Previously in EU28 there was a issue reported by Evonik that the result processing thread died and caused probe discovery result not processing, but we don't know why due to insufficient information. To figure out the reason, we printed out more details in the logs ([Issue 2230863 - Results processing thread died and caused probe discovery results not processing](https://internal.almoctane.com/ui/entity-navigation?p=97002/20001&entityType=work_item&id=2230863)) in 24.3.2.
|
|
|
|
Later after US26 upgrade to 24.3.2, we observed the same problem of result processing for Salesforce. Thanks to the more information in the log, we now know the root cause (and fixed accordingly [Issue 2386070 - Results processing component is not initialized when farm restart due to NPE and caused EFS credits used up](https://internal.almoctane.com/ui/entity-navigation?p=97002/32001&entityType=work_item&id=2386070)). But it brings another side effect that the EFS credits will be used up soon after upgrading to 24.3.2. The workaround for the side effect is to restart the problematic tenants.
|
|
|
|
After discussion, we concluded that to avoid the same problem (EFS credits used up after upgrade) for other farms, it's better that SaaS Ops can have a check after upgrade. If the problem is reproduced, then perform below steps to work around it.
|
|
|
|
### Steps:
|
|
|
|
1. Prerequisite: SaaS UCMDB has been upgraded to 24.3.2
|
|
2. Find the writer pod: kubectl get po -n cms --show-labels | grep writer
|
|
3. Go to the writer efs, choose one of the following based on step 2:
|
|
1. cd /mnt/cms/var/vols/itom/cms/log\_volume/ucmdb/server/itom-ucmdb-0
|
|
cd /mnt/cms/var/vols/itom/cms/log\_volume/ucmdb/server/itom-ucmdb-1
|
|
4. vi error.log, check if there are lots of logs like below (more than 100 times per minute)
|
|
1. 2024-09-23 06:44:12,962 ERROR \[Process Results Thread (customer 105492793; idx 1)\]\[System\] (:) - Unexpected exception happened in result processing
|
|
java.lang.NullPointerException: null
|
|
at com.hp.ucmdb.discovery.server.framework.discovery.manager.AutoDiscoveryProbeManagerImpl$ProcessResultsThread.authorizedRun(AutoDiscoveryProbeManagerImpl.java:1611) ~\[discovery-server.jar:11.8.5.12\]
|
|
at com.hp.ucmdb.discovery.server.utils.AbstractAuthorizedThread$1.executeInContext(AbstractAuthorizedThread.java:26) ~\[discovery-server.jar:11.8.5.12\]
|
|
at com.hp.ucmdb.discovery.server.utils.AbstractAuthorizedThread$1.executeInContext(AbstractAuthorizedThread.java:23) ~\[discovery-server.jar:11.8.5.12\]
|
|
at com.mercury.topaz.cmdb.shared.manage.AuthorizationContextUtils.executeInSystemAuthorizationContext(AuthorizationContextUtils.java:24) ~\[cmdb-framework.jar:11.8.5.12\]
|
|
5. if no such errors, then it's fine; if there are such errors, restart the customer mentioned in the log in jmx
|
|
6. check the error.log again to confirm if the error persists
|