133 lines
8.5 KiB
Markdown
133 lines
8.5 KiB
Markdown
# How-to-fix-inability-to-add-new-Data-Domains_716270786
|
||
## Introduction
|
||
|
||
This runbook will fix a customer's inability to add new Data Domains if they are receiving the following error "Metadata element DataDomains.<customName> is invalid. Enumeration descriptor is invalid, since the orders defined for its values are not greater than or equal to 1. Refer to customization"
|
||
|
||
The problem occurred due to a defect for Dev2Prod process in RangeEnumerationValueOrdersPreProcessorForImport. In SMAX versions priori to 25.2, when importing DataDomains fields, the code would assign sequential order\_key values starting at 1 to newly imported DataDomains. It would persist any existing order\_keys, so generally if all newly imported DataDomains appeared at the end of the list, all was well. However, if Dev and Prod environments were out of sync, i.e. manually added domains to both environments in a different order, then the new DataDomain would be assigned an existing order\_key. This was handled later by the function "mergePreValidatedEnumerations()" and would assign a random order\_key between 10000-32767. This worked fine, although the DataDomains may be imported out of order, until 25.2 changed the logic.
|
||
|
||
In 25.2 the DataDomains no longer got sequential IDs starting at 1, but from the highest order\_key already present. This meant they would no longer be duplicate keys to generate random ones. However, some unlucky customers that had existing order\_keys that were generated significantly high were now limited by the SQL/Java limits on "smallInt", maxing out at 32767. If a customer's DataDomains max order key reached 32767, the next DataDomain would try to be added with 32767+1, which would overflow and become -32768 and no longer be valid.
|
||
|
||
No new customers will get random order\_keys anymore, so this will not be a problem for any customer whose max order\_key for DataDomains is below 31767, since the max number of enum items is 1000. However, to be safe from future changes, we should fix any customer whose max order\_key is over 30000. We must remap the DataDomains order\_keys in enumeration\_value, enum\_value\_<tenantId> and entities\_<tenantId> via SQL.
|
||
|
||
**Note: This script applies to SMAX versions from 25.2 and higher. Back up your data in the following EMS tables before running this script: enumeration\_value, enum\_value\_<tenantId> and entities\_<tenantId>**
|
||
|
||
## Resolution
|
||
|
||
Pre-requisites:
|
||
|
||
- EMS database access
|
||
- RabbitMQ Management console or REST API access
|
||
|
||
Steps:
|
||
|
||
1. Download the SQL scripts and update <tenantId> throughout each of them.
|
||
1. [find\_data\_domains.sql](attachments/716270786/716270787.sql)
|
||
2. [update\_data\_domains.sql](attachments/716270786/716270788.sql)
|
||
2. Run find\_data\_domains.sql and record the output. You will get:
|
||
1. A helper SQL to find all entities that will need updating that can be used in pre- and post-validation, for example:
|
||
```
|
||
NOTICE:
|
||
--- HELPER SQL TO BE RUN INDEPENDENTLY TO VIEW ALL ENTITIES NEEDING UPDATE ---
|
||
select DISTINCT entity_id, entity_type_id, ed.name as entity_type, schar0, data_domains, e.usmallint0 as Company_dd, e.usmallint1 as Idea_dd, last_update_time from entities_100000002 e
|
||
join entity_descriptor ed on e.entity_type_id = ed.id WHERE
|
||
EXISTS (
|
||
SELECT 1
|
||
FROM unnest(e.data_domains) AS dd
|
||
WHERE dd > 10000
|
||
) OR
|
||
(e.entity_type_id = 498 AND e.usmallint0 >= 10000) OR
|
||
(e.entity_type_id = 435 AND e.usmallint1 >= 10000);
|
||
```
|
||
2. If Data Domains has been used in custom fields on any entities:
|
||
1. A SELECT statement to copy into update\_data\_domains.sql for pre-validation
|
||
```
|
||
NOTICE:
|
||
--- SELECT STATEMENT TO ADD TO update_data_domains.sql for COUNT ---
|
||
SELECT COUNT(*) INTO row_count FROM entities_100000002 e WHERE
|
||
(e.entity_type_id = 498 AND e.usmallint0 = ANY(old_keys)) OR
|
||
(e.entity_type_id = 435 AND e.usmallint1 = ANY(old_keys));
|
||
```
|
||
2. A series of UPDATE statements to copy into update\_data\_domains.sql for updating data domains on custom fields
|
||
```
|
||
NOTICE:
|
||
--- UPDATE STATEMENTS TO ADD TO update_data_domains.sql for mapping new order_keys ---
|
||
UPDATE entities_100000002 e SET usmallint0 = t.new_key FROM temp_new_keys t WHERE usmallint0 = t.old_key AND e.entity_type_id = 498;
|
||
UPDATE entities_100000002 e SET usmallint1 = t.new_key FROM temp_new_keys t WHERE usmallint1 = t.old_key AND e.entity_type_id = 435;
|
||
```
|
||
3. If the Data Domains have not been used in any custom fields, then you will get a simple notice and simply remove the RAISE EXCEPTION in the SELECT and UPDATE locations in the update\_data\_domains.sql
|
||
```
|
||
NOTICE: No relevant entity-property mappings found. You may remove the SELECT AND UPDATE RAISE EXCEPTIONS in update_data_domains.
|
||
```
|
||
3. Run update\_data\_domains.sql with doUpdate=false (the default) and SAVE and review the output.
|
||
1. You will observe the list of enums that will be updated, for example:
|
||
```
|
||
NOTICE: 2025-09-02 16:52:49.399163+00 - ℹ️ Detected DataDomains that require updating:
|
||
NOTICE: No key conflicts detected.
|
||
NOTICE: Name: NickTest3_c, OldKey: 30009, NewKey: 1009
|
||
NOTICE: Name: NickTest4_c, OldKey: 30010, NewKey: 1010
|
||
NOTICE: Name: NickTest5_c, OldKey: 30011, NewKey: 1011
|
||
```
|
||
2. And you will observe information about how many entities will need to be updated:
|
||
```
|
||
NOTICE: 2025-09-02 16:52:49.612168+00 - ℹ️ data_domains system field on 2 entities requires updating.
|
||
NOTICE: 2025-09-02 16:52:49.614872+00 - ℹ️ data_domains custom field on 3 entities requires updating.
|
||
NOTICE: 2025-09-02 16:52:49.614958+00 - ℹ️ Have you saved the above output? Have you taken a DB backup? If you are ready to proceed with the update, change doUpdate to true and re-run the SQL.
|
||
```
|
||
4. Ensure you have taken back-ups of **enumeration\_value, enum\_value\_<tenantId> and entities\_<tenantId>**
|
||
5. Modify the update\_data\_domains.sql with doUpdate=true and run the script
|
||
1. You will observe the same output as the previous execution, along with some additional update results. For example:
|
||
```
|
||
NOTICE: 2025-09-02 17:18:45.558558+00 - ✅ Enum update completed: 409 enum rows updated, 409 tenant rows updated.
|
||
NOTICE: 2025-09-02 17:18:45.903909+00 - ✅ Entities data_domain system field update completed: 5 updated.
|
||
NOTICE: 2025-09-02 17:18:45.907732+00 - ✅ Entities update completed: 1 updated.
|
||
```
|
||
6. You can validate the success execution with the following queries:
|
||
1. Validate enumeration\_values table returns no results with the following SQL (updated for your tenant):
|
||
```
|
||
SELECT * FROM enumeration_value
|
||
WHERE enumeration_descriptor_name LIKE '%DataDomain%'
|
||
AND tenant_id = '<tenantId>'
|
||
AND order_key > 10000
|
||
```
|
||
2. Validate the enum\_value\_<tenantId> table returns no results with the following SQL (updated for your tenant):
|
||
```
|
||
SELECT * FROM enum_value_<tenantId>
|
||
WHERE enumeration_descriptor_name LIKE '%DataDomain%'
|
||
AND order_key > 10000
|
||
```
|
||
3. Validate the entities\_<tenantId> table returns no results by executing the HELPER SQL generated in Step 2.a
|
||
7. Refresh the Data Domains cache in RabbitMQ:
|
||
1. If you have access to the RabbitMQ Management Console, navigate to the PLATFORM\_EVENT\_EXCHANGE and publish the following message, with the payload updated to your tenantId.
|
||
|
||
2. If you do not have access to the RabbitMQ Management UI, you can trigger this over REST API.
|
||
1. Login to the RabbitMQ pods and execute the following, updating the username, password, host and tenantId
|
||
```
|
||
curl -u <username>:<password> -H "content-type:application/json" \
|
||
-X POST http://<host>:15672/api/exchanges/xservices4/PLATFORM_EVENT_EXCHANGE/publish \
|
||
-d '{ "vhost": "xservices4",
|
||
"name": "PLATFORM_EVENT_EXCHANGE",
|
||
"properties": {
|
||
"delivery_mode": 2,
|
||
"headers": {
|
||
"__TypeId__": "java.lang.String"
|
||
},
|
||
"type": "Metadata.Update"
|
||
},
|
||
"routing_key": "Metadata.Update",
|
||
"delivery_mode": "2",
|
||
"payload": "<tenantId>",
|
||
"payload_encoding": "string",
|
||
"headers": {
|
||
"__TypeId__": "java.lang.String"
|
||
},
|
||
"props": {
|
||
"type": "Metadata.Update"
|
||
}
|
||
}'
|
||
```
|
||
8. Validate the cache has been refreshed by monitoring the gateway logs for the following log line:
|
||
```
|
||
2025-08-29T12:04:26.623-0400|INFO |java-client-amqp-consumers-5-thread-13|||||||| Metadata: 100000002 cache is invalidated
|
||
2025-08-29T12:04:26.623-0400|INFO |java-client-amqp-consumers-5-thread-13|||||||| Metadata: invalidateCache was called for tenant id 100000002
|
||
```
|