How-to-fix-inability-to-add-new-Data-Domains_716270786

Introduction

This runbook will fix a customer's inability to add new Data Domains if they are receiving the following error "Metadata element DataDomains. is invalid. Enumeration descriptor is invalid, since the orders defined for its values are not greater than or equal to 1. Refer to customization"

The problem occurred due to a defect for Dev2Prod process in RangeEnumerationValueOrdersPreProcessorForImport. In SMAX versions priori to 25.2, when importing DataDomains fields, the code would assign sequential order_key values starting at 1 to newly imported DataDomains. It would persist any existing order_keys, so generally if all newly imported DataDomains appeared at the end of the list, all was well. However, if Dev and Prod environments were out of sync, i.e. manually added domains to both environments in a different order, then the new DataDomain would be assigned an existing order_key. This was handled later by the function "mergePreValidatedEnumerations()" and would assign a random order_key between 10000-32767. This worked fine, although the DataDomains may be imported out of order, until 25.2 changed the logic.

In 25.2 the DataDomains no longer got sequential IDs starting at 1, but from the highest order_key already present. This meant they would no longer be duplicate keys to generate random ones. However, some unlucky customers that had existing order_keys that were generated significantly high were now limited by the SQL/Java limits on "smallInt", maxing out at 32767. If a customer's DataDomains max order key reached 32767, the next DataDomain would try to be added with 32767+1, which would overflow and become -32768 and no longer be valid.

No new customers will get random order_keys anymore, so this will not be a problem for any customer whose max order_key for DataDomains is below 31767, since the max number of enum items is 1000. However, to be safe from future changes, we should fix any customer whose max order_key is over 30000. We must remap the DataDomains order_keys in enumeration_value, enum_value_ and entities_ via SQL.

Note: This script applies to SMAX versions from 25.2 and higher. Back up your data in the following EMS tables before running this script: enumeration_value, enum_value_ and entities_

Resolution

Pre-requisites:

EMS database access
RabbitMQ Management console or REST API access

Steps:

Download the SQL scripts and update throughout each of them.
1. find_data_domains.sql 2. update_data_domains.sql

Run find_data_domains.sql and record the output. You will get:

A helper SQL to find all entities that will need updating that can be used in pre- and post-validation, for example:

NOTICE:  
--- HELPER SQL TO BE RUN INDEPENDENTLY TO VIEW ALL ENTITIES NEEDING UPDATE ---
select DISTINCT entity_id, entity_type_id, ed.name as entity_type, schar0, data_domains, e.usmallint0 as Company_dd, e.usmallint1 as Idea_dd,  last_update_time from entities_100000002 e
        join entity_descriptor ed on e.entity_type_id = ed.id WHERE 
        EXISTS (
            SELECT 1
            FROM unnest(e.data_domains) AS dd
            WHERE dd > 10000
        ) OR 
        (e.entity_type_id = 498 AND e.usmallint0 >= 10000) OR
        (e.entity_type_id = 435 AND e.usmallint1 >= 10000);

If Data Domains has been used in custom fields on any entities:

A SELECT statement to copy into update_data_domains.sql for pre-validation

NOTICE:  
--- SELECT STATEMENT TO ADD TO update_data_domains.sql for COUNT ---
    SELECT COUNT(*) INTO row_count FROM entities_100000002 e WHERE
        (e.entity_type_id = 498 AND e.usmallint0 = ANY(old_keys)) OR
        (e.entity_type_id = 435 AND e.usmallint1 = ANY(old_keys));

 2. A series of UPDATE statements to copy into update\_data\_domains.sql for updating data domains on custom fields

NOTICE:  
--- UPDATE STATEMENTS TO ADD TO update_data_domains.sql for mapping new order_keys ---
        UPDATE entities_100000002 e SET usmallint0 = t.new_key FROM temp_new_keys t WHERE usmallint0 = t.old_key AND e.entity_type_id = 498;
        UPDATE entities_100000002 e SET usmallint1 = t.new_key FROM temp_new_keys t WHERE usmallint1 = t.old_key AND e.entity_type_id = 435;

If the Data Domains have not been used in any custom fields, then you will get a simple notice and simply remove the RAISE EXCEPTION in the SELECT and UPDATE locations in the update_data_domains.sql

NOTICE:  No relevant entity-property mappings found. You may remove the SELECT AND UPDATE RAISE EXCEPTIONS in update_data_domains.

Run update_data_domains.sql with doUpdate=false (the default) and SAVE and review the output.

You will observe the list of enums that will be updated, for example:

NOTICE:  2025-09-02 16:52:49.399163+00 - ℹ️ Detected DataDomains that require updating:
NOTICE:  No key conflicts detected.
NOTICE:  Name: NickTest3_c, OldKey: 30009, NewKey: 1009
NOTICE:  Name: NickTest4_c, OldKey: 30010, NewKey: 1010
NOTICE:  Name: NickTest5_c, OldKey: 30011, NewKey: 1011

And you will observe information about how many entities will need to be updated:

NOTICE:  2025-09-02 16:52:49.612168+00 - ℹ️ data_domains system field on 2 entities requires updating.
NOTICE:  2025-09-02 16:52:49.614872+00 - ℹ️ data_domains custom field on 3 entities requires updating.
NOTICE:  2025-09-02 16:52:49.614958+00 - ℹ️ Have you saved the above output? Have you taken a DB backup? If you are ready to proceed with the update, change doUpdate to true and re-run the SQL.

Ensure you have taken back-ups of enumeration_value, enum_value_ and entities_

Modify the update_data_domains.sql with doUpdate=true and run the script

You will observe the same output as the previous execution, along with some additional update results. For example:

NOTICE:  2025-09-02 17:18:45.558558+00 - ✅ Enum update completed: 409 enum rows updated, 409 tenant rows updated.
NOTICE:  2025-09-02 17:18:45.903909+00 - ✅ Entities data_domain system field update completed: 5 updated.
NOTICE:  2025-09-02 17:18:45.907732+00 - ✅ Entities update completed: 1 updated.

You can validate the success execution with the following queries:
1. Validate enumeration_values table returns no results with the following SQL (updated for your tenant):
```
SELECT * FROM enumeration_value 
WHERE enumeration_descriptor_name LIKE '%DataDomain%' 
  AND tenant_id = '<tenantId>'
  AND order_key > 10000
```
  1. Validate the enum_value_ table returns no results with the following SQL (updated for your tenant):
```
SELECT * FROM enum_value_<tenantId> 
WHERE enumeration_descriptor_name LIKE '%DataDomain%' 
  AND order_key > 10000
```
  1. Validate the entities_ table returns no results by executing the HELPER SQL generated in Step 2.a

Refresh the Data Domains cache in RabbitMQ:

If you have access to the RabbitMQ Management Console, navigate to the PLATFORM_EVENT_EXCHANGE and publish the following message, with the payload updated to your tenantId.

2. If you do not have access to the RabbitMQ Management UI, you can trigger this over REST API.

curl -u <username>:<password> -H "content-type:application/json" \
  -X POST http://<host>:15672/api/exchanges/xservices4/PLATFORM_EVENT_EXCHANGE/publish \
  -d '{ "vhost": "xservices4",
  "name": "PLATFORM_EVENT_EXCHANGE",
  "properties": {
    "delivery_mode": 2,
    "headers": {
      "__TypeId__": "java.lang.String"
    },
    "type": "Metadata.Update"
  },
  "routing_key": "Metadata.Update",
  "delivery_mode": "2",
  "payload": "<tenantId>",
  "payload_encoding": "string",
  "headers": {
    "__TypeId__": "java.lang.String"
  },
  "props": {
    "type": "Metadata.Update"
  }
  }'

Validate the cache has been refreshed by monitoring the gateway logs for the following log line:

2025-08-29T12:04:26.623-0400|INFO |java-client-amqp-consumers-5-thread-13|||||||| Metadata: 100000002 cache is invalidated
2025-08-29T12:04:26.623-0400|INFO |java-client-amqp-consumers-5-thread-13|||||||| Metadata: invalidateCache was called for tenant id 100000002

8.7 KiB Raw Blame History Unescape Escape

How-to-fix-inability-to-add-new-Data-Domains_716270786

Introduction

Resolution

8.7 KiB

Raw Blame History