How-to-check-native-SACM-notificaiton-queue-in-SaaS_686074669

Description

This is a guide to query the CI notification queue depth of native SACM.

The queue is built in offline platform pod memory, and program prints relevant information in logs. Given that said, we will leverage log analysis, such as OpenSearch, to find required information. Also, the information will be reset, after the offline platform pod restarts.

A few key metrics since offline platform pod started:

Total incoming notifications
Proceeded CI create notifications
Dropped CI udpate notifications
Proceeded CI update notifications
Dropped CI remove notifications
Proceeded CI remove notifications

Current Q depth = Total incoming notificaitons - Proceeded CI create notifications - Dropped CI update notifications - Proceed CI update notifications - Dropped CI remove notofications - Proceeded CI remove notifications

*: Let's take EU8 as a sample, and we can do the similar check for other farms

Total incoming notifications

Query key word - "The current batch of notifications:" By default, the first log is the latest log, It will be printed every time a notification is received.（ActualEnqueueSize）

The total number of notifications received after starting from the offline pod，Abbreviated as A for later use in calculations. Value is 3812985

https://eu8-logs.itsma-ng.com/_plugin/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22The%20current%20batch%20of%20notifications:%22'),sort:!())

Proceeded CI create notifications

Query key word - "Start running in ems, global id" AddCount Processed ADD Notification. Abbreviated as B,Value is 12165

https://eu8-logs.itsma-ng.com/_plugin/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Start%20running%20in%20ems,%20global%20id%22'),sort:!())

Dropped CI update notifications

Query key word - "Intercepted UPDATE notification total count:" Intercepted Update Notification Abbreviated as C,Value is 34879

https://eu8-logs.itsma-ng.com/_plugin/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Intercepted%20UPDATE%20notification%20total%20count%20:%22'),sort:!())

Proceed CI udpate notifications

Query key word - "in ems. UpdateCount:" Processed UPDATE Notification Abbreviated as D,Value is 3557709

Dropped CI remove notifications

Query key word - "Intercepted REMOVE notification total count:" Intercepted Remove Notification. Abbreviated as E, Value is 6661

https://eu8-logs.itsma-ng.com/_plugin/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Intercepted%20REMOVE%20notification%20total%20count%20:%22'),sort:!())

Proceeded CI remove notifications

Query key word - "in ems. RemoveCount:" Processed REMOVE Notification Abbreviated as F, Value is 11423

Summary

According to the formula described in the beginning of this doc, the total number of unprocessed notifications in the Q is A-B-C-D-E-F. As of 2:36 Nov-9th, there are 3812985-12165-34879-3557709-6661-11423=190148 notifications.

6.0 KiB Raw Blame History Unescape Escape