76 lines
6.0 KiB
Markdown
76 lines
6.0 KiB
Markdown
# How-to-check-native-SACM-notificaiton-queue-in-SaaS_686074669
|
||
## Description
|
||
|
||
This is a guide to query the CI notification queue depth of native SACM.
|
||
|
||
The queue is built in offline platform pod memory, and program prints relevant information in logs. Given that said, we will leverage log analysis, such as OpenSearch, to find required information. Also, the information will be reset, after the offline platform pod restarts.
|
||
|
||
A few key metrics since offline platform pod started:
|
||
|
||
1. Total incoming notifications
|
||
2. Proceeded CI create notifications
|
||
3. Dropped CI udpate notifications
|
||
4. Proceeded CI update notifications
|
||
5. Dropped CI remove notifications
|
||
6. Proceeded CI remove notifications
|
||
|
||
**Current Q depth = Total incoming notificaitons - Proceeded CI create notifications - Dropped CI update notifications - Proceed CI update notifications - Dropped CI remove notofications - Proceeded CI remove notifications**
|
||
|
||
\*: Let's take EU8 as a sample, and we can do the similar check for other farms
|
||
|
||
---
|
||
|
||
## Total incoming notifications
|
||
|
||
Query key word - "The current batch of notifications:" By default, the first log is the latest log, It will be printed every time a notification is received.（ActualEnqueueSize）
|
||
|
||
The total number of notifications received after starting from the offline pod，Abbreviated as A for later use in calculations. Value is 3812985
|
||
|
||
https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22The%20current%20batch%20of%20notifications:%22'),sort:!())
|
||
|
||
|
||
|
||
## Proceeded CI create notifications
|
||
|
||
Query key word - "Start running in ems, global id" AddCount Processed ADD Notification. Abbreviated as B,Value is 12165
|
||
|
||
https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Start%20running%20in%20ems,%20global%20id%22'),sort:!())
|
||
|
||
|
||
|
||
## Dropped CI update notifications
|
||
|
||
Query key word - "Intercepted UPDATE notification total count:" Intercepted Update Notification Abbreviated as C,Value is 34879
|
||
|
||
https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Intercepted%20UPDATE%20notification%20total%20count%20:%22'),sort:!())
|
||
|
||
|
||
|
||
## Proceed CI udpate notifications
|
||
|
||
Query key word - "in ems. UpdateCount:" Processed UPDATE Notification Abbreviated as D,Value is 3557709
|
||
|
||
https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22in%20ems.%20UpdateCount%20:%22'),sort:!())
|
||
|
||
|
||
|
||
## Dropped CI remove notifications
|
||
|
||
Query key word - "Intercepted REMOVE notification total count:" Intercepted Remove Notification. Abbreviated as E, Value is 6661
|
||
|
||
https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Intercepted%20REMOVE%20notification%20total%20count%20:%22'),sort:!())
|
||
|
||
|
||
|
||
## Proceeded CI remove notifications
|
||
|
||
Query key word - "in ems. RemoveCount:" Processed REMOVE Notification Abbreviated as F, Value is 11423
|
||
|
||
https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22in%20ems.%20RemoveCount%20:%22'),sort:!())
|
||
|
||
|
||
|
||
## Summary
|
||
|
||
According to the formula described in the beginning of this doc, the total number of unprocessed notifications in the Q is A-B-C-D-E-F. As of 2:36 Nov-9th, there are 3812985-12165-34879-3557709-6661-11423=190148 notifications.
|