# How-to-check-native-SACM-notificaiton-queue-in-SaaS_686074669
## Description

This is a guide to query the CI notification queue depth of native SACM.

The queue is built in offline platform pod memory, and program prints relevant information in logs. Given that said, we will leverage log analysis, such as OpenSearch, to find required information. Also, the information will be reset, after the offline platform pod restarts.

A few key metrics since offline platform pod started:

1. Total incoming notifications
2. Proceeded CI create notifications
3. Dropped CI udpate notifications
4. Proceeded CI update notifications
5. Dropped CI remove notifications
6. Proceeded CI remove notifications

**Current Q depth = Total incoming notificaitons - Proceeded CI create notifications - Dropped CI update notifications - Proceed CI update notifications - Dropped CI remove notofications - Proceeded CI remove notifications**

\*: Let's take EU8 as a sample, and we can do the similar check for other farms

---

## Total incoming notifications

Query key word - "The current batch of notifications:" By default, the first log is the latest log, It will be printed every time a notification is received.（ActualEnqueueSize）

The total number of notifications received after starting from the offline pod，Abbreviated as A for later use in calculations. Value is 3812985

https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22The%20current%20batch%20of%20notifications:%22'),sort:!())

![](attachments/686074669/686074635.png)

## Proceeded CI create notifications

Query key word - "Start running in ems, global id" AddCount Processed ADD Notification. Abbreviated as B,Value is 12165

https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Start%20running%20in%20ems,%20global%20id%22'),sort:!())

![](attachments/686074669/686074636.png)

## Dropped CI update notifications

Query key word - "Intercepted UPDATE notification total count:" Intercepted Update Notification Abbreviated as C,Value is 34879

https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Intercepted%20UPDATE%20notification%20total%20count%20:%22'),sort:!())

![](attachments/686074669/686074640.png)

## Proceed CI udpate notifications

Query key word - "in ems. UpdateCount:" Processed UPDATE Notification Abbreviated as D,Value is 3557709

https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22in%20ems.%20UpdateCount%20:%22'),sort:!())

![](attachments/686074669/686074645.png)

## Dropped CI remove notifications

Query key word - "Intercepted REMOVE notification total count:" Intercepted Remove Notification. Abbreviated as E, Value is 6661

https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22Intercepted%20REMOVE%20notification%20total%20count%20:%22'),sort:!())

![](attachments/686074669/686074647.png)

## Proceeded CI remove notifications

Query key word - "in ems. RemoveCount:" Processed REMOVE Notification Abbreviated as F, Value is 11423

https://eu8-logs.itsma-ng.com/\_plugin/kibana/app/discover#/?\_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fd,to:now%2Fd))&\_a=(columns:!(\_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,key:service.keyword,negate:!f,params:(query:xService-platform-offline),type:phrase),query:(match\_phrase:(service.keyword:xService-platform-offline)))),index:cc7f93d0-59ad-11ec-b5a7-9dc618af91ac,interval:auto,query:(language:kuery,query:'%22in%20ems.%20RemoveCount%20:%22'),sort:!())

![](attachments/686074669/686074649.png)

## Summary

According to the formula described in the beginning of this doc, the total number of unprocessed notifications in the Q is A-B-C-D-E-F. As of 2:36 Nov-9th, there are 3812985-12165-34879-3557709-6661-11423=190148 notifications.
