nexus/knowledgebase/csd-wiki/ICSD/Monitoring-Database_686083870.md

Monitoring-Database_686083870

Introduction

This page presents all the monitoring used to cover database level monitoring, not limited to DB instance level, application level monitoring.

Detailed monitors

Instance level monitoring

• Locks - blocked sessions(TBD) - check every 1 minute
  • Graph (To be reviewed)

    SELECT count(bl.pid) AS blocked_pid_count FROM pg_catalog.pg_locks bl JOIN pg_catalog.pg_stat_activity a ON a.pid = bl.pid JOIN pg_catalog.pg_locks kl ON kl.transactionid = bl.transactionid AND kl.pid != bl.pid JOIN pg_catalog.pg_stat_activity ka ON ka.pid = kl.pid WHERE NOT bl.granted and a.query_start <= (now() - interval '5 minutes');
    

    • Table

    SELECT bl.pid AS blocked_pid, a.usename AS blocked_user, a.query_start AS blocked_start_time , a.query AS blocked_statement, now() - a.query_start AS blocked_duration,ka.query AS blocking_statement, ka.query_start AS blocking_start_time , now() - ka.query_start AS blocking_duration, kl.pid AS blocking_pid, ka.usename AS blocking_user FROM pg_catalog.pg_locks bl JOIN pg_catalog.pg_stat_activity a ON a.pid = bl.pid JOIN pg_catalog.pg_locks kl ON kl.transactionid = bl.transactionid AND kl.pid != bl.pid JOIN pg_catalog.pg_stat_activity ka ON ka.pid = kl.pid WHERE NOT bl.granted and a.query_start <= (now() - interval '5 minutes');
    

• Long active queries (TBD) - check every 10 minutes
  • Graph

    SELECT count(pid) AS Long_Query_count FROM pg_catalog.pg_stat_activity WHERE (now() - pg_catalog.pg_stat_activity.query_start) > interval '30 minutes' and state = 'active';
    

    • Table

    SELECT pid, now() - pg_catalog.pg_stat_activity.query_start AS duration, state FROM pg_catalog.pg_stat_activity WHERE (now() - pg_catalog.pg_stat_activity.query_start) > interval '30 minutes' and state = 'active';
    

• Capture RDS top 10 query (TBD) - check every 10 minutes
  1. Clean stat_statement bi-weekly 2. capture during runtime if CPU is more than 97% for 60 mins 3. Table

     Select (select datname from pg_database where oid=dbid) as db_name, query as short_query, round(total_time::numeric, 2) as total_time, calls, round(mean_time::numeric, 2) as mean, round((100 * total_time / sum(total_time::numeric) over ())::numeric, 2) as percentage_overall from pg_stat_statements order by total_time desc limit 20;
     

     4. Graph (Optional)

     Select (select datname from pg_database where oid=dbid) as db_name, query as short_query, round((100 * total_time / sum(total_time::numeric) over ())::numeric, 2) as percentage_overall from pg_stat_statements order by total_time desc limit 20;
     

• Dead tuple - logical database level (TBD) - check every 10 minutes
  • Table

    -- Top bloating tables
    select relname, n_dead_tup, last_vacuum, last_autovacuum from
    pg_catalog.pg_stat_all_tables
    where n_dead_tup > 0 order by n_dead_tup desc limit 20;
    

    • Graph (Optional)

    -- Top bloating tables
    select relname, n_dead_tup
    pg_catalog.pg_stat_all_tables
    where n_dead_tup > 0 order by n_dead_tup desc limit 20;
    

Database level customer metrics

1. NativeSACM Transaction Context Queue - check every 10 minutes

   select count(*) from transaction_context_$tenant where entity::jsonb->>'entity_type' in('Device','SystemElement','ServiceComponent','ActualService') ;
   

2. NativeSACM Transaction Context Queue retries - check every 10 minutes

   select count(*) from transaction_context_$tenant where retry_count=2 and entity::jsonb->>'entity_type' in('Device','SystemElement','ServiceComponent','ActualService') ;
   

3. SLT Job queue - check every 10 minutes

   select count(1) from transaction_context_$tenant a, transaction_etl_job_$tenant b where a.transaction_timestamp > b.last_taken and a.flag0 & 1 = 1 and b.job_name = 'SLT';
   

4. SLT Delay time (This may need to be removed if it's the same as SLT Job Queue) - check every 10 minutes

   SELECT now() - last_taken as delay FROM maas_admin.transaction_etl_job_$tenant where job_name = 'SLT';"