If your temBoard environment grows and grows, it will become more likely that you run into issues which you haven’t had before. One of those is the following ERROR. In the temboard.log you will see the full glory of this issue.
$ cat /var/log/temboard/temboard.log | grep ERROR | grep history
2021-04-09 05:14:14,186 temboardui[604]: [monitoring] ERROR: Could not history montitoring tables
2021-04-09 05:14:14,186 temboardui[604]: [monitoring] ERROR: 'SELECT * FROM history_tables()' {}
2021-04-09 05:14:14,230 temboardui[604]: [taskmanager] ERROR: PL/pgSQL function history_tables() line 14 at EXECUTE
2021-04-09 05:14:14,230 temboardui[604]: [taskmanager] ERROR: 'SELECT * FROM history_tables()' {}
2021-04-09 05:14:14,230 temboardui[604]: [taskmanager] ERROR: File "/usr/lib/python2.7/site-packages/temboardui/plugins/monitoring/__init__.py", line 113, in history_tables_worker
2021-04-09 05:14:14,230 temboardui[604]: [taskmanager] ERROR: PL/pgSQL function history_tables() line 14 at EXECUTE
2021-04-09 05:14:14,230 temboardui[604]: [taskmanager] ERROR: 'SELECT * FROM history_tables()' {}
2021-04-09 08:14:03,720 temboardui[3941]: [monitoring] ERROR: Could not history montitoring tables
2021-04-09 08:14:03,721 temboardui[3941]: [monitoring] ERROR: PL/pgSQL function history_tables() line 14 at EXECUTE
2021-04-09 08:14:03,721 temboardui[3941]: [monitoring] ERROR: 'SELECT * FROM history_tables()' {}
2021-04-09 08:14:03,721 temboardui[3941]: [monitoring] ERROR: File "/usr/lib/python2.7/site-packages/temboardui/plugins/monitoring/__init__.py", line 100, in history_tables_worker
2021-04-09 08:14:03,721 temboardui[3941]: [monitoring] ERROR: res = conn.execute("SELECT * FROM history_tables()")
2021-04-09 08:14:03,721 temboardui[3941]: [monitoring] ERROR: PL/pgSQL function history_tables() line 14 at EXECUTE
2021-04-09 08:14:03,721 temboardui[3941]: [monitoring] ERROR: 'SELECT * FROM history_tables()' {}
2021-04-09 08:14:03,732 temboardui[3941]: [taskmanager] ERROR: PL/pgSQL function history_tables() line 14 at EXECUTE
2021-04-09 08:14:03,732 temboardui[3941]: [taskmanager] ERROR: 'SELECT * FROM history_tables()' {}
2021-04-09 08:14:03,732 temboardui[3941]: [taskmanager] ERROR: File "/usr/lib/python2.7/site-packages/temboardui/plugins/monitoring/__init__.py", line 113, in history_tables_worker
2021-04-09 08:14:03,732 temboardui[3941]: [taskmanager] ERROR: PL/pgSQL function history_tables() line 14 at EXECUTE
2021-04-09 08:14:03,732 temboardui[3941]: [taskmanager] ERROR: 'SELECT * FROM history_tables()' {}
It will show up in the PostgreSQL log as well.
2021-04-09 11:14:20.269 CEST [7346] ERROR: deadlock detected
2021-04-09 11:14:20.269 CEST [7346] DETAIL: Process 7346 waits for ShareLock on relation 16820 of database 16566; blocked by process 7699.
Process 7699 waits for RowExclusiveLock on relation 16778 of database 16566; blocked by process 7346.
Process 7346: SELECT * FROM history_tables()
Process 7699:
INSERT INTO monitoring.metric_sessions_current
VALUES ('2021-04-09T11:08:37.532164+02:00', 119, 'template1', (NULL, 0, 0, 0, 0, 0, 0, 0, 0))
2021-04-09 11:14:20.269 CEST [7346] HINT: See server log for query details.
2021-04-09 11:14:20.269 CEST [7346] CONTEXT: SQL statement "LOCK TABLE metric_xacts_current IN SHARE MODE"
PL/pgSQL function history_tables() line 14 at EXECUTE
2021-04-09 11:14:20.269 CEST [7346] STATEMENT: SELECT * FROM history_tables()
2021-04-09 11:14:20.275 CEST [7721] LOG: duration: 321704.730 ms statement:
INSERT INTO monitoring.metric_sessions_current
VALUES ('2021-04-09T11:08:46.608808+02:00', 65, 'template1', (NULL, 0, 0, 0, 0, 0, 0, 0, 0))
2021-04-09 11:14:20.275 CEST [7603] LOG: duration: 322836.309 ms statement:
INSERT INTO monitoring.metric_sessions_current
VALUES ('2021-04-09T11:08:48.085377+02:00', 15, 'template1', (NULL, 0, 0, 0, 0, 0, 0, 0, 0))
It looks like a locking issue. As long as the temBoard service is running, you will not be able to fix the issue. However, there is very simple workaround which you can use to fix the issue. It is simply stopping the service, run the command “SELECT * FROM history_tables()” manually, and start the service again.
$ sudo systemctl stop temboard.service
$ sudo systemctl status temboard.service
? temboard.service - temBoard Web UI
Loaded: loaded (/usr/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/temboard.service.d
+-override.conf
Active: failed (Result: exit-code) since Wed 2021-04-14 11:04:23 CEST; 5s ago
Process: 1560 ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /etc/temboard/temboard.conf (code=exited, status=1/FAILURE)
Main PID: 1560 (code=exited, status=1/FAILURE)
Apr 14 11:04:22 host01 env[1560]: web terminated.
Apr 14 11:04:22 host01 systemd[1]: Stopping temBoard Web UI...
Apr 14 11:04:22 host01 env[1560]: scheduler terminated.
Apr 14 11:04:22 host01 env[1560]: worker pool terminated.
Apr 14 11:04:22 host01 env[1560]: worker pool terminated.
Apr 14 11:04:22 host01 env[1560]: worker pool terminated.
Apr 14 11:04:23 host01 systemd[1]: temboard.service: main process exited, code=exited, status=1/FAILURE
Apr 14 11:04:23 host01 systemd[1]: Stopped temBoard Web UI.
Apr 14 11:04:23 host01 systemd[1]: Unit temboard.service entered failed state.
Apr 14 11:04:23 host01 systemd[1]: temboard.service failed.
Now run the command manually until the history tables are cleared.
(postgres@[local]:58888)[temboard] set search_path = monitoring ;
SET
(postgres@[local]:58888)[temboard] SELECT * FROM history_tables();
tblname | nb_rows
---------------------------------------+---------
metric_sessions_history | 6168
metric_xacts_history | 6168
metric_locks_history | 6168
metric_blocks_history | 6168
metric_bgwriter_history | 1364
metric_db_size_history | 6168
metric_tblspc_size_history | 2728
metric_filesystems_size_history | 3927
metric_temp_files_size_delta_history | 6104
metric_wal_files_history | 702
metric_cpu_history | 280
metric_process_history | 280
metric_memory_history | 280
metric_loadavg_history | 280
metric_vacuum_analyze_history | 0
metric_replication_lag_history | 0
metric_replication_connection_history | 646
metric_heap_bloat_history | 1348
metric_btree_bloat_history | 1348
(19 rows)
(postgres@[local]:58888)[temboard] SELECT * FROM history_tables();
tblname | nb_rows
---------------------------------------+---------
metric_sessions_history | 0
metric_xacts_history | 0
metric_locks_history | 0
metric_blocks_history | 0
metric_bgwriter_history | 0
metric_db_size_history | 0
metric_tblspc_size_history | 0
metric_filesystems_size_history | 0
metric_temp_files_size_delta_history | 0
metric_wal_files_history | 0
metric_cpu_history | 0
metric_process_history | 0
metric_memory_history | 0
metric_loadavg_history | 0
metric_vacuum_analyze_history | 0
metric_replication_lag_history | 0
metric_replication_connection_history | 0
metric_heap_bloat_history | 0
metric_btree_bloat_history | 0
(19 rows)
And afterwards start the service again.
$ sudo systemctl start temboard.service
$ sudo systemctl status temboard.service
? temboard.service - temBoard Web UI
Loaded: loaded (/usr/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/temboard.service.d
+-override.conf
Active: active (running) since Wed 2021-04-14 11:05:02 CEST; 7s ago
Main PID: 6561 (temboard)
CGroup: /system.slice/temboard.service
+-6561 temboard: web
+-6576 temboard: worker pool
+-6577 temboard: scheduler
+-6823 temboard: worker pool
+-6836 temboard: worker pool
+-6848 temboard: worker pool
+-6856 temboard: worker pool
+-6858 temboard: worker pool
+-6859 temboard: worker pool
+-6925 temboard: worker pool
+-6927 temboard: worker pool
+-6941 temboard: worker pool
+-6943 temboard: worker pool
+-6946 temboard: worker pool
+-6947 temboard: worker pool
Apr 14 11:05:09 host01 env[6561]: End of collector worker.
Apr 14 11:05:09 host01 env[6561]: End of collector worker.
Apr 14 11:05:09 host01 env[6561]: Starting collector worker for host02.example.ch:44025
Apr 14 11:05:09 host01 env[6561]: Starting collector worker for host02.example.ch:44026
Apr 14 11:05:09 host01 env[6561]: Starting collector worker for host02.example.ch:44029
Apr 14 11:05:09 host01 env[6561]: Failed to preprocess alerting check 'wal_files_archive': list index out of range
Apr 14 11:05:09 host01 env[6561]: Failed to preprocess alerting check 'wal_files_total': list index out of range
Apr 14 11:05:09 host01 env[6561]: Failed to preprocess alerting check 'replication_lag': list index out of range
Apr 14 11:05:09 host01 env[6561]: Starting collector worker for host02.example.ch:44030
Apr 14 11:05:09 host01 env[6561]: End of collector worker.
That’s it. From now on, the ERROR should not appear in the temboard.log anymore.
Conclusion
Always keep an eye on your temboard.log file and scan it for errors.