25.2. Sakana#
Sakana runs alongside GNU Taler monitoring. It checks GNU Taler exchanges for problems that an availability probe does not see, and it checks Sentol’s own Prometheus and Alertmanager from the outside: it is the monitor that monitors the monitoring. Its checks and its reports are configured separately from Sentol’s Prometheus probes and Alertmanager routing.
The source is at git://git.taler.net/sakana.git, and the sakana
package is on deb.taler.net. sakana(1) documents the command,
sakana.conf(5) every configuration option, and info sakana is the
full manual.
25.2.1. Configuration#
Sakana reads /etc/sakana/sakana.conf. Every section other than
[sakana] describes one monitored system; KIND selects exchange
(the default), prometheus or alertmanager, and BASE_URL is the
system’s base URL, not an endpoint. The [sakana] section holds defaults
for all systems, notably:
FREQUENCY: how often each system is checked;TOLERANCE: how many consecutive failures of one error class are tolerated before a report is made;COMMAND: the command that receives each report on its standard input. The first line of a report is a summary suitable as a mail subject, and/usr/share/sakana/sakana-notify-maildelivers reports by email.
Each error class of each system has its own tolerance, so a long-standing warning never delays the report of a new outage. Once a system passes every check again, Sakana sends a brief “service restored” report.
The file may hold credentials (AUTH_BASIC, AUTH_BEARER, ENV_*),
so keep it mode 0640, owned by root:sakana. Run sakana --test to
check a configuration without contacting anything.
25.2.2. Monitoring exchanges#
To monitor an exchange, we want to not merely monitor that the REST API is
operational, but also check that the /keys response is not outdated.
Besides failures to connect, failed TLS handshakes, HTTP errors and timeouts,
Sakana reports:
a TLS certificate that expires within 5 days (
TLS_RENEWAL_HORIZON), by when certbot should have renewed it;signing keys of which none is valid beyond the next 30 days (
SIGNKEY_EXPIRATION_HORIZON);denominations of which none can be withdrawn beyond the next 14 days (
DENOMINATION_EXPIRATION_HORIZON). This horizon has to stay below theLOOKAHEAD_SIGNof the monitored exchange, or a correctly rotating exchange warns on every check;global fees or wire fees that are not configured for the current year (and, in December, for the next year).
Exchanges to monitor include:
exchange.taler-ops.ch
exchange.e.netzbon-basel.ch
exchange.demo.taler.net
exchange.head.taler.net
exchange.test.taler.net
25.2.3. Monitoring the monitoring#
When an Alertmanager stops working, the symptom is silence, and silence
looks the same as everything being in order. Sakana therefore also checks
Sentol’s Prometheus (KIND = prometheus) and Alertmanager
(KIND = alertmanager) and reports through its own COMMAND, not
through Alertmanager.
For both, Sakana reports an instance that is running but not ready, and one
that restarted since the previous check. For Prometheus, it also reports a
failed configuration reload, a corrupted time series database and rules that
cannot be evaluated. For Alertmanager, it also reports a gossip cluster with
fewer peers than CLUSTER_PEERS requires.
BASE_URL is the prefix that the service’s --web.external-url names,
since that is where its /-/healthy endpoint lives. Sentol’s web interfaces
require a password (see GNU Taler monitoring), so these sections also need
AUTH_BASIC.