DD 102: systemd service restart policy

23.103. DD 102: systemd service restart policy#

Design status:

Draft

Implementation status:

Not started

DD shepherd:

Florian Dold

Historical contributors:

Florian Dold

First published:

2026-09-03

Last substantive change:

2026-09-05

23.103.1. Summary#

Long-running Taler services should recover from transient daemon failures without operator intervention. They should stop on invalid configuration or another known permanent failure, and repeated crashes must be visible to operators.

The proposed policy retries indefinitely with a fixed ten-second delay. Exit status 6 means that the service is not configured correctly; exit status 9 means that the service encountered another permanent failure. Neither status causes an automatic restart. Associated activation sockets stop with the service so that incoming traffic cannot start it again after a permanent failure or an explicit stop.

This recovery guarantee assumes that systemd can start the daemon. Failures in systemd prerequisites, such as a socket that cannot bind, require monitoring and operator repair.

Systemd 257 in Debian Trixie supports an increasing restart delay, but does not reset that delay after the service has been healthy for a long time. If a service reaches a five-minute delay, runs successfully for days, and then crashes again, the next restart still waits five minutes. Services restarted regularly through RuntimeMaxSec= also advance the counter. A fixed delay avoids this persistent state.

23.103.2. Motivation#

Taler’s systemd units currently use different restart policies. Some permit only five starts in five seconds and some rely on systemd’s default start-rate limit. Once that limit is reached, Restart=always no longer restarts the service. A temporary dependency outage can therefore require manual recovery.

Other units restart after only a few milliseconds or seconds, producing a tight loop during a longer outage. Some configure RestartSteps= without the required maximum delay, so the setting has no effect.

Exit handling is inconsistent as well. Native Taler services commonly use status 6 for configuration errors and status 9 for failures that should not be retried, but not all units recognize both statuses and non-C services do not always return them.

Socket activation and unit dependencies also affect recovery. Incoming traffic can start a service again after RestartPreventExitStatus= has suppressed its automatic restart, without observing RestartSec=. In the other direction, a failed dependency with Requires= and After= can prevent the daemon from starting at all, leaving its restart policy unused.

An unlimited restart policy also needs monitoring. In particular, RestartMode=direct skips the failed/inactive transition during automatic restarts and does not invoke OnFailure= units. A service can therefore remain in a crash loop without triggering monitoring based only on its current state.

23.103.3. Requirements#

  • Long-running services keep retrying after transient daemon failures, including temporary loss of reconnectable dependencies.

  • Restarts do not form a tight loop.

  • Invalid configuration and known permanent failures are not retried.

  • Exit statuses have the same meaning in every implementation language.

  • An explicit systemctl stop stops the service and its activation sockets.

  • Operators can detect stopped services, failed prerequisites, and crash loops.

  • The policy works with Debian Trixie.

23.103.4. Proposed Solution#

23.103.4.1. Scope#

The policy applies to long-running product services shipped or deployed by Taler, Anastasis, LibEuFin, Donau, Paivana, Challenger, and related repositories, including distribution-specific copies of their units. The socket-lifetime rules below also apply to their associated activation sockets.

The automatic recovery guarantee covers failures handled by the daemon and its service restart policy. It does not cover failures in systemd prerequisites, such as failed assertions, socket bind failures, or dependency start-job failures. These may leave a service inactive without ever running its main process. Deployments must detect these cases and provide operator recovery; disabling the service start-rate limiter does not resolve them.

The service baseline does not apply to one-shot initialization commands, timer-triggered jobs, garbage collection jobs, or other processes that are expected to finish.

23.103.4.2. Unit policy#

Long-running services use this baseline:

[Unit]
StartLimitIntervalSec=0

[Service]
Restart=always
RestartSec=10s
RestartPreventExitStatus=6 9

StartLimitIntervalSec=0 disables the service’s start-rate limiter. This is needed because Restart= remains subject to start-rate limiting.

Restart=always restarts the service after clean and unsuccessful exits, signals, timeouts, and watchdog failures. It does not override an explicit systemctl stop.

RestartSec=10s avoids a busy loop and has no backoff state to carry across unrelated failures. Units must not add StartLimitBurst, a non-zero StartLimitIntervalSec, the legacy StartLimitInterval, RestartSteps, or RestartMaxDelaySec. A unit may prevent additional exit statuses when its program documents them as permanent failures.

23.103.4.3. Socket lifetime#

For a service with activation sockets, apply the baseline above and add the following to example.service:

[Unit]
Requires=example.socket

[Service]
RestartMode=direct

Its example.socket uses this lifetime relationship, in addition to its existing listen address and access settings:

[Unit]
BindsTo=example.service

[Socket]
Accept=no
Service=example.service

Apply this relationship to every activation socket of the service, including the corresponding instances of template units. The service requires all sockets whose file descriptors it needs. Do not add After=example.service to a socket: systemd already orders activation sockets before their services, so the opposite ordering would introduce a cycle.

BindsTo= stops the sockets when the service stops or fails permanently. RestartMode=direct is required for this profile: it avoids the transient failed/inactive state during automatic restarts, so those restarts keep the sockets available. Pending connections wait for the next restart; they must not shorten the ten-second delay. PartOf= alone does not provide the required behavior on permanent failure.

Starting a socket with this relationship also starts the daemon. These are long-running services, rather than daemons started only when traffic arrives. After exit status 6 or 9, the service remains failed and its sockets stop. An explicit systemctl stop example.service also stops the sockets, so subsequent traffic cannot reactivate the service. After repairing the failure, systemctl start example.service starts the service and its required sockets.

RestartPreventExitStatus= only suppresses automatic service restarts. It does not reject independent start requests, including socket activation or an operator’s explicit start. Deployments must not add unconditional activation mechanisms, such as Upholds= or a handler that always starts the service, where they would override permanent exits or explicit stops. Socket trigger limits are separate from the service start-rate limiter and are not a substitute for the lifetime relationship above.

23.103.4.4. Reconnectable dependencies#

For dependencies that the daemon can reconnect to, such as a database or security module, request local startup through a shared product target and order startup with After= on the consumer. For example, example.target uses Wants=example.service example-dependency.service, while example.service uses After=example-dependency.service. A failed dependency then does not prevent the consumer from starting.

Replace Requires= or BindsTo= relationships from the consumer that would block its startup or stop it when a reconnectable dependency goes away. A failed start job with Requires= and After= does not schedule an automatic restart of the consumer; starting the dependency later does not automatically start that service either.

Do not simply replace these relationships with Wants= on the consumer. An automatic restart of the consumer also pulls in its wanted units, which can start a dependency again after a permanent exit or explicit stop. With Wants= on the shared target instead, consumer restarts do not reactivate the dependency. Start the product target to request startup of the group; starting the consumer alone does not start its reconnectable dependencies.

The daemon must handle an unavailable dependency by reconnecting internally or returning a restartable status, such as 1. Temporary unavailability must not be classified as invalid configuration or a permanent failure merely because it occurs during startup.

Keep hard dependencies on resources required to start the daemon, including its activation sockets. Failures in these prerequisites are outside the automatic recovery guarantee. Operators must repair the prerequisite and explicitly start the affected service. If a prerequisite has hit its own start or trigger limit, reset its failed state before starting it again.

23.103.4.5. Why the delay is fixed#

Systemd 257 calculates its increasing delay from the number of automatic restarts. It has no setting equivalent to “reset the delay after ten minutes of successful operation”. Time spent active does not reduce the counter. The counter is reset only by an explicit manager action such as systemctl reset-failed or by stopping and starting the unit.

This matters even for services that rarely fail. For example, after enough failures to reach a five-minute cap:

  1. the service recovers and runs for a week;

  2. it encounters one unrelated transient failure;

  3. systemd still waits five minutes before restarting it.

It is worse for a service using RuntimeMaxSec= as an intentional recycling mechanism: every scheduled automatic restart advances the same counter, so the normal restart delay eventually reaches the cap.

Systemd 258 adds RESTART_RESET=1 to its service notification protocol. A daemon can send this after it considers itself healthy, but systemd 257 ignores it and every daemon would need explicit support. The policy can be revisited once the minimum supported systemd version and the applications can provide a reliable reset.

23.103.4.6. Exit-status contract#

The main process uses these common statuses:

6

Configuration is missing or invalid. Retrying the same configuration cannot succeed.

9

The service has identified another permanent failure and asks not to be restarted.

All other failures are restartable. Implementations in C, Kotlin, Rust, Go, and other languages must map configuration failures to status 6 rather than a generic status 1. Before returning 6 or 9, the program should log a clear diagnostic.

RestartPreventExitStatus applies only to the main service process. It does not affect ExecStartPre=. Configuration validation that decides whether the service should be retried must therefore be reflected in the main process’s exit status.

23.103.4.7. Application-level retry#

Restarting the process is the fallback when it cannot continue. If a daemon can keep serving useful work or report degraded health while reconnecting to a dependency, it should remain running and retry that operation itself. Domain-specific waits need not use the systemd restart interval.

A service should not wrap its main worker in an internal relaunch loop. Such a loop hides the worker’s exit status and restart count from systemd. If the worker’s termination makes the service unavailable, propagate the result and let systemd restart it.

A failed synchronization pass is not necessarily worker termination. Keep recoverable API and database-operation retries inside the worker, with suitable backoff that resets after success. Reconnect failed sessions without discarding healthy clients or independent work. For example, Wise balances have independent polling/retry loops; Cyclos polling continues while its optional notification listener reconnects. The adapters’ HTTP servers are separate services, so a worker restart does not restart HTTP serving. Propagate terminal failures such as an unusable closed database pool, and do not catch panics merely to relaunch the application.

23.103.4.8. Detecting crash loops#

A service that crashes repeatedly may still appear active whenever monitoring checks it, because systemd keeps restarting it. Checking only whether the unit is active or failed therefore misses crash loops. Deployments must also monitor how often a service restarts. Three restarts within fifteen minutes is a useful default alert threshold.

RestartMode=direct makes failed-state monitoring even less useful: systemd goes directly from a process failure to a restart without marking the unit as failed or invoking OnFailure= units. Existing units may keep this setting, and services with activation sockets must use it as described above.

Services that stop after status 6 or 9 remain in the failed state and must be covered by failed-unit monitoring. Also monitor services that are expected to be running but are inactive, and failed prerequisite and socket units: a failed dependency start job need not mark the dependent service as failed. Expected state must account for intentional operator stops and maintenance.

The service’s NRestarts property counts automatic restarts, not independent activations. Socket-triggered starts can therefore form a tight loop while NRestarts stays zero. Restart-frequency monitoring does not replace the activation controls above. Journal output is useful for diagnosis but is not an alert.

The relevant behavior is documented in systemd.service(5), systemd.unit(5), systemd.socket(5), and the systemd 258 RESTART_RESET notification.

23.103.5. Test Plan#

Run systemd-analyze verify against the changed service and socket units together, including their dependency graph. Run the following acceptance tests on systemd 257 in Debian Trixie with representative services and all their associated activation sockets:

  • Exit statuses 0 and 1, termination by signals, and intentional RuntimeMaxSec= recycling trigger automatic restarts with a ten-second delay after termination. Activation sockets stay available during that delay, and incoming traffic does not cause an earlier restart.

  • Exit statuses 6 and 9 leave the service failed and stop every associated socket. Test with connections already pending and with subsequent traffic; neither may cause further starts. Numeric exit statuses 6 and 9 must not suppress restarts after SIGABRT or SIGKILL.

  • Explicitly stopping the service stops its sockets and prevents further starts under traffic, including when stopped during the restart delay. Explicitly starting it again restores the service and its sockets.

  • Starting an associated socket starts the daemon without waiting for traffic.

  • Starting a product target with a failed reconnectable dependency does not prevent the consumer from starting. When the dependency recovers, internal reconnection or the next automatic restart restores useful service without an operator restarting the consumer. If the dependency exits with status 6 or 9 or is explicitly stopped, repeated consumer restarts must not start it again.

  • Failed prerequisites, including a socket bind failure and a dependency start-job failure, trigger an alert even if the service remains inactive. Repairing the prerequisite and explicitly starting the service restores it.

  • Repeated automatic restarts trigger the deployment’s crash-loop alert; permanent failures trigger its failed-unit alert.

Preliminary tests of the socket-lifetime profile on systemd 259 confirmed that statuses 6 and 9 stop the socket, transient failures preserve it without shortening the restart delay under traffic, and an explicit stop stops both units. Shared-target tests also confirmed that consumers recover when a dependency becomes available, while consumer restarts leave a dependency that exited with status 6 stopped. These checks do not replace the systemd 257 acceptance tests above.

23.103.6. Definition of Done#

  • [ ] All checked-in long-running product units follow the baseline policy.

  • [ ] Associated activation sockets follow the lifetime profile, including

    permanent failure, explicit stop, and template-instance behavior.

  • [ ] Distribution and deployment copies match their product units.

  • [ ] Service implementations return status 6 for invalid or missing

    configuration and reserve status 9 for known permanent failures.

  • [ ] Reconnectable dependencies do not block daemon startup or propagate

    stops; their temporary unavailability remains restartable, and consumer restarts do not reactivate stopped dependencies.

  • [ ] Worker wrappers do not hide service termination from systemd.

  • [ ] Supported deployments detect unexpectedly inactive services, failed

    services and prerequisites, and repeated restarts, and document operator recovery from prerequisite failures.

  • [ ] The service and socket acceptance tests pass on systemd 257.

  • [ ] The accepted policy is added to the appropriate developer or operations

    reference manual.

23.103.7. Alternatives#

23.103.7.1. Finite start-rate limit#

Systemd’s default limit protects the host and leaves a persistently failing unit in a clear failed state. It also abandons recovery after a temporary outage, so it does not meet the main requirement.

23.103.7.2. Systemd exponential delay#

An increasing delay with a cap reduces load during a long outage. On systemd 257, however, the delay does not reset after a healthy period. This makes an old failure history determine recovery time for a later, unrelated failure. It should be reconsidered when a reliable healthy-runtime reset is available.

23.103.7.3. Timer, wrapper, or external orchestrator#

A timer or wrapper can implement a custom reset rule, but duplicates systemd supervision and complicates stop, status, and exit-code handling. An external orchestrator can provide richer restart policies, but requiring one would change Taler’s deployment model. Deployments that already use an orchestrator may translate the behavior specified here into its native policy.

23.103.8. Drawbacks#

A ten-second delay is not adaptive. A single crash recovers more slowly than with the shortest current settings, while a long outage causes more attempts than capped exponential backoff. It also adds ten seconds to an intentional RuntimeMaxSec restart.

Unlimited retries can consume resources and produce repeated log messages, which is why restart monitoring is required. Incorrectly returning status 6 or 9 can also turn a recoverable failure into one that waits for an operator.

Binding activation sockets to the service starts the daemon as soon as a socket starts, losing startup deferred until incoming traffic. A permanent failure or explicit stop also closes those sockets, so clients cannot queue connections until an operator starts the service again. Failures in systemd prerequisites still require monitoring and operator recovery.

23.103.9. Discussion / Q&A#

23.103.9.1. Why restart after a clean exit?#

These units represent services expected to remain available. Processes that are expected to finish belong in one-shot or timer-driven units.

23.103.9.2. Why not treat every startup failure as permanent?#

Startup may depend on a database, network service, or credential agent that is temporarily unavailable. Only failures classified by the program as invalid configuration or permanent should suppress restart.