Ttyjfyk

I configured the service - calc_mem.service

as the following

Restart=on-failure
RestartSec=5
StartLimitInterval=400
StartLimitBurst=3

the configuration above should do the following from my understanding

the service have 3 retries when service exit with error

and before service start it will wait 5 seconds

also I found that "Restart" can be also:

Restart=always

I can understand that need to restart the service on failure
but what is the meaning of Restart=always ?

in which case we need to set - Restart=always

The systemd.service man page has a description of the values Restart= takes, and a table of what options cause a restart when. Always pretty much does what it says on the lid:

If set to always, the service will be restarted regardless of whether it exited cleanly or not, got terminated abnormally by a signal, or hit a timeout.

I don't know for sure what situation they had in mind for that feature, but we might hypothesise e.g. a service configured to only run for a fixed period of time or to serve fixed number of requests and to then stop to avoid any possible resource leaks. Having systemd do the restarting makes for a cleaner implementation of the service itself.

In some sense, we might also ask why not include that option in systemd. Since it is capable of restarting services on failure, they might as well include the option of restarting the service always, just in case someone needs it. To provide tools, not policy.

Note also that a "successful exit" here is defined rather broadly:

If set to on-success, it will be restarted only when the service process exits cleanly. In this context, a clean exit means an exit code of 0, or one of the signals SIGHUP, SIGINT, SIGTERM or SIGPIPE, [...]

SIGHUP is a common way of asking a process to restart, but it unhandled, it
terminates the process. So having Restart=always (or Restart=on-success) allows to use SIGHUP for restarting, even without the service itself supporting that.

Also, as far as I can read the man page, always doesn't mean it would override the limits set by StartLimitInterval and StartLimitBurst:

Note that service restart is subject to unit start rate limiting configured with StartLimitIntervalSec= and StartLimitBurst=, see systemd.unit(5) for details. A restarted service enters the failed state only after the start limits are reached.

If set to on-failure, the service will be restarted when the process exits with a non-zero exit code, is terminated by a signal (including on core dump, but excluding the aforementioned four signals), when an operation (such as service reload) times out, and when the configured watchdog timeout is triggered. [...] If set to always, the service will be restarted regardless of whether it exited cleanly or not, got terminated abnormally by a signal, or hit a timeout.

Excerpt from https://www.freedesktop.org/software/systemd/man/systemd.service.html

So if you set on-failure, it won't get restarted on clean exit.

@JdeBP suggested there is another way to look at this question.

Restart=always is simpler. Easier to implement, easier to understand. Why would we ever want check whether the service terminated with exit code 0 (EXIT_SUCCESS) ? There might even have been a weird bug / error in the service, that caused it to terminate with exit code 0 when it should not have done so.

Answer 1: There are some units that must not use Restart=always. In particular, if the service exits after an idle timeout.

Intriguingly, it would not matter so much if a bug/error causes such a service to exit "successfully", when it should not have done so. Because an idle timeout implies that the service has already been set up to start automatically when a new request is made.

However Restart=on-failure might be used for a service which can exit on idle in some configurations, but not in others. systemd-networkd uses it for this reason.

Answer 2: System administration practices may include killing or messaging service processes to stop them. Sometimes people use a plain kill command, but there are also scripts like apachectl. The advantage of Restart=on-failure is that it is less risky for systemd to recommend using it (as the man page does).

However systemd is left in a strange position, where they also support Restart=always, and that is what they like to set for the majority of long-running services inside the systemd project... This does not seem very helpful when you are trying to learn about systemd service definitions.