# Keep a service running with mehen run A server, a bot or a resident agent can take its variables from `mehen run` like any other command ([Project variables with mehen run](https://docs.mehen.dev/guides/projects.md)). A run has a hard end, though: it lives at most 12 hours, or 2 hours when an approval started it, however often it renews. So a service under `mehen run` stops now and then, and a supervisor starts it again. The fresh run reads every variable again, so a changed variable or a rotated credential reaches the service at the next start. `mehen run` makes that stop clean, gives it an exit code the supervisor can act on, and writes one line on stderr naming the reason. ## How a run ends **At its maximum life.** The vault answers every run and every renewal with `endsAt`, the run's hard end. A minute before the stop, `mehen run` warns on stderr that the run reaches its maximum life and that renewal cannot extend it. Then: 1. `--stop-grace` seconds before `endsAt` (10 by default, whole seconds from 1 to 300), it sends SIGTERM to the child: to its process group when the child runs detached (stdin is not a terminal, as under launchd or systemd), otherwise to the child and its descendants. The run is still live, so the local relay and the child's placeholders keep working while it shuts down. 2. When the child exits, or at `endsAt` if it has not, `mehen run` sends SIGKILL to whatever is left, closes the relay and ends the run. 3. `mehen run` exits 7. **When authority is lost.** If the vault refuses a renewal, or no renewal succeeds before the last confirmed expiry, the run is no use any more. `mehen run` closes the relay first, sends SIGTERM, and sends SIGKILL 10 seconds later whatever `--stop-grace` says, so a revoked machine does not keep real values in memory for longer. If that happens during the maximum-life grace, the relay closes at once and SIGKILL follows 10 seconds later, or at `endsAt` if that is sooner, with the refusal's exit code. Every refusal of a renewal ends the run at once, except `rate_limited`, `upstream_failed` and `refresh_retrying`, which `mehen run` retries with backoff. A network failure is retried too, until the last confirmed expiry: about 15 minutes after the last successful renewal. A Mac that sleeps for longer than that wakes to a run that has expired. **When the child exits by itself,** `mehen run` ends the run and exits with the child's own code. A SIGTERM or SIGINT sent to `mehen run` is passed to the child, so a supervisor's stop works as it would without Mehen. ## Exit codes When `mehen run` stops the child itself, it exits with one of [the documented exit codes](https://docs.mehen.dev/reference/errors.md#exit-codes) instead of the child's, and writes one stderr line naming the reason, for example `mehen: run run_3kq... has reached its maximum life; its child processes were stopped.` | Exit | Why the child was stopped | What the supervisor does | | --- | --- | --- | | 7 | The run reached its maximum life; its sources or credentials changed (`conflict`); or the vault could not renew it in time. | Start it again. | | 6 | This machine's identity is no longer accepted (`device_revoked`, `unauthenticated`, `proof_invalid`), or its workload ended. | Stop. A human must enrol or repair the machine. | | 5 | Refused by rule (`forbidden`, `remote_mismatch`). | Stop. Retrying will not help. | | 4 | A human must act (`frozen`, `owner_inactive`, `project_unshared`). | Restarting fails until they have. | | 3 | A protected environment needs an approval to start. | A human approves each start. | | 1 | The run had already ended or expired (`not_found`), for example after a long sleep. | Start it again. | A protected environment does not suit a service: every run needs a human's approval, so a restart stops at exit 3 until someone approves it. Run services in an unprotected environment. ## launchd on macOS A LaunchAgent in `~/Library/LaunchAgents/dev.example.api.plist`, loaded as the user who enrolled the machine, so `mehen` finds its key: ```xml Label dev.example.api ProgramArguments /opt/homebrew/bin/mehen run --env dev --stop-grace 30 -- bun run start WorkingDirectory /Users/sam/code/example-api KeepAlive ThrottleInterval 10 ExitTimeOut 60 StandardErrorPath /Users/sam/Library/Logs/example-api.log ``` ```sh launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/dev.example.api.plist ``` - `WorkingDirectory` is the service's folder: `mehen run` finds `mehen.toml` from there. - `KeepAlive` starts it again after every exit, at most once every `ThrottleInterval` seconds. launchd cannot skip a restart by exit code, so a revoked machine retries every 10 seconds until someone unloads the job; watch the log for exit 6. - `ExitTimeOut` is how long launchd waits after its own SIGTERM before it sends SIGKILL. Keep it above the time the child needs to stop. If launchd kills `mehen` first, the child's group can outlive it until the run expires. ## systemd on Linux A user unit in `~/.config/systemd/user/example-api.service`, so it runs as the user who enrolled the machine: ```ini [Unit] Description=example API under mehen run Wants=network-online.target After=network-online.target [Service] WorkingDirectory=/home/sam/code/example-api ExecStart=/usr/local/bin/mehen run --env dev --stop-grace 30 -- bun run start Restart=always RestartSec=5 RestartPreventExitStatus=5 6 KillMode=control-group TimeoutStopSec=60 [Install] WantedBy=default.target ``` ```sh systemctl --user daemon-reload systemctl --user enable --now example-api ``` - `Restart=always` with `RestartSec=5` starts it again after the maximum life and every other stop. - `RestartPreventExitStatus=5 6` stops the restarts when the vault refuses by rule or no longer accepts the machine, so a revoked machine does not loop. - `KillMode=control-group` makes sure a stop reaches the child and anything it started, even if `mehen` is gone. - `TimeoutStopSec` is how long systemd waits after its own SIGTERM. Keep it above the time the child needs to stop.