Guides›Keep a service running with mehen run

Keep a service running with mehen run#

How do I run a long-lived process under mehen run with launchd or systemd, and what do its exit codes mean?

A server, a bot or a resident agent can take its variables from mehen run like any other command (Project variables with mehen run). A run has a hard end, though: it lives at most 12 hours, or 2 hours when an approval started it, however often it renews. So a service under mehen run stops now and then, and a supervisor starts it again. The fresh run reads every variable again, so a changed variable or a rotated credential reaches the service at the next start.

mehen run makes that stop clean, gives it an exit code the supervisor can act on, and writes one line on stderr naming the reason.

How a run ends#

At its maximum life. The vault answers every run and every renewal with endsAt, the run's hard end. A minute before the stop, mehen run warns on stderr that the run reaches its maximum life and that renewal cannot extend it. Then:

1. --stop-grace seconds before endsAt (10 by default, whole seconds from 1 to 300), it sends SIGTERM to the child: to its process group when the child runs detached (stdin is not a terminal, as under launchd or systemd), otherwise to the child and its descendants. The run is still live, so the local relay and the child's placeholders keep working while it shuts down.
2. When the child exits, or at endsAt if it has not, mehen run sends SIGKILL to whatever is left, closes the relay and ends the run.
3. mehen run exits 7.

When authority is lost. If the vault refuses a renewal, or no renewal succeeds before the last confirmed expiry, the run is no use any more. mehen run closes the relay first, sends SIGTERM, and sends SIGKILL 10 seconds later whatever --stop-grace says, so a revoked machine does not keep real values in memory for longer. If that happens during the maximum-life grace, the relay closes at once and SIGKILL follows 10 seconds later, or at endsAt if that is sooner, with the refusal's exit code.

Every refusal of a renewal ends the run at once, except rate_limited, upstream_failed and refresh_retrying, which mehen run retries with backoff. A network failure is retried too, until the last confirmed expiry: about 15 minutes after the last successful renewal. A Mac that sleeps for longer than that wakes to a run that has expired.

When the child exits by itself, mehen run ends the run and exits with the child's own code. A SIGTERM or SIGINT sent to mehen run is passed to the child, so a supervisor's stop works as it would without Mehen.

Exit codes#

When mehen run stops the child itself, it exits with one of the documented exit codes instead of the child's, and writes one stderr line naming the reason, for example mehen: run run_3kq... has reached its maximum life; its child processes were stopped.

A protected environment does not suit a service: every run needs a human's approval, so a restart stops at exit 3 until someone approves it. Run services in an unprotected environment.

launchd on macOS#

A LaunchAgent in ~/Library/LaunchAgents/dev.example.api.plist, loaded as the user who enrolled the machine, so mehen finds its key:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>dev.example.api</string>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/homebrew/bin/mehen</string>
    <string>run</string>
    <string>--env</string>
    <string>dev</string>
    <string>--stop-grace</string>
    <string>30</string>
    <string>--</string>
    <string>bun</string>
    <string>run</string>
    <string>start</string>
  </array>
  <key>WorkingDirectory</key>
  <string>/Users/sam/code/example-api</string>
  <key>KeepAlive</key>
  <true/>
  <key>ThrottleInterval</key>
  <integer>10</integer>
  <key>ExitTimeOut</key>
  <integer>60</integer>
  <key>StandardErrorPath</key>
  <string>/Users/sam/Library/Logs/example-api.log</string>
</dict>
</plist>

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/dev.example.api.plist

• WorkingDirectory is the service's folder: mehen run finds mehen.toml from there.
• KeepAlive starts it again after every exit, at most once every ThrottleInterval seconds. launchd cannot skip a restart by exit code, so a revoked machine retries every 10 seconds until someone unloads the job; watch the log for exit 6.
• ExitTimeOut is how long launchd waits after its own SIGTERM before it sends SIGKILL. Keep it above the time the child needs to stop. If launchd kills mehen first, the child's group can outlive it until the run expires.

systemd on Linux#

A user unit in ~/.config/systemd/user/example-api.service, so it runs as the user who enrolled the machine:

[Unit]
Description=example API under mehen run
Wants=network-online.target
After=network-online.target

[Service]
WorkingDirectory=/home/sam/code/example-api
ExecStart=/usr/local/bin/mehen run --env dev --stop-grace 30 -- bun run start
Restart=always
RestartSec=5
RestartPreventExitStatus=5 6
KillMode=control-group
TimeoutStopSec=60

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now example-api

• Restart=always with RestartSec=5 starts it again after the maximum life and every other stop.
• RestartPreventExitStatus=5 6 stops the restarts when the vault refuses by rule or no longer accepts the machine, so a revoked machine does not loop.
• KillMode=control-group makes sure a stop reaches the child and anything it started, even if mehen is gone.
• TimeoutStopSec is how long systemd waits after its own SIGTERM. Keep it above the time the child needs to stop.