Delegation to tools

Procpath itself is only concerned with Procfs [1], but there is a wide range of Linux tools, language-specific or not, from profilers to system call tracers which can provide the key to the problem at hand. These tools typically accept a PID or list of PIDs, and hence benefit from the process tree query capability Procpath provides. It’s a convenience to avoid unnecessary scripting and/or terminal multiplexers in case of many process trees of interest (e.g. Celery nodes).

Procpath has watch command which is analogous to procps watch. In this example watch delegates two process trees to smemstat [2] and py-spy [3].

procpath watch --interval 601 \
  -e TS='date +%s' \
  -e S1='systemctl show --property MainPID redis-server | cut -d "=" -f 2' \
  -e C1='docker inspect -f "{{.State.Pid}}" app_gunicorn_1' \
  -q L1='$..children[?(@.stat.pid == $S1)]..pid' \
  -c 'smemstat -q -o redis-memdiff-$TS.json -p $L1 30 20' \
  -c 'timeout --foreground --signal SIGINT 600 \
      py-spy record --subprocesses --output app-flamegraph-$TS.svg --pid $C1'

Notes:

  1. Typical watch pattern is:

    1. take the root PID from you process supervisor (systemd, Docker, etc)

    2. query all PIDs of its descendant processes

    3. pass the PID list to the analysis tool of choice

  2. The command environment is re-evaluated each --interval seconds

  3. A process is restarted each --interval seconds only if it has stopped and unless --no-restart is provided

  4. A process’s stdout output is forwarded as INFO, and stderr as WARNING logging records

  5. If the analysis tool of choice needs to work continuously and doesn’t have a means to terminate itself, it’s suggested to wrap it into timeout --foreground --signal SIGINT INTERVAL ...

  6. watch expects to be interrupted by SIGINT (Ctrl+C), where it sends SIGINT (by default) to all its descendant processes

  7. watch can run fixed number of repetitions specified by --repeat