Delegation to tools¶
Procpath itself is only concerned with Procfs [1], but there is a wide range of Linux tools, language-specific or not, from profilers to system call tracers which can provide the key to the problem at hand. These tools typically accept a PID or list of PIDs, and hence benefit from the process tree query capability Procpath provides. It’s a convenience to avoid unnecessary scripting and/or terminal multiplexers in case of many process trees of interest (e.g. Celery nodes).
Procpath has watch
command which is analogous to procps
watch
.
In this example watch
delegates two process trees to smemstat
[2]
and py-spy
[3].
procpath watch --interval 601 \
-e TS='date +%s' \
-e S1='systemctl show --property MainPID redis-server | cut -d "=" -f 2' \
-e C1='docker inspect -f "{{.State.Pid}}" app_gunicorn_1' \
-q L1='$..children[?(@.stat.pid == $S1)]..pid' \
-c 'smemstat -q -o redis-memdiff-$TS.json -p $L1 30 20' \
-c 'timeout --foreground --signal SIGINT 600 \
py-spy record --subprocesses --output app-flamegraph-$TS.svg --pid $C1'
Notes:
Typical
watch
pattern is:take the root PID from you process supervisor (systemd, Docker, etc)
query all PIDs of its descendant processes
pass the PID list to the analysis tool of choice
The command environment is re-evaluated each
--interval
secondsA process is restarted each
--interval
seconds only if it has stopped and unless--no-restart
is providedA process’s
stdout
output is forwarded asINFO
, andstderr
asWARNING
logging recordsIf the analysis tool of choice needs to work continuously and doesn’t have a means to terminate itself, it’s suggested to wrap it into
timeout --foreground --signal SIGINT INTERVAL ...
watch
expects to be interrupted by SIGINT (Ctrl+C), where it sends SIGINT (by default) to all its descendant processeswatch
can run fixed number of repetitions specified by--repeat