Sharing and ops¶
Procpath commands are typically multi-line shell commands, and when it comes to sharing them as such, it can become unwieldy. The spectrum here can go from having a couple of queries to diagnose your workstation you’d like to share with your colleagues, to distributing a part of a commercial product’s, say delivered on premises of the customers as systemd services, troubleshooting operations procedure.
Playbook¶
To make writing and sharing of command bundles easy, Procpath comes with
another convenience layer – playbooks. Procpath playbooks are a Python
configparser
representation of its command-line interface, with a few bits
of custom semantics. It looks like this:
[stack]
environment:
L=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query: $..children[?(@.stat.pid in [$L])]
procfile_list: stat
# this section inherits some options, and overrides one of them
[stack:status:query]
extends: stack
sql_query: SELECT SUM(status_vmrss) total FROM record
procfile_list: stat,status
[stack:stat:query]
extends: stack
sql_query: SELECT SUM(stat_rss) * 4 total FROM record
Here’s how playbooks are read and interpreted:
A CLI minus-separated argument is written as an underscore-separated option.
The option value delimiter is
:
. A comment is prefixed with#
.A multi-value option is written one per line. A long line can be broken up by placing a backslash before the newline.
A section name can be compound. Its segments are delimited by
:
. If a section represents a command, its last segment must be the command’s name.A section inherits from other sections via
extends
option.Single-value option search stops, going from the command section up, on the first match.
A multi-value option is joined across the section’s and its parent sections’ values.
A playbook can be saved as a .procpath
file and run like:
procpath play -f example.procpath '*:query'
For the playbook CLI, see the listing of play command.
Advanced usage¶
CLI option override¶
Setting and/or overriding options via CLI:
[python:record]
environment:
PIDS=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query: $..children[?(@.stat.pid in [$PIDS] and 'python' in @.stat.comm)]
interval: 10
recnum: 30
[python:plot]
query_name:
cpu
rss
database_file
is required for both record
and plot
. It can be set
via CLI like the following. Hence this will record the database and make
CPU vs RSS plot out of it:
procpath play -f demo.procpath -o 'database_file=db.sqlite' '*'
Escalated privileges¶
Running playbook with escalated privileges:
[python:watch]
environment:
DT=date +"%Y%m%dT%H%M%S"
STACK=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query:
PIDS=$..children[?(@.stat.pid in [$STACK] and 'python' in @.stat.comm)]..pid
interval: 10
repeat: 30
command:
procpath record -i 1 -d db_$DT.sqlite \
'$..children[?(@.stat.pid in [$PIDS])]'
echo $PIDS | tr ',' '\n' | xargs -P0 -I{} -- \
py-spy record --idle --pid {} -o py_{}_$DT.svg
py-spy
typically requires escalated privileges to access the target
Python process’ memory. xargs -P0
can be used to spawn py-spy
per
PID, because py-spy
doesn’t support multiple targets natively. A playbook
running py-spy
with sudo
can be run like the following:
sudo env "PATH=$PATH" procpath play -f demo.procpath python:watch
Alternatively sudo ln -s ~/.local/bin/procpath /usr/local/bin/procpath
so
root
under sudo
can run it by default.
Target life-cycle¶
A playbook can cover full target process measurement life-cycle i.e. start
the process of interest, run procpath record
against it, and automatically
stop with the target process:
[watch]
environment:
DT=date +"%Y%m%dT%H%M%S"
interval: 1
no_restart: 1
command:
xz -9 /some/big/database.sqlite
procpath record -i 0.1 -f stat -d xz_$DT.sqlite --stop-without-result \
-p $WPS1 "$..children[?(@.stat.pid == $WPS1)]"
Similar approach can be used when the target process is run from a detached Docker container.
[redoc:watch]
interval: 1
no_restart: 1
command:
rm -f redoc_build.sqlite
docker run --rm -d -v $PWD:/tmp/build --name=redoc \
ghcr.io/redocly/redoc/cli:v2.0.0-rc.76 \
build -o /tmp/redoc.html /tmp/build/openapi.json
[redoc:record]
environment:
ROOT=docker inspect -f '{{.State.Pid}}' redoc
database: redoc_build.sqlite
query: $..children[?(@.stat.pid in [$ROOT])]
stop_without_result: true
interval: 0.025
[redoc:plot]
database_file: redoc_build.sqlite
plot_file: redoc_rss.svg
query_name:
rss
Known limitations¶
There is a known limitation for item 3 regarding the use of multiline strings with escaped newlines. Use an empty string to end such string, like:
[test:watch]
interval: 1
no_restart: 1
command:
docker run --rm -e TEST=42 debian:bullseye \
bash -c "\
echo 'Debian container'; \
cat /etc/os-release; \
echo \$TEST; \
"\
""