Sharing and ops

Procpath commands are typically multi-line shell commands, and when it comes to sharing them as such, it can become unwieldy. The spectrum here can go from having a couple of queries to diagnose your workstation you’d like to share with your colleagues, to distributing a part of a commercial product’s, say delivered on premises of the customers as systemd services, troubleshooting operations procedure.

Playbook

To make writing and sharing of command bundles easy, Procpath comes with another convenience layer – playbooks. Procpath playbooks are a Python configparser representation of its command-line interface, with a few bits of custom semantics. It looks like this:

[stack]
environment:
  L=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
    docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query: $..children[?(@.stat.pid in [$L])]
procfile_list: stat

# this section inherits some options, and overrides one of them
[stack:status:query]
extends: stack
sql_query: SELECT SUM(status_vmrss) total FROM record
procfile_list: stat,status

[stack:stat:query]
extends: stack
sql_query: SELECT SUM(stat_rss) * 4 total FROM record

Here’s how playbooks are read and interpreted:

  1. A CLI minus-separated argument is written as an underscore-separated option.

  2. The option value delimiter is :. A comment is prefixed with #.

  3. A multi-value option is written one per line. A long line can be broken up by placing a backslash before the newline.

  4. A section name can be compound. Its segments are delimited by :. If a section represents a command, its last segment must be the command’s name.

  5. A section inherits from other sections via extends option.

  6. Single-value option search stops, going from the command section up, on the first match.

  7. A multi-value option is joined across the section’s and its parent sections’ values.

A playbook can be saved as a .procpath file and run like:

procpath play -f example.procpath '*:query'

For the playbook CLI, see the listing of play command.

Advanced usage

CLI option override

Setting and/or overriding options via CLI:

[python:record]
environment:
  PIDS=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
       docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query: $..children[?(@.stat.pid in [$PIDS] and 'python' in @.stat.comm)]
interval: 10
recnum: 30

[python:plot]
query_name:
  cpu
  rss

database_file is required for both record and plot. It can be set via CLI like the following. Hence this will record the database and make CPU vs RSS plot out of it:

procpath play -f demo.procpath -o 'database_file=db.sqlite' '*'

Escalated privileges

Running playbook with escalated privileges:

[python:watch]
environment:
  DT=date +"%Y%m%dT%H%M%S"
  STACK=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
        docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query:
  PIDS=$..children[?(@.stat.pid in [$STACK] and 'python' in @.stat.comm)]..pid
interval: 10
repeat: 30
command:
  procpath record -i 1 -d db_$DT.sqlite \
    '$..children[?(@.stat.pid in [$PIDS])]'
  echo $PIDS | tr ',' '\n' | xargs -P0 -I{} -- \
    py-spy record --idle --pid {} -o py_{}_$DT.svg

py-spy typically requires escalated privileges to access the target Python process’ memory. xargs -P0 can be used to spawn py-spy per PID, because py-spy doesn’t support multiple targets natively. A playbook running py-spy with sudo can be run like the following:

sudo env "PATH=$PATH" procpath play -f demo.procpath python:watch

Alternatively sudo ln -s ~/.local/bin/procpath /usr/local/bin/procpath so root under sudo can run it by default.

Target life-cycle

A playbook can cover full target process measurement life-cycle i.e. start the process of interest, run procpath record against it, and automatically stop with the target process:

[watch]
environment:
  DT=date +"%Y%m%dT%H%M%S"
interval: 1
no_restart: 1
command:
  xz -9 /some/big/database.sqlite
  procpath record -i 0.1 -f stat -d xz_$DT.sqlite --stop-without-result \
    -p $WPS1 "$..children[?(@.stat.pid == $WPS1)]"

Similar approach can be used when the target process is run from a detached Docker container.

[redoc:watch]
interval: 1
no_restart: 1
command:
  rm -f redoc_build.sqlite
  docker run --rm -d -v $PWD:/tmp/build --name=redoc \
    ghcr.io/redocly/redoc/cli:v2.0.0-rc.76 \
    build -o /tmp/redoc.html /tmp/build/openapi.json

[redoc:record]
environment:
  ROOT=docker inspect -f '{{.State.Pid}}' redoc
database: redoc_build.sqlite
query: $..children[?(@.stat.pid in [$ROOT])]
stop_without_result: true
interval: 0.025

[redoc:plot]
database_file: redoc_build.sqlite
plot_file: redoc_rss.svg
query_name:
  rss

Known limitations

There is a known limitation for item 3 regarding the use of multiline strings with escaped newlines. Use an empty string to end such string, like:

[test:watch]
interval: 1
no_restart: 1
command:
  docker run --rm -e TEST=42 debian:bullseye \
    bash -c "\
      echo 'Debian container'; \
      cat /etc/os-release; \
      echo \$TEST; \
    "\
    ""