Forms of I/O: file, shell, serial, queue and browser sources, each behind an allow-list; release 0.7.0
Claude · 2026-10-26 · de8967b238aeb07f46304b54721a412f3b5a2b7a · parents 0be7dde
An observe or act node may now name a file, a program, a serial line, an MQTT queue or a browser instead of http, and every one follows the rule hosts set: the tree lists what the source may reach (paths, commands, devices, topics; hosts for a browser and a broker), pyrtan check refuses a literal outside the list, the runtime refuses a templated one before the adapter is asked, and nothing else is reachable. The result is a plain table under `into`; every source leaves `body`, so extract reads it unchanged; the trace line reads the same (`shell df -P / -> exit 0`).
src/pyrtan/adapters.py holds the five: a file read (whole or a tail) or write; one program by name with its arguments as a list, no shell in between; a serial exchange over termios; MQTT 3.1.1 at QoS 0 over a socket (subscribe and wait for one message, or publish); and a page rendered by Playwright's Chromium on a worker thread, with wait_for and actions, where a navigation outside hosts is aborted. Playwright is the one optional extra (pyrtan[browser], PYRTAN_BROWSER). --egress public now also closes files, programs and serial lines: a hosted box is not lent out.
One fixture answers for every source, keyed by adapter_key, so the five new examples (disk-space, log-errors, modem-check, room-temperature, rendered-headline) run offline, pyrtan run --record writes such records, and the conformance vectors cover all six sources in both runtimes (37/37). pyrtan.js reads HTML with a port of html.py now, not the host's DOM, so css extracts agree on a service, in a browser and in node; a member answers a non-http source with source-unsupported, and the hub hands members only http trees. pyrtan list names a tree's sources.
Tests: allow-lists, keys, each adapter live (a pty for serial, a small MQTT broker for queue, a local page and Chromium for browser, skipped without Playwright), egress, record and replay through the command and a service. Docs: _meta/io.md, tree-format, design, README, deploy.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YChr78vDHtQJ15HNbsaRLZ
Changes
README.md | 49 +- _meta/design.md | 27 +- _meta/io.md | 138 ++++ _meta/members.md | 5 +- _meta/tree-format.md | 33 +- deploy/README.md | 8 + deploy/openmediavault/docker-compose.yml | 9 + examples/disk-space/disk-space.toml | 75 ++ examples/log-errors/log-errors.toml | 65 ++ examples/modem-check/modem-check.toml | 53 ++ examples/rendered-headline/rendered-headline.toml | 57 ++ examples/room-temperature/room-temperature.toml | 65 ++ pyproject.toml | 2 +- src/pyrtan/__init__.py | 2 +- src/pyrtan/adapters.py | 680 +++++++++++++++++++ src/pyrtan/catalog.py | 2 +- src/pyrtan/cli.py | 13 +- src/pyrtan/hub.py | 7 +- src/pyrtan/js/pyrtan.js | 351 ++++++++-- src/pyrtan/runtime.py | 52 +- src/pyrtan/service.py | 4 +- src/pyrtan/transport.py | 37 +- src/pyrtan/tree.py | 191 +++++- tests/conformance/make_vectors.py | 20 + tests/conformance/vectors.json | 1509 +++++++++++++++++++++++++++++++++++++++++ tests/fixtures/examples.json | 329 ++++++++- tests/test_adapters.py | 579 ++++++++++++++++ tests/test_catalog.py | 6 +- tests/test_examples.py | 51 ++ tests/test_federation.py | 14 +- tests/test_service_hub.py | 4 +- uv.lock | 2 +- 32 files changed, 4298 insertions(+), 141 deletions(-)
diff --git a/README.md b/README.md index 6dd1235..2e12737 100644 --- a/README.md +++ b/README.md@@ -52,6 +52,10 @@ uv tool install git+https://github.com/b-i-t-8/pyrtan # or: pip install gitpyrtan --version ```+Trees that drive a browser need the one optional extra, Playwright:+`pip install 'pyrtan[browser]'` and `playwright install chromium` (or+point `PYRTAN_BROWSER` at a Chromium you have).+From a checkout, `uv sync` then `uv run pyrtan …`, and `uv run pytest`. ## A tree@@ -114,6 +118,38 @@ list with a limit, through another tree or a local sub-flow), `delegate``done` and `fail`. Trees do not loop; `pyrtan check` refuses a cycle. The whole format is in [`_meta/tree-format.md`](_meta/tree-format.md).+## Forms of I/O++HTTP is one source. An `observe` or `act` node may instead name a file,+a program, a serial line, a queue or a browser, and every one of them+follows the rule `hosts` set: the tree lists what the source may reach,+`pyrtan check` refuses a literal outside the list, the runtime refuses a+templated one, and nothing else is reachable.++```toml+commands = ["df"]+[nodes.df]+kind = "observe"+shell = { command = "df", args = ["-P", "{{ inputs.mount }}"] } # one program, arguments as a list, no shell in between+into = "df" # status (exit code), ok, stdout, stderr, body, json, lines+extract = [{ into = "percent", regex = "(\\d+)%", group = 1 }]+```++| Source | Allow-list | Example |+| :-- | :-- | :-- |+| `file = { path, tail }` or `{ path, write / append }` | `paths` | [`log-errors`](examples/log-errors/log-errors.toml): error lines in the tail of a log |+| `shell = { command, args }` | `commands` | [`disk-space`](examples/disk-space/disk-space.toml): how full a filesystem is |+| `serial = { device, baud, write, read_until }` | `devices` | [`modem-check`](examples/modem-check/modem-check.toml): does the modem answer AT |+| `queue = { url, topic }` or `{ …, publish }` | `hosts`, `topics` | [`room-temperature`](examples/room-temperature/room-temperature.toml): a sensor's last reading over MQTT |+| `browser = { url, wait_for, actions }` | `hosts` | [`rendered-headline`](examples/rendered-headline/rendered-headline.toml): the heading of a page that needs a script to render |++The trace reads the same whichever source a node used (`shell df -P / ->+exit 0`, `file /var/log/syslog -> 48213 B`), and one fixture file answers+for all of them, so every example runs offline in the test suite and the+same vectors hold in the JavaScript runtime. A service started with+`--egress public` refuses files, programs and serial lines outright: a+hosted box is not lent out. The design is in [`_meta/io.md`](_meta/io.md).+## The command | Command | What it does |@@ -369,15 +405,16 @@ site/ pyrtan.com: the landing page, the seed list, the catalog## Status-0.6.0. The core, the service, the hub, the command, the console, the+0.7.0. The core, the service, the hub, the command, the console, thecatalog client, the language rules, federation between hubs, the gossiped directory, members (a browser or a phone as part of the swarm), the blobs they keep and the trees they run (`pyrtan.js`, conformant with-the Python runtime) are built and tested; HTTP is the one form of I/O the core speaks; a swarm can-require a token, keep its gaps, refuse private addresses, and follow-other swarms. Not yet built: further adapters (a browser for pages that need-a script to render; files, shells, serial lines, message queues),-persistent storage for unanswered sentences, `pyrtan watch`, and hosting.+the Python runtime), and six forms of I/O (HTTP, a browser, files,+programs, serial lines, MQTT queues), each behind an allow-list, are+built and tested; a swarm can require a token, keep its gaps, refuse+private addresses and its own box, and follow other swarms. Not yet+built: persistent storage for unanswered sentences, `pyrtan watch`, and+hosting.See [`_meta/design.md`](_meta/design.md) for the runtime's design and [`_meta/platform.md`](_meta/platform.md) for pyrtan.com. diff --git a/_meta/design.md b/_meta/design.md index 2107c4a..b965f93 100644 --- a/_meta/design.md +++ b/_meta/design.md@@ -166,11 +166,12 @@ the standard library's client, a recorded fixture that raises on anyrequest it did not foresee, and a recorder that writes such fixtures. Every example tree is tested offline through the second, and `pyrtan run --record` produces fixtures through the third. The transport-is also the seam where forms of I/O plug in. A node names a source (`http`-today), the runtime enforces that source's allow-list, and the transport-moves the bytes. A file, shell, serial or queue adapter is a new node-source with the same three parts, and the same recorded and recording-wrappers around it.+is also the seam where forms of I/O plug in. A node names a source, the+runtime enforces that source's allow-list, and the adapter moves the+bytes. Since v0.7 `adapters.py` holds five more (`browser`, `file`,+`shell`, `serial`, `queue`), each a node table with a list of its own+kind, and the recorded and recording fixtures answer for all of them+under one key scheme (`io.md`).**Growth** (`grow.py`) is where a dead end becomes the next tree. A sentence no tree answers is recorded as *unanswered*, counted, and@@ -296,13 +297,15 @@ Five trees, all against real public APIs, all tested offline:answers sentences through its hub's mailbox (`Ask`/`Answer`); a hub hands its own trees and `--handout` ones to the devices that joined with its token. Design in [`members.md`](members.md) §7.-7. **v0.7: more forms of I/O.** First the browser: `observe` with- `browser = { url, wait_for, actions }` on Playwright as an optional- extra, for pages that need a script to render, with the same host rule- and the same extraction. Then file, shell, serial and queue adapters,- each a node source with its own allow-list (paths, commands, devices,- topics), so a tree reads a log, runs a named command or polls a device- by the same rules it reads a page, and the trace reads the same.+7. **v0.7, built: more forms of I/O.** `browser = { url, wait_for,+ actions }` on Playwright as the one optional extra, for pages that need+ a script to render, with the same host rule and the same extraction;+ and `file`, `shell`, `serial` and `queue` (MQTT) as node sources, each+ with its own allow-list (`paths`, `commands`, `devices`, `topics`), so+ a tree reads a log, runs a named program or polls a device by the same+ rules it reads a page, and the trace reads the same. One fixture+ answers for every source, in both runtimes. Design in+ [`io.md`](io.md).8. **v0.8: the catalog grows up.** Namespaces, publishing through pull requests with the checker as the gate, and opt-in gap reporting from hubs back to a tree's author, carried as another federation topic. diff --git a/_meta/io.md b/_meta/io.md new file mode 100644 index 0000000..4e20b2e --- /dev/null +++ b/_meta/io.md@@ -0,0 +1,138 @@+# Design — Forms of I/O: file, shell, serial, queue, browser++| | |+|---|---|+| **Status** | **Built, v0.7.** Five sources beside `http`, each a node table with an allow-list of its own kind, the same retry rule, the same trace, the same fixtures; the examples cover each; the JavaScript runtime refuses them on a member and answers them from a fixture. |+| **Date** | 2026-10-25 |+| **Decision** | A node names a *source* and what it wants from it; the tree lists what that source may reach; the runtime refuses anything outside the list before the adapter is asked; the adapter moves the bytes and leaves a plain table in state. Files, shells and serial lines are the box itself, and a hosted service keeps them closed. |+| **Where** | `src/pyrtan/adapters.py` (the adapters, the allow-list matchers, the fixture keys), `src/pyrtan/tree.py` (the lists, `Tree.allowed`, the per-source checks), `src/pyrtan/runtime.py` (one path for every source), `src/pyrtan/transport.py` (fixtures answer for every source), `src/pyrtan/js/pyrtan.js` (the same on a member). |++## 0. The brief++"Act on all forms of I/O." v0.1 spoke HTTP and said the shape would hold+for any other form: an adapter names what it may reach, a node observes+or acts through it, a `decide` reads what came back. v0.7 is that+sentence made true five times, and nothing more: no new node kind, no+new template syntax, no way to reach anything a tree did not list.++## 1. The shape, once++An `observe` or `act` node has exactly one of `http`, `browser`, `file`,+`shell`, `serial`, `queue`, `from` or `emit`. The runtime renders the+source's table with the node's scope, asks the tree whether the rendered+request is inside its allow-list, and only then asks the adapter. An+observe retries on a transport error (not on an answer the author can+decide on); an act is attempted once; a policy refusal is never retried.+The result is a table under `into`, `extract` reads it (every source+leaves `body`, so `regex` and `css` extracts work unchanged), and the+trace line reads `source what -> brief`, the way an HTTP line reads+`GET url -> status`.++| Source | Allow-list | What it names | What comes back |+| :-- | :-- | :-- | :-- |+| `http` | `hosts` | a URL | `status`, `ok`, `headers`, `body`, `json`, `url`, `elapsed_ms` |+| `browser` | `hosts` | a URL, `wait_for`, `actions` | `url` (where it ended up), `status`, `ok`, `title`, `body` (rendered), `elapsed_ms` |+| `file` | `paths` | a path, `tail`; `write` or `append` | `path`, `exists`, `size`, `modified`, `body`, `json`, `lines`; `written` |+| `shell` | `commands` | a program and `args` | `command`, `args`, `status` (exit code), `ok`, `stdout`, `stderr`, `body`, `json`, `lines` |+| `serial` | `devices` | a device, `baud`, `write`, `read_until` or `read` | `device`, `baud`, `sent`, `body`, `bytes`, `complete`, `json`, `lines` |+| `queue` | `hosts` and `topics` | a broker URL and a topic; `publish` | `received`, `message` (`topic`, `body`, `json`), `body`, `json`; `published` |++## 2. The allow-lists++Each list is checked twice, like `hosts`: at load time when the value is+literal (`pyrtan check` refuses `shell = { command = "rm" }` in a tree+whose `commands` is `["df"]`), and at run time when it is templated+(`paths = ["/var/log"]` refuses `/etc/passwd` from a sentence, and+`/var/log/../etc/passwd`, because the path is made absolute first). A+tree that uses a source without its list does not load. The matchers:++- `paths`: absolute paths or globs; a directory covers everything under+ it; `~` is the service's home; `*` alone is everything.+- `commands`: program names (`df` matches `df` and `/usr/bin/df`) or full+ paths (`/usr/bin/df` matches only that). No shell ever reads the line:+ `args` is a list handed to the program as given, so `$HOME`, `;` and+ backticks in an argument are letters.+- `devices`: paths or globs (`/dev/ttyUSB*`).+- `topics`: MQTT filters, `+` (or `*`) one level, `#` the rest; the+ broker's host must be in `hosts` as well.++## 3. What this design refuses++- **A general `exec`.** `shell` runs one program by name with a list of+ arguments, through `subprocess` with `shell=False`. There is no+ `script`, no pipe, no `sh -c` unless the author lists `sh` and writes+ the `-c` out loud, which the checker lets through because the list is+ the author's word and `pyrtan check` shows it.+- **Files beyond what a tree reads at once.** A file over 8 MB is refused+ without `tail`; a tail reads the last megabyte and keeps the last N+ lines. A tree is not a log shipper.+- **Making directories.** `write` and `append` need the directory to+ exist. A tree that wants a place to write is given one by its+ operator.+- **A queue library.** MQTT 3.1.1 over a socket is six packets: connect,+ connack, subscribe, suback, publish, disconnect, at QoS 0. That is what+ a tree needs to read one reading or flip one switch; retained sessions,+ QoS 2 and wildcards in a publish are not a tree's business. `mqtts://`+ uses the standard library's TLS.+- **A browser as a requirement.** Playwright is the one optional+ dependency (`pip install 'pyrtan[browser]'`), and a service without it+ answers a `browser` node with a transport error that says so. The+ browser lives on one worker thread (Playwright's sync API is bound to+ the thread that started it) and the service's request threads queue+ for it; each run gets a fresh context. A page may fetch what it likes,+ as pages do; a top-level navigation to a host outside `hosts`, by+ redirect or by a click in `actions`, is aborted and the node fails.+ `PYRTAN_BROWSER` points at a Chromium when Playwright's own is not+ installed.+- **Lending the box.** `--egress public` was "reach only public+ addresses". It now also means "no files, no programs, no serial lines":+ an operator hosting other people's trees does not hand them the host.+ A queue or a browser under that policy goes through the same address+ check an HTTP request does.+- **Members running these.** A phone has no shell. The hub hands a+ trusted member only trees whose sources are `http`; the JavaScript+ runtime answers any other source with `source-unsupported` unless a+ recorded fixture stands in for it, which is how the conformance+ vectors cover all six.++## 4. Fixtures++One fixture file answers for every source. An HTTP record is keyed `GET+url`; the others by `adapter_key`: `shell df -P /`, `file /var/log/syslog`+(or `file append …`), `serial /dev/ttyUSB0 "AT\r\n"`, `queue mqtt://broker+home/kitchen/temperature` (or `queue publish …`), `browser https://…`.+The record is the table the node would observe, or `{"error": "…"}` for+a transport failure. `pyrtan run --record` writes them from real runs;+the JavaScript runtime builds the same keys, so `tests/fixtures/+examples.json` serves both runtimes and the vectors in `tests/conformance`+cover a shell, a file, a queue, a browser and a serial line without any+of them being present.++## 5. Failure modes, honestly++- A program that hangs holds a service thread for the node's `timeout`+ and is then killed; the node fails with `no answer within`. An author+ who sets `timeout = 600` on a shell node gets what they asked for.+- A serial device that is open elsewhere, or answers nothing, is a+ transport error after the timeout; an observe tries once more. A+ device that answers slowly past the terminator returns `complete =+ false` and what it had.+- A broker that never delivers makes `received = false` after the+ timeout; the tree decides what silence means (the example says "is the+ sensor publishing?"). A broker that needs a client certificate is not+ reachable; username and password are.+- A page that keeps loading past `timeout` fails the node; `wait_until =+ "domcontentloaded"` is the author's lever. A browser that cannot start+ (no Chromium, no display libraries) fails every browser node the same+ way until it can.+- `tail` on a file whose last megabyte is one line gives one line.++## 6. Sequencing++- **v0.7, this:** the five sources, their lists, their fixtures, the+ examples, the JavaScript runtime's own HTML reader (so css extracts+ agree on a service, in a browser and in node).+- **Later:** a `watch` that runs a tree when a file, a topic or a serial+ line speaks (the adapters are ready for it; the trigger is the design+ still owed), and a serial line held open across nodes for devices with+ a session.diff --git a/_meta/members.md b/_meta/members.md index dbd3fa2..5ca5641 100644 --- a/_meta/members.md +++ b/_meta/members.md@@ -163,7 +163,10 @@ gives each trusted member every tree it should run as a `Keep` of`--handout DIR` (`$PYRTAN_HANDOUT`), which are trees the hub hands out and does not run itself. The member's `Kept` confirms; a changed tree is sent again. The member holds them in its store like any blob, so they-survive a reload, and announces their names as its capabilities. The hub+survive a reload, and announces their names as its capabilities. Only+trees whose sources are `http` are handed out: a phone has no files,+programs, serial lines or broker sockets of the box's, and no browser of+its own to drive, so those stay on the hub (`io.md`). The hubtakes the names it handed out and ignores the rest, and the triggers it routes by are its own copy's, never the member's word. diff --git a/_meta/tree-format.md b/_meta/tree-format.md index c0bfe78..236ec42 100644 --- a/_meta/tree-format.md +++ b/_meta/tree-format.md@@ -9,7 +9,11 @@ A tree is one TOML file. This is the whole of it.| `name` | yes | Lowercase letters, digits and dashes. The tree's identity in the swarm. | | `description` | should | One line. Shown by `pyrtan list` and in `/capabilities`. | | `version` | no | Positive integer, default 1. |-| `hosts` | if any node speaks HTTP | The hosts the tree may reach: exact names, `*.example.com`, or `*` for a tree whose host comes from its input. |+| `hosts` | if any node speaks HTTP, drives a browser or talks to a queue | The hosts the tree may reach: exact names, `*.example.com`, or `*` for a tree whose host comes from its input. |+| `paths` | if any node reads or writes a file | Absolute paths or globs (`/var/log/*`); a directory covers everything under it; `*` is everything. |+| `commands` | if any node runs a program | Program names (`df`) or full paths (`/usr/bin/df`); `*` is everything. |+| `devices` | if any node opens a serial line | Device paths or globs (`/dev/ttyUSB*`). |+| `topics` | if any node talks to a queue | MQTT topic filters: `+` one level, `#` the rest (`home/+/temperature`). || `start` | yes | The first node. | | `tags` | no | Free strings, for people and for the hub's listing. | | `license` | for a catalog | An SPDX identifier such as `MIT`, `Apache-2.0` or `CC0-1.0`: what others may do with the file. A tree without one runs anywhere but is left out of a catalog. |@@ -148,6 +152,33 @@ Selectors support tag, `#id`, `.class`, `[attr]`, `[attr=v]`, `[attr^=v]`,`:contains(text)`, the descendant and `>` combinators, and `,` groups. Anything else fails `pyrtan check`.+### The other sources++Instead of `http`, an `observe` or `act` may name one of these; each+needs the matching allow-list at the top of the tree, checked at load+time when the value is literal and at run time when it is templated. The+result lands under `into` and `extract` reads it the same way (every+source leaves `body`). The design is in [`io.md`](io.md).++```toml+file = { path = "/var/log/syslog", tail = 200 } # observe: path, exists, size, modified, body, json, lines+file = { path = "./notes.txt", append = "visited\n" } # act: … plus written (or write = "…")+shell = { command = "df", args = ["-P", "/"] } # command, args, status (exit code), ok, stdout, stderr, body, json, lines+serial = { device = "/dev/ttyUSB0", baud = 9600, write = "AT\r\n", read_until = "\n" } # device, baud, sent, body, bytes, complete, json, lines+queue = { url = "mqtt://broker.local", topic = "home/kitchen/temperature" } # observe: received, message { topic, body, json }, body, json+queue = { url = "mqtt://broker.local", topic = "home/kitchen/light", publish = "on" } # act: … plus published+browser = { url = "https://app.example/", wait_for = "h1", actions = [{ click = "#more" }, { fill = { selector = "#q", value = "x" } }] } # url, status, ok, title, body (rendered), elapsed_ms+```++`shell` runs one program with its arguments as a list; no shell reads the+line. `serial` reads until `read_until`, `read` bytes or `timeout`.+`queue` is MQTT 3.1.1 at QoS 0; `mqtts://` for TLS, `username` and+`password` if the broker wants them. `browser` needs Playwright+(`pip install 'pyrtan[browser]'`); its actions are `click`, `fill`,+`press`, `select`, `wait_for` and `wait` (ms), and a navigation to a host+outside `hosts` fails the node. A service started with `--egress public`+refuses `file`, `shell` and `serial` altogether.+### `act`: change Same shape as `observe` with any method, or an event: diff --git a/deploy/README.md b/deploy/README.md index 871b347..90721da 100644 --- a/deploy/README.md +++ b/deploy/README.md@@ -81,6 +81,14 @@ rather than the NAS (they need the internet the NAS is kept from, or thenetwork the person is on) go in a `handout` directory next to `trees`, with `PYRTAN_HANDOUT: /handout` in the compose file.+Trees that read the box (a `file` node over `/var/log`, a `shell` node+running `df`, a `serial` node on a USB modem) see the container, not the+NAS: mount the paths and pass the devices through in the compose file+(there are commented lines for it), and the tree's own `paths`,+`commands` and `devices` lists decide what it may touch. Trees that+drive a browser need Playwright, which the image does not carry; run+those from a checkout with `uv sync --extra browser`.+### Without the plugin Over SSH, from the checkout: diff --git a/deploy/openmediavault/docker-compose.yml b/deploy/openmediavault/docker-compose.yml index dca671c..9f92f0c 100644 --- a/deploy/openmediavault/docker-compose.yml +++ b/deploy/openmediavault/docker-compose.yml@@ -57,6 +57,15 @@ services:volumes: - hub-state:/state # - /srv/dev-disk-by-uuid-CHANGE-ME/pyrtan/handout:/handout # with PYRTAN_HANDOUT above: trees the phones run+ # Trees that read files, run programs or open serial lines reach this+ # container, not the NAS: mount what they may see (read-only where you+ # can) and pass the devices through; the tree's own paths/commands/+ # devices lists are the allow-lists. PYRTAN_EGRESS: public closes all+ # three. Browser trees need an image with Playwright (pyrtan[browser]).+ # volumes:+ # - /var/log:/var/log:ro+ # devices:+ # - /dev/ttyUSB0:/dev/ttyUSB0news: <<: *pyrtan diff --git a/examples/disk-space/disk-space.toml b/examples/disk-space/disk-space.toml new file mode 100644 index 0000000..8baa73b --- /dev/null +++ b/examples/disk-space/disk-space.toml@@ -0,0 +1,75 @@+# How full a filesystem is, by asking df on the box this tree runs on. A+# shell node runs one program by name, with its arguments as a list: no+# shell reads the line, so nothing in an argument is ever a command.+# `commands` is the allow-list, as `hosts` is for HTTP; `df` is all this+# tree may run. A service started with --egress public refuses shell nodes+# altogether: a hosted box is not lent out.++name = "disk-space"+version = 1+description = "How full a filesystem is, from df on the box this tree runs on."+tags = ["ops", "box", "read-only"]+license = "MIT"+commands = ["df"]+start = "df"++[[triggers]]+pattern = "[how] full is the disk"+[[triggers]]+pattern = "disk space"+[[triggers]]+pattern = "disk space on {mount:word}"++[inputs.mount]+type = "str"+default = "/"+description = "A mount point, or any path on the filesystem to measure"++[nodes.df]+kind = "observe"+shell = { command = "df", args = ["-P", "{{ inputs.mount }}"] }+into = "df" # state.df = { command, args, status, ok, stdout, stderr, body, json, lines }+extract = [+ { into = "percent", regex = "(\\d+)%", group = 1 },+ { into = "mounted", regex = "%\\s+(\\S+)\\s*$", group = 1 },+]+next = "ran"++[nodes.ran]+kind = "decide"+on = "df.ok"+branches = [{ when = { eq = true }, then = "number" }, { else = "failed" }]++[nodes.number]+kind = "set"+set = { pct = "{{ state.percent | int }}" }+next = "level"++[nodes.level]+kind = "decide"+on = "pct"+branches = [+ { when = { lt = 80 }, then = "fine" },+ { when = { lt = 95 }, then = "tight" },+ { else = "full" },+]++[nodes.fine]+kind = "done"+say = "{{ state.mounted }} is {{ state.pct }}% full."+output = { mounted = "{{ state.mounted }}", percent = "{{ state.pct }}" }++[nodes.tight]+kind = "done"+say = "{{ state.mounted }} is {{ state.pct }}% full; make room soon."+output = { mounted = "{{ state.mounted }}", percent = "{{ state.pct }}" }++[nodes.full]+kind = "fail"+reason = "disk-full"+say = "{{ state.mounted }} is {{ state.pct }}% full."++[nodes.failed]+kind = "fail"+reason = "df-failed"+say = "df could not measure {{ inputs.mount }}: {{ state.df.stderr | strip }}"diff --git a/examples/log-errors/log-errors.toml b/examples/log-errors/log-errors.toml new file mode 100644 index 0000000..a3d88ab --- /dev/null +++ b/examples/log-errors/log-errors.toml@@ -0,0 +1,65 @@+# Error lines in the tail of a log on the box this tree runs on. A file node+# reads a path (whole, or its last `tail` lines); `paths` is the allow-list,+# and a path from the sentence is refused at run time when it falls outside+# /var/log, ".." included.++name = "log-errors"+version = 1+description = "Error lines in the tail of a log file on this box."+tags = ["ops", "box", "read-only"]+license = "MIT"+paths = ["/var/log"]+start = "read"++[[triggers]]+pattern = "errors in the log"+[[triggers]]+pattern = "errors in {path:word}"++[inputs.path]+type = "str"+default = "/var/log/syslog"++[inputs.lines]+type = "int"+default = 200+min = 1+max = 5000+description = "How many lines from the end to look at"++[nodes.read]+kind = "observe"+file = { path = "{{ inputs.path }}", tail = "{{ inputs.lines }}" }+into = "log" # state.log = { path, exists, size, modified, body, json, lines }+extract = [{ into = "errors", regex = "(?im)^.*\\berror\\b.*$", all = true }]+next = "exists"++[nodes.exists]+kind = "decide"+on = "log.exists"+branches = [{ when = { eq = true }, then = "count" }, { else = "missing" }]++[nodes.count]+kind = "set"+set = { n = "{{ state.errors | length }}", seen = "{{ state.log.lines | length }}" }+next = "any"++[nodes.any]+kind = "decide"+on = "n"+branches = [{ when = { eq = 0 }, then = "clean" }, { else = "some" }]++[nodes.clean]+kind = "done"+say = "No error lines in the last {{ state.seen }} of {{ inputs.path }}."+output = { path = "{{ inputs.path }}", errors = 0 }++[nodes.some]+kind = "done"+say = "{{ state.n }} error line(s) in the last {{ state.seen }} of {{ inputs.path }}; latest: {{ state.errors | last | strip }}"+output = { path = "{{ inputs.path }}", errors = "{{ state.n }}", lines = "{{ state.errors }}" }++[nodes.missing]+kind = "fail"+reason = "no-such-file"+say = "{{ inputs.path }} is not on this box."diff --git a/examples/modem-check/modem-check.toml b/examples/modem-check/modem-check.toml new file mode 100644 index 0000000..92ec818 --- /dev/null +++ b/examples/modem-check/modem-check.toml@@ -0,0 +1,53 @@+# Whether a modem on a serial line answers AT. A serial node opens a device+# at a baud rate, writes, and reads until a terminator or the timeout;+# `devices` is the allow-list. An observe retries once on no answer.++name = "modem-check"+version = 1+description = "Whether a modem on a serial line answers AT."+tags = ["hardware", "serial"]+license = "MIT"+devices = ["/dev/ttyUSB*", "/dev/ttyACM*"]+start = "at"++[[triggers]]+pattern = "is the modem up"+[[triggers]]+pattern = "is the modem on {device:word} up"++[inputs.device]+type = "str"+default = "/dev/ttyUSB0"++[nodes.at]+kind = "observe"+serial = { device = "{{ inputs.device }}", baud = 9600, write = "AT\r\n", read_until = "\n" }+into = "s" # state.s = { device, baud, sent, body, bytes, complete, json, lines }+timeout = 3+retries = 1+extract = [{ into = "answer", regex = "(OK|ERROR)" }]+next = "answered"++[nodes.answered]+kind = "decide"+on = "answer"+branches = [+ { when = { eq = "OK" }, then = "yes" },+ { when = { eq = "ERROR" }, then = "error" },+ { else = "odd" },+]++[nodes.yes]+kind = "done"+say = "The modem on {{ inputs.device }} is there."+output = { device = "{{ inputs.device }}" }++[nodes.error]+kind = "fail"+reason = "modem-error"+say = "The modem on {{ inputs.device }} answered ERROR."++[nodes.odd]+kind = "fail"+reason = "odd-answer"+say = "{{ inputs.device }} answered {{ state.s.body | strip }} to AT."diff --git a/examples/rendered-headline/rendered-headline.toml b/examples/rendered-headline/rendered-headline.toml new file mode 100644 index 0000000..a742032 --- /dev/null +++ b/examples/rendered-headline/rendered-headline.toml@@ -0,0 +1,57 @@+# The main heading of a page that needs a script to render: a browser node+# opens it in a real browser (Playwright's Chromium, the one optional+# dependency: pip install 'pyrtan[browser]'), waits for a selector, and+# hands the rendered document to the same extraction an http node uses.+# The page may fetch what it likes; it may not take the tree to a host+# outside `hosts`.++name = "rendered-headline"+version = 1+description = "The main heading of a page that needs a script to render, through a real browser."+tags = ["web", "browser", "read-only"]+license = "MIT"+hosts = ["*"]+start = "open"++[[triggers]]+pattern = "headline of {url:url}"+[[triggers]]+pattern = "what does {url:url} say"++[inputs.url]+type = "str"+pattern = "^https?://.+$"++[nodes.open]+kind = "observe"+browser = { url = "{{ inputs.url }}", wait_for = "h1" }+into = "page" # state.page = { url, status, ok, title, body, elapsed_ms }+timeout = 30+retries = 0+extract = [{ into = "heading", css = "h1" }]+next = "loaded"++[nodes.loaded]+kind = "decide"+on = "page.ok"+branches = [{ when = { eq = true }, then = "found" }, { else = "not-ok" }]++[nodes.found]+kind = "decide"+on = "heading"+branches = [{ when = { exists = true }, then = "say" }, { else = "none" }]++[nodes.say]+kind = "done"+say = "{{ state.page.title }} — {{ state.heading | strip }}"+output = { url = "{{ state.page.url }}", title = "{{ state.page.title }}", heading = "{{ state.heading | strip }}" }++[nodes.none]+kind = "fail"+reason = "no-heading"+say = "{{ state.page.url }} rendered without an h1."++[nodes.not-ok]+kind = "fail"+reason = "unexpected-response"+say = "{{ inputs.url }} answered {{ state.page.status }}."diff --git a/examples/room-temperature/room-temperature.toml b/examples/room-temperature/room-temperature.toml new file mode 100644 index 0000000..c6381db --- /dev/null +++ b/examples/room-temperature/room-temperature.toml@@ -0,0 +1,65 @@+# The temperature a room's sensor last published, over MQTT. A queue node+# subscribes to a topic and waits for one message (an act node publishes+# one). The broker must be in `hosts`, the topic under `topics`; both are+# checked at run time when they come from the sentence.++name = "room-temperature"+version = 1+description = "The temperature a room's sensor last published, over MQTT."+tags = ["home", "mqtt"]+license = "MIT"+hosts = ["broker.local"]+topics = ["home/+/temperature"]+start = "wait"++[[triggers]]+pattern = "[what's the] temperature in the {room:word}"+[[triggers]]+pattern = "how warm is the {room:word}"++[inputs.room]+type = "str"+default = "kitchen"+pattern = "^[a-z0-9-]+$"++[nodes.wait]+kind = "observe"+queue = { url = "mqtt://broker.local", topic = "home/{{ inputs.room }}/temperature" }+into = "q" # state.q = { url, topic, received, message = { topic, body, json }, body, json }+timeout = 15+retries = 0+next = "got"++[nodes.got]+kind = "decide"+on = "q.received"+branches = [{ when = { eq = true }, then = "reading" }, { else = "quiet" }]++[nodes.reading]+kind = "decide"+on = "q.json.temperature"+branches = [+ { when = { lt = 16 }, then = "cold" },+ { when = { lt = 26 }, then = "fine" },+ { when = { gte = 26 }, then = "warm" },+]++[nodes.cold]+kind = "done"+say = "{{ inputs.room }}: {{ state.q.json.temperature }}°C, cold."+output = { room = "{{ inputs.room }}", temperature = "{{ state.q.json.temperature }}", topic = "{{ state.q.message.topic }}" }++[nodes.fine]+kind = "done"+say = "{{ inputs.room }}: {{ state.q.json.temperature }}°C."+output = { room = "{{ inputs.room }}", temperature = "{{ state.q.json.temperature }}", topic = "{{ state.q.message.topic }}" }++[nodes.warm]+kind = "done"+say = "{{ inputs.room }}: {{ state.q.json.temperature }}°C, warm."+output = { room = "{{ inputs.room }}", temperature = "{{ state.q.json.temperature }}", topic = "{{ state.q.message.topic }}" }++[nodes.quiet]+kind = "fail"+reason = "no-reading"+say = "Nothing on home/{{ inputs.room }}/temperature within 15 s; is the sensor publishing?"diff --git a/pyproject.toml b/pyproject.toml index 15bafee..3bced09 100644 --- a/pyproject.toml +++ b/pyproject.toml@@ -1,6 +1,6 @@[project] name = "pyrtan"-version = "0.6.0"+version = "0.7.0"description = "Open-source total automation: a core set of expandable services that act on all forms of I/O through hardcoded, inspectable decision trees." readme = "README.md" license = "AGPL-3.0-or-later" diff --git a/src/pyrtan/__init__.py b/src/pyrtan/__init__.py index f0a53a2..dee4ef0 100644 --- a/src/pyrtan/__init__.py +++ b/src/pyrtan/__init__.py@@ -33,4 +33,4 @@ __all__ = ["Response", ]-__version__ = "0.6.0"+__version__ = "0.7.0"diff --git a/src/pyrtan/adapters.py b/src/pyrtan/adapters.py new file mode 100644 index 0000000..a2ecf90 --- /dev/null +++ b/src/pyrtan/adapters.py@@ -0,0 +1,680 @@+"""The other forms of I/O: a browser, a file, a shell, a serial line, a queue.++HTTP is the form the core spoke first (``transport.py``). Every other form+arrives here by the same rule: a node names a *source* and what it wants+from it, the tree lists what that source may reach, the runtime refuses+anything outside the list, and the adapter moves the bytes. The trace reads+the same whichever source a node used.++ [nodes.df] observe shell = { command = "df", args = ["-P", "/"] }+ [nodes.log] observe file = { path = "/var/log/syslog", tail = 200 }+ [nodes.note] act file = { path = "./notes.txt", append = "..." }+ [nodes.modem] observe serial = { device = "/dev/ttyUSB0", baud = 9600, write = "AT\\r\\n", read_until = "\\n" }+ [nodes.sensor] observe queue = { url = "mqtt://broker.local", topic = "home/kitchen/temperature" }+ [nodes.switch] act queue = { url = "mqtt://broker.local", topic = "home/kitchen/light", publish = "on" }+ [nodes.page] observe browser = { url = "https://example.com/app", wait_for = "h1", actions = [{ click = "#more" }] }++and at the top of the tree the allow-list of the same kind: ``hosts`` for+``http``, ``browser`` and a queue's broker; ``paths`` for ``file``;+``commands`` for ``shell``; ``devices`` for ``serial``; ``topics`` for+``queue``. A tree without the list cannot use the source; ``pyrtan check``+says so.++What each source leaves in state (the node's ``into``):++``file`` ``path``, ``exists``, ``size``, ``modified``, ``body``, ``json``, ``lines``;+ after a write, ``written`` as well.+``shell`` ``command``, ``args``, ``status`` (the exit code), ``ok``, ``stdout``,+ ``stderr``, ``body`` (stdout again, so ``extract`` reads it by+ default), ``json`` (stdout parsed), ``lines``.+``serial`` ``device``, ``baud``, ``sent``, ``body``, ``bytes``, ``complete``, ``json``, ``lines``.+``queue`` ``url``, ``topic``, ``received``, ``message`` (``topic`` and ``body``),+ ``body``, ``json``; after a publish, ``published``.+``browser`` ``url`` (where the page ended up), ``status``, ``ok``, ``title``,+ ``body`` (the rendered document), ``elapsed_ms``.++Files, shells and serial lines are on the box the service runs on. A+service started with ``--egress public`` refuses them outright, the way it+refuses private addresses: an operator hosting other people's trees does+not lend them the box. A queue's broker and a browser's page go through the+same egress check as an HTTP request.++The browser adapter needs Playwright (``pip install 'pyrtan[browser]'``),+the one optional dependency; everything else here is the standard library.+A queue is MQTT 3.1.1 over a socket, the few packets a tree needs: connect,+subscribe and wait for one message, or publish one; QoS 0.++:class:`LocalAdapters` is the real thing. A recorded fixture (see+``transport.RecordedTransport``) answers for every source as well, keyed by+:func:`adapter_key`, so a tree over any of them is tested offline the same+way an HTTP tree is.+"""++from __future__ import annotations++import datetime as _dt+import fnmatch+import json+import os+import queue as _queue+import select+import socket+import ssl+import struct+import subprocess+import threading+import time+import urllib.parse+from typing import Any, Callable++from .transport import EgressRefused, TransportError, check_egress++SOURCES = ("http", "browser", "file", "shell", "serial", "queue")+LOCAL_SOURCES = ("file", "shell", "serial") # on the box itself+ALLOW_LISTS = {"http": ("hosts",), "browser": ("hosts",), "file": ("paths",), "shell": ("commands",), "serial": ("devices",), "queue": ("hosts", "topics")}+LIST_NAMES = ("hosts", "paths", "commands", "devices", "topics")+MAX_READ = 8 * 1024 * 1024 # a file or a command's output larger than this is refused+TAIL_WINDOW = 1024 * 1024 # how far back a tail looks+MQTT_PORTS = {"mqtt": 1883, "mqtts": 8883}+BROWSER_ACTIONS = ("click", "fill", "press", "select", "wait", "wait_for")++Allowed = Callable[[str], bool]+++# ---------------------------------------------------------------- allow-lists+++def path_allowed(patterns: list[str], path: str) -> bool:+ """``paths`` entries are absolute paths or globs; a directory covers+ everything beneath it; ``*`` covers everything."""+ target = absolute_path(path)+ for pat in patterns:+ if pat == "*":+ return True+ pat = absolute_path(pat)+ if any(c in pat for c in "*?["):+ if fnmatch.fnmatchcase(target, pat):+ return True+ elif target == pat or target.startswith(pat.rstrip("/") + "/"):+ return True+ return False+++def absolute_path(path: str) -> str:+ return os.path.abspath(os.path.expanduser(str(path)))+++def command_allowed(commands: list[str], command: str) -> bool:+ """``commands`` entries are program names (matched against the name the+ node gives, or the last part of a path it gives) or full paths."""+ command = str(command)+ for entry in commands:+ if entry == "*":+ return True+ if "/" in entry:+ if absolute_path(command) == absolute_path(entry):+ return True+ elif command == entry or os.path.basename(command) == entry:+ return True+ return False+++def device_allowed(devices: list[str], device: str) -> bool:+ device = str(device)+ return any(d == "*" or fnmatch.fnmatchcase(device, d) for d in devices)+++def topic_allowed(topics: list[str], topic: str) -> bool:+ """``topics`` entries are MQTT filters: ``+`` matches one level (so does+ ``*``, for people who think in globs), ``#`` the rest."""+ parts = str(topic).split("/")+ for entry in topics:+ if entry in ("*", "#"):+ return True+ pat = entry.split("/")+ ok = True+ for i, seg in enumerate(pat):+ if seg == "#":+ ok = ok and i <= len(parts)+ break+ if i >= len(parts) or (seg not in ("+", "*") and seg != parts[i]):+ ok = False+ break+ else:+ ok = ok and len(pat) == len(parts)+ if ok:+ return True+ return False+++def broker(url: str) -> tuple[str, str, int]:+ """``(scheme, host, port)`` of a queue URL, or a :class:`ValueError`."""+ parsed = urllib.parse.urlsplit(str(url))+ if parsed.scheme not in MQTT_PORTS or not parsed.hostname:+ raise ValueError(f"{url!r} is not mqtt://host[:port] or mqtts://host[:port]")+ return parsed.scheme, parsed.hostname, parsed.port or MQTT_PORTS[parsed.scheme]+++# ---------------------------------------------------------------- keys and words+++def adapter_key(source: str, spec: dict[str, Any]) -> str:+ """How a fixture names one request to a source. The JavaScript runtime+ builds the same key, so one fixture serves both."""+ if source == "file":+ mode = "write " if "write" in spec else "append " if "append" in spec else ""+ return f"file {mode}{spec.get('path')}"+ if source == "shell":+ return "shell " + " ".join([str(spec.get("command"))] + [_arg(a) for a in (spec.get("args") or [])])+ if source == "serial":+ write = spec.get("write")+ return f"serial {spec.get('device')}" + (f" {json.dumps(str(write))}" if write not in (None, "") else "")+ if source == "queue":+ mode = "publish " if "publish" in spec else ""+ return f"queue {mode}{spec.get('url')} {spec.get('topic')}"+ if source == "browser":+ return f"browser {spec.get('url')}"+ raise ValueError(source)+++def _arg(a: Any) -> str:+ return a if isinstance(a, str) else json.dumps(a)+++def what(source: str, spec: dict[str, Any]) -> str:+ """The words a trace uses for one request: what was reached."""+ if source == "file":+ return str(spec.get("path"))+ if source == "shell":+ return " ".join([str(spec.get("command"))] + [_arg(a) for a in (spec.get("args") or [])])+ if source == "serial":+ return str(spec.get("device"))+ if source == "queue":+ return f"{spec.get('topic')} at {spec.get('url')}"+ return str(spec.get("url"))+++def brief_result(source: str, observed: dict[str, Any]) -> str:+ """The words a trace uses for what came back."""+ if source in ("http", "browser"):+ return str(observed.get("status"))+ if source == "file":+ if "written" in observed:+ return f"wrote {observed['written']} B"+ return f"{observed.get('size')} B" if observed.get("exists") else "missing"+ if source == "shell":+ return f"exit {observed.get('status')}"+ if source == "serial":+ return f"{observed.get('bytes')} B"+ if source == "queue":+ if "published" in observed:+ return f"published {observed['published']} B"+ return "1 message" if observed.get("received") else "no message"+ return "ok"+++def _json(text: str | None) -> Any:+ if not text:+ return None+ try:+ return json.loads(text)+ except (json.JSONDecodeError, TypeError):+ return None+++def _iso(ts: float) -> str:+ return _dt.datetime.fromtimestamp(ts, _dt.timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")+++# ---------------------------------------------------------------- the real adapters+++class LocalAdapters:+ """The sources as this box reaches them. ``egress`` is the service's+ policy; ``browser`` an executable for Playwright's Chromium, if not the+ one Playwright installed."""++ def __init__(self, egress: str = "any", browser: str | None = None) -> None:+ self.egress = egress+ self._browser: Browser | None = None+ self._browser_path = browser or os.environ.get("PYRTAN_BROWSER") or None+ self._lock = threading.Lock()++ def perform(self, source: str, spec: dict[str, Any], timeout: float, allowed: Allowed | None = None) -> dict[str, Any]:+ if source in LOCAL_SOURCES and self.egress != "any":+ raise EgressRefused(f"{source}: this service reaches only public addresses (--egress {self.egress}); files, shells and serial lines are its own box, and stay closed")+ if source == "file":+ return read_or_write_file(spec)+ if source == "shell":+ return run_command(spec, timeout)+ if source == "serial":+ return serial_exchange(spec, timeout)+ if source == "queue":+ return mqtt(spec, timeout, self.egress)+ if source == "browser":+ with self._lock:+ if self._browser is None:+ self._browser = Browser(self._browser_path)+ return self._browser.perform(spec, timeout, allowed or (lambda url: True), self.egress)+ raise TransportError(f"no adapter for {source!r}")++ def close(self) -> None:+ if self._browser is not None:+ self._browser.close()+++# ---------------------------------------------------------------- file+++def read_or_write_file(spec: dict[str, Any]) -> dict[str, Any]:+ path = absolute_path(spec.get("path", ""))+ if "write" in spec or "append" in spec:+ data = spec["write"] if "write" in spec else spec["append"]+ if not isinstance(data, str):+ data = json.dumps(data, ensure_ascii=False) + "\n"+ if not os.path.isdir(os.path.dirname(path) or "."):+ raise TransportError(f"{path}: the directory does not exist; a tree writes files, it does not make directories")+ try:+ with open(path, "a" if "append" in spec else "w", encoding="utf-8") as f:+ f.write(data)+ except OSError as e:+ raise TransportError(f"{path}: {e.strerror or e}") from e+ st = os.stat(path)+ return {"path": path, "exists": True, "written": len(data.encode("utf-8")), "size": st.st_size, "modified": _iso(st.st_mtime)}+ if not os.path.exists(path):+ return {"path": path, "exists": False, "size": None, "modified": None, "body": None, "json": None, "lines": []}+ try:+ st = os.stat(path)+ tail = spec.get("tail")+ if tail is not None:+ tail = int(tail)+ with open(path, "rb") as f:+ f.seek(max(0, st.st_size - TAIL_WINDOW))+ raw = f.read()+ lines = raw.decode("utf-8", errors="replace").splitlines()+ if st.st_size > TAIL_WINDOW and lines:+ lines = lines[1:] # the first line is a fragment+ lines = lines[-tail:] if tail > 0 else []+ text = "\n".join(lines)+ else:+ if st.st_size > MAX_READ:+ raise TransportError(f"{path}: {st.st_size} bytes is more than a tree reads at once ({MAX_READ}); give 'tail'")+ with open(path, "rb") as f:+ text = f.read().decode("utf-8", errors="replace")+ lines = text.splitlines()+ except OSError as e:+ raise TransportError(f"{path}: {e.strerror or e}") from e+ return {"path": path, "exists": True, "size": st.st_size, "modified": _iso(st.st_mtime), "body": text, "json": _json(text), "lines": lines}+++# ---------------------------------------------------------------- shell+++def run_command(spec: dict[str, Any], timeout: float) -> dict[str, Any]:+ """One program, by name, with its arguments as a list: no shell reads+ the line, so nothing in an argument is ever a command."""+ argv = [str(spec.get("command", ""))] + [_arg(a) for a in (spec.get("args") or [])]+ stdin = spec.get("stdin")+ if stdin is not None and not isinstance(stdin, str):+ stdin = json.dumps(stdin)+ env = dict(os.environ)+ env.update({str(k): str(v) for k, v in (spec.get("env") or {}).items()})+ cwd = spec.get("cwd")+ started = time.monotonic()+ try:+ p = subprocess.run(argv, input=stdin, capture_output=True, text=True, timeout=timeout, cwd=str(cwd) if cwd else None, env=env, shell=False, errors="replace")+ except FileNotFoundError:+ raise TransportError(f"{argv[0]}: not found on this box") from None+ except subprocess.TimeoutExpired:+ raise TransportError(f"{argv[0]}: no answer within {timeout:g} s") from None+ except OSError as e:+ raise TransportError(f"{argv[0]}: {e.strerror or e}") from e+ out, err = p.stdout[:MAX_READ], p.stderr[:MAX_READ]+ return {"command": argv[0], "args": argv[1:], "status": p.returncode, "ok": p.returncode == 0, "stdout": out, "stderr": err, "body": out, "json": _json(out), "lines": out.splitlines(), "elapsed_ms": int((time.monotonic() - started) * 1000)}+++# ---------------------------------------------------------------- serial+++def serial_exchange(spec: dict[str, Any], timeout: float) -> dict[str, Any]:+ """Open a serial device at a baud rate, write what the node says, read+ until a terminator, a byte count or the timeout."""+ try:+ import termios+ except ImportError: # pragma: no cover - Windows+ raise TransportError("serial lines need a POSIX system") from None+ device = str(spec.get("device", ""))+ baud = int(spec.get("baud", 9600))+ write = spec.get("write")+ until = spec.get("read_until", "\n" if spec.get("read") is None else None)+ count = int(spec.get("read") or 0)+ speed = getattr(termios, f"B{baud}", None)+ if speed is None:+ raise TransportError(f"{device}: {baud} baud is not a rate this system knows")+ try:+ fd = os.open(device, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)+ except OSError as e:+ raise TransportError(f"{device}: {e.strerror or e}") from e+ sent = 0+ buf = b""+ try:+ attrs = termios.tcgetattr(fd)+ attrs[0] &= ~(termios.IXON | termios.IXOFF | termios.ICRNL | termios.INLCR | termios.IGNCR | termios.ISTRIP) # iflag+ attrs[1] &= ~termios.OPOST # oflag+ attrs[2] = (attrs[2] & ~(termios.PARENB | termios.CSTOPB | termios.CSIZE)) | termios.CS8 | termios.CLOCAL | termios.CREAD # cflag+ attrs[3] &= ~(termios.ICANON | termios.ECHO | termios.ECHOE | termios.ISIG | termios.IEXTEN) # lflag+ attrs[4] = attrs[5] = speed+ termios.tcsetattr(fd, termios.TCSANOW, attrs)+ if write not in (None, ""):+ data = str(write).encode("utf-8")+ while sent < len(data):+ _, ready, _ = select.select([], [fd], [], timeout)+ if not ready:+ raise TransportError(f"{device}: could not write within {timeout:g} s")+ sent += os.write(fd, data[sent:])+ deadline = time.monotonic() + timeout+ terminator = str(until).encode("utf-8") if until else None+ complete = False+ while True:+ remaining = deadline - time.monotonic()+ if remaining <= 0:+ break+ ready, _, _ = select.select([fd], [], [], remaining)+ if not ready:+ break+ try:+ chunk = os.read(fd, 4096)+ except BlockingIOError:+ continue+ except OSError as e:+ if buf:+ break+ raise TransportError(f"{device}: {e.strerror or e}") from e+ if not chunk:+ break+ buf += chunk+ if (terminator and terminator in buf) or (count and len(buf) >= count) or len(buf) >= MAX_READ:+ complete = True+ break+ finally:+ os.close(fd)+ if not buf and (write not in (None, "") or terminator or count):+ raise TransportError(f"{device}: no answer within {timeout:g} s")+ text = buf.decode("utf-8", errors="replace")+ return {"device": device, "baud": baud, "sent": sent, "body": text, "bytes": len(buf), "complete": complete, "json": _json(text.strip()), "lines": text.splitlines()}+++# ---------------------------------------------------------------- queue (MQTT 3.1.1)+++def _mqtt_str(s: str) -> bytes:+ b = s.encode("utf-8")+ return struct.pack(">H", len(b)) + b+++def _mqtt_packet(head: int, body: bytes) -> bytes:+ n = len(body)+ length = b""+ while True:+ byte = n % 128+ n //= 128+ length += bytes([byte | (0x80 if n else 0)])+ if not n:+ break+ return bytes([head]) + length + body+++def _mqtt_read(sock: socket.socket, deadline: float) -> tuple[int, int, bytes] | None:+ """One packet: ``(type, flags, body)``, or None at the deadline."""++ def take(n: int) -> bytes | None:+ out = b""+ while len(out) < n:+ remaining = deadline - time.monotonic()+ if remaining <= 0:+ return None+ sock.settimeout(remaining)+ try:+ chunk = sock.recv(n - len(out))+ except (socket.timeout, TimeoutError):+ return None+ if not chunk:+ raise TransportError("the broker closed the connection")+ out += chunk+ return out++ head = take(1)+ if head is None:+ return None+ n = 0+ shift = 1+ while True:+ b = take(1)+ if b is None:+ return None+ n += (b[0] & 0x7F) * shift+ shift *= 128+ if not b[0] & 0x80:+ break+ body = take(n) if n else b""+ if body is None:+ return None+ return head[0] >> 4, head[0] & 0x0F, body+++def mqtt(spec: dict[str, Any], timeout: float, egress: str = "any") -> dict[str, Any]:+ """Connect, then either publish one message (``publish``) or subscribe+ and wait for one. QoS 0, clean session, no retained state of our own."""+ url = str(spec.get("url", ""))+ topic = str(spec.get("topic", ""))+ try:+ scheme, host, port = broker(url)+ except ValueError as e:+ raise TransportError(str(e)) from None+ check_egress(f"http://{host}:{port}/", egress)+ deadline = time.monotonic() + timeout+ try:+ sock = socket.create_connection((host, port), timeout=timeout)+ except OSError as e:+ raise TransportError(f"{url}: {e.strerror or e}") from e+ try:+ if scheme == "mqtts":+ sock = ssl.create_default_context().wrap_socket(sock, server_hostname=host)+ client_id = str(spec.get("client_id") or f"pyrtan-{os.getpid()}-{int(time.time() * 1000) % 100000}")+ user, password = spec.get("username"), spec.get("password")+ flags = 0x02 | (0x80 if user else 0) | (0x40 if password else 0)+ keepalive = min(65535, max(60, int(timeout) + 30))+ body = _mqtt_str("MQTT") + bytes([4, flags]) + struct.pack(">H", keepalive) + _mqtt_str(client_id)+ if user:+ body += _mqtt_str(str(user))+ if password:+ body += _mqtt_str(str(password))+ sock.sendall(_mqtt_packet(0x10, body))+ ack = _mqtt_read(sock, deadline)+ if ack is None or ack[0] != 2 or len(ack[2]) < 2:+ raise TransportError(f"{url}: no CONNACK within {timeout:g} s")+ if ack[2][1] != 0:+ raise TransportError(f"{url}: connection refused (code {ack[2][1]})")+ if "publish" in spec:+ payload = spec["publish"]+ if not isinstance(payload, str):+ payload = json.dumps(payload, ensure_ascii=False)+ data = payload.encode("utf-8")+ sock.sendall(_mqtt_packet(0x30, _mqtt_str(topic) + data))+ sock.sendall(_mqtt_packet(0xE0, b""))+ return {"url": url, "topic": topic, "published": len(data), "body": payload}+ sock.sendall(_mqtt_packet(0x82, struct.pack(">H", 1) + _mqtt_str(topic) + b"\x00"))+ message = None+ while True:+ pkt = _mqtt_read(sock, deadline)+ if pkt is None:+ break+ ptype, pflags, pbody = pkt+ if ptype == 9: # SUBACK+ if len(pbody) >= 3 and pbody[2] == 0x80:+ raise TransportError(f"{url}: the broker refused the subscription to {topic!r}")+ continue+ if ptype == 3: # PUBLISH+ tlen = struct.unpack(">H", pbody[:2])[0]+ mtopic = pbody[2:2 + tlen].decode("utf-8", errors="replace")+ rest = pbody[2 + tlen:]+ qos = (pflags >> 1) & 3+ if qos:+ pid, rest = rest[:2], rest[2:]+ if qos == 1:+ sock.sendall(_mqtt_packet(0x40, pid))+ text = rest.decode("utf-8", errors="replace")+ message = {"topic": mtopic, "body": text, "json": _json(text)}+ break+ try:+ sock.sendall(_mqtt_packet(0xE0, b""))+ except OSError:+ pass+ return {"url": url, "topic": topic, "received": message is not None, "message": message, "body": message["body"] if message else None, "json": message["json"] if message else None}+ finally:+ try:+ sock.close()+ except OSError:+ pass+++# ---------------------------------------------------------------- browser (Playwright)+++class Browser:+ """A page rendered by Chromium, through Playwright. Playwright's sync+ API lives on one thread, so one worker thread owns the browser and the+ service's request threads queue jobs for it. The page may fetch what+ it likes; it may not take the tree to another host: a top-level+ navigation outside the tree's hosts is aborted."""++ def __init__(self, executable: str | None = None) -> None:+ self.executable = executable+ self._jobs: _queue.Queue[tuple[Any, ...] | None] = _queue.Queue()+ self._thread: threading.Thread | None = None+ self._error: str | None = None+ self._ready = threading.Event()+ self._lock = threading.Lock()++ def _start(self) -> None:+ with self._lock:+ if self._thread is None:+ self._thread = threading.Thread(target=self._loop, name="pyrtan-browser", daemon=True)+ self._thread.start()+ self._ready.wait(120)+ if self._error:+ raise TransportError(self._error)++ def _loop(self) -> None:+ try:+ from playwright.sync_api import sync_playwright+ except ImportError:+ self._error = "the browser adapter needs Playwright: pip install 'pyrtan[browser]' (and 'playwright install chromium', or set PYRTAN_BROWSER to a Chromium)"+ self._ready.set()+ return+ try:+ pw = sync_playwright().start()+ browser = pw.chromium.launch(executable_path=self.executable, headless=True)+ except Exception as e:+ self._error = f"the browser could not start: {str(e).splitlines()[0]}"+ self._ready.set()+ return+ self._ready.set()+ while True:+ job = self._jobs.get()+ if job is None:+ break+ spec, timeout, allowed, done, box = job+ try:+ box["result"] = self._render(browser, spec, timeout, allowed)+ except TransportError as e:+ box["error"] = str(e)+ except Exception as e: # Playwright's own errors+ box["error"] = str(e).splitlines()[0]+ finally:+ done.set()+ try:+ browser.close()+ pw.stop()+ except Exception:+ pass++ def _render(self, browser: Any, spec: dict[str, Any], timeout: float, allowed: Allowed) -> dict[str, Any]:+ url = str(spec.get("url", ""))+ started = time.monotonic()+ context = browser.new_context(user_agent=f"pyrtan/{_version()} (+https://github.com/b-i-t-8/pyrtan)")+ try:+ page = context.new_page()+ page.set_default_timeout(timeout * 1000)+ blocked: list[str] = []++ def route(r: Any) -> None:+ req = r.request+ if req.resource_type == "document" and req.is_navigation_request() and not allowed(req.url):+ blocked.append(req.url)+ r.abort()+ else:+ r.continue_()++ page.route("**/*", route)+ try:+ resp = page.goto(url, wait_until=str(spec.get("wait_until", "load")))+ except Exception as e:+ if blocked:+ raise TransportError(f"{url} led to {blocked[-1]}, outside this tree's hosts; the navigation was refused") from None+ raise TransportError(f"{url}: {str(e).splitlines()[0]}") from None+ if blocked:+ raise TransportError(f"{url} led to {blocked[-1]}, outside this tree's hosts; the navigation was refused")+ wait_for = spec.get("wait_for")+ if wait_for:+ page.wait_for_selector(str(wait_for))+ for action in spec.get("actions") or []:+ self._act(page, action)+ if blocked:+ raise TransportError(f"{page.url} led to {blocked[-1]}, outside this tree's hosts; the navigation was refused")+ status = resp.status if resp is not None else None+ return {"url": page.url, "status": status, "ok": status is not None and 200 <= status < 300, "title": page.title(), "body": page.content(), "elapsed_ms": int((time.monotonic() - started) * 1000)}+ finally:+ context.close()++ @staticmethod+ def _act(page: Any, action: dict[str, Any]) -> None:+ if "click" in action:+ page.click(str(action["click"]))+ elif "fill" in action:+ page.fill(str(action["fill"]["selector"]), str(action["fill"]["value"]))+ elif "press" in action:+ page.press(str(action["press"]["selector"]), str(action["press"]["key"]))+ elif "select" in action:+ page.select_option(str(action["select"]["selector"]), str(action["select"]["value"]))+ elif "wait_for" in action:+ page.wait_for_selector(str(action["wait_for"]))+ elif "wait" in action:+ page.wait_for_timeout(float(action["wait"]))+ else:+ raise TransportError(f"a browser action is one of {', '.join(BROWSER_ACTIONS)}, not {sorted(action)}")++ def perform(self, spec: dict[str, Any], timeout: float, allowed: Allowed, egress: str = "any") -> dict[str, Any]:+ url = str(spec.get("url", ""))+ check_egress(url, egress)+ self._start()+ done = threading.Event()+ box: dict[str, Any] = {}+ self._jobs.put((spec, timeout, allowed, done, box))+ if not done.wait(timeout + 30):+ raise TransportError(f"{url}: the browser did not answer within {timeout + 30:g} s")+ if "error" in box:+ raise TransportError(box["error"])+ return box["result"]++ def close(self) -> None:+ if self._thread is not None:+ self._jobs.put(None)+ self._thread.join(10)+ self._thread = None+++def _version() -> str:+ from . import __version__++ return __version__diff --git a/src/pyrtan/catalog.py b/src/pyrtan/catalog.py index 966aa77..eb5c48d 100644 --- a/src/pyrtan/catalog.py +++ b/src/pyrtan/catalog.py@@ -148,7 +148,7 @@ def _fetch(url: str, timeout: float = 20.0) -> bytes:raise CatalogError(f"{url}: {e}") from e if parts.scheme not in ("http", "https"): raise CatalogError(f"{url}: a catalog is reached over https (or a file: path)")- req = urllib.request.Request(url, headers={"User-Agent": "pyrtan-catalog/0.6", "Accept": "application/json, application/toml, text/plain, */*"})+ req = urllib.request.Request(url, headers={"User-Agent": "pyrtan-catalog/0.7", "Accept": "application/json, application/toml, text/plain, */*"})try: with urllib.request.urlopen(req, timeout=timeout) as resp: return resp.read() diff --git a/src/pyrtan/cli.py b/src/pyrtan/cli.py index 1a870dd..644d7fc 100644 --- a/src/pyrtan/cli.py +++ b/src/pyrtan/cli.py@@ -48,6 +48,7 @@ from .page import MANIFEST, PAGE, PYRTAN_JS, SERVICE_WORKERfrom .registry import Registry, default_paths from .runtime import Outcome, Runner from .service import RemoteResolver, http_json, outcome_from_dict, serve+from .adapters import LocalAdaptersfrom .transport import EGRESS_POLICIES, RecordedTransport, RecordingTransport, UrllibTransport from .tree import check, load_tree@@ -74,7 +75,7 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--json", action="store_true", help="print the outcome as JSON") p.add_argument("--no-trace", action="store_true", help="do not print the trace") p.add_argument("--replay", help="answer HTTP from this fixture instead of the network")- p.add_argument("--record", help="save every HTTP response to this fixture")+ p.add_argument("--record", help="save every response (HTTP and the other sources) to this fixture")p.add_argument("--hub", help="forward trees not held here to this hub or service") p.add_argument("--env", action="append", default=[], metavar="K=V", help="a value trees may read as {{ env.K }} (repeatable)") p.add_argument("--egress", choices=EGRESS_POLICIES, default="any", help="'public' refuses private, loopback and link-local addresses")@@ -270,7 +271,8 @@ def cmd_list(args: argparse.Namespace) -> int:print(json.dumps({"trees": reg.capabilities(), "errors": reg.errors}, indent=2, ensure_ascii=False)) return 0 for tree in reg.trees():- print(f"{tree.name} v{tree.version} {tree.description}")+ via = ", ".join(s for s in tree.sources() if s != "http")+ print(f"{tree.name} v{tree.version} {tree.description}" + (f" [{via}]" if via else ""))for t in tree.triggers: print(f" {t.pattern}") for path, err in reg.errors.items():@@ -299,11 +301,12 @@ def cmd_run(args: argparse.Namespace) -> int:for path, err in reg.errors.items(): print(f"pyrtan: {path}: {err}", file=sys.stderr) transport: Any = RecordedTransport.from_file(args.replay) if args.replay else UrllibTransport(egress=args.egress)+ adapters: Any = transport if args.replay else LocalAdapters(egress=args.egress)recorder = None if args.record:- recorder = RecordingTransport(transport)- transport = recorder- runner = Runner(reg, transport, resolver=RemoteResolver(args.hub, args.token) if args.hub else None, env=_kv(args.env))+ recorder = RecordingTransport(transport, adapters)+ transport = adapters = recorder+ runner = Runner(reg, transport, resolver=RemoteResolver(args.hub, args.token) if args.hub else None, env=_kv(args.env), adapters=adapters)if args.tree: outcome = runner.run_tree(args.tree, _kv(args.input)) else: diff --git a/src/pyrtan/hub.py b/src/pyrtan/hub.py index c7cd453..882efff 100644 --- a/src/pyrtan/hub.py +++ b/src/pyrtan/hub.py@@ -151,14 +151,17 @@ class Hub:def handout_trees(self) -> dict[str, Tree]: """The trees a trusted member may run: the ones this hub holds itself,- and the ones given it to hand out (``--handout``), by name."""+ and the ones given it to hand out (``--handout``), by name. A member+ is a browser: a tree that reads a file, runs a program, opens a+ serial line, talks to a broker or drives a browser stays here."""out: dict[str, Tree] = {} for reg in (self.local.service.registry if self.local is not None else None, self.handout_registry): if reg is None: continue reg.reload() for t in reg.trees():- out.setdefault(t.name, t)+ if set(t.sources()) <= {"http"}:+ out.setdefault(t.name, t)return out def member_capabilities(self) -> list[dict[str, Any]]: diff --git a/src/pyrtan/js/pyrtan.js b/src/pyrtan/js/pyrtan.js index d9486e8..eac836c 100644 --- a/src/pyrtan/js/pyrtan.js +++ b/src/pyrtan/js/pyrtan.js@@ -8,8 +8,9 @@* What differs, and is written down here rather than hidden: * - `float` in a condition's `type` means any number here; JSON has no * distinct float, so 4.0 and 4 are the same value.- * - CSS extraction uses the host's selector engine (DOMParser) plus- * `:contains(text)`; in a host without a DOM, a css extract is MISSING.+ * - HTML is read by a port of the Python runtime's own parser and selector+ * subset, not the host's DOM, so a page reads the same everywhere; named+ * character references beyond the common ones are left as written.* - Regular expressions are the host's; the subset trees use is common. * - There is no intent matching here: a member runs the tree its hub * already chose. `delegate` goes to the resolver (the hub).@@ -219,43 +220,170 @@} const holds = (cond, value) => Object.keys(cond).every((k) => test(k, cond[k], value));- // ------------------------------------------------------------ html (host engine)+ // ------------------------------------------------------------ html+ // The Python runtime's own document model and selector subset (html.py),+ // here too, so a page reads the same on a service and on a member, and+ // the same in node: tag, #id, .class, [attr], [attr=v], [attr^=v],+ // [attr$=v], [attr*=v], :first-child, :last-child, :nth-child(n),+ // :contains(text), descendant and > combinators, and , groups.+ const VOID = new Set(["area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr"]);const INLINE = new Set(["a", "abbr", "b", "bdi", "bdo", "cite", "code", "data", "dfn", "em", "i", "kbd", "label", "mark", "q", "s", "samp", "small", "span", "strong", "sub", "sup", "time", "u", "var", "del", "ins"]);- function textOf(el) {- const parts = [];- for (const child of el.childNodes) {- if (child.nodeType === 3) parts.push(child.nodeValue);- else if (child.nodeType === 1) parts.push(INLINE.has(child.tagName.toLowerCase()) ? textOf(child) : " " + textOf(child) + " ");+ const RAW_TEXT = new Set(["script", "style"]);+ const ENTITIES = { amp: "&", lt: "<", gt: ">", quot: '"', apos: "'", nbsp: " ", copy: "©", reg: "®", trade: "™", hellip: "…", mdash: "—", ndash: "–", laquo: "«", raquo: "»", lsquo: "‘", rsquo: "’", ldquo: "“", rdquo: "”", bull: "•", middot: "·", deg: "°", euro: "€", pound: "£", yen: "¥", cent: "¢", times: "×", divide: "÷", plusmn: "±", frac12: "½", eacute: "é", egrave: "è", agrave: "à", ccedil: "ç", ouml: "ö", auml: "ä", uuml: "ü", szlig: "ß", ntilde: "ñ", aring: "å", oslash: "ø", aelig: "æ" };+ function unescapeHtml(text) {+ if (!text.includes("&")) return text;+ return text.replace(/&(#x[0-9a-fA-F]+|#[0-9]+|[a-zA-Z][a-zA-Z0-9]*);?/g, (m, body) => {+ if (body[0] === "#") { const code = body[1] === "x" || body[1] === "X" ? parseInt(body.slice(2), 16) : parseInt(body.slice(1), 10); return Number.isFinite(code) && code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : m; }+ return body in ENTITIES ? ENTITIES[body] : m;+ });+ }+ class Element {+ constructor(tag, attrs, parent) { this.tag = tag; this.attrs = attrs || {}; this.children = []; this.parent = parent || null; }+ get text() {+ const parts = [];+ for (const child of this.children) {+ if (typeof child === "string") parts.push(child);+ else parts.push(INLINE.has(child.tag) ? child.text : " " + child.text + " ");+ }+ return parts.join("").replace(/\s+/g, " ").trim();}- return parts.join("").replace(/\s+/g, " ").trim();+ get classes() { return (this.attrs.class || "").split(/\s+/).filter(Boolean); }+ getAttribute(name) { return name in this.attrs ? this.attrs[name] : null; }+ elements() { return this.children.filter((c) => typeof c !== "string"); }+ *descendants() { for (const child of this.elements()) { yield child; yield* child.descendants(); } }+ select(selector) { return select(this, selector); }}- function selectAll(rootEl, selector) {- // The host's engine, plus :contains(text) and a group of them.- const out = [];- for (const part of splitGroups(selector)) {- const m = /^(.*?)\s*:contains\((?:"([^"]*)"|'([^']*)'|([^)]*))\)\s*$/.exec(part);- if (m) {- const base = m[1].trim() || "*", needle = m[2] !== undefined ? m[2] : m[3] !== undefined ? m[3] : m[4];- for (const el of rootEl.querySelectorAll(base)) if (textOf(el).includes(needle) && !out.includes(el)) out.push(el);- } else {- for (const el of rootEl.querySelectorAll(part)) if (!out.includes(el)) out.push(el);+ const ATTR = /\s*([^\s"'>\/=]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'=<>`]+)))?/y;+ function parseHtml(text) {+ const root = new Element("document");+ const stack = [root];+ let pos = 0;+ const n = text.length;+ const addText = (t) => { if (t) stack[stack.length - 1].children.push(unescapeHtml(t)); };+ while (pos < n) {+ const lt = text.indexOf("<", pos);+ if (lt < 0) { addText(text.slice(pos)); break; }+ addText(text.slice(pos, lt));+ if (text.startsWith("<!--", lt)) { const e = text.indexOf("-->", lt + 4); pos = e < 0 ? n : e + 3; continue; }+ if (text[lt + 1] === "!" || text[lt + 1] === "?") { const e = text.indexOf(">", lt); pos = e < 0 ? n : e + 1; continue; }+ if (text[lt + 1] === "/") {+ const e = text.indexOf(">", lt);+ const tag = text.slice(lt + 2, e < 0 ? n : e).trim().toLowerCase();+ pos = e < 0 ? n : e + 1;+ for (let i = stack.length - 1; i > 0; i--) if (stack[i].tag === tag) { stack.length = i; break; }+ continue;+ }+ const m = /^<([a-zA-Z][^\s\/>]*)/.exec(text.slice(lt, lt + 200));+ if (!m) { addText("<"); pos = lt + 1; continue; }+ const tag = m[1].toLowerCase();+ let p = lt + m[0].length;+ const attrs = {};+ while (p < n) {+ const ws = /\s*/y; ws.lastIndex = p; ws.exec(text); p = ws.lastIndex;+ if (text[p] === ">" || text.startsWith("/>", p) || p >= n) break;+ ATTR.lastIndex = p;+ const am = ATTR.exec(text);+ if (!am) { p++; continue; }+ p = ATTR.lastIndex;+ const name = am[1].toLowerCase();+ if (name !== "/") attrs[name] = unescapeHtml(am[2] !== undefined ? am[2] : am[3] !== undefined ? am[3] : am[4] !== undefined ? am[4] : "");+ }+ const selfClosing = text.startsWith("/>", p);+ pos = p >= n ? n : p + (selfClosing ? 2 : 1);+ const el = new Element(tag, attrs, stack[stack.length - 1]);+ stack[stack.length - 1].children.push(el);+ if (selfClosing || VOID.has(tag)) continue;+ if (RAW_TEXT.has(tag)) {+ const close = text.toLowerCase().indexOf("</" + tag, pos);+ const raw = text.slice(pos, close < 0 ? n : close);+ if (raw) el.children.push(raw);+ if (close < 0) { pos = n; continue; }+ const e = text.indexOf(">", close); pos = e < 0 ? n : e + 1;+ continue;}+ stack.push(el);}+ return root;+ }+ class SelectorError extends Error {}+ const SEL_TOKEN = /([a-zA-Z][a-zA-Z0-9\-]*|\*)|#([\w\-]+)|\.([\w\-]+)|\[([\w\-:]+)(?:([\^$*]?=)(["']?)([^\]"']*)\6)?\]|:(first-child|last-child|nth-child\(\d+\)|contains\((["']?)([^)"']*)\9\))/y;+ function parseCompound(text) {+ const simple = { tag: null, id: null, classes: [], attrs: [], pseudos: [] };+ text = text.trim();+ if (!text) throw new SelectorError("empty selector part");+ let pos = 0;+ while (pos < text.length) {+ SEL_TOKEN.lastIndex = pos;+ const m = SEL_TOKEN.exec(text);+ if (!m) throw new SelectorError(`cannot parse selector near ${pyrepr(text.slice(pos))}`);+ pos = SEL_TOKEN.lastIndex;+ if (m[1]) { if (simple.tag) throw new SelectorError(`two tags in one compound selector: ${pyrepr(text)}`); simple.tag = m[1].toLowerCase(); }+ else if (m[2]) simple.id = m[2];+ else if (m[3]) simple.classes.push(m[3]);+ else if (m[4]) simple.attrs.push([m[4].toLowerCase(), m[5] || "", m[7] === undefined ? null : m[7]]);+ else if (m[8]) simple.pseudos.push(m[8].startsWith("contains(") ? "contains:" + m[10] : m[8]);+ }+ return simple;+ }+ function compileSelector(selector) {+ const groups = [];+ for (let alt of String(selector).split(",")) {+ alt = alt.trim();+ if (!alt) throw new SelectorError(`empty alternative in ${pyrepr(selector)}`);+ const chain = [];+ let comb = " ";+ for (const part of alt.split(/\s*(>)\s*|\s+/)) {+ if (part === undefined || part === "") continue;+ if (part === ">") { comb = ">"; continue; }+ chain.push([comb, parseCompound(part)]);+ comb = " ";+ }+ if (!chain.length) throw new SelectorError(`nothing to match in ${pyrepr(selector)}`);+ groups.push(chain);+ }+ return groups;+ }+ function simpleMatches(simple, el) {+ if (simple.tag && simple.tag !== "*" && el.tag !== simple.tag) return false;+ if (simple.id && el.attrs.id !== simple.id) return false;+ if (simple.classes.length) { const have = new Set(el.classes); if (!simple.classes.every((c) => have.has(c))) return false; }+ for (const [name, op, val] of simple.attrs) {+ if (!(name in el.attrs)) return false;+ const have = el.attrs[name];+ if (op === "=" && have !== val) return false;+ if (op === "^=" && !have.startsWith(val || "")) return false;+ if (op === "$=" && !have.endsWith(val || "")) return false;+ if (op === "*=" && !have.includes(val || "")) return false;+ }+ for (const p of simple.pseudos) {+ if (p.startsWith("contains:")) { if (!el.text.includes(p.slice(9))) return false; continue; }+ const siblings = el.parent ? el.parent.elements() : [el];+ if (p === "first-child") { if (siblings[0] !== el) return false; }+ else if (p === "last-child") { if (siblings[siblings.length - 1] !== el) return false; }+ else if (p.startsWith("nth-child(")) { const k = parseInt(p.slice(10, -1), 10); if (!(k > 0 && k <= siblings.length && siblings[k - 1] === el)) return false; }+ else throw new SelectorError(`unknown pseudo-class :${p}`);+ }+ return true;+ }+ function chainMatches(el, chain) {+ const [comb, simple] = chain[chain.length - 1];+ if (!simpleMatches(simple, el)) return false;+ if (chain.length === 1) return true;+ const rest = chain.slice(0, -1);+ if (comb === ">") return el.parent !== null && el.parent.tag !== "document" && chainMatches(el.parent, rest);+ for (let anc = el.parent; anc !== null && anc.tag !== "document"; anc = anc.parent) if (chainMatches(anc, rest)) return true;+ return false;+ }+ function select(rootEl, selector) {+ const compiled = compileSelector(selector);+ const out = [];+ const seen = new Set();+ for (const el of rootEl.descendants()) for (const chain of compiled) if (chainMatches(el, chain) && !seen.has(el)) { seen.add(el); out.push(el); break; }return out; }- function splitGroups(selector) {- const parts = []; let depth = 0, cur = "";- for (const ch of selector) { if (ch === "(") depth++; if (ch === ")") depth--; if (ch === "," && depth === 0) { parts.push(cur.trim()); cur = ""; } else cur += ch; }- if (cur.trim()) parts.push(cur.trim());- return parts;- }- const html = {- parse(text) {- if (typeof DOMParser === "undefined") return null;- const doc = new DOMParser().parseFromString(text, "text/html");- return doc.documentElement;- },- };+ const selectAll = select;+ const textOf = (el) => el.text;+ const html = { parse: parseHtml, select, compileSelector, Element, SelectorError };// ------------------------------------------------------------ extract function extract(entry, source, cache) {@@ -274,7 +402,7 @@const text = textSource(entry, source); if (isMissing(text)) return MISSING; let doc = cache && cache.get(text);- if (!doc) { doc = html.parse(text); if (!doc) return MISSING; if (cache) cache.set(text, doc); }+ if (!doc) { doc = html.parse(text); if (cache) cache.set(text, doc); }const found = selectAll(doc, entry.css); if (entry.all) return found.map((el) => elementValue(entry, el)); if (!found.length) return MISSING;@@ -304,6 +432,90 @@return textOf(el); }+ // ------------------------------------------------------------ sources and their allow-lists+ // The other forms of I/O (file, shell, serial, queue, browser) live on a+ // service; a member refuses them unless a recorded fixture answers for+ // them. The allow-list checks and the fixture keys are the Python ones.+ const SOURCES = ["http", "browser", "file", "shell", "serial", "queue"];+ const LIST_NAMES = ["paths", "commands", "devices", "topics"];+ const MQTT_PORTS = { mqtt: 1883, mqtts: 8883 };+ function absolutePath(path) {+ path = String(path);+ const parts = path.split("/"), out = [];+ for (const part of parts) { if (part === "" || part === ".") continue; if (part === "..") out.pop(); else out.push(part); }+ return (path.startsWith("/") ? "/" : "") + out.join("/");+ }+ function globMatch(pattern, text) {+ let re = "";+ for (const ch of pattern) re += ch === "*" ? ".*" : ch === "?" ? "." : ch.replace(/[.+^${}()|[\]\\]/g, "\\$&");+ return new RegExp("^" + re + "$").test(text);+ }+ function pathAllowed(patterns, path) {+ const target = absolutePath(path);+ for (let pat of patterns) {+ if (pat === "*") return true;+ pat = absolutePath(pat);+ if (/[*?[]/.test(pat)) { if (globMatch(pat, target)) return true; }+ else if (target === pat || target.startsWith(pat.replace(/\/+$/, "") + "/")) return true;+ }+ return false;+ }+ function commandAllowed(commands, command) {+ command = String(command);+ for (const entry of commands) {+ if (entry === "*") return true;+ if (entry.includes("/")) { if (absolutePath(command) === absolutePath(entry)) return true; }+ else if (command === entry || command.split("/").pop() === entry) return true;+ }+ return false;+ }+ const deviceAllowed = (devices, device) => devices.some((d) => d === "*" || globMatch(d, String(device)));+ function topicAllowed(topics, topic) {+ const parts = String(topic).split("/");+ for (const entry of topics) {+ if (entry === "*" || entry === "#") return true;+ const pat = entry.split("/");+ let ok = true, i = 0;+ for (; i < pat.length; i++) {+ const seg = pat[i];+ if (seg === "#") { ok = ok && i <= parts.length; break; }+ if (i >= parts.length || (seg !== "+" && seg !== "*" && seg !== parts[i])) { ok = false; break; }+ }+ if (i === pat.length) ok = ok && pat.length === parts.length;+ if (ok) return true;+ }+ return false;+ }+ function broker(url) {+ const m = /^(mqtts?):\/\/([^/:?#]+)(?::(\d+))?/.exec(String(url));+ if (!m) throw new Error(`${pyrepr(String(url))} is not mqtt://host[:port] or mqtts://host[:port]`);+ return [m[1], m[2], m[3] ? Number(m[3]) : MQTT_PORTS[m[1]]];+ }+ const arg = (a) => (typeof a === "string" ? a : pyJson(a));+ function adapterKey(source, spec) {+ if (source === "file") return `file ${"write" in spec ? "write " : "append" in spec ? "append " : ""}${spec.path}`;+ if (source === "shell") return "shell " + [String(spec.command)].concat((spec.args || []).map(arg)).join(" ");+ if (source === "serial") { const w = spec.write; return `serial ${spec.device}` + (w === undefined || w === null || w === "" ? "" : ` ${JSON.stringify(String(w))}`); }+ if (source === "queue") return `queue ${"publish" in spec ? "publish " : ""}${spec.url} ${spec.topic}`;+ if (source === "browser") return `browser ${spec.url}`;+ throw new Error(source);+ }+ function what(source, spec) {+ if (source === "file") return String(spec.path);+ if (source === "shell") return [String(spec.command)].concat((spec.args || []).map(arg)).join(" ");+ if (source === "serial") return String(spec.device);+ if (source === "queue") return `${spec.topic} at ${spec.url}`;+ return String(spec.url);+ }+ function briefResult(source, observed) {+ if (source === "http" || source === "browser") return pyStr(observed.status === undefined ? null : observed.status);+ if (source === "file") return "written" in observed ? `wrote ${observed.written} B` : observed.exists ? `${observed.size} B` : "missing";+ if (source === "shell") return `exit ${observed.status}`;+ if (source === "serial") return `${observed.bytes} B`;+ if (source === "queue") return "published" in observed ? `published ${observed.published} B` : observed.received ? "1 message" : "no message";+ return "ok";+ }+// ------------------------------------------------------------ trees const INPUT_TYPES = ["str", "int", "float", "bool", "list", "dict"]; function inferType(v) { return isBool(v) ? "bool" : isNum(v) ? (Number.isInteger(v) ? "int" : "float") : Array.isArray(v) ? "list" : isObj(v) ? "dict" : "str"; }@@ -329,6 +541,7 @@this.raw = raw; this.name = raw.name; this.version = raw.version || 1; this.description = raw.description || ""; this.hosts = raw.hosts || []; this.start = raw.start; this.tags = raw.tags || []; this.license = raw.license || "";+ for (const k of LIST_NAMES) this[k] = raw[k] || [];this.language = raw.language || {}; this.triggers = (raw.triggers || []).map((t) => (typeof t === "string" ? { pattern: t } : t)); this.inputs = {};@@ -352,6 +565,25 @@} return false; }+ allowed(source, spec) {+ if (source === "http" || source === "browser") { const url = String(spec.url === undefined ? "" : spec.url); return this.hostAllowed(url) ? null : ["host-not-allowed", `${url} is outside this tree's hosts ${pyrepr(this.hosts)}.`]; }+ if (source === "file") { const path = absolutePath(spec.path === undefined ? "" : spec.path); return pathAllowed(this.paths, path) ? null : ["path-not-allowed", `${path} is outside this tree's paths ${pyrepr(this.paths)}.`]; }+ if (source === "shell") { const cmd = String(spec.command === undefined ? "" : spec.command); return commandAllowed(this.commands, cmd) ? null : ["command-not-allowed", `${pyrepr(cmd)} is not in this tree's commands ${pyrepr(this.commands)}.`]; }+ if (source === "serial") { const dev = String(spec.device === undefined ? "" : spec.device); return deviceAllowed(this.devices, dev) ? null : ["device-not-allowed", `${dev} is outside this tree's devices ${pyrepr(this.devices)}.`]; }+ if (source === "queue") {+ let host, port;+ try { [, host, port] = broker(spec.url === undefined ? "" : spec.url); } catch (e) { return ["host-not-allowed", e.message]; }+ if (!this.hostAllowed(`mqtt://${host}:${port}/`)) return ["host-not-allowed", `${host} is outside this tree's hosts ${pyrepr(this.hosts)}.`];+ const topic = String(spec.topic === undefined ? "" : spec.topic);+ return topicAllowed(this.topics, topic) ? null : ["topic-not-allowed", `${pyrepr(topic)} is outside this tree's topics ${pyrepr(this.topics)}.`];+ }+ return null;+ }+ sources() {+ const out = new Set();+ for (const n of Object.values(this.nodes)) if (n.kind === "observe" || n.kind === "act") for (const s of SOURCES) if (n.spec[s] !== undefined && n.spec[s] !== null) out.add(s);+ return [...out].sort();+ }bindInputs(given) { given = given || {}; const unknown = Object.keys(given).filter((k) => !(k in this.inputs)).sort();@@ -375,7 +607,9 @@if (s.description) d.description = s.description; inputs[n] = d; }- return { name: this.name, version: this.version, description: this.description, tags: this.tags, hosts: this.hosts, license: this.license, language: this.language, triggers: this.triggers.map((t) => t.pattern), inputs };+ const cap = { name: this.name, version: this.version, description: this.description, tags: this.tags, hosts: this.hosts };+ for (const k of LIST_NAMES) if (this[k].length) cap[k] = this[k];+ return Object.assign(cap, { sources: this.sources(), license: this.license, language: this.language, triggers: this.triggers.map((t) => t.pattern), inputs });} }@@ -415,6 +649,15 @@const headers = {}; for (const [k, v] of Object.entries(rec.headers || {})) headers[k.toLowerCase()] = v; return new Response(rec.status === undefined ? 200 : Number(rec.status), headers, text, url, 0); }+ async perform(source, spec) {+ const key = adapterKey(source, spec);+ this.calls.push([source, key]);+ let rec = this.responses[key];+ if (rec === undefined) throw new TransportError(`no recorded result for ${key}`);+ if (Array.isArray(rec)) rec = rec.length > 1 ? rec.shift() : rec[0];+ if ("error" in rec) throw new TransportError(String(rec.error));+ return JSON.parse(JSON.stringify(rec));+ }} // ------------------------------------------------------------ runtime@@ -462,6 +705,8 @@this.trees = {}; for (const t of opts.trees || []) this.add(t); this.transport = opts.transport || new FetchTransport();+ // The other sources: a fixture answers for them; a live member has none and says so.+ this.adapters = opts.adapters || (typeof this.transport.perform === "function" ? this.transport : null);this.resolver = opts.resolver || null; // {runTree(name, inputs, depth), runIntent(text, depth)} -> outcome dicts this.env = opts.env || {}; this.clock = opts.clock || (() => new Date());@@ -535,27 +780,37 @@return this.observeOrAct(tree, node, scope, trace, false); } async observeOrAct(tree, node, scope, trace, retry) {- const state = scope.state, into = node.get("into", "last"), http = node.get("http");+ const state = scope.state, into = node.get("into", "last");+ const sourceName = SOURCES.find((s) => node.spec[s] !== undefined && node.spec[s] !== null);let source, summary;- if (http !== undefined && http !== null) {- const req = render(http, scope);- const method = String(req.method || "GET").toUpperCase(), url = String(req.url);- if (!tree.hostAllowed(url)) return outcome("error", tree.name, { reason: "host-not-allowed", say: `${node.id}: ${url} is outside this tree's hosts ${pyrepr(tree.hosts)}.` });- const headers = {}; for (const [k, v] of Object.entries(req.headers || {})) headers[String(k)] = String(v);- let body = req.body;- if (body !== undefined && body !== null && typeof body !== "string") { body = JSON.stringify(body); if (!("Content-Type" in headers)) headers["Content-Type"] = "application/json"; }+ if (sourceName !== undefined) {+ const spec = render(node.get(sourceName), scope);+ const problem = tree.allowed(sourceName, spec);+ if (problem) return outcome("error", tree.name, { reason: problem[0], say: `${node.id}: ${problem[1]}` });const timeout = Number(node.get("timeout", DEFAULT_TIMEOUT));+ let label, call;+ if (sourceName === "http") {+ const method = String(spec.method || "GET").toUpperCase(), url = String(spec.url);+ const headers = {}; for (const [k, v] of Object.entries(spec.headers || {})) headers[String(k)] = String(v);+ let body = spec.body;+ if (body !== undefined && body !== null && typeof body !== "string") { body = JSON.stringify(body); if (!("Content-Type" in headers)) headers["Content-Type"] = "application/json"; }+ label = `${method} ${url}`;+ call = async () => (await this.transport.request(method, url, headers, body === undefined ? null : body, timeout)).toState();+ } else {+ label = `${sourceName} ${what(sourceName, spec)}`;+ if (!this.adapters) return outcome("error", tree.name, { reason: "source-unsupported", say: `${node.id}: a ${sourceName} node runs on a service, not on a member; this device has no ${sourceName === "browser" ? "browser of its own to drive" : sourceName === "queue" ? "socket to a broker" : "access to the box's " + (sourceName === "file" ? "files" : sourceName === "shell" ? "programs" : "devices")}.` });+ call = () => this.adapters.perform(sourceName, spec, timeout, (u) => tree.hostAllowed(u));+ }const attempts = retry ? 1 + Number(node.get("retries", DEFAULT_RETRIES)) : 1;- let lastError = null, resp = null;+ let lastError = null, observed = null;for (let attempt = 0; attempt < attempts; attempt++) {- try { resp = await this.transport.request(method, url, headers, body === undefined ? null : body, timeout); break; }+ try { observed = await call(); break; }catch (e) { if (!(e instanceof TransportError)) throw e; lastError = e.message; if (attempt + 1 < attempts) await sleep(Math.min(200 * (attempt + 1), 1000)); } }- if (!resp) { trace.add(step(node.id, node.kind, `${method} ${url} failed: ${lastError}`)); return outcome("error", tree.name, { reason: "transport", say: `${node.id}: ${lastError}` }); }- const observed = resp.toState();+ if (observed === null) { trace.add(step(node.id, node.kind, `${label} failed: ${lastError}`)); return outcome("error", tree.name, { reason: "transport", say: `${node.id}: ${lastError}` }); }put(state, into, observed); source = observed;- summary = `${method} ${url} -> ${resp.status}`;+ summary = `${label} -> ${briefResult(sourceName, observed)}`;} else { const frm = node.get("from"); const sourceVal = resolve(statePath(frm), scope);@@ -650,5 +905,5 @@return outcome(d.status || "error", d.tree || fallbackTree, { say: d.say === undefined ? null : d.say, output: d.output === undefined ? null : d.output, reason: d.reason === undefined ? null : d.reason, question: d.question === undefined ? null : d.question, snapshot: d.snapshot === undefined ? null : d.snapshot, trace, inputs: d.inputs || {} }); }- return { MISSING, Tree, Runner, Trace, FetchTransport, RecordedTransport, Response, render, resolve, holds, extract, outcomeDict, fromDict, pyrepr, pyJson, html, TemplateError, TreeError, TransportError, ConditionError, version: "0.6.0" };+ return { MISSING, Tree, Runner, Trace, FetchTransport, RecordedTransport, Response, render, resolve, holds, extract, outcomeDict, fromDict, pyrepr, pyJson, html, TemplateError, TreeError, TransportError, ConditionError, SOURCES, adapterKey, version: "0.7.0" };}); diff --git a/src/pyrtan/runtime.py b/src/pyrtan/runtime.py index 6c50ba7..f8d21aa 100644 --- a/src/pyrtan/runtime.py +++ b/src/pyrtan/runtime.py@@ -32,6 +32,7 @@ from dataclasses import dataclass, fieldfrom typing import Any, Callable, Protocol from . import conditions+from .adapters import SOURCES, LocalAdapters, brief_result, whatfrom .extract import extract from .grow import no_match_outcome from .registry import Ambiguous, NoMatch, Registry@@ -102,6 +103,11 @@ class Runner:max_steps: int = MAX_STEPS max_depth: int = MAX_DEPTH emitted: list[dict[str, Any]] = field(default_factory=list)+ adapters: Any = None # the other sources (file, shell, serial, queue, browser); a recorded transport answers for them too++ def __post_init__(self) -> None:+ if self.adapters is None:+ self.adapters = self.transport if hasattr(self.transport, "perform") else LocalAdapters()# ------------------------------------------------------------ entry points@@ -225,27 +231,34 @@ class Runner:def _observe_or_act(self, tree: Tree, node: Node, scope: dict[str, Any], trace: Trace, *, retry: bool): state = scope["state"] into = node.get("into", "last")- http = node.get("http")- if http is not None:- req = render(http, scope)- method = str(req.get("method", "GET")).upper()- url = str(req["url"])- if not tree.host_allowed(url):- return Outcome("error", tree.name, reason="host-not-allowed", say=f"{node.id}: {url} is outside this tree's hosts {tree.hosts}.")- headers = {str(k): str(v) for k, v in (req.get("headers") or {}).items()}- body = req.get("body")- if body is not None and not isinstance(body, str):- import json as _json-- body = _json.dumps(body)- headers.setdefault("Content-Type", "application/json")+ source_name = next((s for s in SOURCES if node.get(s) is not None), None)+ if source_name is not None:+ spec = render(node.get(source_name), scope)+ problem = tree.allowed(source_name, spec)+ if problem is not None:+ return Outcome("error", tree.name, reason=problem[0], say=f"{node.id}: {problem[1]}")timeout = float(node.get("timeout", DEFAULT_TIMEOUT))+ if source_name == "http":+ method = str(spec.get("method", "GET")).upper()+ url = str(spec["url"])+ headers = {str(k): str(v) for k, v in (spec.get("headers") or {}).items()}+ body = spec.get("body")+ if body is not None and not isinstance(body, str):+ import json as _json++ body = _json.dumps(body)+ headers.setdefault("Content-Type", "application/json")+ label = f"{method} {url}"+ call = lambda: self.transport.request(method, url, headers, body, timeout).to_state() # noqa: E731+ else:+ label = f"{source_name} {what(source_name, spec)}"+ call = lambda: self.adapters.perform(source_name, spec, timeout, tree.host_allowed) # noqa: E731attempts = 1 + int(node.get("retries", DEFAULT_RETRIES)) if retry else 1 last_error: str | None = None- resp = None+ observed = Nonefor attempt in range(attempts): try:- resp = self.transport.request(method, url, headers, body, timeout)+ observed = call()break except TransportError as e: last_error = str(e)@@ -253,13 +266,12 @@ class Runner:break # a policy refusal does not change on retry if attempt + 1 < attempts: time.sleep(min(0.2 * (attempt + 1), 1.0))- if resp is None:- trace.add(Step(node.id, node.kind, f"{method} {url} failed: {last_error}"))+ if observed is None:+ trace.add(Step(node.id, node.kind, f"{label} failed: {last_error}"))return Outcome("error", tree.name, reason="transport", say=f"{node.id}: {last_error}")- observed = resp.to_state()_put(state, into, observed) source = observed- summary = f"{method} {url} -> {resp.status}"+ summary = f"{label} -> {brief_result(source_name, observed)}"else: frm = node.get("from") source_val = resolve(_state_path(frm), scope) diff --git a/src/pyrtan/service.py b/src/pyrtan/service.py index ac71024..6bbd55f 100644 --- a/src/pyrtan/service.py +++ b/src/pyrtan/service.py@@ -58,6 +58,7 @@ from .grow import Unansweredfrom .registry import Registry from .runtime import Outcome, Runner from .trace import Trace+from .adapters import LocalAdaptersfrom .transport import Transport, UrllibTransport MAX_BODY = 1 << 20@@ -153,7 +154,8 @@ class Service:self.token = token or None self.egress = egress self.started = time.time()- self.runner = Runner(registry, transport or UrllibTransport(egress=egress), resolver=RemoteResolver(self.hub, self.token) if self.hub else None, env=env or {})+ adapters = transport if transport is not None and hasattr(transport, "perform") else LocalAdapters(egress=egress)+ self.runner = Runner(registry, transport or UrllibTransport(egress=egress), resolver=RemoteResolver(self.hub, self.token) if self.hub else None, env=env or {}, adapters=adapters)self.gaps = gaps if isinstance(gaps, GapStore) else GapStore(gaps) self.unanswered = Unanswered() self.runs = 0 diff --git a/src/pyrtan/transport.py b/src/pyrtan/transport.py index dc36905..c86c1e7 100644 --- a/src/pyrtan/transport.py +++ b/src/pyrtan/transport.py@@ -129,7 +129,7 @@ class UrllibTransport:``public`` (refuse private, loopback, link-local and metadata addresses, on the first URL and on every redirect)."""- def __init__(self, user_agent: str = "pyrtan/0.6 (+https://github.com/b-i-t-8/pyrtan)", egress: str = "any") -> None:+ def __init__(self, user_agent: str = "pyrtan/0.7 (+https://github.com/b-i-t-8/pyrtan)", egress: str = "any") -> None:if egress not in EGRESS_POLICIES: raise ValueError(f"egress policy must be one of {EGRESS_POLICIES}, got {egress!r}") self.user_agent = user_agent@@ -189,18 +189,31 @@ class RecordedTransport:def request(self, method: str, url: str, headers: dict[str, str] | None = None, body: str | None = None, timeout: float = 10.0) -> Response: self.calls.append((method.upper(), url))- rec = self.responses.get(_key(method, url))+ rec = self._take(_key(method, url), "response")+ text = rec.get("body", "")+ if not isinstance(text, str):+ text = json.dumps(text)+ return Response(int(rec.get("status", 200)), {k.lower(): v for k, v in rec.get("headers", {}).items()}, text, url, 0)++ def perform(self, source: str, spec: dict[str, Any], timeout: float = 10.0, allowed: Any = None) -> dict[str, Any]:+ """The other sources, from the same fixture: a record under+ :func:`pyrtan.adapters.adapter_key` is what the node observes."""+ from .adapters import adapter_key++ key = adapter_key(source, spec)+ self.calls.append((source, key))+ return json.loads(json.dumps(self._take(key, "result")))++ def _take(self, key: str, noun: str) -> dict[str, Any]:+ rec = self.responses.get(key)if rec is None:- raise TransportError(f"no recorded response for {_key(method, url)}")+ raise TransportError(f"no recorded {noun} for {key}")if isinstance(rec, list): # A sequence: consume in order, repeating the last. rec = rec.pop(0) if len(rec) > 1 else rec[0] if "error" in rec: raise TransportError(str(rec["error"]))- text = rec.get("body", "")- if not isinstance(text, str):- text = json.dumps(text)- return Response(int(rec.get("status", 200)), {k.lower(): v for k, v in rec.get("headers", {}).items()}, text, url, 0)+ return rec@dataclass@@ -209,6 +222,7 @@ class RecordingTransport:fixture for :class:`RecordedTransport`.""" inner: Transport+ adapters: Any = Nonerecorded: dict[str, Any] = field(default_factory=dict) def request(self, method: str, url: str, headers: dict[str, str] | None = None, body: str | None = None, timeout: float = 10.0) -> Response:@@ -216,5 +230,14 @@ class RecordingTransport:self.recorded[_key(method, url)] = {"status": resp.status, "headers": resp.headers, "body": resp.body} return resp+ def perform(self, source: str, spec: dict[str, Any], timeout: float = 10.0, allowed: Any = None) -> dict[str, Any]:+ from .adapters import LocalAdapters, adapter_key++ if self.adapters is None:+ self.adapters = LocalAdapters()+ observed = self.adapters.perform(source, spec, timeout, allowed)+ self.recorded[adapter_key(source, spec)] = json.loads(json.dumps(observed))+ return observed+def save(self, path: str | Path) -> None: Path(path).write_text(json.dumps(self.recorded, indent=2, ensure_ascii=False) + "\n", encoding="utf-8") diff --git a/src/pyrtan/tree.py b/src/pyrtan/tree.py index a2afd1b..8e78fbe 100644 --- a/src/pyrtan/tree.py +++ b/src/pyrtan/tree.py@@ -42,7 +42,7 @@ from dataclasses import dataclass, fieldfrom pathlib import Path from typing import Any-from . import conditions, extract as extractmod+from . import adapters, conditions, extract as extractmodfrom .intent import KINDS as SLOT_KINDS, PRODUCES, Language, Trigger, TriggerError from .template import PLACEHOLDER, placeholders@@ -163,6 +163,10 @@ class Tree:raw: dict[str, Any] = field(default_factory=dict, repr=False) license: str = "" language: Language = field(default_factory=Language)+ paths: list[str] = field(default_factory=list) # what ``file`` nodes may reach+ commands: list[str] = field(default_factory=list) # what ``shell`` nodes may run+ devices: list[str] = field(default_factory=list) # what ``serial`` nodes may open+ topics: list[str] = field(default_factory=list) # what ``queue`` nodes may subscribe to or publish ondef node(self, node_id: str) -> Node: try:@@ -186,6 +190,37 @@ class Tree:return True return False+ def allowed(self, source: str, spec: dict[str, Any]) -> tuple[str, str] | None:+ """Why a rendered request to ``source`` is outside this tree's+ allow-list, as ``(reason, text)``, or None when it is inside."""+ if source in ("http", "browser"):+ url = str(spec.get("url", ""))+ return None if self.host_allowed(url) else ("host-not-allowed", f"{url} is outside this tree's hosts {self.hosts}.")+ if source == "file":+ path = adapters.absolute_path(spec.get("path", ""))+ return None if adapters.path_allowed(self.paths, path) else ("path-not-allowed", f"{path} is outside this tree's paths {self.paths}.")+ if source == "shell":+ cmd = str(spec.get("command", ""))+ return None if adapters.command_allowed(self.commands, cmd) else ("command-not-allowed", f"{cmd!r} is not in this tree's commands {self.commands}.")+ if source == "serial":+ dev = str(spec.get("device", ""))+ return None if adapters.device_allowed(self.devices, dev) else ("device-not-allowed", f"{dev} is outside this tree's devices {self.devices}.")+ if source == "queue":+ url = str(spec.get("url", ""))+ try:+ _, host, port = adapters.broker(url)+ except ValueError as e:+ return ("host-not-allowed", str(e))+ if not self.host_allowed(f"mqtt://{host}:{port}/"):+ return ("host-not-allowed", f"{host} is outside this tree's hosts {self.hosts}.")+ topic = str(spec.get("topic", ""))+ return None if adapters.topic_allowed(self.topics, topic) else ("topic-not-allowed", f"{topic!r} is outside this tree's topics {self.topics}.")+ return None++ def sources(self) -> list[str]:+ """The forms of I/O this tree's nodes use."""+ return sorted({s for n in self.nodes.values() if n.kind in ("observe", "act") for s in adapters.SOURCES if n.get(s) is not None})+def bind_inputs(self, given: dict[str, Any]) -> dict[str, Any]: """Apply defaults, coerce types, and refuse unknown or missing inputs.""" unknown = set(given) - set(self.inputs)@@ -211,6 +246,8 @@ class Tree:"description": self.description, "tags": self.tags, "hosts": self.hosts,+ **{k: getattr(self, k) for k in ("paths", "commands", "devices", "topics") if getattr(self, k)},+ "sources": self.sources(),"license": self.license, "language": self.language.to_dict(), "triggers": [t.pattern for t in self.triggers],@@ -251,6 +288,12 @@ def build(raw: dict[str, Any], source: str | None = None) -> Tree:hosts = raw.get("hosts", []) if not isinstance(hosts, list) or not all(isinstance(h, str) and h for h in hosts): raise TreeError(f"{name}: 'hosts' must be a list of host names")+ lists: dict[str, list[str]] = {}+ for key, noun in (("paths", "paths or globs"), ("commands", "program names"), ("devices", "device paths or globs"), ("topics", "topic filters")):+ value = raw.get(key, [])+ if not isinstance(value, list) or not all(isinstance(v, str) and v for v in value):+ raise TreeError(f"{name}: '{key}' must be a list of {noun}")+ lists[key] = valuetags = raw.get("tags", []) if not isinstance(tags, list) or not all(isinstance(t, str) for t in tags): raise TreeError(f"{name}: 'tags' must be a list of strings")@@ -317,7 +360,7 @@ def build(raw: dict[str, Any], source: str | None = None) -> Tree:hint = " (top-level keys such as 'start' must come before the first [table] in TOML)" if _buried(raw, "start") else "" raise TreeError(f"{name}: 'start' must name a node{hint}")- tree = Tree(name, description, version, hosts, triggers, inputs, start, nodes, tags, source, raw, license_.strip(), language)+ tree = Tree(name, description, version, hosts, triggers, inputs, start, nodes, tags, source, raw, license_.strip(), language, **lists)errors = [p for p in check(tree) if p.level == "error"] if errors: raise TreeError(f"{name}: " + "; ".join(f"{p.where}: {p.message}" for p in errors))@@ -397,7 +440,7 @@ def check(tree: Tree) -> list[Problem]:if node.kind == "set" and isinstance(node.get("set"), dict): produced.update(k.split(".")[0] for k in node.get("set"))- uses_http = False+ used: set[str] = set()for node in tree.nodes.values(): where = f"nodes.{node.id}" for target in node.targets():@@ -409,15 +452,15 @@ def check(tree: Tree) -> list[Problem]:err(where, f"a {node.kind} node needs 'next'") if node.kind in ("observe", "act"):- http = node.get("http")+ present = [s for s in adapters.SOURCES if node.get(s) is not None]frm = node.get("from") emit = node.get("emit")- ways = sum(1 for x in (http, frm, emit) if x is not None)+ ways = len(present) + sum(1 for x in (frm, emit) if x is not None)if ways != 1:- err(where, f"an {node.kind} node needs exactly one of 'http', 'from' or 'emit'")- if http is not None:- uses_http = True- _check_http(tree, node, http, err, warn)+ err(where, f"an {node.kind} node needs exactly one of {', '.join(repr(s) for s in adapters.SOURCES)}, 'from' or 'emit'")+ for s in present:+ used.add(s)+ _SOURCE_CHECKS[s](tree, node, node.get(s), err, warn)if emit is not None: if node.kind != "act": err(where, "'emit' is an act, not an observation")@@ -517,16 +560,135 @@ def check(tree: Tree) -> list[Problem]:elif head not in ("item", "index", "tree", "now", "env"): err(where, f"template root {head!r} is not one of inputs, state, item, index, tree, now, env")- if uses_http and not tree.hosts:- err("hosts", "a tree that speaks HTTP must list the hosts it may reach")- if tree.hosts and not uses_http:- info("hosts", "hosts are listed but no node speaks HTTP")+ for list_name, sources in _NEEDS.items():+ have = getattr(tree, list_name)+ using = sorted(used & sources)+ if using and not have:+ if using == ["http"]:+ err("hosts", "a tree that speaks HTTP must list the hosts it may reach")+ else:+ err(list_name, f"a tree with {', '.join(using)} nodes must list the {list_name} it may reach")+ if have and not using:+ info(list_name, f"{list_name} are listed but no node uses them")if tree.start in tree.nodes: problems.extend(_check_graph(tree)) return problems+_NEEDS = {"hosts": {"http", "browser", "queue"}, "paths": {"file"}, "commands": {"shell"}, "devices": {"serial"}, "topics": {"queue"}}+++def _literal(value: Any) -> str | None:+ """The value when it is a string with no placeholder, else None."""+ return value if isinstance(value, str) and not PLACEHOLDER.search(value) else None+++def _check_browser(tree: Tree, node: Node, spec: Any, err, warn) -> None:+ where = f"nodes.{node.id}.browser"+ if not isinstance(spec, dict) or not isinstance(spec.get("url"), str) or not spec["url"]:+ err(where, "'browser' is a table with 'url' and optionally 'wait_for', 'wait_until', 'actions'")+ return+ head = PLACEHOLDER.split(spec["url"], 1)[0]+ parsed = urllib.parse.urlsplit(head)+ if parsed.scheme and parsed.hostname:+ if parsed.scheme not in ("http", "https"):+ err(where, f"scheme {parsed.scheme!r} is not http or https")+ if not tree.host_allowed(head):+ err(where, f"host {parsed.hostname!r} is not in this tree's hosts")+ else:+ warn(where, "the host is decided at run time; the runtime refuses hosts outside the list")+ actions = spec.get("actions", [])+ if not isinstance(actions, list):+ err(where, "'actions' is a list of tables")+ return+ for i, a in enumerate(actions):+ keys = set(a) & set(adapters.BROWSER_ACTIONS) if isinstance(a, dict) else set()+ if len(keys) != 1:+ err(f"{where}.actions[{i}]", f"an action is a table with exactly one of {', '.join(adapters.BROWSER_ACTIONS)}")+ elif keys & {"fill", "press", "select"}:+ k = keys.pop()+ need = "value" if k != "press" else "key"+ if not isinstance(a[k], dict) or "selector" not in a[k] or need not in a[k]:+ err(f"{where}.actions[{i}]", f"'{k}' is a table with 'selector' and '{need}'")+++def _check_file(tree: Tree, node: Node, spec: Any, err, warn) -> None:+ where = f"nodes.{node.id}.file"+ if not isinstance(spec, dict) or not isinstance(spec.get("path"), str) or not spec["path"]:+ err(where, "'file' is a table with 'path' and optionally 'tail' (to read), 'write' or 'append' (to change)")+ return+ writes = "write" in spec or "append" in spec+ if writes and node.kind == "observe":+ err(where, "a write changes things; make this an act node")+ if not writes and node.kind == "act":+ warn(where, "an act that only reads; an observe node would retry it safely")+ if "write" in spec and "append" in spec:+ err(where, "'write' or 'append', not both")+ if spec.get("tail") is not None and not isinstance(spec["tail"], int) and not (isinstance(spec["tail"], str) and PLACEHOLDER.search(spec["tail"])):+ err(where, "'tail' is a number of lines")+ path = _literal(spec["path"])+ if path is None:+ warn(where, "the path is decided at run time; the runtime refuses paths outside the list")+ elif tree.paths and not adapters.path_allowed(tree.paths, path):+ err(where, f"{path!r} is not under this tree's paths")+++def _check_shell(tree: Tree, node: Node, spec: Any, err, warn) -> None:+ where = f"nodes.{node.id}.shell"+ if not isinstance(spec, dict) or not isinstance(spec.get("command"), str) or not spec["command"]:+ err(where, "'shell' is a table with 'command' (a program name) and optionally 'args' (a list), 'stdin', 'env', 'cwd'")+ return+ if spec.get("args") is not None and not isinstance(spec["args"], list):+ err(where, "'args' is a list; no shell reads the line, so there is nothing to quote")+ cmd = _literal(spec["command"])+ if cmd is None:+ warn(where, "the command is decided at run time; the runtime refuses commands outside the list")+ elif tree.commands and not adapters.command_allowed(tree.commands, cmd):+ err(where, f"{cmd!r} is not in this tree's commands")+++def _check_serial(tree: Tree, node: Node, spec: Any, err, warn) -> None:+ where = f"nodes.{node.id}.serial"+ if not isinstance(spec, dict) or not isinstance(spec.get("device"), str) or not spec["device"]:+ err(where, "'serial' is a table with 'device' and optionally 'baud', 'write', 'read_until' or 'read'")+ return+ if spec.get("baud") is not None and not isinstance(spec["baud"], int):+ err(where, "'baud' is a number")+ dev = _literal(spec["device"])+ if dev is None:+ warn(where, "the device is decided at run time; the runtime refuses devices outside the list")+ elif tree.devices and not adapters.device_allowed(tree.devices, dev):+ err(where, f"{dev!r} is not in this tree's devices")+++def _check_queue(tree: Tree, node: Node, spec: Any, err, warn) -> None:+ where = f"nodes.{node.id}.queue"+ if not isinstance(spec, dict) or not isinstance(spec.get("url"), str) or not isinstance(spec.get("topic"), str):+ err(where, "'queue' is a table with 'url' (mqtt://host[:port]), 'topic' and, to send, 'publish'")+ return+ if "publish" in spec and node.kind == "observe":+ err(where, "a publish changes things; make this an act node")+ if "publish" not in spec and node.kind == "act":+ warn(where, "an act that only waits for a message; an observe node would retry it safely")+ url = _literal(spec["url"])+ if url is None:+ warn(where, "the broker is decided at run time; the runtime refuses hosts outside the list")+ else:+ try:+ _, host, port = adapters.broker(url)+ except ValueError as e:+ err(where, str(e))+ else:+ if tree.hosts and not tree.host_allowed(f"mqtt://{host}:{port}/"):+ err(where, f"host {host!r} is not in this tree's hosts")+ topic = _literal(spec["topic"])+ if topic is None:+ warn(where, "the topic is decided at run time; the runtime refuses topics outside the list")+ elif tree.topics and not adapters.topic_allowed(tree.topics, topic):+ err(where, f"{topic!r} is not under this tree's topics")++def _check_http(tree: Tree, node: Node, http: Any, err, warn) -> None: where = f"nodes.{node.id}.http" if not isinstance(http, dict):@@ -557,6 +719,9 @@ def _check_http(tree: Tree, node: Node, http: Any, err, warn) -> None:err(where, "'headers' is a table")+_SOURCE_CHECKS = {"http": _check_http, "browser": _check_browser, "file": _check_file, "shell": _check_shell, "serial": _check_serial, "queue": _check_queue}++def _check_graph(tree: Tree) -> list[Problem]: problems: list[Problem] = [] # Reachability from start. diff --git a/tests/conformance/make_vectors.py b/tests/conformance/make_vectors.py index db87505..cb360fe 100644 --- a/tests/conformance/make_vectors.py +++ b/tests/conformance/make_vectors.py@@ -43,6 +43,26 @@ CASES: list[dict] = [{"intent": "weather in Paris"}, {"tree": "hacker-news-item", "inputs": {"id": 101}}, {"tree": "hacker-news", "inputs": {"count": 2}},+ # The other sources, answered by the same fixture.+ {"intent": "how full is the disk"},+ {"intent": "disk space on /data"},+ {"intent": "disk space on /nowhere"},+ {"intent": "errors in the log"},+ {"intent": "errors in /var/log/quiet.log"},+ {"intent": "errors in /var/log/nothing.log"},+ {"intent": "errors in /etc/passwd"},+ {"intent": "temperature in the kitchen"},+ {"intent": "how warm is the attic"},+ {"intent": "temperature in the garage"},+ {"intent": "temperature in the hall"},+ {"tree": "room-temperature", "inputs": {"room": "a-b"}},+ {"intent": "headline of https://app.example/dashboard"},+ {"intent": "what does https://app.example/empty say"},+ {"intent": "headline of https://app.example/gone"},+ {"intent": "is the modem up"},+ {"intent": "is the modem on /dev/ttyUSB1 up"},+ {"intent": "is the modem on /dev/ttyACM0 up"},+ {"intent": "is the modem on /dev/ttyS0 up"},] diff --git a/tests/conformance/vectors.json b/tests/conformance/vectors.json index 7962818..45f5b80 100644 --- a/tests/conformance/vectors.json +++ b/tests/conformance/vectors.json@@ -1299,9 +1299,1103 @@"count": 2 }, "tree": "hacker-news"+ },+ {+ "expect": {+ "inputs": {+ "mount": "/"+ },+ "output": {+ "mounted": "/",+ "percent": 71+ },+ "say": "/ is 71% full.",+ "status": "done",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "ran",+ "node": "df",+ "observed": {+ "mounted": "/",+ "percent": "71"+ },+ "summary": "shell df -P / -> exit 0; percent=71, mounted=/"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "number",+ "node": "ran",+ "observed": true,+ "summary": "state.df.ok = True"+ },+ {+ "depth": 0,+ "kind": "set",+ "next": "level",+ "node": "number",+ "observed": {+ "pct": 71+ },+ "summary": "pct=71"+ },+ {+ "chosen": "lt 80",+ "depth": 0,+ "kind": "decide",+ "next": "fine",+ "node": "level",+ "observed": 71,+ "summary": "state.pct = 71"+ },+ {+ "depth": 0,+ "kind": "done",+ "next": null,+ "node": "fine",+ "summary": "/ is 71% full."+ }+ ],+ "tree": "disk-space"+ },+ "tree": "disk-space"+ },+ "inputs": {},+ "intent": "how full is the disk",+ "tree": "disk-space"+ },+ {+ "expect": {+ "inputs": {+ "mount": "/data"+ },+ "reason": "disk-full",+ "say": "/data is 96% full.",+ "status": "failed",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "ran",+ "node": "df",+ "observed": {+ "mounted": "/data",+ "percent": "96"+ },+ "summary": "shell df -P /data -> exit 0; percent=96, mounted=/data"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "number",+ "node": "ran",+ "observed": true,+ "summary": "state.df.ok = True"+ },+ {+ "depth": 0,+ "kind": "set",+ "next": "level",+ "node": "number",+ "observed": {+ "pct": 96+ },+ "summary": "pct=96"+ },+ {+ "chosen": "else",+ "depth": 0,+ "kind": "decide",+ "next": "full",+ "node": "level",+ "observed": 96,+ "summary": "state.pct = 96"+ },+ {+ "depth": 0,+ "kind": "fail",+ "next": null,+ "node": "full",+ "summary": "disk-full: /data is 96% full."+ }+ ],+ "tree": "disk-space"+ },+ "tree": "disk-space"+ },+ "inputs": {+ "mount": "/data"+ },+ "intent": "disk space on /data",+ "tree": "disk-space"+ },+ {+ "expect": {+ "inputs": {+ "mount": "/nowhere"+ },+ "reason": "df-failed",+ "say": "df could not measure /nowhere: df: /nowhere: No such file or directory",+ "status": "failed",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "ran",+ "node": "df",+ "observed": {+ "mounted": null,+ "percent": null+ },+ "summary": "shell df -P /nowhere -> exit 1; percent=None, mounted=None"+ },+ {+ "chosen": "else",+ "depth": 0,+ "kind": "decide",+ "next": "failed",+ "node": "ran",+ "observed": false,+ "summary": "state.df.ok = False"+ },+ {+ "depth": 0,+ "kind": "fail",+ "next": null,+ "node": "failed",+ "summary": "df-failed: df could not measure /nowhere: df: /nowhere: No such file or directory"+ }+ ],+ "tree": "disk-space"+ },+ "tree": "disk-space"+ },+ "inputs": {+ "mount": "/nowhere"+ },+ "intent": "disk space on /nowhere",+ "tree": "disk-space"+ },+ {+ "expect": {+ "inputs": {+ "lines": 200,+ "path": "/var/log/syslog"+ },+ "output": {+ "errors": 2,+ "lines": [+ "Oct 25 12:00:00 nas pyrtan[1200]: error: weather: open-meteo answered 503",+ "Oct 25 12:01:02 nas pyrtan[1200]: ERROR site-status: broken.example answered 500"+ ],+ "path": "/var/log/syslog"+ },+ "say": "2 error line(s) in the last 5 of /var/log/syslog; latest: Oct 25 12:01:02 nas pyrtan[1200]: ERROR site-status: broken.example answered 500",+ "status": "done",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "exists",+ "node": "read",+ "observed": {+ "errors": [+ "Oct 25 12:00:00 nas pyrtan[1200]: error: weather: open-meteo answered 503",+ "Oct 25 12:01:02 nas pyrtan[1200]: ERROR site-status: broken.example answered 500"+ ]+ },+ "summary": "file /var/log/syslog -> 48213 B; errors=['Oct 25 12:00:00 nas pyrtan[1200]: error: weather: open-meteo answered 503', '…"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "count",+ "node": "exists",+ "observed": true,+ "summary": "state.log.exists = True"+ },+ {+ "depth": 0,+ "kind": "set",+ "next": "any",+ "node": "count",+ "observed": {+ "n": 2,+ "seen": 5+ },+ "summary": "n=2, seen=5"+ },+ {+ "chosen": "else",+ "depth": 0,+ "kind": "decide",+ "next": "some",+ "node": "any",+ "observed": 2,+ "summary": "state.n = 2"+ },+ {+ "depth": 0,+ "kind": "done",+ "next": null,+ "node": "some",+ "summary": "2 error line(s) in the last 5 of /var/log/syslog; latest: Oct 25 12:01:02 nas pyrtan[1200]: ERROR site-status: broken.example answered 500"+ }+ ],+ "tree": "log-errors"+ },+ "tree": "log-errors"+ },+ "inputs": {},+ "intent": "errors in the log",+ "tree": "log-errors"+ },+ {+ "expect": {+ "inputs": {+ "lines": 200,+ "path": "/var/log/quiet.log"+ },+ "output": {+ "errors": 0,+ "path": "/var/log/quiet.log"+ },+ "say": "No error lines in the last 1 of /var/log/quiet.log.",+ "status": "done",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "exists",+ "node": "read",+ "observed": {+ "errors": []+ },+ "summary": "file /var/log/quiet.log -> 71 B; errors=[]"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "count",+ "node": "exists",+ "observed": true,+ "summary": "state.log.exists = True"+ },+ {+ "depth": 0,+ "kind": "set",+ "next": "any",+ "node": "count",+ "observed": {+ "n": 0,+ "seen": 1+ },+ "summary": "n=0, seen=1"+ },+ {+ "chosen": "eq 0",+ "depth": 0,+ "kind": "decide",+ "next": "clean",+ "node": "any",+ "observed": 0,+ "summary": "state.n = 0"+ },+ {+ "depth": 0,+ "kind": "done",+ "next": null,+ "node": "clean",+ "summary": "No error lines in the last 1 of /var/log/quiet.log."+ }+ ],+ "tree": "log-errors"+ },+ "tree": "log-errors"+ },+ "inputs": {+ "path": "/var/log/quiet.log"+ },+ "intent": "errors in /var/log/quiet.log",+ "tree": "log-errors"+ },+ {+ "expect": {+ "inputs": {+ "lines": 200,+ "path": "/var/log/nothing.log"+ },+ "reason": "no-such-file",+ "say": "/var/log/nothing.log is not on this box.",+ "status": "failed",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "exists",+ "node": "read",+ "observed": {+ "errors": null+ },+ "summary": "file /var/log/nothing.log -> missing; errors=None"+ },+ {+ "chosen": "else",+ "depth": 0,+ "kind": "decide",+ "next": "missing",+ "node": "exists",+ "observed": false,+ "summary": "state.log.exists = False"+ },+ {+ "depth": 0,+ "kind": "fail",+ "next": null,+ "node": "missing",+ "summary": "no-such-file: /var/log/nothing.log is not on this box."+ }+ ],+ "tree": "log-errors"+ },+ "tree": "log-errors"+ },+ "inputs": {+ "path": "/var/log/nothing.log"+ },+ "intent": "errors in /var/log/nothing.log",+ "tree": "log-errors"+ },+ {+ "expect": {+ "inputs": {+ "lines": 200,+ "path": "/etc/passwd"+ },+ "reason": "path-not-allowed",+ "say": "read: /etc/passwd is outside this tree's paths ['/var/log'].",+ "status": "error",+ "trace": {+ "gaps": [],+ "steps": [],+ "tree": "log-errors"+ },+ "tree": "log-errors"+ },+ "inputs": {+ "path": "/etc/passwd"+ },+ "intent": "errors in /etc/passwd",+ "tree": "log-errors"+ },+ {+ "expect": {+ "inputs": {+ "room": "kitchen"+ },+ "output": {+ "room": "kitchen",+ "temperature": 21.5,+ "topic": "home/kitchen/temperature"+ },+ "say": "kitchen: 21.5°C.",+ "status": "done",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "got",+ "node": "wait",+ "summary": "queue home/kitchen/temperature at mqtt://broker.local -> 1 message"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "reading",+ "node": "got",+ "observed": true,+ "summary": "state.q.received = True"+ },+ {+ "chosen": "lt 26",+ "depth": 0,+ "kind": "decide",+ "next": "fine",+ "node": "reading",+ "observed": 21.5,+ "summary": "state.q.json.temperature = 21.5"+ },+ {+ "depth": 0,+ "kind": "done",+ "next": null,+ "node": "fine",+ "summary": "kitchen: 21.5°C."+ }+ ],+ "tree": "room-temperature"+ },+ "tree": "room-temperature"+ },+ "inputs": {+ "room": "kitchen"+ },+ "intent": "temperature in the kitchen",+ "tree": "room-temperature"+ },+ {+ "expect": {+ "inputs": {+ "room": "attic"+ },+ "output": {+ "room": "attic",+ "temperature": 31,+ "topic": "home/attic/temperature"+ },+ "say": "attic: 31°C, warm.",+ "status": "done",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "got",+ "node": "wait",+ "summary": "queue home/attic/temperature at mqtt://broker.local -> 1 message"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "reading",+ "node": "got",+ "observed": true,+ "summary": "state.q.received = True"+ },+ {+ "chosen": "gte 26",+ "depth": 0,+ "kind": "decide",+ "next": "warm",+ "node": "reading",+ "observed": 31,+ "summary": "state.q.json.temperature = 31"+ },+ {+ "depth": 0,+ "kind": "done",+ "next": null,+ "node": "warm",+ "summary": "attic: 31°C, warm."+ }+ ],+ "tree": "room-temperature"+ },+ "tree": "room-temperature"+ },+ "inputs": {+ "room": "attic"+ },+ "intent": "how warm is the attic",+ "tree": "room-temperature"+ },+ {+ "expect": {+ "inputs": {+ "room": "garage"+ },+ "reason": "no-reading",+ "say": "Nothing on home/garage/temperature within 15 s; is the sensor publishing?",+ "status": "failed",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "got",+ "node": "wait",+ "summary": "queue home/garage/temperature at mqtt://broker.local -> no message"+ },+ {+ "chosen": "else",+ "depth": 0,+ "kind": "decide",+ "next": "quiet",+ "node": "got",+ "observed": false,+ "summary": "state.q.received = False"+ },+ {+ "depth": 0,+ "kind": "fail",+ "next": null,+ "node": "quiet",+ "summary": "no-reading: Nothing on home/garage/temperature within 15 s; is the sensor publishing?"+ }+ ],+ "tree": "room-temperature"+ },+ "tree": "room-temperature"+ },+ "inputs": {+ "room": "garage"+ },+ "intent": "temperature in the garage",+ "tree": "room-temperature"+ },+ {+ "expect": {+ "inputs": {+ "room": "hall"+ },+ "output": {+ "gap": {+ "node": "reading",+ "observed": null,+ "on": "state.q.json.temperature",+ "tree": "room-temperature"+ }+ },+ "reason": "gap",+ "say": "reading: no branch for state.q.json.temperature = <missing>.",+ "status": "error",+ "trace": {+ "gaps": [+ {+ "node": "reading",+ "observed": null,+ "on": "state.q.json.temperature",+ "tree": "room-temperature"+ }+ ],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "got",+ "node": "wait",+ "summary": "queue home/hall/temperature at mqtt://broker.local -> 1 message"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "reading",+ "node": "got",+ "observed": true,+ "summary": "state.q.received = True"+ },+ {+ "depth": 0,+ "kind": "decide",+ "next": null,+ "node": "reading",+ "summary": "state.q.json.temperature = <missing>: no branch"+ }+ ],+ "tree": "room-temperature"+ },+ "tree": "room-temperature"+ },+ "inputs": {+ "room": "hall"+ },+ "intent": "temperature in the hall",+ "tree": "room-temperature"+ },+ {+ "expect": {+ "inputs": {+ "room": "a-b"+ },+ "reason": "transport",+ "say": "wait: no recorded result for queue mqtt://broker.local home/a-b/temperature",+ "status": "error",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": null,+ "node": "wait",+ "summary": "queue home/a-b/temperature at mqtt://broker.local failed: no recorded result for queue mqtt://broker.local home/a-b/temperature"+ }+ ],+ "tree": "room-temperature"+ },+ "tree": "room-temperature"+ },+ "inputs": {+ "room": "a-b"+ },+ "tree": "room-temperature"+ },+ {+ "expect": {+ "inputs": {+ "url": "https://app.example/dashboard"+ },+ "output": {+ "heading": "Welcome back, Barry",+ "title": "Dashboard",+ "url": "https://app.example/dashboard"+ },+ "say": "Dashboard — Welcome back, Barry",+ "status": "done",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "loaded",+ "node": "open",+ "observed": {+ "heading": "Welcome back, Barry"+ },+ "summary": "browser https://app.example/dashboard -> 200; heading=Welcome back, Barry"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "found",+ "node": "loaded",+ "observed": true,+ "summary": "state.page.ok = True"+ },+ {+ "chosen": "exists True",+ "depth": 0,+ "kind": "decide",+ "next": "say",+ "node": "found",+ "observed": "Welcome back, Barry",+ "summary": "state.heading = Welcome back, Barry"+ },+ {+ "depth": 0,+ "kind": "done",+ "next": null,+ "node": "say",+ "summary": "Dashboard — Welcome back, Barry"+ }+ ],+ "tree": "rendered-headline"+ },+ "tree": "rendered-headline"+ },+ "inputs": {+ "url": "https://app.example/dashboard"+ },+ "intent": "headline of https://app.example/dashboard",+ "tree": "rendered-headline"+ },+ {+ "expect": {+ "inputs": {+ "url": "https://app.example/empty"+ },+ "reason": "no-heading",+ "say": "https://app.example/empty rendered without an h1.",+ "status": "failed",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "loaded",+ "node": "open",+ "observed": {+ "heading": null+ },+ "summary": "browser https://app.example/empty -> 200; heading=None"+ },+ {+ "chosen": "eq True",+ "depth": 0,+ "kind": "decide",+ "next": "found",+ "node": "loaded",+ "observed": true,+ "summary": "state.page.ok = True"+ },+ {+ "chosen": "else",+ "depth": 0,+ "kind": "decide",+ "next": "none",+ "node": "found",+ "summary": "state.heading = <missing>"+ },+ {+ "depth": 0,+ "kind": "fail",+ "next": null,+ "node": "none",+ "summary": "no-heading: https://app.example/empty rendered without an h1."+ }+ ],+ "tree": "rendered-headline"+ },+ "tree": "rendered-headline"+ },+ "inputs": {+ "url": "https://app.example/empty"+ },+ "intent": "what does https://app.example/empty say",+ "tree": "rendered-headline"+ },+ {+ "expect": {+ "inputs": {+ "url": "https://app.example/gone"+ },+ "reason": "unexpected-response",+ "say": "https://app.example/gone answered 404.",+ "status": "failed",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "loaded",+ "node": "open",+ "observed": {+ "heading": "Not found"+ },+ "summary": "browser https://app.example/gone -> 404; heading=Not found"+ },+ {+ "chosen": "else",+ "depth": 0,+ "kind": "decide",+ "next": "not-ok",+ "node": "loaded",+ "observed": false,+ "summary": "state.page.ok = False"+ },+ {+ "depth": 0,+ "kind": "fail",+ "next": null,+ "node": "not-ok",+ "summary": "unexpected-response: https://app.example/gone answered 404."+ }+ ],+ "tree": "rendered-headline"+ },+ "tree": "rendered-headline"+ },+ "inputs": {+ "url": "https://app.example/gone"+ },+ "intent": "headline of https://app.example/gone",+ "tree": "rendered-headline"+ },+ {+ "expect": {+ "inputs": {+ "device": "/dev/ttyUSB0"+ },+ "output": {+ "device": "/dev/ttyUSB0"+ },+ "say": "The modem on /dev/ttyUSB0 is there.",+ "status": "done",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "answered",+ "node": "at",+ "observed": {+ "answer": "OK"+ },+ "summary": "serial /dev/ttyUSB0 -> 4 B; answer=OK"+ },+ {+ "chosen": "eq 'OK'",+ "depth": 0,+ "kind": "decide",+ "next": "yes",+ "node": "answered",+ "observed": "OK",+ "summary": "state.answer = OK"+ },+ {+ "depth": 0,+ "kind": "done",+ "next": null,+ "node": "yes",+ "summary": "The modem on /dev/ttyUSB0 is there."+ }+ ],+ "tree": "modem-check"+ },+ "tree": "modem-check"+ },+ "inputs": {},+ "intent": "is the modem up",+ "tree": "modem-check"+ },+ {+ "expect": {+ "inputs": {+ "device": "/dev/ttyUSB1"+ },+ "reason": "transport",+ "say": "at: /dev/ttyUSB1: no answer within 3 s",+ "status": "error",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": null,+ "node": "at",+ "summary": "serial /dev/ttyUSB1 failed: /dev/ttyUSB1: no answer within 3 s"+ }+ ],+ "tree": "modem-check"+ },+ "tree": "modem-check"+ },+ "inputs": {+ "device": "/dev/ttyUSB1"+ },+ "intent": "is the modem on /dev/ttyUSB1 up",+ "tree": "modem-check"+ },+ {+ "expect": {+ "inputs": {+ "device": "/dev/ttyACM0"+ },+ "reason": "modem-error",+ "say": "The modem on /dev/ttyACM0 answered ERROR.",+ "status": "failed",+ "trace": {+ "gaps": [],+ "steps": [+ {+ "depth": 0,+ "kind": "observe",+ "next": "answered",+ "node": "at",+ "observed": {+ "answer": "ERROR"+ },+ "summary": "serial /dev/ttyACM0 -> 16 B; answer=ERROR"+ },+ {+ "chosen": "eq 'ERROR'",+ "depth": 0,+ "kind": "decide",+ "next": "error",+ "node": "answered",+ "observed": "ERROR",+ "summary": "state.answer = ERROR"+ },+ {+ "depth": 0,+ "kind": "fail",+ "next": null,+ "node": "error",+ "summary": "modem-error: The modem on /dev/ttyACM0 answered ERROR."+ }+ ],+ "tree": "modem-check"+ },+ "tree": "modem-check"+ },+ "inputs": {+ "device": "/dev/ttyACM0"+ },+ "intent": "is the modem on /dev/ttyACM0 up",+ "tree": "modem-check"+ },+ {+ "expect": {+ "inputs": {+ "device": "/dev/ttyS0"+ },+ "reason": "device-not-allowed",+ "say": "at: /dev/ttyS0 is outside this tree's devices ['/dev/ttyUSB*', '/dev/ttyACM*'].",+ "status": "error",+ "trace": {+ "gaps": [],+ "steps": [],+ "tree": "modem-check"+ },+ "tree": "modem-check"+ },+ "inputs": {+ "device": "/dev/ttyS0"+ },+ "intent": "is the modem on /dev/ttyS0 up",+ "tree": "modem-check"} ], "trees": {+ "disk-space": {+ "commands": [+ "df"+ ],+ "description": "How full a filesystem is, from df on the box this tree runs on.",+ "inputs": {+ "mount": {+ "default": "/",+ "description": "A mount point, or any path on the filesystem to measure",+ "type": "str"+ }+ },+ "license": "MIT",+ "name": "disk-space",+ "nodes": {+ "df": {+ "extract": [+ {+ "group": 1,+ "into": "percent",+ "regex": "(\\d+)%"+ },+ {+ "group": 1,+ "into": "mounted",+ "regex": "%\\s+(\\S+)\\s*$"+ }+ ],+ "into": "df",+ "kind": "observe",+ "next": "ran",+ "shell": {+ "args": [+ "-P",+ "{{ inputs.mount }}"+ ],+ "command": "df"+ }+ },+ "failed": {+ "kind": "fail",+ "reason": "df-failed",+ "say": "df could not measure {{ inputs.mount }}: {{ state.df.stderr | strip }}"+ },+ "fine": {+ "kind": "done",+ "output": {+ "mounted": "{{ state.mounted }}",+ "percent": "{{ state.pct }}"+ },+ "say": "{{ state.mounted }} is {{ state.pct }}% full."+ },+ "full": {+ "kind": "fail",+ "reason": "disk-full",+ "say": "{{ state.mounted }} is {{ state.pct }}% full."+ },+ "level": {+ "branches": [+ {+ "then": "fine",+ "when": {+ "lt": 80+ }+ },+ {+ "then": "tight",+ "when": {+ "lt": 95+ }+ },+ {+ "else": "full"+ }+ ],+ "kind": "decide",+ "on": "pct"+ },+ "number": {+ "kind": "set",+ "next": "level",+ "set": {+ "pct": "{{ state.percent | int }}"+ }+ },+ "ran": {+ "branches": [+ {+ "then": "number",+ "when": {+ "eq": true+ }+ },+ {+ "else": "failed"+ }+ ],+ "kind": "decide",+ "on": "df.ok"+ },+ "tight": {+ "kind": "done",+ "output": {+ "mounted": "{{ state.mounted }}",+ "percent": "{{ state.pct }}"+ },+ "say": "{{ state.mounted }} is {{ state.pct }}% full; make room soon."+ }+ },+ "start": "df",+ "tags": [+ "ops",+ "box",+ "read-only"+ ],+ "triggers": [+ {+ "pattern": "[how] full is the disk"+ },+ {+ "pattern": "disk space"+ },+ {+ "pattern": "disk space on {mount:word}"+ }+ ],+ "version": 1+ },"github-release": { "description": "The latest release of a GitHub repository.", "hosts": [@@ -1679,6 +2773,421 @@], "version": 1 },+ "log-errors": {+ "description": "Error lines in the tail of a log file on this box.",+ "inputs": {+ "lines": {+ "default": 200,+ "description": "How many lines from the end to look at",+ "max": 5000,+ "min": 1,+ "type": "int"+ },+ "path": {+ "default": "/var/log/syslog",+ "type": "str"+ }+ },+ "license": "MIT",+ "name": "log-errors",+ "nodes": {+ "any": {+ "branches": [+ {+ "then": "clean",+ "when": {+ "eq": 0+ }+ },+ {+ "else": "some"+ }+ ],+ "kind": "decide",+ "on": "n"+ },+ "clean": {+ "kind": "done",+ "output": {+ "errors": 0,+ "path": "{{ inputs.path }}"+ },+ "say": "No error lines in the last {{ state.seen }} of {{ inputs.path }}."+ },+ "count": {+ "kind": "set",+ "next": "any",+ "set": {+ "n": "{{ state.errors | length }}",+ "seen": "{{ state.log.lines | length }}"+ }+ },+ "exists": {+ "branches": [+ {+ "then": "count",+ "when": {+ "eq": true+ }+ },+ {+ "else": "missing"+ }+ ],+ "kind": "decide",+ "on": "log.exists"+ },+ "missing": {+ "kind": "fail",+ "reason": "no-such-file",+ "say": "{{ inputs.path }} is not on this box."+ },+ "read": {+ "extract": [+ {+ "all": true,+ "into": "errors",+ "regex": "(?im)^.*\\berror\\b.*$"+ }+ ],+ "file": {+ "path": "{{ inputs.path }}",+ "tail": "{{ inputs.lines }}"+ },+ "into": "log",+ "kind": "observe",+ "next": "exists"+ },+ "some": {+ "kind": "done",+ "output": {+ "errors": "{{ state.n }}",+ "lines": "{{ state.errors }}",+ "path": "{{ inputs.path }}"+ },+ "say": "{{ state.n }} error line(s) in the last {{ state.seen }} of {{ inputs.path }}; latest: {{ state.errors | last | strip }}"+ }+ },+ "paths": [+ "/var/log"+ ],+ "start": "read",+ "tags": [+ "ops",+ "box",+ "read-only"+ ],+ "triggers": [+ {+ "pattern": "errors in the log"+ },+ {+ "pattern": "errors in {path:word}"+ }+ ],+ "version": 1+ },+ "modem-check": {+ "description": "Whether a modem on a serial line answers AT.",+ "devices": [+ "/dev/ttyUSB*",+ "/dev/ttyACM*"+ ],+ "inputs": {+ "device": {+ "default": "/dev/ttyUSB0",+ "type": "str"+ }+ },+ "license": "MIT",+ "name": "modem-check",+ "nodes": {+ "answered": {+ "branches": [+ {+ "then": "yes",+ "when": {+ "eq": "OK"+ }+ },+ {+ "then": "error",+ "when": {+ "eq": "ERROR"+ }+ },+ {+ "else": "odd"+ }+ ],+ "kind": "decide",+ "on": "answer"+ },+ "at": {+ "extract": [+ {+ "into": "answer",+ "regex": "(OK|ERROR)"+ }+ ],+ "into": "s",+ "kind": "observe",+ "next": "answered",+ "retries": 1,+ "serial": {+ "baud": 9600,+ "device": "{{ inputs.device }}",+ "read_until": "\n",+ "write": "AT\r\n"+ },+ "timeout": 3+ },+ "error": {+ "kind": "fail",+ "reason": "modem-error",+ "say": "The modem on {{ inputs.device }} answered ERROR."+ },+ "odd": {+ "kind": "fail",+ "reason": "odd-answer",+ "say": "{{ inputs.device }} answered {{ state.s.body | strip }} to AT."+ },+ "yes": {+ "kind": "done",+ "output": {+ "device": "{{ inputs.device }}"+ },+ "say": "The modem on {{ inputs.device }} is there."+ }+ },+ "start": "at",+ "tags": [+ "hardware",+ "serial"+ ],+ "triggers": [+ {+ "pattern": "is the modem up"+ },+ {+ "pattern": "is the modem on {device:word} up"+ }+ ],+ "version": 1+ },+ "rendered-headline": {+ "description": "The main heading of a page that needs a script to render, through a real browser.",+ "hosts": [+ "*"+ ],+ "inputs": {+ "url": {+ "pattern": "^https?://.+$",+ "type": "str"+ }+ },+ "license": "MIT",+ "name": "rendered-headline",+ "nodes": {+ "found": {+ "branches": [+ {+ "then": "say",+ "when": {+ "exists": true+ }+ },+ {+ "else": "none"+ }+ ],+ "kind": "decide",+ "on": "heading"+ },+ "loaded": {+ "branches": [+ {+ "then": "found",+ "when": {+ "eq": true+ }+ },+ {+ "else": "not-ok"+ }+ ],+ "kind": "decide",+ "on": "page.ok"+ },+ "none": {+ "kind": "fail",+ "reason": "no-heading",+ "say": "{{ state.page.url }} rendered without an h1."+ },+ "not-ok": {+ "kind": "fail",+ "reason": "unexpected-response",+ "say": "{{ inputs.url }} answered {{ state.page.status }}."+ },+ "open": {+ "browser": {+ "url": "{{ inputs.url }}",+ "wait_for": "h1"+ },+ "extract": [+ {+ "css": "h1",+ "into": "heading"+ }+ ],+ "into": "page",+ "kind": "observe",+ "next": "loaded",+ "retries": 0,+ "timeout": 30+ },+ "say": {+ "kind": "done",+ "output": {+ "heading": "{{ state.heading | strip }}",+ "title": "{{ state.page.title }}",+ "url": "{{ state.page.url }}"+ },+ "say": "{{ state.page.title }} — {{ state.heading | strip }}"+ }+ },+ "start": "open",+ "tags": [+ "web",+ "browser",+ "read-only"+ ],+ "triggers": [+ {+ "pattern": "headline of {url:url}"+ },+ {+ "pattern": "what does {url:url} say"+ }+ ],+ "version": 1+ },+ "room-temperature": {+ "description": "The temperature a room's sensor last published, over MQTT.",+ "hosts": [+ "broker.local"+ ],+ "inputs": {+ "room": {+ "default": "kitchen",+ "pattern": "^[a-z0-9-]+$",+ "type": "str"+ }+ },+ "license": "MIT",+ "name": "room-temperature",+ "nodes": {+ "cold": {+ "kind": "done",+ "output": {+ "room": "{{ inputs.room }}",+ "temperature": "{{ state.q.json.temperature }}",+ "topic": "{{ state.q.message.topic }}"+ },+ "say": "{{ inputs.room }}: {{ state.q.json.temperature }}°C, cold."+ },+ "fine": {+ "kind": "done",+ "output": {+ "room": "{{ inputs.room }}",+ "temperature": "{{ state.q.json.temperature }}",+ "topic": "{{ state.q.message.topic }}"+ },+ "say": "{{ inputs.room }}: {{ state.q.json.temperature }}°C."+ },+ "got": {+ "branches": [+ {+ "then": "reading",+ "when": {+ "eq": true+ }+ },+ {+ "else": "quiet"+ }+ ],+ "kind": "decide",+ "on": "q.received"+ },+ "quiet": {+ "kind": "fail",+ "reason": "no-reading",+ "say": "Nothing on home/{{ inputs.room }}/temperature within 15 s; is the sensor publishing?"+ },+ "reading": {+ "branches": [+ {+ "then": "cold",+ "when": {+ "lt": 16+ }+ },+ {+ "then": "fine",+ "when": {+ "lt": 26+ }+ },+ {+ "then": "warm",+ "when": {+ "gte": 26+ }+ }+ ],+ "kind": "decide",+ "on": "q.json.temperature"+ },+ "wait": {+ "into": "q",+ "kind": "observe",+ "next": "got",+ "queue": {+ "topic": "home/{{ inputs.room }}/temperature",+ "url": "mqtt://broker.local"+ },+ "retries": 0,+ "timeout": 15+ },+ "warm": {+ "kind": "done",+ "output": {+ "room": "{{ inputs.room }}",+ "temperature": "{{ state.q.json.temperature }}",+ "topic": "{{ state.q.message.topic }}"+ },+ "say": "{{ inputs.room }}: {{ state.q.json.temperature }}°C, warm."+ }+ },+ "start": "wait",+ "tags": [+ "home",+ "mqtt"+ ],+ "topics": [+ "home/+/temperature"+ ],+ "triggers": [+ {+ "pattern": "[what's the] temperature in the {room:word}"+ },+ {+ "pattern": "how warm is the {room:word}"+ }+ ],+ "version": 1+ },"site-status": { "description": "Whether a web site answers, and how quickly.", "hosts": [ diff --git a/tests/fixtures/examples.json b/tests/fixtures/examples.json index 6735aa7..53969bd 100644 --- a/tests/fixtures/examples.json +++ b/tests/fixtures/examples.json@@ -1,17 +1,316 @@{- "GET https://hacker-news.firebaseio.com/v0/topstories.json": {"status": 200, "body": [101, 102, 103]},- "GET https://hacker-news.firebaseio.com/v0/item/101.json": {"status": 200, "body": {"id": 101, "type": "story", "title": "Pyrtan 0.1", "url": "https://example.org/p", "score": 42, "by": "alice"}},- "GET https://hacker-news.firebaseio.com/v0/item/102.json": {"status": 200, "body": {"id": 102, "type": "story", "title": "Ask HN: self post", "score": 7, "by": "x"}},- "GET https://hacker-news.firebaseio.com/v0/item/103.json": {"status": 200, "body": {"id": 103, "type": "comment", "by": "y", "text": "a comment"}},- "GET https://hacker-news.firebaseio.com/v0/item/999.json": {"status": 200, "body": null},- "GET https://geocoding-api.open-meteo.com/v1/search?name=Oslo&count=1&language=en&format=json": {"status": 200, "body": {"results": [{"name": "Oslo", "country": "Norway", "latitude": 59.91, "longitude": 10.75}]}},- "GET https://geocoding-api.open-meteo.com/v1/search?name=Nowhereville&count=1&language=en&format=json": {"status": 200, "body": {"generationtime_ms": 0.5}},- "GET https://api.open-meteo.com/v1/forecast?latitude=59.91&longitude=10.75¤t=temperature_2m,relative_humidity_2m,wind_speed_10m,weather_code&temperature_unit=celsius": {"status": 200, "body": {"current_units": {"temperature_2m": "°C"}, "current": {"temperature_2m": 4.2, "relative_humidity_2m": 80, "wind_speed_10m": 12.3, "weather_code": 61}}},- "GET https://api.open-meteo.com/v1/forecast?latitude=59.91&longitude=10.75¤t=temperature_2m,relative_humidity_2m,wind_speed_10m,weather_code&temperature_unit=fahrenheit": {"status": 200, "body": {"current_units": {"temperature_2m": "°F"}, "current": {"temperature_2m": 39.6, "relative_humidity_2m": 80, "wind_speed_10m": 12.3, "weather_code": 0}}},- "GET https://example.com/": {"status": 200, "body": "<html><head><title>Example Domain</title></head><body><h1>Example Domain</h1></body></html>"},- "GET https://broken.example/": {"status": 503, "body": "down"},- "GET https://teapot.example/": {"status": 418, "body": "short and stout"},- "GET https://api.github.com/repos/b-i-t-8/pyrtan/releases/latest": {"status": 200, "body": {"tag_name": "v0.1.0", "name": "First", "published_at": "2026-10-24T00:00:00Z", "html_url": "https://github.com/b-i-t-8/pyrtan/releases/tag/v0.1.0", "prerelease": false}},- "GET https://api.github.com/repos/b-i-t-8/nothing/releases/latest": {"status": 404, "body": {"message": "Not Found"}},- "GET https://api.github.com/repos/b-i-t-8/busy/releases/latest": {"status": 403, "body": {"message": "API rate limit exceeded"}}+ "GET https://hacker-news.firebaseio.com/v0/topstories.json": {+ "status": 200,+ "body": [+ 101,+ 102,+ 103+ ]+ },+ "GET https://hacker-news.firebaseio.com/v0/item/101.json": {+ "status": 200,+ "body": {+ "id": 101,+ "type": "story",+ "title": "Pyrtan 0.1",+ "url": "https://example.org/p",+ "score": 42,+ "by": "alice"+ }+ },+ "GET https://hacker-news.firebaseio.com/v0/item/102.json": {+ "status": 200,+ "body": {+ "id": 102,+ "type": "story",+ "title": "Ask HN: self post",+ "score": 7,+ "by": "x"+ }+ },+ "GET https://hacker-news.firebaseio.com/v0/item/103.json": {+ "status": 200,+ "body": {+ "id": 103,+ "type": "comment",+ "by": "y",+ "text": "a comment"+ }+ },+ "GET https://hacker-news.firebaseio.com/v0/item/999.json": {+ "status": 200,+ "body": null+ },+ "GET https://geocoding-api.open-meteo.com/v1/search?name=Oslo&count=1&language=en&format=json": {+ "status": 200,+ "body": {+ "results": [+ {+ "name": "Oslo",+ "country": "Norway",+ "latitude": 59.91,+ "longitude": 10.75+ }+ ]+ }+ },+ "GET https://geocoding-api.open-meteo.com/v1/search?name=Nowhereville&count=1&language=en&format=json": {+ "status": 200,+ "body": {+ "generationtime_ms": 0.5+ }+ },+ "GET https://api.open-meteo.com/v1/forecast?latitude=59.91&longitude=10.75¤t=temperature_2m,relative_humidity_2m,wind_speed_10m,weather_code&temperature_unit=celsius": {+ "status": 200,+ "body": {+ "current_units": {+ "temperature_2m": "°C"+ },+ "current": {+ "temperature_2m": 4.2,+ "relative_humidity_2m": 80,+ "wind_speed_10m": 12.3,+ "weather_code": 61+ }+ }+ },+ "GET https://api.open-meteo.com/v1/forecast?latitude=59.91&longitude=10.75¤t=temperature_2m,relative_humidity_2m,wind_speed_10m,weather_code&temperature_unit=fahrenheit": {+ "status": 200,+ "body": {+ "current_units": {+ "temperature_2m": "°F"+ },+ "current": {+ "temperature_2m": 39.6,+ "relative_humidity_2m": 80,+ "wind_speed_10m": 12.3,+ "weather_code": 0+ }+ }+ },+ "GET https://example.com/": {+ "status": 200,+ "body": "<html><head><title>Example Domain</title></head><body><h1>Example Domain</h1></body></html>"+ },+ "GET https://broken.example/": {+ "status": 503,+ "body": "down"+ },+ "GET https://teapot.example/": {+ "status": 418,+ "body": "short and stout"+ },+ "GET https://api.github.com/repos/b-i-t-8/pyrtan/releases/latest": {+ "status": 200,+ "body": {+ "tag_name": "v0.1.0",+ "name": "First",+ "published_at": "2026-10-24T00:00:00Z",+ "html_url": "https://github.com/b-i-t-8/pyrtan/releases/tag/v0.1.0",+ "prerelease": false+ }+ },+ "GET https://api.github.com/repos/b-i-t-8/nothing/releases/latest": {+ "status": 404,+ "body": {+ "message": "Not Found"+ }+ },+ "GET https://api.github.com/repos/b-i-t-8/busy/releases/latest": {+ "status": 403,+ "body": {+ "message": "API rate limit exceeded"+ }+ },+ "shell df -P /": {+ "command": "df",+ "args": [+ "-P",+ "/"+ ],+ "status": 0,+ "ok": true,+ "stdout": "Filesystem 1024-blocks Used Available Capacity Mounted on\n/dev/sda1 100000000 71000000 29000000 71% /\n",+ "stderr": "",+ "json": null,+ "lines": [+ "Filesystem 1024-blocks Used Available Capacity Mounted on",+ "/dev/sda1 100000000 71000000 29000000 71% /"+ ],+ "elapsed_ms": 4,+ "body": "Filesystem 1024-blocks Used Available Capacity Mounted on\n/dev/sda1 100000000 71000000 29000000 71% /\n"+ },+ "shell df -P /data": {+ "command": "df",+ "args": [+ "-P",+ "/data"+ ],+ "status": 0,+ "ok": true,+ "stdout": "Filesystem 1024-blocks Used Available Capacity Mounted on\n/dev/sdb1 400000000 384000000 16000000 96% /data\n",+ "stderr": "",+ "json": null,+ "lines": [+ "Filesystem 1024-blocks Used Available Capacity Mounted on",+ "/dev/sdb1 400000000 384000000 16000000 96% /data"+ ],+ "elapsed_ms": 4,+ "body": "Filesystem 1024-blocks Used Available Capacity Mounted on\n/dev/sdb1 400000000 384000000 16000000 96% /data\n"+ },+ "shell df -P /nowhere": {+ "command": "df",+ "args": [+ "-P",+ "/nowhere"+ ],+ "status": 1,+ "ok": false,+ "stdout": "",+ "stderr": "df: /nowhere: No such file or directory\n",+ "json": null,+ "lines": [],+ "elapsed_ms": 3,+ "body": ""+ },+ "file /var/log/syslog": {+ "path": "/var/log/syslog",+ "exists": true,+ "size": 48213,+ "modified": "2026-10-25T12:01:02Z",+ "body": "Oct 25 11:58:01 nas systemd[1]: Started Daily apt download activities.\nOct 25 11:59:14 nas smartd[812]: Device: /dev/sda, SMART Usage Attribute: 194 Temperature_Celsius changed from 112 to 111\nOct 25 12:00:00 nas pyrtan[1200]: error: weather: open-meteo answered 503\nOct 25 12:00:30 nas kernel: usb 1-1: new high-speed USB device number 4\nOct 25 12:01:02 nas pyrtan[1200]: ERROR site-status: broken.example answered 500",+ "json": null,+ "lines": [+ "Oct 25 11:58:01 nas systemd[1]: Started Daily apt download activities.",+ "Oct 25 11:59:14 nas smartd[812]: Device: /dev/sda, SMART Usage Attribute: 194 Temperature_Celsius changed from 112 to 111",+ "Oct 25 12:00:00 nas pyrtan[1200]: error: weather: open-meteo answered 503",+ "Oct 25 12:00:30 nas kernel: usb 1-1: new high-speed USB device number 4",+ "Oct 25 12:01:02 nas pyrtan[1200]: ERROR site-status: broken.example answered 500"+ ]+ },+ "file /var/log/quiet.log": {+ "path": "/var/log/quiet.log",+ "exists": true,+ "size": 71,+ "modified": "2026-10-25T09:00:00Z",+ "body": "Oct 25 09:00:00 nas cron[400]: (root) CMD (run-parts /etc/cron.hourly)",+ "json": null,+ "lines": [+ "Oct 25 09:00:00 nas cron[400]: (root) CMD (run-parts /etc/cron.hourly)"+ ]+ },+ "file /var/log/nothing.log": {+ "path": "/var/log/nothing.log",+ "exists": false,+ "size": null,+ "modified": null,+ "body": null,+ "json": null,+ "lines": []+ },+ "queue mqtt://broker.local home/kitchen/temperature": {+ "url": "mqtt://broker.local",+ "topic": "home/kitchen/temperature",+ "received": true,+ "message": {+ "topic": "home/kitchen/temperature",+ "body": "{\"temperature\": 21.5, \"unit\": \"C\"}",+ "json": {+ "temperature": 21.5,+ "unit": "C"+ }+ },+ "body": "{\"temperature\": 21.5, \"unit\": \"C\"}",+ "json": {+ "temperature": 21.5,+ "unit": "C"+ }+ },+ "queue mqtt://broker.local home/attic/temperature": {+ "url": "mqtt://broker.local",+ "topic": "home/attic/temperature",+ "received": true,+ "message": {+ "topic": "home/attic/temperature",+ "body": "{\"temperature\": 31}",+ "json": {+ "temperature": 31+ }+ },+ "body": "{\"temperature\": 31}",+ "json": {+ "temperature": 31+ }+ },+ "queue mqtt://broker.local home/garage/temperature": {+ "url": "mqtt://broker.local",+ "topic": "home/garage/temperature",+ "received": false,+ "message": null,+ "body": null,+ "json": null+ },+ "queue mqtt://broker.local home/hall/temperature": {+ "url": "mqtt://broker.local",+ "topic": "home/hall/temperature",+ "received": true,+ "message": {+ "topic": "home/hall/temperature",+ "body": "warm-ish",+ "json": null+ },+ "body": "warm-ish",+ "json": null+ },+ "browser https://app.example/dashboard": {+ "url": "https://app.example/dashboard",+ "status": 200,+ "ok": true,+ "title": "Dashboard",+ "body": "<!doctype html><html><head><title>Dashboard</title></head><body><div id=\"root\"><nav>Home</nav><h1>Welcome back, Barry</h1><p>3 alerts</p></div></body></html>",+ "elapsed_ms": 812+ },+ "browser https://app.example/empty": {+ "url": "https://app.example/empty",+ "status": 200,+ "ok": true,+ "title": "Empty",+ "body": "<html><head><title>Empty</title></head><body><div id=\"root\"></div></body></html>",+ "elapsed_ms": 300+ },+ "browser https://app.example/gone": {+ "url": "https://app.example/gone",+ "status": 404,+ "ok": false,+ "title": "Not found",+ "body": "<html><body><h1>Not found</h1></body></html>",+ "elapsed_ms": 200+ },+ "serial /dev/ttyUSB0 \"AT\\r\\n\"": {+ "device": "/dev/ttyUSB0",+ "baud": 9600,+ "sent": 4,+ "body": "OK\r\n",+ "bytes": 4,+ "complete": true,+ "json": null,+ "lines": [+ "OK"+ ]+ },+ "serial /dev/ttyUSB1 \"AT\\r\\n\"": {+ "error": "/dev/ttyUSB1: no answer within 3 s"+ },+ "serial /dev/ttyACM0 \"AT\\r\\n\"": {+ "device": "/dev/ttyACM0",+ "baud": 9600,+ "sent": 4,+ "body": "+CME ERROR: 10\r\n",+ "bytes": 16,+ "complete": true,+ "json": null,+ "lines": [+ "+CME ERROR: 10"+ ]+ }} diff --git a/tests/test_adapters.py b/tests/test_adapters.py new file mode 100644 index 0000000..418d635 --- /dev/null +++ b/tests/test_adapters.py@@ -0,0 +1,579 @@+"""The other forms of I/O: file, shell, serial, queue, browser; their+allow-lists, their fixtures, and the egress policy over them."""++from __future__ import annotations++import http.server+import json+import os+import socket+import struct+import sys+import threading+import time+from pathlib import Path++import pytest++from pyrtan import Registry, RecordedTransport, Runner+from pyrtan.adapters import (+ LocalAdapters,+ _mqtt_packet,+ _mqtt_read,+ _mqtt_str,+ adapter_key,+ command_allowed,+ device_allowed,+ path_allowed,+ topic_allowed,+)+from pyrtan.cli import main+from pyrtan.transport import EgressRefused, RecordingTransport, TransportError+from pyrtan.tree import TreeError, check, load_tree_text+++def registry(tmp_path: Path, **trees: str) -> Registry:+ d = tmp_path / "trees"+ d.mkdir(exist_ok=True)+ for name, text in trees.items():+ (d / f"{name}.toml").write_text(text)+ reg = Registry.from_paths([d])+ assert not reg.errors, reg.errors+ return reg+++# ---------------------------------------------------------------- allow-lists+++def test_allow_lists():+ assert path_allowed(["/var/log/*"], "/var/log/syslog") and path_allowed(["/var/log"], "/var/log/nginx/access.log")+ assert not path_allowed(["/var/log"], "/var/logs") and not path_allowed(["/var/log/*.log"], "/etc/passwd") and not path_allowed([], "/x")+ assert path_allowed(["*"], "/anything") and path_allowed(["~/notes"], os.path.expanduser("~/notes/today.md"))+ assert command_allowed(["df", "uptime"], "df") and command_allowed(["df"], "/usr/bin/df") and command_allowed(["/usr/bin/df"], "/usr/bin/df")+ assert not command_allowed(["df"], "dfx") and not command_allowed(["/usr/bin/df"], "/bin/df") and command_allowed(["*"], "rm")+ assert device_allowed(["/dev/ttyUSB*"], "/dev/ttyUSB0") and not device_allowed(["/dev/ttyUSB*"], "/dev/ttyACM0")+ assert topic_allowed(["home/+/temperature"], "home/kitchen/temperature") and not topic_allowed(["home/+/temperature"], "home/kitchen/humidity")+ assert topic_allowed(["home/#"], "home/a/b/c") and topic_allowed(["home/#"], "home") and not topic_allowed(["home/#"], "work/a")+ assert topic_allowed(["home/*"], "home/a") and not topic_allowed(["home/*"], "home/a/b") and topic_allowed(["#"], "x/y") and not topic_allowed(["home/a"], "home/a/b")+++def test_adapter_keys_are_stable():+ assert adapter_key("file", {"path": "/var/log/syslog", "tail": 3}) == "file /var/log/syslog"+ assert adapter_key("file", {"path": "/tmp/x", "append": "hi"}) == "file append /tmp/x"+ assert adapter_key("shell", {"command": "df", "args": ["-P", "/", 2]}) == "shell df -P / 2"+ assert adapter_key("serial", {"device": "/dev/ttyUSB0", "write": "AT\r\n"}) == 'serial /dev/ttyUSB0 "AT\\r\\n"'+ assert adapter_key("queue", {"url": "mqtt://b", "topic": "t", "publish": "on"}) == "queue publish mqtt://b t"+ assert adapter_key("browser", {"url": "https://x/"}) == "browser https://x/"+++# ---------------------------------------------------------------- file+++FILE_TREE = """+name = "notes"+description = "reads and appends a notes file"+paths = ["{root}"]+start = "read"++[inputs.path]+type = "str"++[nodes.read]+kind = "observe"+file = {{ path = "{{{{ inputs.path }}}}", tail = 2 }}+into = "f"+extract = [{{ into = "last", regex = "(?m)^(.*)$", group = 1, all = true }}]+next = "exists"++[nodes.exists]+kind = "decide"+on = "f.exists"+branches = [{{ when = {{ eq = true }}, then = "note" }}, {{ else = "none" }}]++[nodes.note]+kind = "act"+file = {{ path = "{{{{ inputs.path }}}}", append = "visited at {{{{ now.iso }}}}\\n" }}+into = "w"+next = "say"++[nodes.say]+kind = "done"+say = "{{{{ state.f.lines | length }}}} line(s), {{{{ state.f.size }}}} B; wrote {{{{ state.w.written }}}} B"++[nodes.none]+kind = "fail"+reason = "no-file"+say = "{{{{ inputs.path }}}} is not there"+"""+++def test_file_read_tail_and_append(tmp_path):+ reg = registry(tmp_path, notes=FILE_TREE.format(root=str(tmp_path / "data")))+ (tmp_path / "data").mkdir()+ f = tmp_path / "data" / "notes.txt"+ f.write_text("one\ntwo\nthree\n")+ out = Runner(reg).run_tree("notes", {"path": str(f)})+ assert out.status == "done", out.say+ assert out.say.startswith("2 line(s), 14 B; wrote ")+ assert f.read_text().startswith("one\ntwo\nthree\nvisited at 20")+ assert out.trace.steps[0].summary == f"file {f} -> 14 B; last=['two', 'three']"+ assert out.trace.steps[2].summary.startswith(f"file {f} -> wrote ")+ out = Runner(reg).run_tree("notes", {"path": str(tmp_path / "data" / "missing.txt")})+ assert out.status == "failed" and out.reason == "no-file" and "missing" in out.trace.steps[0].summary+ # Outside the tree's paths: refused at run time, by the runtime, before the adapter.+ out = Runner(reg).run_tree("notes", {"path": "/etc/hostname"})+ assert out.status == "error" and out.reason == "path-not-allowed" and "outside this tree's paths" in out.say+ # And ".." does not escape.+ out = Runner(reg).run_tree("notes", {"path": str(tmp_path / "data" / ".." / "secret")})+ assert out.reason == "path-not-allowed"+ # A whole file, and JSON parsed.+ reg2 = registry(tmp_path, j='''+name = "j"+paths = ["*"]+start = "r"+[nodes.r]+kind = "observe"+file = { path = "%s" }+into = "f"+next = "d"+[nodes.d]+kind = "done"+say = "{{ state.f.json.a }}"+''' % (tmp_path / "x.json"))+ (tmp_path / "x.json").write_text('{"a": 42}')+ assert Runner(reg2).run_tree("j").say == "42"+++def test_file_checker():+ base = '''+name = "f"+paths = ["/var/log"]+start = "r"+[nodes.r]+kind = "observe"+file = { path = "/var/log/syslog" }+next = "d"+[nodes.d]+kind = "done"+'''+ tree = load_tree_text(base)+ assert tree.sources() == ["file"] and tree.capability()["paths"] == ["/var/log"] and tree.capability()["sources"] == ["file"]+ with pytest.raises(TreeError, match="make this an act"):+ load_tree_text(base.replace('path = "/var/log/syslog"', 'path = "/var/log/syslog", append = "x"'))+ with pytest.raises(TreeError, match="not under this tree's paths"):+ load_tree_text(base.replace("/var/log/syslog", "/etc/passwd"))+ with pytest.raises(TreeError, match="must list the paths"):+ load_tree_text(base.replace('paths = ["/var/log"]', ""))+ with pytest.raises(TreeError, match="exactly one of"):+ load_tree_text(base.replace('file = { path = "/var/log/syslog" }', 'file = { path = "/var/log/syslog" }\nshell = { command = "df" }'))+ notes = [str(p) for p in check(load_tree_text(base.replace('paths = ["/var/log"]', 'paths = ["/var/log"]\ncommands = ["df"]')))]+ assert any("commands are listed but no node uses them" in n for n in notes)+ notes = [str(p) for p in check(load_tree_text(base.replace("/var/log/syslog", "/var/log/{{ inputs.name }}").replace('start = "r"', 'start = "r"\n[inputs.name]\ntype = "str"')))]+ assert any("decided at run time" in n for n in notes)+++# ---------------------------------------------------------------- shell+++SHELL_TREE = """+name = "say"+description = "runs a program"+commands = ["printf", "{py}", "false", "sleep"]+start = "run"++[inputs.text]+type = "str"+default = "hi"++[nodes.run]+kind = "observe"+shell = {{ command = "printf", args = ["%s|%s", "{{{{ inputs.text }}}}", "$HOME;ls"] }}+into = "p"+retries = 0+next = "ok"++[nodes.ok]+kind = "decide"+on = "p.ok"+branches = [{{ when = {{ eq = true }}, then = "say" }}, {{ else = "bad" }}]++[nodes.say]+kind = "done"+say = "{{{{ state.p.stdout }}}}"++[nodes.bad]+kind = "fail"+reason = "exit"+say = "exit {{{{ state.p.status }}}}: {{{{ state.p.stderr | strip }}}}"+"""+++def test_shell_runs_one_program_without_a_shell(tmp_path):+ reg = registry(tmp_path, say=SHELL_TREE.format(py=sys.executable))+ out = Runner(reg).run_tree("say", {"text": "a b"})+ assert out.status == "done" and out.say == "a b|$HOME;ls" # the arguments reached printf as given: no expansion, no second command+ assert out.trace.steps[0].summary == "shell printf %s|%s a b $HOME;ls -> exit 0"+ # Exit status is observed, not an error; stderr too; stdout as JSON when it is.+ reg2 = registry(tmp_path, say=SHELL_TREE.format(py=sys.executable).replace('shell = { command = "printf", args = ["%s|%s", "{{ inputs.text }}", "$HOME;ls"] }', 'shell = { command = "%s", args = ["-c", "import sys; print(\'{\\"n\\": 3}\'); sys.stderr.write(\'careful\'); sys.exit(2)"] }' % sys.executable).replace('say = "{{ state.p.stdout }}"', 'say = "{{ state.p.json.n }}"'))+ out = Runner(reg2).run_tree("say")+ assert out.status == "failed" and out.say == "exit 2: careful"+ # A program not on the box, or too slow, is a transport error (an observe retries; here retries = 0).+ reg3 = registry(tmp_path, say=SHELL_TREE.format(py=sys.executable).replace('command = "printf"', 'command = "{{ inputs.text }}"'))+ out = Runner(reg3).run_tree("say", {"text": "no-such-program-xyz"})+ assert out.status == "error" and out.reason == "command-not-allowed"+ out = Runner(reg3).run_tree("say", {"text": "sleep"}) # sleep "%s|%s" ... exits non-zero: fine, that is observed+ assert out.status == "failed"+ reg4 = registry(tmp_path, say=SHELL_TREE.format(py=sys.executable).replace('shell = { command = "printf", args = ["%s|%s", "{{ inputs.text }}", "$HOME;ls"] }', 'shell = { command = "sleep", args = ["5"] }\ntimeout = 0.3'))+ t0 = time.time()+ out = Runner(reg4).run_tree("say")+ assert out.status == "error" and out.reason == "transport" and "no answer within" in out.say and time.time() - t0 < 3+ with pytest.raises(TreeError, match="not in this tree's commands"):+ load_tree_text(SHELL_TREE.format(py=sys.executable).replace('command = "printf"', 'command = "rm"'))+ with pytest.raises(TreeError, match="must list the commands"):+ load_tree_text(SHELL_TREE.format(py=sys.executable).replace('commands = ["printf", "%s", "false", "sleep"]' % sys.executable, ""))+++# ---------------------------------------------------------------- serial+++SERIAL_TREE = """+name = "modem"+description = "asks a modem whether it is there"+devices = ["/dev/pts/*", "/dev/ttyUSB*"]+start = "at"++[inputs.device]+type = "str"++[nodes.at]+kind = "observe"+serial = { device = "{{ inputs.device }}", baud = 9600, write = "AT\\r\\n", read_until = "\\n" }+into = "s"+timeout = 2+retries = 0+extract = [{ into = "answer", regex = "(OK|ERROR)" }]+next = "ok"++[nodes.ok]+kind = "decide"+on = "answer"+branches = [{ when = { eq = "OK" }, then = "there" }, { else = "odd" }]++[nodes.there]+kind = "done"+say = "{{ inputs.device }} answers ({{ state.s.bytes }} B, sent {{ state.s.sent }})"++[nodes.odd]+kind = "fail"+reason = "odd-answer"+say = "{{ state.s.body | strip }}"+"""+++@pytest.mark.skipif(not hasattr(os, "openpty"), reason="no pty on this platform")+def test_serial_exchange_over_a_pty(tmp_path):+ reg = registry(tmp_path, modem=SERIAL_TREE)+ master, slave = os.openpty()+ device = os.ttyname(slave)+ got: list[bytes] = []++ def modem():+ buf = b""+ while b"\n" not in buf:+ buf += os.read(master, 64)+ got.append(buf)+ os.write(master, b"OK\r\n")++ t = threading.Thread(target=modem, daemon=True)+ t.start()+ try:+ out = Runner(reg).run_tree("modem", {"device": device})+ t.join(2)+ assert out.status == "done", out.say+ assert out.say == f"{device} answers (4 B, sent 4)" and got == [b"AT\r\n"]+ assert out.trace.steps[0].summary == f"serial {device} -> 4 B; answer=OK"+ # Nothing answers: a transport error, not a hang.+ t0 = time.time()+ out = Runner(reg).run_tree("modem", {"device": device})+ assert out.status == "error" and out.reason == "transport" and "no answer within" in out.say and time.time() - t0 < 5+ finally:+ os.close(master)+ os.close(slave)+ assert Runner(reg).run_tree("modem", {"device": "/dev/ttyACM0"}).reason == "device-not-allowed"+ out = Runner(reg).run_tree("modem", {"device": "/dev/ttyUSB99"})+ assert out.status == "error" and out.reason == "transport"+ with pytest.raises(TreeError, match="not in this tree's devices"):+ load_tree_text(SERIAL_TREE.replace('device = "{{ inputs.device }}"', 'device = "/dev/ttyACM0"'))+++# ---------------------------------------------------------------- queue (MQTT)+++class Broker:+ """Just enough of an MQTT 3.1.1 broker: one client at a time, a canned+ message for whoever subscribes, a record of what is published."""++ def __init__(self, message: bytes | None = b'{"temperature": 23.5}', refuse: int = 0):+ self.message = message+ self.refuse = refuse+ self.published: list[tuple[str, bytes]] = []+ self.subscribed: list[str] = []+ self.sock = socket.socket()+ self.sock.bind(("127.0.0.1", 0))+ self.sock.listen(5)+ self.port = self.sock.getsockname()[1]+ self.url = f"mqtt://127.0.0.1:{self.port}"+ threading.Thread(target=self._serve, daemon=True).start()++ def _serve(self):+ while True:+ try:+ conn, _ = self.sock.accept()+ except OSError:+ return+ threading.Thread(target=self._client, args=(conn,), daemon=True).start()++ def _client(self, conn: socket.socket):+ deadline = time.monotonic() + 10+ try:+ while True:+ pkt = _mqtt_read(conn, deadline)+ if pkt is None:+ return+ ptype, flags, body = pkt+ if ptype == 1:+ conn.sendall(_mqtt_packet(0x20, bytes([0, self.refuse])))+ if self.refuse:+ return+ elif ptype == 8:+ pid = body[:2]+ tlen = struct.unpack(">H", body[2:4])[0]+ topic = body[4:4 + tlen].decode()+ self.subscribed.append(topic)+ conn.sendall(_mqtt_packet(0x90, pid + b"\x00"))+ if self.message is not None:+ conn.sendall(_mqtt_packet(0x30, _mqtt_str(topic.replace("+", "kitchen")) + self.message))+ elif ptype == 3:+ tlen = struct.unpack(">H", body[:2])[0]+ self.published.append((body[2:2 + tlen].decode(), body[2 + tlen:]))+ elif ptype == 14:+ return+ finally:+ conn.close()++ def close(self):+ self.sock.close()+++QUEUE_TREE = """+name = "sensor"+description = "the last temperature a room reported"+hosts = ["127.0.0.1"]+topics = ["home/+/temperature", "home/+/light"]+start = "wait"++[inputs.room]+type = "str"+default = "kitchen"++[inputs.broker]+type = "str"++[nodes.wait]+kind = "observe"+queue = { url = "{{ inputs.broker }}", topic = "home/{{ inputs.room }}/temperature" }+into = "q"+timeout = 2+retries = 0+next = "got"++[nodes.got]+kind = "decide"+on = "q.received"+branches = [{ when = { eq = true }, then = "warm" }, { else = "quiet" }]++[nodes.warm]+kind = "decide"+on = "q.json.temperature"+branches = [{ when = { gt = 25 }, then = "cool-it" }, { else = "fine" }]++[nodes.cool-it]+kind = "act"+queue = { url = "{{ inputs.broker }}", topic = "home/{{ inputs.room }}/light", publish = { on = false } }+into = "sent"+next = "fine"++[nodes.fine]+kind = "done"+say = "{{ inputs.room }}: {{ state.q.json.temperature }} from {{ state.q.message.topic }}"++[nodes.quiet]+kind = "fail"+reason = "no-reading"+say = "nothing on home/{{ inputs.room }}/temperature within 2 s"+"""+++def test_queue_subscribe_wait_and_publish(tmp_path):+ reg = registry(tmp_path, sensor=QUEUE_TREE)+ b = Broker()+ try:+ out = Runner(reg).run_tree("sensor", {"broker": b.url})+ assert out.status == "done", out.say+ assert out.say == "kitchen: 23.5 from home/kitchen/temperature" and b.subscribed == ["home/kitchen/temperature"] and b.published == []+ assert out.trace.steps[0].summary == f"queue home/kitchen/temperature at {b.url} -> 1 message"+ hot = Broker(message=b'{"temperature": 31}')+ out = Runner(reg).run_tree("sensor", {"broker": hot.url})+ for _ in range(50): # the broker reads the publish on its own thread+ if hot.published:+ break+ time.sleep(0.02)+ assert out.status == "done" and hot.published == [("home/kitchen/light", b'{"on": false}')]+ assert out.trace.steps[3].summary == f"queue home/kitchen/light at {hot.url} -> published 13 B"+ hot.close()+ quiet = Broker(message=None)+ t0 = time.time()+ out = Runner(reg).run_tree("sensor", {"broker": quiet.url})+ assert out.status == "failed" and out.reason == "no-reading" and time.time() - t0 < 5+ quiet.close()+ refusing = Broker(refuse=5)+ out = Runner(reg).run_tree("sensor", {"broker": refusing.url})+ assert out.status == "error" and "refused (code 5)" in out.say+ refusing.close()+ # Allow-lists: the broker's host in hosts, the topic in topics.+ assert Runner(reg).run_tree("sensor", {"broker": "mqtt://broker.example"}).reason == "host-not-allowed"+ assert Runner(reg).run_tree("sensor", {"broker": b.url, "room": "a/b"}).reason == "topic-not-allowed"+ assert Runner(reg).run_tree("sensor", {"broker": "http://127.0.0.1/"}).reason == "host-not-allowed"+ finally:+ b.close()+ with pytest.raises(TreeError, match="make this an act"):+ load_tree_text(QUEUE_TREE.replace('topic = "home/{{ inputs.room }}/temperature" }', 'topic = "home/{{ inputs.room }}/temperature", publish = "x" }'))+ with pytest.raises(TreeError, match="must list the topics"):+ load_tree_text(QUEUE_TREE.replace('topics = ["home/+/temperature", "home/+/light"]', ""))+ with pytest.raises(TreeError, match="not under this tree's topics"):+ load_tree_text(QUEUE_TREE.replace('topic = "home/{{ inputs.room }}/light"', 'topic = "work/light"'))+++# ---------------------------------------------------------------- egress, fixtures, the command+++def test_egress_public_keeps_the_box_closed(tmp_path):+ hosted = LocalAdapters(egress="public")+ for source, spec in (("file", {"path": "/etc/hostname"}), ("shell", {"command": "uptime"}), ("serial", {"device": "/dev/ttyUSB0"})):+ with pytest.raises(EgressRefused, match="its own box"):+ hosted.perform(source, spec, 1.0)+ with pytest.raises(EgressRefused):+ hosted.perform("queue", {"url": "mqtt://127.0.0.1", "topic": "t"}, 1.0)+ with pytest.raises(EgressRefused):+ hosted.perform("browser", {"url": "http://10.0.0.1/"}, 1.0)+ # Through a tree: the refusal is an error the trace shows, and an observe does not retry it.+ reg = registry(tmp_path, say=SHELL_TREE.format(py=sys.executable))+ out = Runner(reg, adapters=hosted).run_tree("say")+ assert out.status == "error" and out.reason == "transport" and "its own box" in out.say+++def test_fixtures_record_and_replay_every_source(tmp_path, capsys):+ reg = registry(tmp_path, say=SHELL_TREE.format(py=sys.executable), notes=FILE_TREE.format(root=str(tmp_path)))+ (tmp_path / "n.txt").write_text("a\nb\n")+ rec = RecordingTransport(None, LocalAdapters()) # type: ignore[arg-type]+ runner = Runner(reg, rec)+ assert runner.run_tree("say", {"text": "x"}).status == "done"+ assert runner.run_tree("notes", {"path": str(tmp_path / "n.txt")}).status == "done"+ assert set(rec.recorded) == {"shell printf %s|%s x $HOME;ls", f"file {tmp_path / 'n.txt'}", f"file append {tmp_path / 'n.txt'}"}+ rec.save(tmp_path / "fx.json")+ replay = RecordedTransport.from_file(tmp_path / "fx.json")+ out = Runner(reg, replay).run_tree("say", {"text": "x"})+ assert out.status == "done" and out.say == "x|$HOME;ls" and replay.calls == [("shell", "shell printf %s|%s x $HOME;ls")]+ out = Runner(reg, replay).run_tree("say", {"text": "unrecorded"})+ assert out.status == "error" and "no recorded result for shell printf %s|%s unrecorded" in out.say+ # The command records and replays the same way.+ code = main(["run", "--tree", "say", "--input", "text=cli", "--trees", str(tmp_path / "trees"), "--record", str(tmp_path / "cli.json"), "--json", "--no-trace"])+ assert code == 0 and json.loads(capsys.readouterr().out)["say"] == "cli|$HOME;ls"+ assert "shell printf %s|%s cli $HOME;ls" in json.loads((tmp_path / "cli.json").read_text())+ code = main(["run", "--tree", "say", "--input", "text=cli", "--trees", str(tmp_path / "trees"), "--replay", str(tmp_path / "cli.json"), "--json", "--no-trace"])+ assert code == 0 and json.loads(capsys.readouterr().out)["say"] == "cli|$HOME;ls"+ # A service with --egress public refuses the box; a recorded transport answers for the sources too.+ from pyrtan.service import Service++ svc = Service(reg, egress="public")+ assert svc.run({"tree": "say", "inputs": {}})["reason"] == "transport"+ svc = Service(reg, transport=replay)+ assert svc.run({"tree": "say", "inputs": {"text": "x"}})["status"] == "done"+++# ---------------------------------------------------------------- browser+++PAGE = b"""<!doctype html><html><head><title>App</title></head><body>+<div id="root">loading</div><a id="away" href="%s">away</a>+<script>setTimeout(() => { document.getElementById("root").innerHTML = "<h1>Rendered</h1><button id=more>more</button>";+document.getElementById("more").onclick = () => { document.getElementById("root").insertAdjacentHTML("beforeend", "<p class=extra>extra</p>"); }; }, 50);</script>+</body></html>"""++BROWSER_TREE = """+name = "app"+description = "a page that needs a script to render"+hosts = ["127.0.0.1"]+start = "open"++[inputs.url]+type = "str"++[nodes.open]+kind = "observe"+browser = { url = "{{ inputs.url }}", wait_for = "h1", actions = [{ click = "#more" }, { wait_for = ".extra" }] }+into = "page"+timeout = 15+retries = 0+extract = [{ into = "heading", css = "h1" }, { into = "extra", css = "p.extra" }]+next = "say"++[nodes.say]+kind = "done"+say = "{{ state.page.title }}: {{ state.heading }} / {{ state.extra }} ({{ state.page.status }})"+"""+++@pytest.fixture+def pages():+ class H(http.server.BaseHTTPRequestHandler):+ def do_GET(self):+ self.send_response(200)+ self.send_header("Content-Type", "text/html")+ self.end_headers()+ self.wfile.write(PAGE % f"http://localhost:{self.server.server_port}/elsewhere".encode())++ def log_message(self, *a):+ pass++ srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), H)+ threading.Thread(target=srv.serve_forever, daemon=True).start()+ yield f"http://127.0.0.1:{srv.server_port}"+ srv.shutdown()+++def test_browser_renders_acts_and_stays_on_its_hosts(tmp_path, pages):+ pytest.importorskip("playwright")+ adapters = LocalAdapters()+ try:+ adapters.perform("browser", {"url": pages + "/"}, 15.0)+ except TransportError as e:+ pytest.skip(str(e))+ reg = registry(tmp_path, app=BROWSER_TREE)+ try:+ out = Runner(reg, adapters=adapters).run_tree("app", {"url": pages + "/app"})+ assert out.status == "done", out.say+ assert out.say == "App: Rendered / extra (200)"+ assert out.trace.steps[0].summary == f"browser {pages}/app -> 200; heading=Rendered, extra=extra"+ # A click that would take the page to another host is refused; so is a URL outside hosts, before any browser starts.+ reg2 = registry(tmp_path, app=BROWSER_TREE.replace("{ click = \"#more\" }", "{ click = \"#away\" }"))+ out = Runner(reg2, adapters=adapters).run_tree("app", {"url": pages + "/app"})+ assert out.status == "error" and out.reason == "transport" and "outside this tree's hosts" in out.say+ out = Runner(reg, adapters=adapters).run_tree("app", {"url": "http://localhost:1/"})+ assert out.reason == "host-not-allowed"+ finally:+ adapters.close()+ with pytest.raises(TreeError, match="exactly one of click"):+ load_tree_text(BROWSER_TREE.replace('{ click = "#more" }', '{ tap = "#more" }'))+ with pytest.raises(TreeError, match="not in this tree's hosts"):+ load_tree_text(BROWSER_TREE.replace('url = "{{ inputs.url }}"', 'url = "https://example.com/"'))diff --git a/tests/test_catalog.py b/tests/test_catalog.py index 8319aa4..7831913 100644 --- a/tests/test_catalog.py +++ b/tests/test_catalog.py@@ -39,7 +39,7 @@ def served(catalog_dir):def test_build_index_describes_every_tree(catalog_dir): index = json.loads((catalog_dir / "index.json").read_text())- assert index["pyrtan-catalog"] == 1 and index["count"] == 5+ assert index["pyrtan-catalog"] == 1 and index["count"] == 10weather = next(e for e in index["trees"] if e["name"] == "weather") assert weather["path"] == "weather/weather.toml" and weather["license"] == "MIT" assert weather["sha256"] == hashlib.sha256((catalog_dir / "weather/weather.toml").read_bytes()).hexdigest()@@ -67,7 +67,7 @@ def test_build_index_leaves_out_what_does_not_belong(tmp_path):def test_search_and_find(served): cat = Catalog(served)- assert {e["name"] for e in cat.search("")} == {"weather", "hacker-news", "hacker-news-item", "site-status", "github-release"}+ assert {e["name"] for e in cat.search("")} == {"weather", "hacker-news", "hacker-news-item", "site-status", "github-release", "disk-space", "log-errors", "room-temperature", "rendered-headline", "modem-check"}assert [e["name"] for e in cat.search("open-meteo")] == ["weather"] assert [e["name"] for e in cat.search("news stories")] == ["hacker-news"] assert cat.search("nothing like this") == []@@ -129,7 +129,7 @@ def test_not_a_catalog(served, tmp_path):def test_cli_index_search_pull(catalog_dir, served, tmp_path, capsys): out = tmp_path / "idx.json" assert main(["index", str(catalog_dir), "--out", str(out)]) == 0- assert json.loads(out.read_text())["count"] == 5+ assert json.loads(out.read_text())["count"] == 10assert main(["search", "weather", "--catalog", served]) == 0 assert "weather@1" in capsys.readouterr().out assert main(["search", "zzz", "--catalog", served]) == 1 diff --git a/tests/test_examples.py b/tests/test_examples.py index 1e02dd4..d6741b8 100644 --- a/tests/test_examples.py +++ b/tests/test_examples.py@@ -86,3 +86,54 @@ def test_github_release(runner):def test_fixture_fails_loudly_on_unforeseen_request(runner): out = runner.run_intent("weather in Paris") assert out.status == "error" and out.reason == "transport"+++def test_disk_space_over_a_shell(runner):+ out = runner.run_intent("how full is the disk")+ assert out.status == "done" and out.say == "/ is 71% full." and out.output == {"mounted": "/", "percent": 71}+ assert out.trace.steps[1].summary == "shell df -P / -> exit 0; percent=71, mounted=/"+ assert runner.run_intent("disk space on /data").reason == "disk-full"+ failed = runner.run_intent("disk space on /nowhere")+ assert failed.status == "failed" and failed.reason == "df-failed" and "No such file" in failed.say+++def test_log_errors_from_a_file(runner):+ out = runner.run_intent("errors in the log")+ assert out.status == "done" and out.say.startswith("2 error line(s) in the last 5 of /var/log/syslog; latest: ") and out.output["errors"] == 2+ assert runner.run_intent("errors in /var/log/quiet.log").say == "No error lines in the last 1 of /var/log/quiet.log."+ assert runner.run_intent("errors in /var/log/nothing.log").reason == "no-such-file"+ # The path from the sentence is checked against the tree's paths before anything is read.+ out = runner.run_intent("errors in /etc/passwd")+ assert out.status == "error" and out.reason == "path-not-allowed" and [s.kind for s in out.trace.steps] == ["intent"]+++def test_room_temperature_from_a_queue(runner):+ assert runner.run_intent("temperature in the kitchen").say == "kitchen: 21.5°C."+ assert runner.run_intent("how warm is the attic").say == "attic: 31°C, warm."+ assert runner.run_intent("temperature in the garage").reason == "no-reading"+ gap = runner.run_intent("temperature in the hall") # a payload that is not JSON: a gap, not a guess+ assert gap.status == "error" and gap.reason == "gap" and gap.trace.gaps[0]["node"] == "reading"+ assert runner.run_tree("room-temperature", {"room": "a-b"}).status == "error" # fixture has nothing for it+++def test_rendered_headline_through_a_browser(runner):+ out = runner.run_intent("headline of https://app.example/dashboard")+ assert out.status == "done" and out.say == "Dashboard — Welcome back, Barry" and out.output["heading"] == "Welcome back, Barry"+ assert out.trace.steps[1].summary == "browser https://app.example/dashboard -> 200; heading=Welcome back, Barry"+ assert runner.run_intent("what does https://app.example/empty say").reason == "no-heading"+ assert runner.run_intent("headline of https://app.example/gone").reason == "unexpected-response"+++def test_modem_check_over_serial(runner):+ assert runner.run_intent("is the modem up").say == "The modem on /dev/ttyUSB0 is there."+ silent = runner.run_intent("is the modem on /dev/ttyUSB1 up")+ assert silent.status == "error" and silent.reason == "transport" and "no answer within 3 s" in silent.say+ assert runner.run_intent("is the modem on /dev/ttyACM0 up").reason == "modem-error"+ assert runner.run_intent("is the modem on /dev/ttyS0 up").reason == "device-not-allowed"+++def test_capabilities_name_the_sources_and_lists(examples):+ caps = {t["name"]: t for t in examples.capabilities()}+ assert caps["disk-space"]["sources"] == ["shell"] and caps["disk-space"]["commands"] == ["df"] and "paths" not in caps["disk-space"]+ assert caps["room-temperature"]["sources"] == ["queue"] and caps["room-temperature"]["topics"] == ["home/+/temperature"]+ assert caps["weather"]["sources"] == ["http"] and caps["modem-check"]["devices"] == ["/dev/ttyUSB*", "/dev/ttyACM*"]diff --git a/tests/test_federation.py b/tests/test_federation.py index b764380..6634bde 100644 --- a/tests/test_federation.py +++ b/tests/test_federation.py@@ -675,6 +675,18 @@ def test_trusted_member_runs_the_trees_its_hub_hands_out(tmp_path):(tmp_path / "h").mkdir() (tmp_path / "t" / "alpha.toml").write_text(ECHO.format(name="alpha", word="alpha")) (tmp_path / "h" / "gamma.toml").write_text(ECHO.format(name="gamma", word="gamma"))+ (tmp_path / "h" / "box.toml").write_text('''+name = "box"+commands = ["uptime"]+start = "up"+[nodes.up]+kind = "observe"+shell = { command = "uptime" }+next = "say"+[nodes.say]+kind = "done"+say = "{{ state.last.stdout }}"+''')from pyrtan.hub import serve_hub hub, srv = serve_hub(port=0, quiet=True, name="solo", trees=[tmp_path / "t"], handout=[tmp_path / "h"], federate=True, token="tok", poll_every=60)@@ -692,7 +704,7 @@ def test_trusted_member_runs_the_trees_its_hub_hands_out(tmp_path):# The hub hands out its own trees and the --handout ones at once, in the mailbox after the Accept and the Announce. acts = phone.poll() assert [a["type"] for a in acts] == ["Accept", "Announce", "Keep", "Keep"]- assert sorted(phone.trees) == ["alpha", "gamma"] and phone.trees["gamma"]["name"] == "gamma"+ assert sorted(phone.trees) == ["alpha", "gamma"] and phone.trees["gamma"]["name"] == "gamma" # not "box": a phone has no shellassert all(h["confirmed"] for h in fed._handed[phone.id].values()) assert fed.hand_out() == 0 # nothing new to give # What the member says it runs is filtered to wh
The diff is cut here; the rest is in the clone.