The catalog grows up: namespaces, publishing by pull request through the gate, gap reports gossiped back to authors; release 0.8.0
Claude · 2026-10-26 · 5ec462740cf1c355d86032135148cad715fde327 · parents 62cd55b
A catalog tree now says who publishes it: a `namespace` line, and the catalog id is namespace/name. The examples are the pyrtan namespace; everyone else publishes under catalog/<namespace>/<name>.toml with an OWNERS file of GitHub logins. `pyrtan pull weather` takes the one there is and asks when two namespaces have it; `pyrtan pull barry/weather` says which. `pyrtan index DIR... --copy OUT` builds one index over several directories and lays the files out as namespace/name.toml, the shape the site serves; entries carry id, namespace, sources and the allow-lists.
Publishing is a pull request. `pyrtan publish FILE --into CHECKOUT --owner LOGIN` places the file, writes OWNERS for a new namespace and runs the gate (catalog.gate, site/gate.py, the new catalog workflow on every pull request): every tree loads clean with a licence and a namespace, sits where its namespace and name say, ids are unique, only a namespace's owners change it, a changed file bumps its version. The pages build runs the gate before indexing.
Gaps come back to the author as a federation topic. A hub started with --report-gaps ($PYRTAN_REPORT_GAPS) shapes the gaps its catalog trees met (namespace/name@version, node, the path decided on, the value no branch covered, count, when; never inputs; long or non-scalar values become their type) into its announces; every federated hub keeps what it hears with the directory's merge rules (newer wins, no future, 30 days, 2000 rows), passes it on, persists it, and serves /reports publicly with counts and reporters summed. site/snapshot.py bakes rendezvous/reports.json from the seeds hourly, the catalog page shows each tree's gaps beside it, and `pyrtan reports NAMESPACE/NAME` reads a hub or the baked file. A tree without a namespace reports nothing.
Every example is bumped to version 2 (the file changed: it names its namespace), as the gate asks of anyone. pyrtan.js mirrors the namespace. Tests: namespaced index and pulls, the gate's every refusal, publish, the command, reports travelling with announces, shaping, persistence, the snapshot and the command. Docs: catalog/README.md, README, site, CONTRIBUTING, tree format, design, platform §2.8, deploy.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YChr78vDHtQJ15HNbsaRLZ
Changes
.github/workflows/catalog.yml | 31 +++++ .github/workflows/pages.yml | 5 +- CONTRIBUTING.md | 26 +++- README.md | 60 ++++++--- _meta/design.md | 10 +- _meta/platform.md | 53 +++++++- _meta/tree-format.md | 1 + catalog/README.md | 21 +++ deploy/openmediavault/docker-compose.yml | 1 + examples/disk-space/disk-space.toml | 3 +- examples/github/github-release.toml | 3 +- examples/hacker-news/hacker-news-item.toml | 3 +- examples/hacker-news/hacker-news.toml | 3 +- examples/log-errors/log-errors.toml | 3 +- examples/modem-check/modem-check.toml | 3 +- examples/rendered-headline/rendered-headline.toml | 3 +- examples/room-temperature/room-temperature.toml | 3 +- examples/site-status/site-status.toml | 3 +- examples/weather/weather.toml | 3 +- pyproject.toml | 2 +- site/README.md | 25 +++- site/build.sh | 15 ++- site/catalog/index.html | 17 ++- site/gate.py | 69 ++++++++++ site/snapshot.py | 57 ++++++++ src/pyrtan/__init__.py | 2 +- src/pyrtan/catalog.py | 295 +++++++++++++++++++++++++++++++++++------- src/pyrtan/cli.py | 103 ++++++++++++--- src/pyrtan/federation.py | 104 ++++++++++++++- src/pyrtan/hub.py | 37 +++++- src/pyrtan/js/pyrtan.js | 9 +- src/pyrtan/service.py | 2 +- src/pyrtan/transport.py | 2 +- src/pyrtan/tree.py | 12 +- tests/conformance/vectors.json | 30 +++-- tests/test_catalog.py | 139 +++++++++++++++++++- tests/test_federation.py | 95 +++++++++++++- uv.lock | 2 +- 38 files changed, 1097 insertions(+), 158 deletions(-)
diff --git a/.github/workflows/catalog.yml b/.github/workflows/catalog.yml new file mode 100644 index 0000000..d1f16e9 --- /dev/null +++ b/.github/workflows/catalog.yml@@ -0,0 +1,31 @@+# The catalog gate: every pull request that touches a tree runs the same+# checks `pyrtan publish` ran locally, with what git knows added: which+# files changed, who opened the pull request, and what version each tree+# had on main. A maintainer merges when it is green; the pages workflow+# publishes on merge.++name: catalog++on:+ pull_request:+ paths:+ - "catalog/**"+ - "examples/**"+ - "site/gate.py"+ - "src/pyrtan/catalog.py"++permissions:+ contents: read++jobs:+ gate:+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v4+ with:+ fetch-depth: 0+ - uses: astral-sh/setup-uv@v5+ - run: uv sync+ - run: uv run pytest -q+ - run: uv run pyrtan check examples catalog --quiet+ - run: uv run python site/gate.py --base "origin/${{ github.base_ref }}" --author "${{ github.event.pull_request.user.login }}"diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index 4330fcd..80ca37e 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml@@ -1,7 +1,8 @@# pyrtan.com: this repository's GitHub Pages site. On every push to main # and once an hour, build the landing page, the console (from this-# checkout's pyrtan), the catalog (from examples/), the seed list and a-# snapshot of the directory as the seeds know it, and publish them. Set the+# checkout's pyrtan), the catalog (from examples/ and catalog/, through the+# gate), the seed list and a snapshot of the directory and of the gap+# reports as the seeds know them, and publish them. Set the# repository's Pages source to "GitHub Actions" and its custom domain to # pyrtan.com; site/CNAME carries the domain. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e41fba5..904012f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md@@ -25,11 +25,27 @@ A tree that reads a page should prefer an API or a feed when the site offersone, and should decide on the status code before it decides on the body. A tree meant for the catalog at pyrtan.com carries a `license` line (an-SPDX identifier; `MIT`, `Apache-2.0` or `CC0-1.0` are the usual choices).-The catalog *is* `examples/`: the `pages` workflow indexes it and-publishes it at <https://pyrtan.com/catalog/> on every push to `main`,-so a tree merged here with a licence and a test is pullable the same-day. See [`site/README.md`](site/README.md).+SPDX identifier; `MIT`, `Apache-2.0` or `CC0-1.0` are the usual choices)+and a `namespace` line (who publishes it; the catalog id is+`namespace/name`). The repository's own examples are the `pyrtan`+namespace. Everyone else publishes under `catalog/<namespace>/`:++ pyrtan publish my-tree.toml --into . --owner <your GitHub login>+ git checkout -b catalog/<namespace>-my-tree && git add catalog && git commit && git push -u origin HEAD++then open a pull request. The `catalog` workflow runs the gate+(`site/gate.py`): the tree loads clean, has a licence and a namespace,+sits in its namespace's directory under its own name, no other tree has+its id, only the namespace's `OWNERS` change it, and a changed file bumps+its `version`. A test and a recording are welcome but not required for+a namespace of your own; the gate is the review. A maintainer merges,+and the `pages` workflow publishes it at <https://pyrtan.com/catalog/>+the same day. See [`catalog/README.md`](catalog/README.md) and+[`site/README.md`](site/README.md).++Once it is out there, `pyrtan reports <namespace>/my-tree` shows the gaps+hubs that opted in have met with it: the node, the value no branch+covered, how often. That is the list of branches to write next.## Changing the runtime diff --git a/README.md b/README.md index 2e12737..353753c 100644 --- a/README.md +++ b/README.md@@ -268,21 +268,44 @@ shows who follows whom. The design is in## The catalog Trees other people wrote are at <https://pyrtan.com/catalog/>, a directory-of static files anyone can mirror.+of static files anyone can mirror. Every tree there is `namespace/name`:+the namespace is who published it, so two people can each have a+`weather`.``` pyrtan search weather-pyrtan pull weather --dir trees+pyrtan pull pyrtan/weather --dir trees # a bare name is enough when one namespace has itpyrtan serve trees ``` A pull checks the file's hash against the catalog's index, loads it as a-tree, and checks the name and version inside before writing it; a file you-have edited is not overwritten without `--force`. Point `--catalog` (or-`$PYRTAN_CATALOG`) at a directory or URL of your own to use a private-catalog; `pyrtan index DIR` builds one from any folder of trees. A tree in-a catalog says what others may do with it in a `license` line (an SPDX-identifier such as `MIT`).+tree, and checks the name, namespace and version inside before writing+it; a file you have edited is not overwritten without `--force`. Point+`--catalog` (or `$PYRTAN_CATALOG`) at a directory or URL of your own to+use a private catalog; `pyrtan index DIR...` builds one from any folders+of trees (`--copy` lays them out as `namespace/name.toml`). A tree in a+catalog says what others may do with it in a `license` line (an SPDX+identifier such as `MIT`) and who publishes it in a `namespace` line.++**Publishing is a pull request.** `pyrtan publish my-tree.toml --into+<checkout of this repository> --owner <your GitHub login>` puts the file+under `catalog/<namespace>/`, writes the namespace's `OWNERS` if it is+new, and runs the gate; push the branch and open a pull request. The+`catalog` workflow runs the same gate (every tree loads clean with a+licence and a namespace, sits where its namespace and name say, ids are+unique, only a namespace's owners change it, a changed file bumps its+`version`), a maintainer merges, and the next build serves it. No+account, no API, no model in the review: [`catalog/README.md`](catalog/README.md).++**Gaps come back to the author.** A hub started with `--report-gaps`+puts the gaps its catalog trees meet (the tree's `namespace/name@version`,+the node, the path decided on and the value no branch covered, with a+count; never the inputs) into its federation announces. Every federated+hub carries and gossips them like the directory, `/reports` shows them,+pyrtan.com bakes them hourly into `rendezvous/reports.json`, and the+catalog page shows each tree's beside it, so `pyrtan reports+pyrtan/weather` tells an author where reality disagreed with their tree+across every swarm that opted in.## Leaving the LAN@@ -405,16 +428,17 @@ site/ pyrtan.com: the landing page, the seed list, the catalog## Status-0.7.0. The core, the service, the hub, the command, the console, the-catalog client, the language rules, federation between hubs, the-gossiped directory, members (a browser or a phone as part of the swarm),-the blobs they keep and the trees they run (`pyrtan.js`, conformant with-the Python runtime), and six forms of I/O (HTTP, a browser, files,-programs, serial lines, MQTT queues), each behind an allow-list, are-built and tested; a swarm can require a token, keep its gaps, refuse-private addresses and its own box, and follow other swarms. Not yet-built: persistent storage for unanswered sentences, `pyrtan watch`, and-hosting.+0.8.0. The core, the service, the hub, the command, the console, the+catalog (namespaces, publishing by pull request through the gate, gap+reports gossiped back to authors), the language rules, federation+between hubs, the gossiped directory, members (a browser or a phone as+part of the swarm), the blobs they keep and the trees they run+(`pyrtan.js`, conformant with the Python runtime), and six forms of I/O+(HTTP, a browser, files, programs, serial lines, MQTT queues), each+behind an allow-list, are built and tested; a swarm can require a token,+keep its gaps, refuse private addresses and its own box, and follow+other swarms. Not yet built: persistent storage for unanswered+sentences, `pyrtan watch`, and hosting.See [`_meta/design.md`](_meta/design.md) for the runtime's design and [`_meta/platform.md`](_meta/platform.md) for pyrtan.com. diff --git a/_meta/design.md b/_meta/design.md index b965f93..5341a4b 100644 --- a/_meta/design.md +++ b/_meta/design.md@@ -306,9 +306,13 @@ Five trees, all against real public APIs, all tested offline:rules it reads a page, and the trace reads the same. One fixture answers for every source, in both runtimes. Design in [`io.md`](io.md).-8. **v0.8: the catalog grows up.** Namespaces, publishing through pull- requests with the checker as the gate, and opt-in gap reporting from- hubs back to a tree's author, carried as another federation topic.+8. **v0.8, built: the catalog grows up.** Namespaces (`namespace/name`,+ the tree's own line), publishing through pull requests with the+ checker as the gate (`pyrtan publish`, `site/gate.py`, the `catalog`+ workflow, `OWNERS` per namespace), and opt-in gap reporting+ (`--report-gaps`) carried as a federation topic like the directory,+ baked hourly into pyrtan.com and shown beside each catalog tree.+ Design in [`platform.md`](platform.md) §2.8.9. **Beside the swarm, never inside it:** an authoring aid that reads gaps and drafts the branch for a person to accept. diff --git a/_meta/platform.md b/_meta/platform.md index 9f58ac2..1024a7f 100644 --- a/_meta/platform.md +++ b/_meta/platform.md@@ -2,7 +2,7 @@| | | |---|---|-| **Status** | **Phase 1 built, v0.2.** Tokens, CORS, persistent gaps, the egress policy, the catalog client and the console are in this repository and tested. The site is this repository's GitHub Pages deployment, described in [`../site/README.md`](../site/README.md). Hosting is designed, not built. |+| **Status** | **Phases 1 and 2 built (v0.2, v0.8).** Tokens, CORS, persistent gaps, the egress policy, the catalog client and the console (Phase 1); namespaces, publishing by pull request through the gate, and gap reports gossiped back to authors (Phase 2, §2.8). The site is this repository's GitHub Pages deployment, described in [`../site/README.md`](../site/README.md). Hosting is designed, not built. || **Date** | 2026-10-25 | | **Decision** | pyrtan.com is three separate things on one invariant: a static catalog of trees anyone can pull, a static console that talks only to the public hub API, and (later) paid hosting that runs the unmodified open-source hub per tenant. A caller cannot tell a hosted hub from a self-hosted one, so every piece works for both. | | **Where** | This repository for everything open: the runtime, the client, the console, the catalog (`examples/`) and the site (`site/`, published by the `pages` workflow). A private repository for the control plane, when there is one. |@@ -100,6 +100,46 @@ one; `pyrtan index` leaves out a tree without one unless told otherwise.Trees are data, not part of the program, so the AGPL does not reach them; a declared licence is what lets a stranger use what they pulled.+Since v0.8 a catalog tree also carries a `namespace`: who publishes it.+The id is `namespace/name`; `pyrtan pull weather` takes the one there is+and asks when two namespaces have it; `pyrtan pull barry/weather` says+which. The index is built over several directories with `--copy`, which+lays the files out as `namespace/name.toml` so the served catalog has one+shape whatever the sources look like.++### 2.8 Publishing and gap reports (`catalog.py`, `federation.py`; built, v0.8)++**Publishing is a pull request.** The repository holds `examples/` (the+`pyrtan` namespace) and `catalog/<namespace>/<name>.toml`, each namespace+with an `OWNERS` file of GitHub logins. `pyrtan publish FILE --into+CHECKOUT --owner LOGIN` places the file, writes `OWNERS` for a new+namespace, and runs the gate; the `catalog` workflow runs the same gate+on every pull request with what git adds: the changed files, the+author, the version each tree had on `main`. The gate (`catalog.gate`,+`site/gate.py`) refuses a tree that does not load clean, lacks a licence+or a namespace, sits under the wrong directory or file name, shares an id+with another, is changed by someone not in its namespace's `OWNERS`, or+changed without a version bump. A maintainer merges; the `pages`+workflow publishes. There is no account, no API and no model in the+review; the "first small API" Phase 2 expected turned out to be GitHub's+pull request, which already exists.++**Gaps come back to the author** as a federation topic, the way the+directory does. A hub started with `--report-gaps` shapes the gaps its+catalog trees met into reports (`namespace/name@version`, node, the path+decided on, the value no branch covered, count, when; a value longer+than eighty characters or not a scalar becomes its type; never the+inputs or the rest of the state) and puts them in its announces. Every+federated hub keeps what it hears, one row per reporter and gap, with+the merge rules the directory uses (newer wins, nothing from the future,+thirty days, two thousand rows), passes it on, and serves `/reports`+publicly with counts and reporters summed. The site's snapshot asks the+seeds for `/reports` hourly and bakes `rendezvous/reports.json`; the+catalog page shows each tree's gaps beside it; `pyrtan reports+NAMESPACE/NAME` reads a hub or the baked file. A tree without a namespace+reports nothing: there is nobody to report to. Opting in is the+operator's, per hub, and off by default.+### 2.5 The console (`page.py`; built) One HTML file, no build, no external assets. Served by a hub or service@@ -152,11 +192,12 @@ fixtures and runs the same commands on their own box.repository. The site: pyrtan.com as static pages, `/console/` from `pyrtan console --out`, `/catalog/` from the catalog repository's CI. Hosting needs nothing yet.-2. **Phase 2.** Namespaces in the catalog (`someone/tree`), publishing- through pull requests with the checker as the gate, and opt-in gap- reporting: a hub that holds a catalog tree can send that tree's gaps to- the catalog, so an author learns where reality disagreed with their- tree across every hub that runs it. This needs the first small API.+2. **Phase 2 (built, v0.8).** Namespaces in the catalog+ (`someone/tree`), publishing through pull requests with the checker as+ the gate, and opt-in gap reporting carried through the federation to+ pyrtan.com, so an author learns where reality disagreed with their+ tree across every hub that runs it. No API was needed after all: the+ pull request and the gossip were enough (§2.8).3. **Phase 3.** The control plane: per-tenant hubs, secrets, hosted watch, retention, billing. diff --git a/_meta/tree-format.md b/_meta/tree-format.md index 236ec42..f92d95c 100644 --- a/_meta/tree-format.md +++ b/_meta/tree-format.md@@ -17,6 +17,7 @@ A tree is one TOML file. This is the whole of it.| `start` | yes | The first node. | | `tags` | no | Free strings, for people and for the hub's listing. | | `license` | for a catalog | An SPDX identifier such as `MIT`, `Apache-2.0` or `CC0-1.0`: what others may do with the file. A tree without one runs anywhere but is left out of a catalog. |+| `namespace` | for a catalog | Who publishes the tree (lowercase letters, digits, dashes). The catalog id is `namespace/name`; a hub reports a tree's gaps to its author only when it has one. || `triggers` | no | Sentences the tree answers to (below). | | `inputs` | no | What the tree needs (below). | | `nodes` | yes | The nodes (below). | diff --git a/catalog/README.md b/catalog/README.md new file mode 100644 index 0000000..82fdda5 --- /dev/null +++ b/catalog/README.md@@ -0,0 +1,21 @@+# The catalog++Trees published here are served at <https://pyrtan.com/catalog/> beside+the repository's own examples (which are the `pyrtan` namespace). One+directory per namespace, one `<name>.toml` per tree, and an `OWNERS` file+naming the GitHub logins that may change the namespace:++ catalog/+ barry/+ OWNERS # one login per line+ lan-printer.toml # namespace = "barry", license = "MIT"++`pyrtan publish my-tree.toml --into <this checkout> --owner <your login>`+puts a tree in the right place and runs the gate; then open a pull+request. The `catalog` workflow runs the same gate on the pull request:+every tree loads clean, carries `license` and `namespace`, sits in its+namespace's directory under its own name, ids are unique, only owners+change a namespace, and a changed file bumps its `version`. A maintainer+merges; the `pages` workflow publishes on merge. There is no account and+no API: the pull request is the publishing. See+[`site/README.md`](../site/README.md).diff --git a/deploy/openmediavault/docker-compose.yml b/deploy/openmediavault/docker-compose.yml index 9f92f0c..ee04cda 100644 --- a/deploy/openmediavault/docker-compose.yml +++ b/deploy/openmediavault/docker-compose.yml@@ -39,6 +39,7 @@ x-pyrtan: &pyrtan# PYRTAN_ADVERTISE: "https://hub.example.net" # PYRTAN_FOLLOW: "seed" # PYRTAN_LISTED: "true"+ # PYRTAN_REPORT_GAPS: "true" # opt in: the gaps your catalog trees meet (tree, node, value; never inputs) reach their authors via pyrtan.comPYRTAN_STATE: /state/federation.json PYRTAN_STORE: /state/store # blobs: the hub's own and the ones it keeps for others PYRTAN_REPLICAS: "2" # how many members or hubs should keep each of its blobs diff --git a/examples/disk-space/disk-space.toml b/examples/disk-space/disk-space.toml index 8baa73b..3e17af2 100644 --- a/examples/disk-space/disk-space.toml +++ b/examples/disk-space/disk-space.toml@@ -6,7 +6,8 @@# altogether: a hosted box is not lent out. name = "disk-space"-version = 1+version = 2+namespace = "pyrtan"description = "How full a filesystem is, from df on the box this tree runs on." tags = ["ops", "box", "read-only"] license = "MIT" diff --git a/examples/github/github-release.toml b/examples/github/github-release.toml index d32f374..c167882 100644 --- a/examples/github/github-release.toml +++ b/examples/github/github-release.toml@@ -2,7 +2,8 @@# optional token from the service's env and a decision on the status code. name = "github-release"-version = 1+version = 2+namespace = "pyrtan"description = "The latest release of a GitHub repository." tags = ["github", "read-only"] license = "MIT" diff --git a/examples/hacker-news/hacker-news-item.toml b/examples/hacker-news/hacker-news-item.toml index eadbb09..b855e98 100644 --- a/examples/hacker-news/hacker-news-item.toml +++ b/examples/hacker-news/hacker-news-item.toml@@ -1,5 +1,6 @@name = "hacker-news-item"-version = 1+version = 2+namespace = "pyrtan"description = "One Hacker News item by id: title, url, score, author." tags = ["news", "read-only"] license = "MIT" diff --git a/examples/hacker-news/hacker-news.toml b/examples/hacker-news/hacker-news.toml index 826c223..b37362c 100644 --- a/examples/hacker-news/hacker-news.toml +++ b/examples/hacker-news/hacker-news.toml@@ -3,7 +3,8 @@# can reuse it. name = "hacker-news"-version = 1+version = 2+namespace = "pyrtan"description = "Top stories on Hacker News, from its public API." tags = ["news", "read-only"] license = "MIT" diff --git a/examples/log-errors/log-errors.toml b/examples/log-errors/log-errors.toml index a3d88ab..2dcc4c7 100644 --- a/examples/log-errors/log-errors.toml +++ b/examples/log-errors/log-errors.toml@@ -4,7 +4,8 @@# /var/log, ".." included. name = "log-errors"-version = 1+version = 2+namespace = "pyrtan"description = "Error lines in the tail of a log file on this box." tags = ["ops", "box", "read-only"] license = "MIT" diff --git a/examples/modem-check/modem-check.toml b/examples/modem-check/modem-check.toml index 92ec818..27b5a07 100644 --- a/examples/modem-check/modem-check.toml +++ b/examples/modem-check/modem-check.toml@@ -3,7 +3,8 @@# `devices` is the allow-list. An observe retries once on no answer. name = "modem-check"-version = 1+version = 2+namespace = "pyrtan"description = "Whether a modem on a serial line answers AT." tags = ["hardware", "serial"] license = "MIT" diff --git a/examples/rendered-headline/rendered-headline.toml b/examples/rendered-headline/rendered-headline.toml index a742032..268455d 100644 --- a/examples/rendered-headline/rendered-headline.toml +++ b/examples/rendered-headline/rendered-headline.toml@@ -6,7 +6,8 @@# outside `hosts`. name = "rendered-headline"-version = 1+version = 2+namespace = "pyrtan"description = "The main heading of a page that needs a script to render, through a real browser." tags = ["web", "browser", "read-only"] license = "MIT" diff --git a/examples/room-temperature/room-temperature.toml b/examples/room-temperature/room-temperature.toml index c6381db..1b5a2a6 100644 --- a/examples/room-temperature/room-temperature.toml +++ b/examples/room-temperature/room-temperature.toml@@ -4,7 +4,8 @@# checked at run time when they come from the sentence. name = "room-temperature"-version = 1+version = 2+namespace = "pyrtan"description = "The temperature a room's sensor last published, over MQTT." tags = ["home", "mqtt"] license = "MIT" diff --git a/examples/site-status/site-status.toml b/examples/site-status/site-status.toml index 3d016af..f3f4a1e 100644 --- a/examples/site-status/site-status.toml +++ b/examples/site-status/site-status.toml@@ -3,7 +3,8 @@# allows any host on purpose, with the wildcard spelled out. name = "site-status"-version = 1+version = 2+namespace = "pyrtan"description = "Whether a web site answers, and how quickly." tags = ["ops", "read-only"] license = "MIT" diff --git a/examples/weather/weather.toml b/examples/weather/weather.toml index 3e65419..1ea8aeb 100644 --- a/examples/weather/weather.toml +++ b/examples/weather/weather.toml@@ -3,7 +3,8 @@# own decision so a failure says which hop failed. name = "weather"-version = 1+version = 2+namespace = "pyrtan"description = "Current weather for a place, from Open-Meteo." tags = ["weather", "read-only"] license = "MIT" diff --git a/pyproject.toml b/pyproject.toml index 3bced09..52fd365 100644 --- a/pyproject.toml +++ b/pyproject.toml@@ -1,6 +1,6 @@[project] name = "pyrtan"-version = "0.7.0"+version = "0.8.0"description = "Open-source total automation: a core set of expandable services that act on all forms of I/O through hardcoded, inspectable decision trees." readme = "README.md" license = "AGPL-3.0-or-later" diff --git a/site/README.md b/site/README.md index 3ee41b6..74d4c8f 100644 --- a/site/README.md +++ b/site/README.md@@ -8,9 +8,10 @@ static directory and publishes it.| :-- | :-- | :-- | | `/` | the landing page, with the live directory read from the seeds in the visitor's browser | `site/index.html` | | `/console/` | the console (installable: manifest, service worker and `pyrtan.js`, the tree runtime, beside it); point it at any hub started with `--cors https://pyrtan.com`; its Member tab joins that hub, and runs its trees when joined with the token | `pyrtan console --out` |-| `/catalog/` | the trees people can `pyrtan pull`, and `index.json` the command reads | `pyrtan index examples`, and the tree files |+| `/catalog/` | the trees people can `pyrtan pull`, as `namespace/name.toml`, and `index.json` the command reads | `pyrtan index examples catalog --copy`, through `site/gate.py` || `/rendezvous/seeds.json` | hubs to follow first | `site/rendezvous/seeds.json` | | `/rendezvous/directory.json` | a snapshot of the directory as the seeds know it, refreshed hourly | `site/snapshot.py` |+| `/rendezvous/reports.json` | the gap reports the seeds carry for catalog trees, refreshed hourly | `site/snapshot.py --reports` |`site/build.sh out` builds the same thing locally; open `out/index.html`.@@ -38,12 +39,22 @@ writes `trees/weather.toml`.## The catalog-The catalog is `examples/`. A tree goes in by pull request to this-repository, under `examples/<name>/`, with a `license` line and a test-in `tests/test_examples.py` against a recording (see-[`CONTRIBUTING.md`](../CONTRIBUTING.md)). `pyrtan index` refuses a tree-with no licence or a duplicate name, and the workflow runs the tests-before it publishes, so what is at `/catalog/` always passes.+The catalog is `examples/` (the `pyrtan` namespace) and `catalog/`+(everyone else's, one directory per namespace with an `OWNERS` file),+served as `/catalog/<namespace>/<name>.toml` beside `index.json`. A tree+goes in by pull request: `pyrtan publish my-tree.toml --into <checkout>+--owner <login>` places it and runs the gate, the `catalog` workflow runs+the same gate on the pull request (`site/gate.py`: clean, licensed,+namespaced, in the right place, unique id, changed only by an owner,+version bumped), and a maintainer merges. The `pages` workflow runs the+gate and the tests again before it publishes, so what is at `/catalog/`+always passes. See [`catalog/README.md`](../catalog/README.md) and+[`CONTRIBUTING.md`](../CONTRIBUTING.md).++`rendezvous/reports.json` is baked beside the directory snapshot: the gap+reports the seeds carry (from hubs that opted in with `--report-gaps`),+merged hourly, which the catalog page shows beside each tree and `pyrtan+reports` reads when given no hub.## The seeds diff --git a/site/build.sh b/site/build.sh index 5811e0f..e2f584f 100755 --- a/site/build.sh +++ b/site/build.sh@@ -1,6 +1,6 @@#!/bin/sh # Assemble pyrtan.com into one static directory: the landing page, the-# console, the catalog (built from examples/), the seed list. Run from the+# console, the catalog (built from examples/ and catalog/), the seed list. Run from the# repository root with pyrtan installed (`uv run site/build.sh`); the Pages # workflow runs the same thing. set -eu@@ -15,12 +15,15 @@ cp site/rendezvous/seeds.json "$out/rendezvous/"python3 site/snapshot.py site/rendezvous/seeds.json "$out/rendezvous/directory.json" \ --previous https://pyrtan.com/rendezvous/directory.json \ --previous https://b-i-t-8.github.io/pyrtan/rendezvous/directory.json \- ${SNAPSHOT_PREVIOUS:+--previous "$SNAPSHOT_PREVIOUS"}+ ${SNAPSHOT_PREVIOUS:+--previous "$SNAPSHOT_PREVIOUS"} \+ --reports "$out/rendezvous/reports.json" \+ --previous-reports https://pyrtan.com/rendezvous/reports.json \+ --previous-reports https://b-i-t-8.github.io/pyrtan/rendezvous/reports.jsoncp site/catalog/index.html "$out/catalog/" pyrtan console --out "$out/console/index.html"-pyrtan check examples --quiet-pyrtan index examples --out "$out/catalog/index.json"-# Every tree file at the path the index names, next to it.-(cd examples && find . -name '*.toml' -not -name 'pyproject.toml' -exec sh -c 'mkdir -p "$0/$(dirname "$1")" && cp "$1" "$0/$1"' "$out/catalog" {} \;)+pyrtan check examples catalog --quiet+python3 site/gate.py+# The index and every tree file beside it, as <namespace>/<name>.toml.+pyrtan index examples catalog --copy "$out/catalog" --require-namespacetouch "$out/.nojekyll" echo "built $out: $(find "$out" -type f | wc -l) files" diff --git a/site/catalog/index.html b/site/catalog/index.html index 9ab8a95..892c031 100644 --- a/site/catalog/index.html +++ b/site/catalog/index.html@@ -25,26 +25,33 @@<body> <main> <h1>Pyrtan catalog</h1>- <p class="lede">Trees you can pull into your own swarm. Each is one TOML file, checked by the repository's tests, with the licence its author chose. <code>pyrtan pull NAME</code> verifies the file against <a href="index.json">index.json</a> before writing it.</p>+ <p class="lede">Trees you can pull into your own swarm. Each is one TOML file, checked by the repository's tests, with the licence its author chose, under the namespace of whoever published it. <code>pyrtan pull NAMESPACE/NAME</code> verifies the file against <a href="index.json">index.json</a> before writing it. Where hubs that opted in have met a value a tree had no branch for, the <a href="../rendezvous/reports.json">gap reports</a> say so beside it.</p><input type="search" id="q" placeholder="filter: weather, github, read-only…" autocomplete="off"> <div id="list"><p class="mute">Loading index.json…</p></div>- <footer>To add a tree, open a pull request that puts it under <code>examples/</code> with a <code>license</code> line. <a href="../">pyrtan.com</a> · <a href="https://github.com/b-i-t-8/pyrtan">source</a></footer>+ <footer>To publish a tree: <code>pyrtan publish my-tree.toml --into <checkout> --owner <your GitHub login></code>, then open a pull request; the gate checks it, a maintainer merges, and it is here on the next build. <a href="../">pyrtan.com</a> · <a href="https://github.com/b-i-t-8/pyrtan/tree/main/catalog">how</a> · <a href="https://github.com/b-i-t-8/pyrtan">source</a></footer></main> <script> const esc = (s) => String(s).replace(/[&<>"]/g, c => ({"&":"&","<":"<",">":">",'"':"""}[c]));-let trees = [];+let trees = [], reports = [];+const idOf = (t) => t.id || (t.namespace ? `${t.namespace}/${t.name}` : t.name);function render() { const q = document.getElementById("q").value.toLowerCase().split(/\s+/).filter(Boolean); const list = document.getElementById("list"); list.innerHTML = "";- const shown = trees.filter(t => { const hay = [t.name, t.description, ...(t.tags||[]), ...(t.hosts||[]), ...(t.triggers||[])].join(" ").toLowerCase(); return q.every(w => hay.includes(w)); });+ const shown = trees.filter(t => { const hay = [idOf(t), t.description, ...(t.tags||[]), ...(t.hosts||[]), ...(t.sources||[]), ...(t.triggers||[])].join(" ").toLowerCase(); return q.every(w => hay.includes(w)); });for (const t of shown) {+ const id = idOf(t);+ const reach = (t.hosts||[]).concat((t.sources||[]).filter(s => s !== "http"));+ const mine = reports.filter(r => String(r.id).split("@")[0] === id);+ const seen = mine.reduce((n, r) => n + (r.count||1), 0), hubs = Math.max(0, ...mine.map(r => r.reporters||1));+ const gaps = mine.length ? `<details><summary class="mute">${seen} gap${seen === 1 ? "" : "s"} reported by ${hubs} hub${hubs === 1 ? "" : "s"}: a value no branch covered</summary>${mine.map(r => `<div><code>${esc(r.node)}</code> ${esc(r.on)} = <code>${esc(JSON.stringify(r.observed))}</code> <span class="mute">×${esc(r.count||1)}${r.id.includes("@") ? " · v" + esc(r.id.split("@")[1]) : ""}</span></div>`).join("")}</details>` : "";const d = document.createElement("div"); d.className = "card";- d.innerHTML = `<h2><code>${esc(t.name)}</code><small>v${esc(t.version)} · ${esc(t.license||"no licence")}${(t.hosts||[]).length ? " · " + esc(t.hosts.join(", ")) : ""}</small></h2><div class="mute">${esc(t.description||"")}</div>${(t.triggers||[]).map(p => `<div><code>${esc(p)}</code></div>`).join("")}<pre>pyrtan pull ${esc(t.name)}</pre><div class="mute"><a href="${esc(t.path)}">${esc(t.path)}</a> · sha256 ${esc((t.sha256||"").slice(0, 12))}…</div>`;+ d.innerHTML = `<h2><code>${esc(id)}</code><small>v${esc(t.version)} · ${esc(t.license||"no licence")}${reach.length ? " · " + esc(reach.join(", ")) : ""}</small></h2><div class="mute">${esc(t.description||"")}</div>${(t.triggers||[]).map(p => `<div><code>${esc(p)}</code></div>`).join("")}<pre>pyrtan pull ${esc(id)}</pre>${gaps}<div class="mute"><a href="${esc(t.path)}">${esc(t.path)}</a> · sha256 ${esc((t.sha256||"").slice(0, 12))}…</div>`;list.appendChild(d); } if (!shown.length) list.innerHTML = `<p class="mute">Nothing matches.</p>`; } fetch("index.json").then(r => r.json()).then(i => { trees = i.trees || []; document.title = `Pyrtan catalog · ${trees.length} trees`; render(); }).catch(e => { document.getElementById("list").innerHTML = `<p class="mute">Could not load index.json: ${esc(e)}</p>`; });+fetch("../rendezvous/reports.json").then(r => r.ok ? r.json() : null).then(j => { if (j && j.reports) { reports = j.reports; render(); } }).catch(() => {});document.getElementById("q").addEventListener("input", render); </script> </body> diff --git a/site/gate.py b/site/gate.py new file mode 100644 index 0000000..398a320 --- /dev/null +++ b/site/gate.py@@ -0,0 +1,69 @@+"""The catalog gate, as the workflows run it.++ uv run python site/gate.py [--base origin/main] [--author LOGIN] [--changed FILE ...]++Runs :func:`pyrtan.catalog.gate` on this checkout: every tree under+``examples/`` and ``catalog/`` loads clean with a licence and a namespace,+sits where its namespace and name say, ids are unique, each namespace has+owners. With ``--base`` it asks git which catalog files the pull request+changed and what version each had before, so a change must come from an+owner (``--author``, the pull request's login) and must bump the version.+Exit 1 with the problems listed, 0 when the gate is open.+"""++from __future__ import annotations++import argparse+import subprocess+import sys+import tomllib+from pathlib import Path++ROOT = Path(__file__).resolve().parents[1]+sys.path.insert(0, str(ROOT / "src"))++from pyrtan.catalog import gate # noqa: E402+++def git(*args: str) -> str | None:+ try:+ return subprocess.run(["git", *args], cwd=ROOT, capture_output=True, text=True, check=True).stdout+ except (subprocess.CalledProcessError, FileNotFoundError):+ return None+++def main(argv: list[str] | None = None) -> int:+ ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])+ ap.add_argument("--base", help="the base ref of the pull request (e.g. origin/main); changes and prior versions come from git")+ ap.add_argument("--author", help="the pull request's GitHub login; must own any namespace it changes")+ ap.add_argument("--changed", nargs="*", default=None, help="changed paths, when not taken from git")+ args = ap.parse_args(argv)+ changed = args.changed+ base: dict[str, tuple[int, bytes]] = {}+ if args.base:+ out = git("diff", "--name-only", f"{args.base}...HEAD")+ if out is None:+ print(f"gate: cannot diff against {args.base}; checking the tree without change information", file=sys.stderr)+ else:+ changed = [line.strip() for line in out.splitlines() if line.strip()] if changed is None else changed+ for path in changed:+ if not (path.startswith(("catalog/", "examples/")) and path.endswith(".toml")):+ continue+ prior = subprocess.run(["git", "show", f"{args.base}:{path}"], cwd=ROOT, capture_output=True)+ if prior.returncode != 0:+ continue # new on this branch+ try:+ version = int(tomllib.loads(prior.stdout.decode("utf-8")).get("version", 1))+ except Exception:+ version = 1+ base[path] = (version, prior.stdout)+ problems = gate(ROOT, changed=changed, author=args.author, base=base)+ for p in problems:+ print(f"gate: {p}")+ scope = f"{len(changed)} changed file(s)" if changed is not None else "the whole checkout"+ print(f"gate: {'refused' if problems else 'open'} ({scope}" + (f", author {args.author}" if args.author else "") + ")")+ return 1 if problems else 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/site/snapshot.py b/site/snapshot.py index 4d9998a..fb19ebd 100644 --- a/site/snapshot.py +++ b/site/snapshot.py@@ -13,6 +13,12 @@ Standard library only; no pyrtan import, so it runs before anything isinstalled. python3 site/snapshot.py SEEDS OUT [--previous URL_OR_FILE ...]+ [--reports OUT --previous-reports URL_OR_FILE ...]++With ``--reports`` it also asks each seed for ``/reports``, the gap+reports hubs that opted in (``--report-gaps``) gossip for catalog trees,+merges them the same way, and writes ``rendezvous/reports.json``; the+catalog page shows them beside each tree.""" from __future__ import annotations@@ -27,6 +33,7 @@ from pathlib import Pathfrom typing import Any TTL = 7 * 86400+REPORT_TTL = 30 * 86400TIMEOUT = 8.0@@ -58,15 +65,52 @@ def merge(into: dict[str, dict[str, Any]], entries: Any, via: str, now: float) -return taken+def merge_reports(into: dict[str, dict[str, Any]], rows: Any, now: float) -> int:+ """One row per gap: the hub's view already sums reporters, so a newer+ or larger row for the same gap replaces an older one."""+ taken = 0+ for r in rows if isinstance(rows, list) else []:+ if not isinstance(r, dict) or not isinstance(r.get("id"), str) or "@" not in r["id"] or not isinstance(r.get("node"), str):+ continue+ try:+ last = min(float(r.get("last", 0)), now)+ count = max(1, int(r.get("count", 1)))+ reporters = max(1, int(r.get("reporters", 1)))+ except (TypeError, ValueError):+ continue+ if now - last >= REPORT_TTL:+ continue+ key = json.dumps([r["id"], r["node"], r.get("on"), r.get("observed")], sort_keys=True, ensure_ascii=False, default=str)+ current = into.get(key)+ if current is not None and (current["last"], current["count"]) >= (last, count):+ continue+ observed = r.get("observed")+ into[key] = {"id": r["id"][:120], "node": r["node"][:80], "on": str(r.get("on", ""))[:120], "observed": observed if isinstance(observed, (str, int, float, bool)) or observed is None else str(observed)[:80], "count": count, "reporters": reporters, "last": last}+ taken += 1+ return taken++def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) ap.add_argument("seeds", help="seeds.json, a path or URL") ap.add_argument("out", help="where to write directory.json") ap.add_argument("--previous", action="append", default=[], help="an earlier directory.json to keep entries from (URL or path; repeatable, first that answers wins)")+ ap.add_argument("--reports", default=None, help="also write the gap reports the seeds carry to this reports.json")+ ap.add_argument("--previous-reports", action="append", default=[], help="an earlier reports.json to keep rows from (URL or path; repeatable, first that answers wins)")args = ap.parse_args(argv) now = time.time() hubs: dict[str, dict[str, Any]] = {} report: dict[str, Any] = {"answered": [], "silent": [], "previous": None}+ reports: dict[str, dict[str, Any]] = {}+ if args.reports:+ for source in args.previous_reports:+ try:+ prev = fetch(source)+ except Exception:+ continue+ if isinstance(prev, dict) and isinstance(prev.get("reports"), list):+ merge_reports(reports, prev["reports"], now)+ breakfor source in args.previous: try:@@ -94,6 +138,13 @@ def main(argv: list[str] | None = None) -> int:report["answered"].append(seed) else: report["silent"].append({"seed": seed, "error": "not a directory"})+ if args.reports:+ try:+ r = fetch(f"{seed}/reports")+ if isinstance(r, dict):+ merge_reports(reports, r.get("reports"), now)+ except Exception:+ passout = { "pyrtan-directory": 1,@@ -105,6 +156,12 @@ def main(argv: list[str] | None = None) -> int:path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(out, indent=2) + "\n", encoding="utf-8") print(f"snapshot: {len(out['hubs'])} hub(s); {len(report['answered'])} seed(s) answered, {len(report['silent'])} silent" + (f"; kept entries from {report['previous']}" if report["previous"] else ""))+ if args.reports:+ rows = sorted(reports.values(), key=lambda r: (-r["count"], -r["last"], r["id"]))+ rpath = Path(args.reports)+ rpath.parent.mkdir(parents=True, exist_ok=True)+ rpath.write_text(json.dumps({"pyrtan-reports": 1, "as_of": out["as_of"], "count": len(rows), "reports": rows}, indent=2) + "\n", encoding="utf-8")+ print(f"snapshot: {len(rows)} gap report(s) -> {rpath}")return 0 diff --git a/src/pyrtan/__init__.py b/src/pyrtan/__init__.py index dee4ef0..745db83 100644 --- a/src/pyrtan/__init__.py +++ b/src/pyrtan/__init__.py@@ -33,4 +33,4 @@ __all__ = ["Response", ]-__version__ = "0.7.0"+__version__ = "0.8.0"diff --git a/src/pyrtan/catalog.py b/src/pyrtan/catalog.py index eb5c48d..2502f93 100644 --- a/src/pyrtan/catalog.py +++ b/src/pyrtan/catalog.py@@ -11,12 +11,16 @@ on every merge.``index.json``:: {"pyrtan-catalog": 1, "generated": "2026-10-25T12:00:00Z",- "trees": [{"name": "weather", "version": 3, "description": "...",- "tags": [...], "hosts": [...], "triggers": [...],- "license": "MIT", "path": "weather/weather.toml",- "sha256": "...", "size": 1234}]}--``path`` is relative to the index. A pull fetches the file, checks its+ "trees": [{"id": "pyrtan/weather", "namespace": "pyrtan", "name": "weather",+ "version": 3, "description": "...", "tags": [...], "hosts": [...],+ "sources": ["http"], "triggers": [...], "license": "MIT",+ "path": "pyrtan/weather.toml", "sha256": "...", "size": 1234}]}++``path`` is relative to the index. A tree's ``id`` is ``namespace/name``:+the namespace is who published it (the tree's own ``namespace`` line), so+two people can each have a ``weather``; ``pyrtan pull weather`` takes the+one there is and asks when there are two, ``pyrtan pull barry/weather``+says which. A pull fetches the file, checks its``sha256`` against the index, loads it as a tree, checks that the name and version inside match the entry, and only then writes it. A catalog cannot hand out a file other than the one its index describes, and a tree a@@ -47,7 +51,7 @@ from .tree import Tree, TreeError, check, load_tree, load_tree_textDEFAULT_CATALOG = "https://pyrtan.com/catalog/" INDEX = "index.json" FORMAT = 1-NAME_AT_VERSION = re.compile(r"^([a-z0-9][a-z0-9-]*)(?:@(\d+))?$")+SPEC = re.compile(r"^(?:([a-z0-9][a-z0-9-]*)/)?([a-z0-9][a-z0-9-]*)(?:@(\d+))?$")class CatalogError(RuntimeError):@@ -70,11 +74,15 @@ def entry_for(tree: Tree, data: bytes, path: str) -> dict[str, Any]:"""One index entry: the tree's capability plus where its file is and what it hashes to.""" cap = tree.capability() return {+ "id": tree.id,+ "namespace": tree.namespace,"name": tree.name, "version": tree.version, "description": tree.description, "tags": tree.tags, "hosts": tree.hosts,+ **{k: cap[k] for k in ("paths", "commands", "devices", "topics") if k in cap},+ "sources": cap["sources"],"triggers": cap["triggers"], "inputs": cap["inputs"], "license": tree.license,@@ -93,39 +101,59 @@ class Problem:return f"{self.path}: {self.message}"-def build_index(root: str | Path, *, require_license: bool = True) -> tuple[dict[str, Any], list[Problem]]:- """Walk ``root`` for tree files and describe each in an index.+def build_index(root: str | Path | list[str | Path], *, require_license: bool = True, copy_to: str | Path | None = None, require_namespace: bool = False) -> tuple[dict[str, Any], list[Problem]]:+ """Walk ``root`` (one directory, or several) for tree files and describe+ each in an index.++ With one root and no ``copy_to``, paths are relative to that root and+ the index belongs beside it. With ``copy_to``, every tree file is copied+ there as ``<namespace>/<name>.toml`` (``_/<name>.toml`` without a+ namespace) and the index describes that layout: the shape the public+ catalog is served in, from any number of source directories.Returns the index and the problems that kept a file out of it: a file- that does not load, two files with the same name, or (when required) a- tree with no ``license``. The caller decides whether problems fail the- build; the catalog's CI does.+ that does not load, two files with the same id, (when required) a tree+ with no ``license`` or no ``namespace``. The caller decides whether+ problems fail the build; the catalog's CI does."""- root = Path(root)+ roots = [Path(r) for r in (root if isinstance(root, list) else [root])]+ if len(roots) > 1 and copy_to is None:+ raise ValueError("an index over several directories needs copy_to: its paths must be relative to one place")trees: list[dict[str, Any]] = [] problems: list[Problem] = [] seen: dict[str, str] = {}- for path in sorted(root.rglob("*.toml")):- if path.name == "pyproject.toml" or any(part.startswith(".") for part in path.relative_to(root).parts):- continue- rel = path.relative_to(root).as_posix()- try:- tree = load_tree(path)- except TreeError as e:- problems.append(Problem(rel, str(e)))- continue- errors = [p for p in check(tree) if p.level == "error"]- if errors:- problems.append(Problem(rel, "; ".join(f"{p.where}: {p.message}" for p in errors)))- continue- if tree.name in seen:- problems.append(Problem(rel, f"a tree named {tree.name!r} is already at {seen[tree.name]}"))- continue- if require_license and not tree.license:- problems.append(Problem(rel, "no 'license': a catalog tree must say what others may do with it"))- continue- seen[tree.name] = rel- trees.append(entry_for(tree, path.read_bytes(), rel))+ for base in roots:+ for path in sorted(base.rglob("*.toml")):+ if path.name == "pyproject.toml" or any(part.startswith(".") for part in path.relative_to(base).parts):+ continue+ rel = path.relative_to(base).as_posix()+ where = rel if len(roots) == 1 else f"{base}/{rel}"+ try:+ tree = load_tree(path)+ except TreeError as e:+ problems.append(Problem(where, str(e)))+ continue+ errors = [p for p in check(tree) if p.level == "error"]+ if errors:+ problems.append(Problem(where, "; ".join(f"{p.where}: {p.message}" for p in errors)))+ continue+ if tree.id in seen:+ problems.append(Problem(where, f"a tree with id {tree.id!r} is already at {seen[tree.id]}"))+ continue+ if require_license and not tree.license:+ problems.append(Problem(where, "no 'license': a catalog tree must say what others may do with it"))+ continue+ if require_namespace and not tree.namespace:+ problems.append(Problem(where, "no 'namespace': a catalog tree says who publishes it"))+ continue+ seen[tree.id] = where+ data = path.read_bytes()+ if copy_to is not None:+ rel = f"{tree.namespace or '_'}/{tree.name}.toml"+ target = Path(copy_to) / rel+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_bytes(data)+ trees.append(entry_for(tree, data, rel))index = { "pyrtan-catalog": FORMAT, "generated": _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),@@ -148,7 +176,7 @@ def _fetch(url: str, timeout: float = 20.0) -> bytes:raise CatalogError(f"{url}: {e}") from e if parts.scheme not in ("http", "https"): raise CatalogError(f"{url}: a catalog is reached over https (or a file: path)")- req = urllib.request.Request(url, headers={"User-Agent": "pyrtan-catalog/0.7", "Accept": "application/json, application/toml, text/plain, */*"})+ req = urllib.request.Request(url, headers={"User-Agent": "pyrtan-catalog/0.8", "Accept": "application/json, application/toml, text/plain, */*"})try: with urllib.request.urlopen(req, timeout=timeout) as resp: return resp.read()@@ -188,32 +216,41 @@ class Catalog:return self._index def entries(self) -> list[dict[str, Any]]:- return [e for e in self.index["trees"] if isinstance(e, dict) and isinstance(e.get("name"), str)]+ out = []+ for e in self.index["trees"]:+ if isinstance(e, dict) and isinstance(e.get("name"), str):+ e.setdefault("namespace", "")+ e.setdefault("id", f"{e['namespace']}/{e['name']}" if e["namespace"] else e["name"])+ out.append(e)+ return outdef search(self, query: str = "") -> list[dict[str, Any]]:- """Entries whose name, description, tags, hosts or triggers mention- every word of ``query``. An empty query lists everything."""+ """Entries whose id, description, tags, hosts, sources or triggers+ mention every word of ``query``. An empty query lists everything."""words = [w.lower() for w in query.split()] out: list[dict[str, Any]] = [] for e in self.entries():- hay = " ".join([e["name"], str(e.get("description", "")), " ".join(e.get("tags", [])), " ".join(e.get("hosts", [])), " ".join(e.get("triggers", []))]).lower()+ hay = " ".join([e["id"], str(e.get("description", "")), " ".join(e.get("tags", [])), " ".join(e.get("hosts", [])), " ".join(e.get("sources", [])), " ".join(e.get("triggers", []))]).lower()if all(w in hay for w in words): out.append(e) return out def find(self, spec: str) -> dict[str, Any]:- """The entry for ``name`` or ``name@version``."""- m = NAME_AT_VERSION.match(spec)+ """The entry for ``[namespace/]name[@version]``. A bare name is enough+ when one namespace has it; with two, the caller is told which."""+ m = SPEC.match(spec)if not m:- raise CatalogError(f"{spec!r}: expected NAME or NAME@VERSION")- name, version = m.group(1), m.group(2)- for e in self.entries():- if e["name"] == name and (version is None or e.get("version") == int(version)):+ raise CatalogError(f"{spec!r}: expected [NAMESPACE/]NAME[@VERSION]")+ namespace, name, version = m.group(1), m.group(2), m.group(3)+ have = [e for e in self.entries() if e["name"] == name and (namespace is None or e["namespace"] == namespace)]+ if not have:+ raise CatalogError(f"{spec!r} is not in the catalog at {self.base}")+ if len({e["id"] for e in have}) > 1:+ raise CatalogError(f"{name!r} is published by more than one namespace: {', '.join(sorted({e['id'] for e in have}))}; say which")+ for e in have:+ if version is None or e.get("version") == int(version):return e- have = [e for e in self.entries() if e["name"] == name]- if have and version is not None:- raise CatalogError(f"{name}: the catalog has version {have[0].get('version')}, not {version}")- raise CatalogError(f"{name!r} is not in the catalog at {self.base}")+ raise CatalogError(f"{have[0]['id']}: the catalog has version {have[0].get('version')}, not {version}")def fetch(self, entry: dict[str, Any]) -> tuple[Tree, bytes]: """The tree file an entry describes, verified against the entry."""@@ -230,6 +267,8 @@ class Catalog:raise CatalogError(f"{entry['name']}: the file does not load as a tree: {e}") from e if tree.name != entry["name"]: raise CatalogError(f"{entry['name']}: the file says it is {tree.name!r}; refusing it")+ if entry.get("namespace") and tree.namespace != entry["namespace"]:+ raise CatalogError(f"{entry.get('id', entry['name'])}: the file says its namespace is {tree.namespace!r}; refusing it")if entry.get("version") is not None and tree.version != entry["version"]: raise CatalogError(f"{entry['name']}: the file is version {tree.version}, the index says {entry['version']}; refusing it") return tree, data@@ -249,3 +288,161 @@ class Catalog:into.mkdir(parents=True, exist_ok=True) target.write_bytes(data) return target+++# ---------------------------------------------------------------- publishing++OWNERS = "OWNERS"+EXAMPLES_NAMESPACE = "pyrtan"+++def owners(namespace_dir: str | Path) -> list[str]:+ """The GitHub logins in ``catalog/<namespace>/OWNERS``: one per line,+ ``#`` starts a comment."""+ path = Path(namespace_dir) / OWNERS+ if not path.is_file():+ return []+ out = []+ for line in path.read_text(encoding="utf-8").splitlines():+ line = line.split("#", 1)[0].strip()+ if line:+ out.append(line.lstrip("@").lower())+ return out+++def gate(root: str | Path, *, changed: list[str] | None = None, author: str | None = None, base: dict[str, tuple[int, bytes]] | None = None) -> list[Problem]:+ """The checks a pull request to the catalog repository must pass. The+ repository is ``root``: its ``examples/`` are the ``pyrtan`` namespace,+ its ``catalog/<namespace>/<name>.toml`` files everyone else's, each+ namespace with an ``OWNERS`` file of GitHub logins.++ Every tree must load clean, carry ``license`` and ``namespace``, sit in+ the directory its namespace names under the file its name names, and+ ids must be unique across both places. Given ``changed`` (paths the+ pull request touches) and ``author`` (its GitHub login), a change under+ ``catalog/<namespace>/`` must come from one of that namespace's owners.+ Given ``base`` (``path -> (version, bytes)`` on the base branch), a tree+ whose file changed must have bumped its version. Returns the problems;+ none means the gate is open.+ """+ import tempfile++ root = Path(root)+ problems: list[Problem] = []+ examples = root / "examples"+ catalog = root / "catalog"+ roots = [d for d in (examples, catalog) if d.is_dir()]+ if not roots:+ return [Problem(str(root), "no examples/ or catalog/ directory: is this a checkout of the pyrtan repository?")]+ with tempfile.TemporaryDirectory() as tmp:+ _, problems = build_index(roots, require_license=True, require_namespace=True, copy_to=tmp)+ if examples.is_dir():+ for path in sorted(examples.rglob("*.toml")):+ try:+ tree = load_tree(path)+ except TreeError:+ continue # already reported+ if tree.namespace != EXAMPLES_NAMESPACE:+ problems.append(Problem(path.relative_to(root).as_posix(), f"a tree under examples/ is published by {EXAMPLES_NAMESPACE!r}, not {tree.namespace!r}"))+ if catalog.is_dir():+ for ns_dir in sorted(catalog.iterdir()):+ rel = ns_dir.relative_to(root).as_posix()+ if ns_dir.name.startswith(".") or ns_dir.name == "README.md":+ continue+ if not ns_dir.is_dir():+ problems.append(Problem(rel, "only namespace directories belong directly under catalog/"))+ continue+ if not re.match(r"^[a-z0-9][a-z0-9-]*$", ns_dir.name):+ problems.append(Problem(rel, "a namespace is lowercase letters, digits and dashes"))+ if not owners(ns_dir):+ problems.append(Problem(f"{rel}/{OWNERS}", "a namespace needs an OWNERS file with at least one GitHub login"))+ for path in sorted(ns_dir.rglob("*")):+ if not path.is_file() or path.name.startswith("."):+ continue+ prel = path.relative_to(root).as_posix()+ if path.name == OWNERS and path.parent == ns_dir:+ continue+ if path.suffix != ".toml" or path.parent != ns_dir:+ problems.append(Problem(prel, "only <name>.toml files and OWNERS belong in a namespace directory"))+ continue+ try:+ tree = load_tree(path)+ except TreeError:+ continue+ if tree.namespace != ns_dir.name:+ problems.append(Problem(prel, f"the tree says namespace {tree.namespace!r}; it sits under catalog/{ns_dir.name}/"))+ if path.stem != tree.name:+ problems.append(Problem(prel, f"the tree is named {tree.name!r}; its file must be {tree.name}.toml"))+ if changed and author:+ login = author.lstrip("@").lower()+ for c in changed:+ parts = Path(c).as_posix().split("/")+ if len(parts) >= 3 and parts[0] == "catalog":+ own = owners(catalog / parts[1])+ if own and login not in own:+ problems.append(Problem(c, f"{author} is not in catalog/{parts[1]}/{OWNERS}; only a namespace's owners change it"))+ for c, (base_version, base_bytes) in (base or {}).items():+ path = root / c+ if not path.is_file():+ continue # a removal+ if path.read_bytes() == base_bytes:+ continue+ try:+ tree = load_tree(path)+ except TreeError:+ continue+ if tree.version <= base_version:+ problems.append(Problem(c, f"the file changed but version is still {tree.version}; bump it so pulls notice"))+ return problems+++def publish(file: str | Path, checkout: str | Path, *, owners: list[str] | None = None) -> tuple[Path, list[str]]:+ """Put a tree into ``checkout/catalog/<namespace>/<name>.toml``, write+ OWNERS for a new namespace, and run the gate. Returns the placed path+ and notes for the person. The pull request they open is the publishing."""+ logins = [o.lstrip("@").lower() for o in (owners or [])]+ src = Path(file)+ try:+ tree = load_tree(src)+ except TreeError as e:+ raise CatalogError(str(e)) from e+ errors = [p for p in check(tree) if p.level == "error"]+ if errors:+ raise CatalogError(f"{src}: " + "; ".join(f"{p.where}: {p.message}" for p in errors))+ if not tree.namespace:+ raise CatalogError(f"{src}: add a line namespace = \"you\" (lowercase; it is who publishes the tree, and the catalog id becomes you/{tree.name})")+ if not tree.license:+ raise CatalogError(f"{src}: add a line license = \"MIT\" (an SPDX identifier: what others may do with it)")+ catalog = Path(checkout) / "catalog"+ if not catalog.is_dir():+ raise CatalogError(f"{checkout}: no catalog/ directory; is this a checkout of github.com/b-i-t-8/pyrtan?")+ ns_dir = catalog / tree.namespace+ notes: list[str] = []+ if not ns_dir.is_dir():+ if not logins:+ raise CatalogError(f"catalog/{tree.namespace}/ is new: say who owns it with --owner YOUR-GITHUB-LOGIN")+ ns_dir.mkdir(parents=True)+ (ns_dir / OWNERS).write_text("# GitHub logins allowed to change this namespace, one per line.\n" + "".join(f"{o}\n" for o in logins), encoding="utf-8")+ notes.append(f"new namespace catalog/{tree.namespace}/ owned by {', '.join(logins)}")+ elif logins:+ have = globals()["owners"](ns_dir)+ missing = [o for o in logins if o not in have]+ if missing:+ with (ns_dir / OWNERS).open("a", encoding="utf-8") as f:+ f.write("".join(f"{o}\n" for o in missing))+ notes.append(f"added {', '.join(missing)} to catalog/{tree.namespace}/{OWNERS}")+ target = ns_dir / f"{tree.name}.toml"+ data = src.read_bytes()+ if target.exists():+ if target.read_bytes() == data:+ notes.append(f"{target.relative_to(checkout)} already holds this exact file")+ else:+ old = load_tree(target)+ if tree.version <= old.version:+ raise CatalogError(f"catalog/{tree.namespace}/{tree.name}.toml is at version {old.version}; bump this file's version past it")+ notes.append(f"replaces version {old.version} with {tree.version}")+ target.write_bytes(data)+ problems = gate(checkout)+ if problems:+ raise CatalogError("the gate refuses the checkout as it now stands:\n " + "\n ".join(str(p) for p in problems))+ return target.resolve(), notesdiff --git a/src/pyrtan/cli.py b/src/pyrtan/cli.py index 644d7fc..c29bb68 100644 --- a/src/pyrtan/cli.py +++ b/src/pyrtan/cli.py@@ -128,6 +128,7 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--listed", action="store_true", default=os.environ.get("PYRTAN_LISTED", "").lower() in ("1", "true", "yes"), help="put this hub in the directory it gossips, so others can find it (default $PYRTAN_LISTED)") p.add_argument("--store", metavar="DIR", default=os.environ.get("PYRTAN_STORE"), help="where this hub keeps blobs, its own and others' (default $PYRTAN_STORE, else next to --state)") p.add_argument("--replicas", type=int, default=int(os.environ.get("PYRTAN_REPLICAS", "2")), help="how many members or hubs should keep each of this hub's blobs (default $PYRTAN_REPLICAS or 2)")+ p.add_argument("--report-gaps", action="store_true", default=os.environ.get("PYRTAN_REPORT_GAPS", "").lower() in ("1", "true", "yes"), help="put the gaps this swarm's catalog trees meet (tree id, node, value; never inputs) into its announces, so they reach the tree's author through the federation and pyrtan.com (default $PYRTAN_REPORT_GAPS)")p.add_argument("--quiet", action="store_true") p = sub.add_parser("follow", help="follow another hub, through a running federated hub of yours")@@ -172,6 +173,11 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--json", action="store_true") _token_arg(p, "the token the service or hub requires, if any")+ p = sub.add_parser("reports", help="gaps reported across the federation for catalog trees, as a hub (or pyrtan.com) carries them")+ p.add_argument("id", nargs="?", default=None, metavar="NAMESPACE/NAME", help="only this tree (any version)")+ p.add_argument("--at", default=os.environ.get("PYRTAN_AT"), help="a federated hub (default $PYRTAN_AT, else the snapshot at pyrtan.com)")+ p.add_argument("--json", action="store_true")+p = sub.add_parser("unanswered", help="sentences nobody answered, each with the tree that would") p.add_argument("--at", help="a running service or hub") p.add_argument("--json", action="store_true")@@ -189,15 +195,22 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--json", action="store_true") p = sub.add_parser("pull", help="fetch a tree from a catalog, verified against its index")- p.add_argument("spec", nargs="+", metavar="NAME[@VERSION]")+ p.add_argument("spec", nargs="+", metavar="[NAMESPACE/]NAME[@VERSION]")p.add_argument("--dir", default=None, help="where to write (default: the first of $PYRTAN_TREES or ./trees)") p.add_argument("--catalog", default=None, help=f"catalog URL or directory (default $PYRTAN_CATALOG or {catalog_url()})") p.add_argument("--force", action="store_true", help="replace a file that exists and differs")- p = sub.add_parser("index", help="build a catalog index.json from a directory of trees")- p.add_argument("dir")- p.add_argument("--out", default=None, help="write the index here (default: DIR/index.json; '-' for stdout)")+ p = sub.add_parser("index", help="build a catalog index.json from directories of trees")+ p.add_argument("dir", nargs="+")+ p.add_argument("--out", default=None, help="write the index here (default: DIR/index.json, or COPY/index.json with --copy; '-' for stdout)")+ p.add_argument("--copy", metavar="DIR", default=None, help="copy every tree file into DIR as <namespace>/<name>.toml and describe that layout (needed with more than one source directory)")p.add_argument("--allow-unlicensed", action="store_true", help="index trees with no 'license' too")+ p.add_argument("--require-namespace", action="store_true", help="leave out trees with no 'namespace'")++ p = sub.add_parser("publish", help="put a tree into a checkout of the catalog repository, ready for a pull request")+ p.add_argument("file", help="the tree file; it must carry 'namespace' and 'license'")+ p.add_argument("--into", required=True, metavar="CHECKOUT", help="a checkout of github.com/b-i-t-8/pyrtan (or any repository with a catalog/ directory)")+ p.add_argument("--owner", action="append", default=[], metavar="LOGIN", help="GitHub login(s) that own the namespace; needed when the namespace is new")p = sub.add_parser("console", help="write the console as one static HTML file, to host anywhere") p.add_argument("--out", required=True, help="the file to write ('-' for stdout)")@@ -205,8 +218,8 @@ def main(argv: list[str] | None = None) -> int:args = parser.parse_args(argv) commands = { "check": cmd_check, "list": cmd_list, "run": cmd_run, "match": cmd_match, "ask": cmd_ask, "serve": cmd_serve, "hub": cmd_hub, "gaps": cmd_gaps, "unanswered": cmd_unanswered, "new": cmd_new,- "search": cmd_search, "pull": cmd_pull, "index": cmd_index, "console": cmd_console,- "follow": cmd_follow, "unfollow": cmd_unfollow, "peers": cmd_peers, "directory": cmd_directory,+ "search": cmd_search, "pull": cmd_pull, "index": cmd_index, "publish": cmd_publish, "console": cmd_console,+ "follow": cmd_follow, "unfollow": cmd_unfollow, "peers": cmd_peers, "directory": cmd_directory, "reports": cmd_reports,"keep": cmd_keep, "kept": cmd_kept, "fetch": cmd_fetch, } return commands[args.command](args)@@ -447,7 +460,7 @@ def cmd_hub(args: argparse.Namespace) -> int:federate=args.federate or bool(args.advertise) or bool(follow) or args.listed, advertise=args.advertise, federation_secret=args.federation_secret, state=args.state, open_follows=not args.closed, follow=follow, listed=args.listed, trees=[Path(t) for t in args.trees], handout=[Path(t) for t in args.handout], env=_kv(args.env), gaps=args.gaps, egress=args.egress,- store=args.store, replicas=args.replicas,+ store=args.store, replicas=args.replicas, report_gaps=args.report_gaps,) fed = f", federated as {hub.federation.id}" + (f" following {len(hub.federation.following)}" if hub.federation.following else "") + (", listed" if hub.federation.listed else "") if hub.federation else "" local = f", holding {len(hub.local.trees)} tree(s)" if hub.local else ""@@ -490,6 +503,34 @@ def cmd_gaps(args: argparse.Namespace) -> int:return 0+def cmd_reports(args: argparse.Namespace) -> int:+ source = f"{args.at.rstrip('/')}/reports" if args.at else (os.environ.get("PYRTAN_REPORTS") or "https://pyrtan.com/rendezvous/reports.json")+ try:+ if source.startswith(("http://", "https://")):+ status, body = http_json("GET", source)+ else:+ status, body = 200, json.loads(Path(source).read_text(encoding="utf-8")) # a baked reports.json on disk+ except Exception as e:+ print(f"pyrtan: {source}: {e}", file=sys.stderr)+ return 3+ if status != 200 or not isinstance(body, dict):+ print(f"pyrtan: {source} answered {status}", file=sys.stderr)+ return 3+ reports = [r for r in body.get("reports", []) if isinstance(r, dict)]+ if args.id:+ reports = [r for r in reports if str(r.get("id", "")).split("@")[0] == args.id]+ if args.json:+ print(json.dumps({"source": source, "reports": reports}, indent=2, ensure_ascii=False))+ return 0+ if not reports:+ print(f"no gap reports at {source}" + (f" for {args.id}" if args.id else ""))+ return 0+ for r in reports:+ who = f", {r['reporters']} hub(s)" if r.get("reporters", 1) > 1 else ""+ print(f"{r['id']} {r['node']}: {r['on']} = {json.dumps(r.get('observed'), ensure_ascii=False)} ×{r.get('count', 1)}{who}")+ return 0++def cmd_follow(args: argparse.Namespace) -> int: url = args.url if not url:@@ -695,8 +736,8 @@ def cmd_search(args: argparse.Namespace) -> int:print(f"nothing in {cat.base} matches {' '.join(args.words)!r}") return 1 for e in found:- hosts = ", ".join(e.get("hosts", []))- print(f"{e['name']}@{e.get('version', '?')} {e.get('description', '')}" + (f" [{hosts}]" if hosts else ""))+ reach = ", ".join(e.get("hosts", []) + [s for s in e.get("sources", []) if s != "http"])+ print(f"{e.get('id', e['name'])}@{e.get('version', '?')} {e.get('description', '')}" + (f" [{reach}]" if reach else ""))for t in e.get("triggers", [])[:3]: print(f" {t}") return 0@@ -714,28 +755,58 @@ def cmd_pull(args: argparse.Namespace) -> int:worst = 1 continue tree = load_tree(target)- print(f"{tree.name}@{tree.version} -> {target}" + (f" (hosts: {', '.join(tree.hosts)})" if tree.hosts else ""))+ reach = ", ".join(tree.hosts + [s for s in tree.sources() if s != "http"])+ print(f"{tree.id}@{tree.version} -> {target}" + (f" (reaches: {reach})" if reach else ""))return worst def cmd_index(args: argparse.Namespace) -> int:- root = Path(args.dir)- if not root.is_dir():- print(f"pyrtan: {root}: not a directory", file=sys.stderr)+ roots = [Path(d) for d in args.dir]+ for root in roots:+ if not root.is_dir():+ print(f"pyrtan: {root}: not a directory", file=sys.stderr)+ return 2+ if len(roots) > 1 and not args.copy:+ print("pyrtan index: more than one directory needs --copy DIR, so the index has one place to point at", file=sys.stderr)return 2- index, problems = build_index(root, require_license=not args.allow_unlicensed)+ index, problems = build_index(roots, require_license=not args.allow_unlicensed, copy_to=args.copy, require_namespace=args.require_namespace)for p in problems: print(f"{p}", file=sys.stderr) text = json.dumps(index, indent=2, ensure_ascii=False) + "\n" if args.out == "-": sys.stdout.write(text) else:- out = Path(args.out) if args.out else root / "index.json"+ out = Path(args.out) if args.out else (Path(args.copy) if args.copy else roots[0]) / "index.json"+ out.parent.mkdir(parents=True, exist_ok=True)out.write_text(text, encoding="utf-8")- print(f"wrote {out}: {index['count']} tree(s)" + (f", {len(problems)} left out" if problems else ""))+ print(f"wrote {out}: {index['count']} tree(s)" + (f", copied to {args.copy}" if args.copy else "") + (f", {len(problems)} left out" if problems else ""))return 1 if problems else 0+def cmd_publish(args: argparse.Namespace) -> int:+ """Copy a tree into a checkout's catalog/<namespace>/, write OWNERS for+ a new namespace, run the gate, and say what to do next. The pull+ request is the publishing; this prepares it."""+ from .catalog import publish++ try:+ placed, notes = publish(args.file, args.into, owners=args.owner)+ except CatalogError as e:+ print(f"pyrtan publish: {e}", file=sys.stderr)+ return 1+ for n in notes:+ print(n)+ print(f"placed {placed}")+ rel = placed.relative_to(Path(args.into).resolve())+ print("next, from that checkout:")+ print(f" git checkout -b catalog/{rel.parent.name}-{rel.stem}")+ print(f" git add {rel.parent}")+ print(f" git commit -m \"catalog: {rel.parent.name}/{rel.stem}\"")+ print(" git push -u origin HEAD")+ print("then open a pull request; the catalog workflow runs the same gate and a maintainer merges.")+ return 0++def cmd_console(args: argparse.Namespace) -> int: if args.out == "-": sys.stdout.write(PAGE) diff --git a/src/pyrtan/federation.py b/src/pyrtan/federation.py index 6c53346..5fe8dfc 100644 --- a/src/pyrtan/federation.py +++ b/src/pyrtan/federation.py@@ -60,6 +60,7 @@ import hashlibimport hmac import json import os+import reimport secrets as _secrets import threading import time@@ -88,6 +89,9 @@ ACTIVITY_TYPES = ("Follow", "Accept", "Reject", "Undo", "Announce", "Keep", "KepASK_TIMEOUT = 30.0 # seconds a hub waits for a member to answer an Ask MEMBER_PREFIX = "urn:pyrtan:member:" # an actor with no inbox: a browser or a phone, which polls a mailbox MAILBOX_MAX = 200 # activities a hub holds for a member that has not polled+REPORT_TTL = 30 * 86400 # a gap report unheard of for this long is dropped+REPORT_MAX = 2000+REPORT_ID = re.compile(r"^[a-z0-9][a-z0-9-]*/[a-z0-9][a-z0-9-]*@\d+$") # namespace/name@version: a catalog treeHOLDER_FRESH = 86400.0 # a holder confirmed within this counts toward the replicas ASK_WINDOW = 600.0 # a member asked to keep within this counts as covered until it confirms SIGNED_PATHS = ("/inbox", "/run", "/intent", "/resume", "/capabilities")@@ -239,6 +243,28 @@ class Remote:return cls(d["url"], d["inbox"], d["secret"], d.get("name", ""), d.get("state", "pending"), since=float(d.get("since", time.time())), kind=d.get("kind", "hub"), trusted=bool(d.get("trusted", False)))+def _shape_report(e: dict[str, Any], reporter: str, now: float) -> dict[str, Any] | None:+ """A report as carried: valid or nothing. ``reporter`` is who said it+ first (the entry's own, else who we heard it from)."""+ tree_id, node, on = e.get("id"), e.get("node"), e.get("on")+ if not (isinstance(tree_id, str) and REPORT_ID.match(tree_id) and isinstance(node, str) and isinstance(on, str)):+ return None+ who = e.get("reporter") if isinstance(e.get("reporter"), str) and e.get("reporter", "").startswith(("http://", "https://")) else reporter+ try:+ count = max(1, int(e.get("count", 1)))+ last = min(float(e.get("last", now)), now)+ except (TypeError, ValueError):+ return None+ if now - last >= REPORT_TTL:+ return None+ observed = e.get("observed")+ if isinstance(observed, str) and len(observed) > 80:+ observed = "<str>"+ elif not isinstance(observed, (str, int, float, bool)) and observed is not None:+ observed = f"<{type(observed).__name__}>"+ return {"reporter": who.rstrip("/"), "id": tree_id, "node": node[:80], "on": on[:120], "observed": observed, "count": count, "last": last}++def heard(seen: float, now: float | None = None) -> str: """How recently a directory entry's hub announced, as gossip carried it.""" age = (now if now is not None else time.time()) - seen@@ -246,8 +272,11 @@ def heard(seen: float, now: float | None = None) -> str:class Federation:- def __init__(self, hub_id: str, *, name: str = "hub", secret: str | None = None, state: str | Path | None = None, open_follows: bool = True, capabilities: Callable[[], list[dict[str, Any]]] | None = None, listed: bool = False, store: BlobStore | str | Path | None = None, replicas: int = 2, auto_keys: dict[str, Callable[[], Any]] | None = None) -> None:+ def __init__(self, hub_id: str, *, name: str = "hub", secret: str | None = None, state: str | Path | None = None, open_follows: bool = True, capabilities: Callable[[], list[dict[str, Any]]] | None = None, listed: bool = False, store: BlobStore | str | Path | None = None, replicas: int = 2, auto_keys: dict[str, Callable[[], Any]] | None = None, report_gaps: bool = False, own_reports: Callable[[], list[dict[str, Any]]] | None = None) -> None:self.id = hub_id.rstrip("/")+ self.report_gaps = report_gaps # opt in: our own catalog trees' gaps travel with our announces+ self.own_reports = own_reports or (lambda: []) # the hub's gaps for catalog trees, shaped for a report+ self.reports: dict[str, dict[str, Any]] = {} # heard gap reports: (reporter, id, node, on, observed) -> entryself.name = name self.secret = secret or None # a shared federation secret: Follows must be signed with it self.open_follows = open_follows # without a secret, whether strangers may follow@@ -298,6 +327,7 @@ class Federation:"followers": [r.to_state() for r in self.followers.values()], "directory": list(self.directory_entries.values()), "keeping": self.keeping,+ "reports": list(self.reports.values()),} self.state_path.parent.mkdir(parents=True, exist_ok=True) self.state_path.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8")@@ -314,6 +344,7 @@ class Federation:for d in data.get("followers", []): self.followers[d["url"]] = Remote.from_state(d) self.merge_directory(data.get("directory", []), via="state")+ self.merge_reports(data.get("reports", []), via="state")keeping = data.get("keeping") if isinstance(keeping, dict): self.keeping = {k: v for k, v in keeping.items() if isinstance(v, dict) and isinstance(v.get("holders"), dict)}@@ -392,6 +423,70 @@ class Federation:hubs.sort(key=lambda h: (h["id"] != self.id, {"up": 0, "stale": 1, "down": 2}[h["link"]], h["name"], h["id"])) return {"hub": self.id, "name": self.name, "listed": self.listed, "count": len(hubs), "hubs": hubs}+ # ------------------------------------------------------------ gap reports+ # A hub that opted in (--report-gaps) puts the gaps its catalog trees met+ # into its announces: the tree's id (namespace/name@version), the node,+ # the path decided on and the value no branch covered, with a count;+ # never the inputs or the rest of the state. Every federated hub keeps+ # what it hears and passes it on, like the directory, so the reports+ # reach the seeds and the snapshot pyrtan.com bakes, where an author+ # reads them beside the tree. Nothing is reported for a tree without a+ # namespace: there is nobody to report to.++ def report_gossip(self) -> list[dict[str, Any]]:+ now = time.time()+ own = []+ if self.report_gaps:+ for r in self.own_reports():+ shaped = _shape_report(r, self.id, now)+ if shaped:+ own.append(shaped)+ with self._lock:+ heard = [r for r in self.reports.values() if r["reporter"] != self.id and now - r["last"] < REPORT_TTL]+ return own + sorted(heard, key=lambda r: -r["last"])++ def merge_reports(self, entries: Any, via: str) -> int:+ """Take reports heard from ``via``: one row per reporter and gap, a+ newer ``last`` replaces an older one, nothing from the future is+ believed, old news is dropped, the table stays bounded."""+ if not isinstance(entries, list):+ return 0+ now = time.time()+ changed = 0+ with self._lock:+ for e in entries:+ shaped = _shape_report(e, via, now) if isinstance(e, dict) else None+ if shaped is None or shaped["reporter"] == self.id:+ continue+ key = json.dumps([shaped["reporter"], shaped["id"], shaped["node"], shaped["on"], shaped["observed"]], sort_keys=True, ensure_ascii=False)+ current = self.reports.get(key)+ if current is not None and (current["last"], current["count"]) >= (shaped["last"], shaped["count"]):+ continue+ self.reports[key] = shaped+ changed += 1+ if len(self.reports) > REPORT_MAX:+ for key in sorted(self.reports, key=lambda k: self.reports[k]["last"])[: len(self.reports) - REPORT_MAX]:+ del self.reports[key]+ if changed:+ self._save()+ return changed++ def reports_view(self) -> dict[str, Any]:+ """Every gap report this hub carries, one row per gap with the+ counts and reporters summed, most seen first."""+ rows: dict[str, dict[str, Any]] = {}+ for r in self.report_gossip():+ key = json.dumps([r["id"], r["node"], r["on"], r["observed"]], sort_keys=True, ensure_ascii=False)+ row = rows.get(key)+ if row is None:+ rows[key] = {"id": r["id"], "node": r["node"], "on": r["on"], "observed": r["observed"], "count": r["count"], "reporters": 1, "last": r["last"]}+ else:+ row["count"] += r["count"]+ row["reporters"] += 1+ row["last"] = max(row["last"], r["last"])+ out = sorted(rows.values(), key=lambda r: (-r["count"], -r["last"], r["id"]))+ return {"hub": self.id, "reporting": self.report_gaps, "count": len(out), "reports": out}+# ------------------------------------------------------------ outgoing def _activity(self, type_: str, object_: Any, **extra: Any) -> dict[str, Any]:@@ -399,6 +494,9 @@ class Federation:def _announcement(self, for_remote: Remote | None = None) -> dict[str, Any]: out = {"type": "Capabilities", "name": self.name, "trees": self.capabilities(), "directory": self.gossip()}+ reports = self.report_gossip()+ if reports:+ out["reports"] = reportsif for_remote is not None: out["kept"] = [b["key"] for b in self.store.list_kept(for_remote.url)] # what we keep for them return out@@ -425,6 +523,7 @@ class Federation:if activity.get("type") == "Announce": # An inbox answers an Announce with its own gossip: one round trip, both directions. self.merge_directory(answer.get("directory"), via=remote.url)+ self.merge_reports(answer.get("reports"), via=remote.url)if activity.get("type") == "Keep" and answer.get("kept") == activity["object"].get("key"): self.confirm(remote.url, activity["object"]["key"], activity["object"]["sha256"]) return ok, None if ok else f"inbox answered {status}: {answer}", answer@@ -835,11 +934,12 @@ class Federation:names = {t.get("name") for t in obj.get("trees", []) if isinstance(t, dict)} remote.trees = [handed[n].capability() for n in sorted(names & set(handed))] taken = self.merge_directory(obj.get("directory", []), via=actor)+ self.merge_reports(obj.get("reports", []), via=actor)# What it keeps for us that we no longer have: ask for it back. wanted = [k for k in obj.get("kept", []) if isinstance(k, str) and not self.store.get_own(k)] for k in wanted[:20]: self.deliver(remote, self._activity("Fetch", {"key": k}))- return 200, {"status": "ok", "trees": len(remote.trees) if remote is followed else 0, "merged": taken, "directory": self.gossip(), "kept": [b["key"] for b in self.store.list_kept(actor)]}+ return 200, {"status": "ok", "trees": len(remote.trees) if remote is followed else 0, "merged": taken, "directory": self.gossip(), "reports": self.report_gossip(), "kept": [b["key"] for b in self.store.list_kept(actor)]}if kind == "Keep": obj = activity.get("object") if not isinstance(obj, dict) or "data" not in obj: diff --git a/src/pyrtan/hub.py b/src/pyrtan/hub.py index 882efff..fbffcc9 100644 --- a/src/pyrtan/hub.py +++ b/src/pyrtan/hub.py@@ -34,6 +34,7 @@ included, so a stranger cannot join the swarm) and presents it to peers.from __future__ import annotations import datetime as _dt+import jsonimport threading from pathlib import Path import time@@ -129,6 +130,8 @@ class Hub:self.hold(local) self.federation: Federation | None = None self.unanswered = Unanswered()+ self._reports: list[dict[str, Any]] = []+ self._reports_at = 0.0self.started = time.time() self._lock = threading.RLock() self._stop = threading.Event()@@ -136,7 +139,7 @@ class Hub:# ------------------------------------------------------------ federation- def federate(self, advertise: str, *, secret: str | None = None, state: str | None = None, open_follows: bool = True, listed: bool = False, store: str | None = None, replicas: int = 2) -> Federation:+ def federate(self, advertise: str, *, secret: str | None = None, state: str | None = None, open_follows: bool = True, listed: bool = False, store: str | None = None, replicas: int = 2, report_gaps: bool = False) -> Federation:"""Give this hub an identity other hubs can follow. ``advertise`` is the URL they reach it at. ``store`` is where it keeps blobs, its own and others'; ``replicas`` how many keepers each of its own should have."""@@ -145,7 +148,7 @@ class Hub:"gaps": lambda: self.gaps()["gaps"], "unanswered": lambda: self.unanswered_all()["unanswered"], }- self.federation = Federation(advertise, name=self.name, secret=secret, state=state, open_follows=open_follows, capabilities=self.announced_capabilities, listed=listed, store=store, replicas=replicas, auto_keys=auto)+ self.federation = Federation(advertise, name=self.name, secret=secret, state=state, open_follows=open_follows, capabilities=self.announced_capabilities, listed=listed, store=store, replicas=replicas, auto_keys=auto, report_gaps=report_gaps, own_reports=self.gap_reports)self.federation.handout = self.handout_trees return self.federation@@ -400,6 +403,29 @@ class Hub:out.extend({**g, "peer": peer.url} for g in body.get("gaps", [])) return {"gaps": out}+ def gap_reports(self) -> list[dict[str, Any]]:+ """This hub's gaps for catalog trees (the ones with a namespace),+ as reports: id, node, on, observed, count, last. Cached a minute;+ the announce heartbeat reads it."""+ now = time.time()+ if now - self._reports_at < 60:+ return self._reports+ ids = {t["name"]: f"{t['namespace']}/{t['name']}@{t.get('version', 1)}" for t in self.announced_capabilities() if t.get("namespace") and t.get("name")}+ rows: dict[str, dict[str, Any]] = {}+ for g in self.gaps()["gaps"]:+ tree_id = ids.get(str(g.get("tree")))+ if tree_id is None:+ continue+ key = json.dumps([tree_id, g.get("node"), g.get("on"), g.get("observed")], sort_keys=True, ensure_ascii=False, default=str)+ row = rows.get(key)+ if row is None:+ rows[key] = {"id": tree_id, "node": g.get("node"), "on": g.get("on"), "observed": g.get("observed"), "count": int(g.get("count", 1)), "last": float(g.get("last_seen", now))}+ else:+ row["count"] += int(g.get("count", 1))+ row["last"] = max(row["last"], float(g.get("last_seen", now)))+ self._reports, self._reports_at = list(rows.values()), now+ return self._reports+def unanswered_all(self) -> dict[str, Any]: """This hub's unanswered sentences, then each peer's own (a peer without a hub keeps its own record; one with a hub leaves it here)."""@@ -448,6 +474,8 @@ class Hub:return 200, fed.actor() if method == "GET" and path == "/directory": return 200, fed.directory()+ if method == "GET" and path == "/reports":+ return 200, fed.reports_view()if method == "POST" and path == "/inbox": return fed.handle_inbox(body, headers, trusted=bool(self.token) and bearer_ok(headers.get("Authorization") or headers.get("authorization"), self.token)) if method == "GET" and path in ("/following", "/followers"):@@ -508,7 +536,7 @@ class Hub:if method == "GET" and path == "/api": endpoints = ["/health", "/peers", "/capabilities", "/intent", "/run", "/resume", "/gaps", "/unanswered", "/register"] if fed is not None:- endpoints += ["/actor", "/inbox", "/mailbox", "/directory", "/follow", "/unfollow", "/listed", "/following", "/followers", "/outbox", "/keep", "/fetch", "/kept/{key}"]+ endpoints += ["/actor", "/inbox", "/mailbox", "/directory", "/reports", "/follow", "/unfollow", "/listed", "/following", "/followers", "/outbox", "/keep", "/fetch", "/kept/{key}"]return 200, {"pyrtan": "hub", "name": self.name, "version": __version__, "auth": "bearer" if self.token else "none", "federation": fed.id if fed else None, "endpoints": endpoints} if method == "POST" and path == "/register": url = body.get("url")@@ -555,12 +583,13 @@ def serve_hub(egress: str = "any", store: str | None = None, replicas: int = 2,+ report_gaps: bool = False,) -> tuple[Hub, Server]: local = Service(Registry.from_paths(trees), name=f"{name}", env=env, gaps=gaps, egress=egress) if trees else None hub = Hub(name=name, poll_every=poll_every, token=token, local=local, handout=Registry.from_paths(handout) if handout else None) server = Server(hub.route, host, port, quiet, token=token, cors=cors, federated=federate or bool(advertise)).start() if federate or advertise:- hub.federate(advertise or server.url, secret=federation_secret, state=state, open_follows=open_follows, listed=listed, store=store or (str(Path(state).with_name("store")) if state else None), replicas=replicas)+ hub.federate(advertise or server.url, secret=federation_secret, state=state, open_follows=open_follows, listed=listed, store=store or (str(Path(state).with_name("store")) if state else None), replicas=replicas, report_gaps=report_gaps)for url in peers or []: hub.add_peer(url) for url in follow or []: diff --git a/src/pyrtan/js/pyrtan.js b/src/pyrtan/js/pyrtan.js index eac836c..ed71cb1 100644 --- a/src/pyrtan/js/pyrtan.js +++ b/src/pyrtan/js/pyrtan.js@@ -540,7 +540,7 @@constructor(raw) { this.raw = raw; this.name = raw.name; this.version = raw.version || 1; this.description = raw.description || "";- this.hosts = raw.hosts || []; this.start = raw.start; this.tags = raw.tags || []; this.license = raw.license || "";+ this.hosts = raw.hosts || []; this.start = raw.start; this.tags = raw.tags || []; this.license = raw.license || ""; this.namespace = raw.namespace || "";for (const k of LIST_NAMES) this[k] = raw[k] || []; this.language = raw.language || {}; this.triggers = (raw.triggers || []).map((t) => (typeof t === "string" ? { pattern: t } : t));@@ -553,6 +553,7 @@for (const [id, spec] of Object.entries(raw.nodes || {})) this.nodes[id] = { id, kind: spec.kind, spec, get: (k, d) => (spec[k] !== undefined ? spec[k] : d), next: spec.next }; } node(id) { const n = this.nodes[id]; if (!n) throw new TreeError(`${this.name}: no node ${pyrepr(id)}`); return n; }+ get id() { return this.namespace ? `${this.namespace}/${this.name}` : this.name; }hostAllowed(url) { let host = ""; try { host = new URL(url).hostname.toLowerCase(); } catch (e) { return false; }@@ -607,7 +608,9 @@if (s.description) d.description = s.description; inputs[n] = d; }- const cap = { name: this.name, version: this.version, description: this.description, tags: this.tags, hosts: this.hosts };+ const cap = { name: this.name, version: this.version };+ if (this.namespace) cap.namespace = this.namespace;+ Object.assign(cap, { description: this.description, tags: this.tags, hosts: this.hosts });for (const k of LIST_NAMES) if (this[k].length) cap[k] = this[k]; return Object.assign(cap, { sources: this.sources(), license: this.license, language: this.language, triggers: this.triggers.map((t) => t.pattern), inputs }); }@@ -905,5 +908,5 @@return outcome(d.status || "error", d.tree || fallbackTree, { say: d.say === undefined ? null : d.say, output: d.output === undefined ? null : d.output, reason: d.reason === undefined ? null : d.reason, question: d.question === undefined ? null : d.question, snapshot: d.snapshot === undefined ? null : d.snapshot, trace, inputs: d.inputs || {} }); }- return { MISSING, Tree, Runner, Trace, FetchTransport, RecordedTransport, Response, render, resolve, holds, extract, outcomeDict, fromDict, pyrepr, pyJson, html, TemplateError, TreeError, TransportError, ConditionError, SOURCES, adapterKey, version: "0.7.0" };+ return { MISSING, Tree, Runner, Trace, FetchTransport, RecordedTransport, Response, render, resolve, holds, extract, outcomeDict, fromDict, pyrepr, pyJson, html, TemplateError, TreeError, TransportError, ConditionError, SOURCES, adapterKey, version: "0.8.0" };}); diff --git a/src/pyrtan/service.py b/src/pyrtan/service.py index 6bbd55f..82d90c8 100644 --- a/src/pyrtan/service.py +++ b/src/pyrtan/service.py@@ -64,7 +64,7 @@ from .transport import Transport, UrllibTransportMAX_BODY = 1 << 20 # What answers without a token: the console, its index, and liveness.-PUBLIC_PATHS = frozenset({"/", "/index.html", "/console", "/api", "/health", "/actor", "/directory", "/manifest.webmanifest", "/sw.js", "/pyrtan.js"})+PUBLIC_PATHS = frozenset({"/", "/index.html", "/console", "/api", "/health", "/actor", "/directory", "/reports", "/manifest.webmanifest", "/sw.js", "/pyrtan.js"})# On a federated hub, a request carrying a signature reaches the route, # which verifies it (see :mod:`pyrtan.federation`). SIGNED_PATHS = frozenset({"/inbox", "/run", "/intent", "/resume", "/capabilities", "/mailbox"}) diff --git a/src/pyrtan/transport.py b/src/pyrtan/transport.py index c86c1e7..ab88770 100644 --- a/src/pyrtan/transport.py +++ b/src/pyrtan/transport.py@@ -129,7 +129,7 @@ class UrllibTransport:``public`` (refuse private, loopback, link-local and metadata addresses, on the first URL and on every redirect)."""- def __init__(self, user_agent: str = "pyrtan/0.7 (+https://github.com/b-i-t-8/pyrtan)", egress: str = "any") -> None:+ def __init__(self, user_agent: str = "pyrtan/0.8 (+https://github.com/b-i-t-8/pyrtan)", egress: str = "any") -> None:if egress not in EGRESS_POLICIES: raise ValueError(f"egress policy must be one of {EGRESS_POLICIES}, got {egress!r}") self.user_agent = user_agent diff --git a/src/pyrtan/tree.py b/src/pyrtan/tree.py index 8e78fbe..ecd0414 100644 --- a/src/pyrtan/tree.py +++ b/src/pyrtan/tree.py@@ -163,6 +163,7 @@ class Tree:raw: dict[str, Any] = field(default_factory=dict, repr=False) license: str = "" language: Language = field(default_factory=Language)+ namespace: str = "" # who published it: the catalog id is namespace/namepaths: list[str] = field(default_factory=list) # what ``file`` nodes may reach commands: list[str] = field(default_factory=list) # what ``shell`` nodes may run devices: list[str] = field(default_factory=list) # what ``serial`` nodes may open@@ -190,6 +191,11 @@ class Tree:return True return False+ @property+ def id(self) -> str:+ """``namespace/name`` in a catalog; just the name without one."""+ return f"{self.namespace}/{self.name}" if self.namespace else self.name+def allowed(self, source: str, spec: dict[str, Any]) -> tuple[str, str] | None: """Why a rendered request to ``source`` is outside this tree's allow-list, as ``(reason, text)``, or None when it is inside."""@@ -243,6 +249,7 @@ class Tree:return { "name": self.name, "version": self.version,+ **({"namespace": self.namespace} if self.namespace else {}),"description": self.description, "tags": self.tags, "hosts": self.hosts,@@ -300,6 +307,9 @@ def build(raw: dict[str, Any], source: str | None = None) -> Tree:license_ = raw.get("license", "") if not isinstance(license_, str): raise TreeError(f"{name}: 'license' must be a string such as \"MIT\" (an SPDX identifier)")+ namespace = raw.get("namespace", "")+ if not isinstance(namespace, str) or (namespace and not NAME.match(namespace)):+ raise TreeError(f"{name}: 'namespace' must be lowercase letters, digits and dashes (who publishes the tree), got {namespace!r}")try: language = Language.from_raw(raw.get("language"), "language")@@ -360,7 +370,7 @@ def build(raw: dict[str, Any], source: str | None = None) -> Tree:hint = " (top-level keys such as 'start' must come before the first [table] in TOML)" if _buried(raw, "start") else "" raise TreeError(f"{name}: 'start' must name a node{hint}")- tree = Tree(name, description, version, hosts, triggers, inputs, start, nodes, tags, source, raw, license_.strip(), language, **lists)+ tree = Tree(name, description, version, hosts, triggers, inputs, start, nodes, tags, source, raw, license_.strip(), language, namespace, **lists)errors = [p for p in check(tree) if p.level == "error"] if errors: raise TreeError(f"{name}: " + "; ".join(f"{p.where}: {p.message}" for p in errors)) diff --git a/tests/conformance/vectors.json b/tests/conformance/vectors.json index 45f5b80..5c96d82 100644 --- a/tests/conformance/vectors.json +++ b/tests/conformance/vectors.json@@ -2282,6 +2282,7 @@}, "license": "MIT", "name": "disk-space",+ "namespace": "pyrtan","nodes": { "df": { "extract": [@@ -2394,7 +2395,7 @@"pattern": "disk space on {mount:word}" } ],- "version": 1+ "version": 2}, "github-release": { "description": "The latest release of a GitHub repository.",@@ -2410,6 +2411,7 @@}, "license": "MIT", "name": "github-release",+ "namespace": "pyrtan","nodes": { "fetch": { "extract": [@@ -2516,7 +2518,7 @@"pattern": "latest {repo:word} release" } ],- "version": 1+ "version": 2}, "hacker-news": { "description": "Top stories on Hacker News, from its public API.",@@ -2534,6 +2536,7 @@}, "license": "MIT", "name": "hacker-news",+ "namespace": "pyrtan","nodes": { "any": { "branches": [@@ -2642,7 +2645,7 @@"pattern": "top {count:int} stories on hacker news" } ],- "version": 1+ "version": 2}, "hacker-news-item": { "description": "One Hacker News item by id: title, url, score, author.",@@ -2658,6 +2661,7 @@}, "license": "MIT", "name": "hacker-news-item",+ "namespace": "pyrtan","nodes": { "fetch": { "extract": [@@ -2771,7 +2775,7 @@"pattern": "hacker news item {id:int}" } ],- "version": 1+ "version": 2}, "log-errors": { "description": "Error lines in the tail of a log file on this box.",@@ -2790,6 +2794,7 @@}, "license": "MIT", "name": "log-errors",+ "namespace": "pyrtan","nodes": { "any": { "branches": [@@ -2885,7 +2890,7 @@"pattern": "errors in {path:word}" } ],- "version": 1+ "version": 2}, "modem-check": { "description": "Whether a modem on a serial line answers AT.",@@ -2901,6 +2906,7 @@}, "license": "MIT", "name": "modem-check",+ "namespace": "pyrtan","nodes": { "answered": { "branches": [@@ -2973,7 +2979,7 @@"pattern": "is the modem on {device:word} up" } ],- "version": 1+ "version": 2}, "rendered-headline": { "description": "The main heading of a page that needs a script to render, through a real browser.",@@ -2988,6 +2994,7 @@}, "license": "MIT", "name": "rendered-headline",+ "namespace": "pyrtan","nodes": { "found": { "branches": [@@ -3070,7 +3077,7 @@"pattern": "what does {url:url} say" } ],- "version": 1+ "version": 2}, "room-temperature": { "description": "The temperature a room's sensor last published, over MQTT.",@@ -3086,6 +3093,7 @@}, "license": "MIT", "name": "room-temperature",+ "namespace": "pyrtan","nodes": { "cold": { "kind": "done",@@ -3186,7 +3194,7 @@"pattern": "how warm is the {room:word}" } ],- "version": 1+ "version": 2}, "site-status": { "description": "Whether a web site answers, and how quickly.",@@ -3202,6 +3210,7 @@}, "license": "MIT", "name": "site-status",+ "namespace": "pyrtan","nodes": { "erroring": { "kind": "fail",@@ -3299,7 +3308,7 @@"pattern": "[check] status of {host:host}" } ],- "version": 1+ "version": 2}, "weather": { "description": "Current weather for a place, from Open-Meteo.",@@ -3331,6 +3340,7 @@}, "license": "MIT", "name": "weather",+ "namespace": "pyrtan","nodes": { "clear": { "kind": "set",@@ -3591,7 +3601,7 @@"pattern": "how [hot|cold|warm] is it in {place}" } ],- "version": 1+ "version": 2} } } diff --git a/tests/test_catalog.py b/tests/test_catalog.py index 7831913..52dd8c5 100644 --- a/tests/test_catalog.py +++ b/tests/test_catalog.py@@ -71,9 +71,9 @@ def test_search_and_find(served):assert [e["name"] for e in cat.search("open-meteo")] == ["weather"] assert [e["name"] for e in cat.search("news stories")] == ["hacker-news"] assert cat.search("nothing like this") == []- assert cat.find("weather")["version"] == 1- assert cat.find("weather@1")["name"] == "weather"- with pytest.raises(CatalogError, match="has version 1, not 4"):+ assert cat.find("weather")["version"] == 2+ assert cat.find("weather@2")["name"] == "weather"+ with pytest.raises(CatalogError, match="has version 2, not 4"):cat.find("weather@4") with pytest.raises(CatalogError, match="not in the catalog"): cat.find("nope")@@ -131,11 +131,140 @@ def test_cli_index_search_pull(catalog_dir, served, tmp_path, capsys):assert main(["index", str(catalog_dir), "--out", str(out)]) == 0 assert json.loads(out.read_text())["count"] == 10 assert main(["search", "weather", "--catalog", served]) == 0- assert "weather@1" in capsys.readouterr().out+ assert "weather@2" in capsys.readouterr().outassert main(["search", "zzz", "--catalog", served]) == 1- assert main(["pull", "site-status", "weather@1", "--catalog", served, "--dir", str(tmp_path / "t")]) == 0+ assert main(["pull", "site-status", "weather@2", "--catalog", served, "--dir", str(tmp_path / "t")]) == 0assert (tmp_path / "t" / "site-status.toml").exists() and (tmp_path / "t" / "weather.toml").exists() assert main(["pull", "weather@9", "--catalog", served, "--dir", str(tmp_path / "t")]) == 1 assert "not 9" in capsys.readouterr().err assert main(["console", "--out", str(tmp_path / "site" / "console.html")]) == 0 assert "<title>Pyrtan</title>" in (tmp_path / "site" / "console.html").read_text()+++# ---------------------------------------------------------------- namespaces, the gate, publishing+++OTHER = '''+name = "weather"+version = 2+namespace = "barry"+description = "someone else's weather"+license = "MIT"+hosts = ["example.com"]+start = "get"++[nodes.get]+kind = "observe"+http = { url = "https://example.com/" }+next = "say"++[nodes.say]+kind = "done"+say = "ok"+'''+++def test_namespaces_in_the_index_and_in_pulls(tmp_path):+ src = tmp_path / "src"+ (src / "examples").mkdir(parents=True)+ shutil.copytree(EXAMPLES, src / "examples", dirs_exist_ok=True)+ (src / "catalog" / "barry").mkdir(parents=True)+ (src / "catalog" / "barry" / "weather.toml").write_text(OTHER)+ out = tmp_path / "out"+ index, problems = build_index([src / "examples", src / "catalog"], copy_to=out, require_namespace=True)+ assert not problems, problems+ ids = {e["id"]: e for e in index["trees"]}+ assert ids["pyrtan/weather"]["path"] == "pyrtan/weather.toml" and ids["barry/weather"]["path"] == "barry/weather.toml"+ assert (out / "pyrtan" / "weather.toml").exists() and (out / "barry" / "weather.toml").read_text() == OTHER+ assert ids["pyrtan/disk-space"]["sources"] == ["shell"] and ids["pyrtan/disk-space"]["commands"] == ["df"]+ (out / "index.json").write_text(json.dumps(index))+ cat = Catalog(str(out))+ # A bare name is enough when one namespace has it; with two, say which.+ assert cat.find("github-release")["id"] == "pyrtan/github-release"+ with pytest.raises(CatalogError, match="more than one namespace: barry/weather, pyrtan/weather"):+ cat.find("weather")+ assert cat.find("barry/weather")["version"] == 2 and cat.find("pyrtan/weather@2")["namespace"] == "pyrtan"+ with pytest.raises(CatalogError, match="has version 2, not 9"):+ cat.find("barry/weather@9")+ with pytest.raises(CatalogError, match="not in the catalog"):+ cat.find("nobody/weather")+ assert [e["id"] for e in cat.search("barry")] == ["barry/weather"]+ assert {e["id"] for e in cat.search("shell")} == {"pyrtan/disk-space"}+ target = cat.pull("barry/weather", tmp_path / "trees")+ assert target.name == "weather.toml" and target.read_text() == OTHER+ # A file whose namespace is not the index's is refused.+ (out / "barry" / "weather.toml").write_text(OTHER.replace('namespace = "barry"', 'namespace = "mallory"'))+ index["trees"] = [dict(e, sha256=hashlib.sha256((out / e["path"]).read_bytes()).hexdigest()) if e["id"] == "barry/weather" else e for e in index["trees"]]+ (out / "index.json").write_text(json.dumps(index))+ with pytest.raises(CatalogError, match="namespace is 'mallory'"):+ Catalog(str(out)).pull("barry/weather", tmp_path / "trees2")+ # Two directories need a place to copy to; one does not.+ with pytest.raises(ValueError):+ build_index([src / "examples", src / "catalog"])+ one, problems = build_index(src / "examples")+ assert not problems and {e["path"] for e in one["trees"]} >= {"weather/weather.toml"}+++def test_gate_and_publish(tmp_path, capsys):+ from pyrtan.catalog import gate, publish++ repo = tmp_path / "repo"+ shutil.copytree(EXAMPLES, repo / "examples")+ (repo / "catalog").mkdir()+ assert gate(repo) == []+ # Publishing needs a namespace and a licence in the file, and an owner for a new namespace.+ tree = tmp_path / "lan-printer.toml"+ tree.write_text(OTHER.replace('name = "weather"', 'name = "lan-printer"').replace('namespace = "barry"\n', ""))+ with pytest.raises(CatalogError, match='namespace = "you"'):+ publish(tree, repo, owners=["barry"])+ tree.write_text(OTHER.replace('name = "weather"', 'name = "lan-printer"'))+ with pytest.raises(CatalogError, match="say who owns it"):+ publish(tree, repo)+ placed, notes = publish(tree, repo, owners=["@Barry"])+ assert placed == (repo / "catalog" / "barry" / "lan-printer.toml").resolve() and notes == ["new namespace catalog/barry/ owned by barry"]+ assert (repo / "catalog" / "barry" / "OWNERS").read_text().endswith("barry\n")+ assert gate(repo) == []+ # The same file again is a no-op; a changed file must bump its version.+ assert publish(tree, repo)[1] == ["catalog/barry/lan-printer.toml already holds this exact file"]+ tree.write_text(tree.read_text().replace('say = "ok"', 'say = "changed"'))+ with pytest.raises(CatalogError, match="bump this file's version past it"):+ publish(tree, repo)+ tree.write_text(tree.read_text().replace("version = 2", "version = 3"))+ assert publish(tree, repo, owners=["alice"])[1] == ["added alice to catalog/barry/OWNERS", "replaces version 2 with 3"]+ # What the gate refuses.+ bad = repo / "catalog" / "barry" / "weather.toml"+ bad.write_text(OTHER.replace('namespace = "barry"', 'namespace = "pyrtan"'))+ problems = [str(p) for p in gate(repo)]+ assert any("catalog/barry/weather.toml: a tree with id 'pyrtan/weather' is already at" in p for p in problems)+ assert any("sits under catalog/barry/" in p for p in problems)+ bad.write_text(OTHER.replace('name = "weather"', 'name = "other"'))+ assert any("its file must be other.toml" in p for p in [str(p) for p in gate(repo)])+ bad.unlink()+ (repo / "catalog" / "barry" / "notes.txt").write_text("x")+ assert any("only <name>.toml files and OWNERS belong" in p for p in [str(p) for p in gate(repo)])+ (repo / "catalog" / "barry" / "notes.txt").unlink()+ (repo / "catalog" / "nobody").mkdir()+ (repo / "catalog" / "nobody" / "t.toml").write_text(OTHER.replace('namespace = "barry"', 'namespace = "nobody"').replace('name = "weather"', 'name = "t"'))+ assert any("needs an OWNERS file" in p for p in [str(p) for p in gate(repo)])+ shutil.rmtree(repo / "catalog" / "nobody")+ (repo / "examples" / "weather" / "weather.toml").write_text((repo / "examples" / "weather" / "weather.toml").read_text().replace('namespace = "pyrtan"', 'namespace = "barry"'))+ assert any("published by 'pyrtan', not 'barry'" in p for p in [str(p) for p in gate(repo)])+ shutil.copy(EXAMPLES / "weather" / "weather.toml", repo / "examples" / "weather" / "weather.toml")+ assert gate(repo) == []+ # Only owners change a namespace; a changed file bumps its version.+ assert [str(p) for p in gate(repo, changed=["catalog/barry/lan-printer.toml"], author="mallory")] == ["catalog/barry/lan-printer.toml: mallory is not in catalog/barry/OWNERS; only a namespace's owners change it"]+ assert gate(repo, changed=["catalog/barry/lan-printer.toml"], author="Alice") == []+ assert gate(repo, changed=["catalog/barry/lan-printer.toml"], author="alice", base={"catalog/barry/lan-printer.toml": (3, b"older bytes")}) != []+ assert gate(repo, changed=["catalog/barry/lan-printer.toml"], author="alice", base={"catalog/barry/lan-printer.toml": (2, b"older bytes")}) == []+ assert gate(repo, base={"catalog/barry/gone.toml": (1, b"x")}) == [] # a removal is fine+ # The command, and the index the site serves.+ assert main(["publish", str(tree), "--into", str(repo)]) == 0+ out = capsys.readouterr().out+ assert "placed" in out and "git checkout -b catalog/barry-lan-printer" in out+ assert main(["index", str(repo / "examples"), str(repo / "catalog"), "--copy", str(tmp_path / "site"), "--require-namespace"]) == 0+ served = json.loads((tmp_path / "site" / "index.json").read_text())+ assert {e["id"] for e in served["trees"]} == {f"pyrtan/{n}" for n in ("weather", "hacker-news", "hacker-news-item", "site-status", "github-release", "disk-space", "log-errors", "room-temperature", "rendered-headline", "modem-check")} | {"barry/lan-printer"}+ assert main(["index", str(repo / "examples"), str(repo / "catalog")]) == 2 # two roots need --copy+ assert main(["search", "lan", "--catalog", str(tmp_path / "site")]) == 0 and "barry/lan-printer@3" in capsys.readouterr().out+ assert main(["pull", "lan-printer", "--catalog", str(tmp_path / "site"), "--dir", str(tmp_path / "pulled")]) == 0+ assert "barry/lan-printer@3 ->" in capsys.readouterr().outdiff --git a/tests/test_federation.py b/tests/test_federation.py index 6634bde..f80a9ba 100644 --- a/tests/test_federation.py +++ b/tests/test_federation.py@@ -12,7 +12,7 @@ from pyrtan.cli import mainfrom pyrtan.federation import Federation, Remote, sign, verify from pyrtan.hub import Hub from pyrtan.service import Server, Service, http_json-from conftest import EXAMPLES, FIXTURES+from conftest import EXAMPLES, FIXTURES, ROOTECHO = """ name = "{name}"@@ -757,3 +757,96 @@ say = "{{ state.last.stdout }}"finally: hub.stop() srv.stop()+++GAPPY = '''+name = "gappy"+version = 2+namespace = "pyrtan"+description = "a tree with a decide that covers one value"+start = "go"++[inputs.v]+type = "int"++[nodes.go]+kind = "set"+set = { x = "{{ inputs.v }}" }+next = "pick"++[nodes.pick]+kind = "decide"+on = "x"+branches = [{ when = { eq = 1 }, then = "one" }]++[nodes.one]+kind = "done"+say = "one"+'''+++def test_gap_reports_travel_with_announces_when_a_hub_opts_in(tmp_path):+ (tmp_path / "g").mkdir()+ (tmp_path / "g" / "gappy.toml").write_text(GAPPY)+ (tmp_path / "g" / "plain.toml").write_text(ECHO.format(name="plain", word="plain"))+ from pyrtan.hub import serve_hub++ a_hub, a_srv = serve_hub(port=0, quiet=True, name="reporter", trees=[tmp_path / "g"], federate=True, report_gaps=True, poll_every=60)+ b_hub, b_srv = serve_hub(port=0, quiet=True, name="carrier", federate=True, poll_every=60, state=str(tmp_path / "b.json"))+ try:+ # Two gaps on the catalog tree (namespace + version), seen twice and once.+ for v in (7, 7, 9):+ out = http_json("POST", f"{a_srv.url}/run", {"tree": "gappy", "inputs": {"v": v}})[1]+ assert out["reason"] == "gap"+ own = sorted(a_hub.gap_reports(), key=lambda r: r["observed"])+ assert [(r["id"], r["node"], r["on"], r["observed"], r["count"]) for r in own] == [("pyrtan/gappy@2", "pick", "state.x", 7, 2), ("pyrtan/gappy@2", "pick", "state.x", 9, 1)]+ _, view = http_json("GET", f"{a_srv.url}/reports") # public, like /directory+ assert view["reporting"] is True and view["count"] == 2 and view["reports"][0] == {"id": "pyrtan/gappy@2", "node": "pick", "on": "state.x", "observed": 7, "count": 2, "reporters": 1, "last": view["reports"][0]["last"]}+ # B follows A: A's announce carries the reports; B keeps them, says who reported, and passes them on.+ assert http_json("POST", f"{b_srv.url}/follow", {"url": a_srv.url})[0] == 200+ _, bview = http_json("GET", f"{b_srv.url}/reports")+ assert bview["reporting"] is False and bview["count"] == 2 and bview["reports"][0]["reporters"] == 1+ assert {r["reporter"] for r in b_hub.federation.reports.values()} == {a_srv.url}+ assert any(r["id"] == "pyrtan/gappy@2" for r in b_hub.federation.report_gossip())+ # The same gap reported by a second hub counts that hub too; B's own announce (not opted in) carries only what it heard.+ b_hub.federation.merge_reports([{"id": "pyrtan/gappy@2", "node": "pick", "on": "state.x", "observed": 7, "count": 5, "last": time.time()}], via="http://other.example")+ row = next(r for r in http_json("GET", f"{b_srv.url}/reports")[1]["reports"] if r["observed"] == 7)+ assert row["count"] == 7 and row["reporters"] == 2+ # Shaping: never an input, no id outside namespace/name@version, no future, no old news, long values summarised.+ now = time.time()+ assert b_hub.federation.merge_reports([{"id": "gappy", "node": "pick", "on": "x", "observed": 1}, {"id": "pyrtan/gappy@2", "node": "pick", "on": "x", "observed": 1, "last": now - 60 * 86400}, "junk"], via="x") == 0+ assert b_hub.federation.merge_reports([{"id": "a/b@1", "node": "n", "on": "o", "observed": "v" * 200, "last": now + 1e6, "count": "3"}], via="http://x") == 1+ shaped = next(r for r in b_hub.federation.reports.values() if r["id"] == "a/b@1")+ assert shaped["observed"] == "<str>" and shaped["count"] == 3 and shaped["last"] <= time.time() and shaped["reporter"] == "http://x"+ assert b_hub.federation.merge_reports([{"id": "a/b@1", "node": "n", "on": "o", "observed": {"k": 1}, "reporter": "https://r.example/"}], via="http://x") == 1+ assert any(r["reporter"] == "https://r.example" and r["observed"] == "<dict>" for r in b_hub.federation.reports.values())+ # Reports survive a restart through the state file.+ b_hub.federation._save()+ again = Federation(b_srv.url, state=str(tmp_path / "b.json"))+ assert {r["id"] for r in again.reports.values()} == {"pyrtan/gappy@2", "a/b@1"}+ # A tree without a namespace reports nothing.+ assert not any(r["id"].endswith("/plain@1") for r in a_hub.gap_reports())+ finally:+ a_hub.stop(); a_srv.stop(); b_hub.stop(); b_srv.stop()+++def test_reports_command_and_snapshot(two, tmp_path, capsys, monkeypatch):+ a, b = two+ a.fed.report_gaps = True+ a.fed.own_reports = lambda: [{"id": "pyrtan/weather@1", "node": "code", "on": "state.code", "observed": 99, "count": 4, "last": time.time()}]+ assert main(["reports", "--at", a.url]) == 0+ assert "pyrtan/weather@1 code: state.code = 99 ×4" in capsys.readouterr().out+ assert main(["reports", "nobody/x", "--at", a.url]) == 0 and "no gap reports" in capsys.readouterr().out+ assert main(["reports", "--at", a.url, "--json"]) == 0 and json.loads(capsys.readouterr().out)["reports"][0]["id"] == "pyrtan/weather@1"+ # The site's snapshot asks the seeds for /reports and bakes reports.json; the command reads that when no hub is given.+ import subprocess, sys as _sys++ seeds = tmp_path / "seeds.json"+ seeds.write_text(json.dumps({"hubs": [a.url]}))+ out = subprocess.run([_sys.executable, "site/snapshot.py", str(seeds), str(tmp_path / "directory.json"), "--reports", str(tmp_path / "reports.json")], capture_output=True, text=True, cwd=str(ROOT))+ assert out.returncode == 0, out.stderr+ baked = json.loads((tmp_path / "reports.json").read_text())+ assert baked["count"] == 1 and baked["reports"][0]["id"] == "pyrtan/weather@1" and baked["reports"][0]["count"] == 4+ monkeypatch.setenv("PYRTAN_REPORTS", str(tmp_path / "reports.json"))+ monkeypatch.delenv("PYRTAN_AT", raising=False)+ assert main(["reports"]) == 0 and "pyrtan/weather@1 code" in capsys.readouterr().outdiff --git a/uv.lock b/uv.lock index b9aeb34..488a654 100644 --- a/uv.lock +++ b/uv.lock@@ -147,7 +147,7 @@ wheels = [[[package]] name = "pyrtan"-version = "0.7.0"+version = "0.8.0"source = { editable = "." } [package.optional-dependencies]