Merge pull request #6 from b-i-t-8/claude/ecstatic-euler-rki9t4
b-i-t-8 · 2026-10-25 · 5367bc2a91751c5b157c2cbebdc8130a9590894f · parents dfd0fda 084dee6
Keep the directory on the swarm, seed it from pyrtan.com, hold trees in the hub (v0.4.0)
Changes
README.md | 28 +++-- _meta/design.md | 10 +- _meta/federation.md | 96 ++++++++++----- _meta/platform.md | 18 +-- deploy/README.md | 18 +-- deploy/openmediavault/docker-compose.yml | 12 ++ deploy/pyrtan.com/README.md | 76 ++++++------ deploy/pyrtan.com/catalog.yml | 2 +- deploy/pyrtan.com/index.html | 47 +++++++- deploy/pyrtan.com/rendezvous/Caddyfile | 7 -- deploy/pyrtan.com/rendezvous/docker-compose.yml | 75 ------------ deploy/pyrtan.com/rendezvous/seeds.json | 5 + deploy/pyrtan.com/site.yml | 4 +- pyproject.toml | 2 +- src/pyrtan/__init__.py | 2 +- src/pyrtan/catalog.py | 2 +- src/pyrtan/cli.py | 91 +++++++++----- src/pyrtan/federation.py | 288 +++++++++++++++++++++++++++++++++----------- src/pyrtan/hub.py | 125 +++++++++++++++---- src/pyrtan/page.py | 11 +- src/pyrtan/transport.py | 2 +- tests/test_federation.py | 146 +++++++++++++++++----- uv.lock | 2 +- 23 files changed, 716 insertions(+), 353 deletions(-)
diff --git a/README.md b/README.md index 90d238a..afe4cab 100644 --- a/README.md +++ b/README.md@@ -124,8 +124,8 @@ whole format is in [`_meta/tree-format.md`](_meta/tree-format.md).| `pyrtan run --tree NAME --input k=v` | Runs a named tree. | | `pyrtan match "…" --trees DIR` | How a sentence reads, without running anything: the rewrites, every trigger's verdict, the winner. The tool for writing triggers. | | `pyrtan serve DIR --port 8701` | Hosts the trees as a service: `/health`, `/capabilities`, `/intent`, `/run`, `/resume`, `/gaps`, `/unanswered`. Reloads trees when their files change. `--gaps FILE` keeps gaps across restarts; `--egress public` refuses private addresses. |-| `pyrtan hub --port 8700` | Routes between services. Services register with `--hub URL`; peers carry a `link` of `up`, `stale` or `down`. `--federate` lets other hubs follow it and it follow them. |-| `pyrtan follow [URL] --at HUB` | Makes your hub follow another (default: the rendezvous at hub.pyrtan.com): it then routes sentences to that hub's trees too. `--listed` asks to be shown in its directory. `unfollow` undoes it; `peers` lists services, hubs followed and followers; `directory` lists hubs that asked to be found. |+| `pyrtan hub --port 8700` | Routes between services. Services register with `--hub URL`; peers carry a `link` of `up`, `stale` or `down`. `--trees DIR` holds trees in the hub itself; `--federate` lets other hubs follow it and it follow them; `--listed` puts it in the directory. |+| `pyrtan follow [URL] --at HUB` | Makes your hub follow another (default: the first live seed from pyrtan.com): it then routes sentences to that hub's trees too. `--listed` also puts your hub in the directory. `unfollow` undoes it; `peers` lists services, hubs followed and followers; `directory` lists every hub that asked to be found, as the swarm knows it. || `pyrtan ask "…" --at URL` | Answers a sentence through a running service or hub. | | `pyrtan gaps --at URL` | Every decision that met a value with no branch: the branches to write next. | | `pyrtan unanswered --at URL` | Every sentence no tree answered, with the tree that would: the trees to write next. `--scaffold DIR` writes them. |@@ -183,12 +183,18 @@ never what it learned from others, and refuses to forward a request thatalready crossed a hub, so there is nothing to loop. **Every message is signed**, and a hub started with `--federation-secret` takes follows only from hubs that know the secret; `--closed` takes none. `--state FILE`-keeps the relationships across restarts. The hub at-<https://hub.pyrtan.com> is the **rendezvous**: it holds the example-trees, anyone may follow it (`pyrtan follow` with no URL does), and-`pyrtan follow --listed` puts your hub in its directory (`pyrtan-directory`) so other operators can find and follow it. It routes to-nobody: finding is the directory's job, following is yours. The answer to a federated+keeps the relationships across restarts.++Hubs find each other without a server. A hub started `--listed` puts an+entry for itself (name, URL, the trees it holds) in every announce it+sends, every hub keeps the entries it hears and passes them on in both+directions of every follow, and `pyrtan directory` (or the console, or+any hub's `/directory`) shows the result. pyrtan.com publishes only+`rendezvous/seeds.json`, a static list of hubs to follow first; `pyrtan+follow` with no URL takes the first that answers. An entry is a hint,+not a route: to use a hub you found, follow it. A hub can also hold trees+itself, `pyrtan hub --trees DIR`, so a whole federated swarm can be one+process on one box. The answer to a federatedsentence carries `via: [the hub it crossed]`, and the console's Peers tab shows who follows whom. The design is in [`_meta/federation.md`](_meta/federation.md).@@ -331,9 +337,9 @@ deploy/ Dockerfile at the root; a compose stack for OpenMediaVault,## Status-0.3.0. The core, the service, the hub, the command, the console, the-catalog client, the language rules and federation between hubs are built-and tested; HTTP is the one form of I/O the core speaks; a swarm can+0.4.0. The core, the service, the hub, the command, the console, the+catalog client, the language rules, federation between hubs and the+gossiped directory are built and tested; HTTP is the one form of I/O the core speaks; a swarm canrequire a token, keep its gaps, refuse private addresses, and follow other swarms. Not yet built: further adapters (a browser for pages that need a script to render; files, shells, serial lines, message queues), diff --git a/_meta/design.md b/_meta/design.md index 867d7ca..1e504f7 100644 --- a/_meta/design.md +++ b/_meta/design.md@@ -282,17 +282,21 @@ Five trees, all against real public APIs, all tested offline:sentence"), and federation in the shape of ActivityPub: a hub follows other hubs and routes to them, one hop, every message signed, following as the only trust. Design in [`federation.md`](federation.md).-4. **v0.4: more forms of I/O.** First the browser: `observe` with+4. **v0.4, built: no server anywhere.** The directory hubs find each+ other by is kept and gossiped by every federated hub, with a static+ seed list at pyrtan.com in place of a rendezvous hub; a hub can hold+ trees itself (`--trees`), so a federated swarm can be one process.+5. **v0.5: more forms of I/O.** First the browser: `observe` with`browser = { url, wait_for, actions }` on Playwright as an optional extra, for pages that need a script to render, with the same host rule and the same extraction. Then file, shell, serial and queue adapters, each a node source with its own allow-list (paths, commands, devices, topics), so a tree reads a log, runs a named command or polls a device by the same rules it reads a page, and the trace reads the same.-5. **v0.5: the catalog grows up.** Namespaces, publishing through pull+6. **v0.6: the catalog grows up.** Namespaces, publishing through pullrequests with the checker as the gate, and opt-in gap reporting from hubs back to a tree's author, carried as another federation topic.-6. **Beside the swarm, never inside it:** an authoring aid that reads gaps+7. **Beside the swarm, never inside it:** an authoring aid that reads gapsand drafts the branch for a person to accept. ## 7. Open questions diff --git a/_meta/federation.md b/_meta/federation.md index c860419..487bc93 100644 --- a/_meta/federation.md +++ b/_meta/federation.md@@ -2,9 +2,9 @@| | | |---|---|-| **Status** | **Built, v0.3.** `src/pyrtan/federation.py`, the hub's federation routes, `pyrtan follow` / `unfollow` / `peers`, the console's federation table, and tests of two and three hubs over real sockets. |+| **Status** | **Built, v0.3; the directory redone in v0.4.** `src/pyrtan/federation.py`, the hub's federation routes, `pyrtan follow` / `unfollow` / `peers` / `directory`, the console's federation and directory tables, and tests of two and three hubs over real sockets. || **Date** | 2026-10-25 |-| **Decision** | A hub may follow other hubs, in the shape of ActivityPub (actor, inbox, Follow/Accept/Undo/Announce), and route sentences to the hubs it follows. Following is trust, in one direction, and federation is one hop. |+| **Decision** | A hub may follow other hubs, in the shape of ActivityPub (actor, inbox, Follow/Accept/Undo/Announce), and route sentences to the hubs it follows. Following is trust, in one direction, and federation is one hop. Hubs find each other through a directory every hub keeps and gossips; nothing central runs. || **Where** | `src/pyrtan/federation.py`, `src/pyrtan/hub.py`. | ## 0. The brief@@ -134,10 +134,11 @@ hears from whom"; the wire format is Pyrtan's own and says so.**A directory of hubs that routes.** A place hubs register with, which then answers for all of them. Rejected: it would route to hubs nobody-chose. What was built instead is the *rendezvous* (§7): a hub anyone may-follow, whose directory lists the followers that asked to be listed, and-which routes to none of them. Discovery is a directory's job; routing-stays an operator's follow.+chose. **A rendezvous hub that holds the directory** (v0.3) was built+and then replaced: it was a server, and the point of a swarm is that+there is none. What stands is §7: a directory every hub keeps and+gossips, a static seed list, and routing that only ever follows an+operator's follow.**Multi-hop routing.** Announcing transitive capabilities with a hop count, and forwarding with a TTL. It would let a sentence cross three@@ -172,30 +173,65 @@ follow. `--closed` takes no follows at all while still following others.restart forgets them and the other side's `link` goes `down` until the follow is redone.-## 7. The rendezvous--A rendezvous is an ordinary federated hub that is public on purpose. The-one at `https://hub.pyrtan.com` (`deploy/pyrtan.com/rendezvous/`) holds-the example trees, takes open follows, follows nobody, and keeps a-directory:--- **To try federation:** `pyrtan follow` with no URL follows it (the- default is `$PYRTAN_RENDEZVOUS` or `https://hub.pyrtan.com`), and your- hub answers "what's the weather in Oslo" through it.-- **To be found:** `pyrtan follow --listed` asks the rendezvous to show- your hub at `GET /directory` (public), with the trees it holds. The- rendezvous reads a listed follower's `/capabilities`, signed with the- pair secret, on its poll; a follower that did not ask is never read.- `pyrtan directory` prints the list; the console shows it under- Directory.-- **What listing is not:** routing. The rendezvous forwards nothing to a- listed hub, and a hub that finds another in the directory follows it- with its own `pyrtan follow`, which is the one act of trust there is.--`listed` travels in the Follow activity and in the state file, so a-restart keeps the directory. Any federated hub can be a rendezvous for-its own circle: the directory is on every federated hub, empty until-someone asks.+## 7. The directory, and why nothing runs at pyrtan.com++v0.3 had a *rendezvous hub*: one public hub at `hub.pyrtan.com` that+kept a list of the followers who asked to be listed. It worked and it+was wrong, for the reason §1 gives: it was a server somebody had to run,+and a swarm that needs a server to find itself is not a swarm. v0.4+replaces it with a directory that every federated hub keeps.++**Entries.** A hub started `--listed` (or told `pyrtan follow --listed`,+or `POST /listed`) makes an entry for itself: its id, name, the names of+the trees it holds, and `seen`, the time it made the entry. That is all a+directory knows about a hub.++**Gossip.** Every `Announce` carries the sender's directory: its own+entry if listed, and every live entry it has heard. Announces now go+both ways: down to followers as before, and *up* from a follower to the+hubs it follows (`announce_up`), signed with the pair secret. The+followed hub routes none of what a follower announces, being followed+grants nothing, but it keeps the directory the announce carries. An+inbox answers an Announce with its own gossip, so one round trip+exchanges both ways. Announces go out when the trees or the directory+change and on a heartbeat (`ANNOUNCE_EVERY`, a minute), which is what+keeps `seen` fresh across the swarm.++**Merging.** A newer `seen` replaces an older one; a hub's own entry is+never overwritten by what others say about it; a `seen` from the future+is clamped to now; an entry unheard of for `DIRECTORY_TTL` (a week) is+dropped; the table is bounded (`DIRECTORY_MAX`, the oldest go first).+Each entry remembers `via`, the hub it was last heard from.++**Reading it.** `GET /directory` (public) on any federated hub lists what+that hub knows, each entry with `link` computed from `seen`: `up` within+ten minutes, `stale` within a day, `down` after. `pyrtan directory` asks+your own hub first and falls back to the seeds; the console shows it on+every federated hub; the page at pyrtan.com asks the seeds from the+visitor's browser.++**Seeds.** pyrtan.com publishes `rendezvous/seeds.json`, a static list+of hub URLs maintained by pull request, the role of a torrent's bootstrap+nodes. `pyrtan follow` with no URL follows the first seed whose `/actor`+answers; `pyrtan hub --follow seed` does the same on start. A seed is an+ordinary federated hub whose operator keeps it up; the first one is the+author's NAS. `$PYRTAN_SEEDS` points at another list (a URL or a file),+`$PYRTAN_RENDEZVOUS` at one hub.++**What an entry is not.** A hub two follows away cannot verify who made+an entry: there are no public keys in the standard library, and HMAC+proves things only between two hubs that share a secret. So an entry is+a *hint*: a name, a URL, a claim about trees. It is verified the one way+that matters, by following the hub, which fetches its `/actor` and runs+the handshake. The directory never routes. Signing entries with a+pure-Python Ed25519 would be three hundred lines of this author's own+cryptography for a property a follow already gives; not taken.++**Reachability.** A hub behind NAT can follow, announce upward, keep the+directory and pass it on, all outbound. It cannot be followed or asked,+because nobody can reach its inbox. To be routable it needs an address:+a port forward behind TLS, or a tunnel (Cloudflare Tunnel, Tailscale+Funnel) that costs nothing and opens no port. The deploy guide says how.## 8. What this design does not claim diff --git a/_meta/platform.md b/_meta/platform.md index 1bf702c..2435761 100644 --- a/_meta/platform.md +++ b/_meta/platform.md@@ -117,21 +117,21 @@ a gap), Trees (capabilities, with a link to the catalog), Peers (hub only;A browser will refuse a `http://` hub from an `https://` page. The console says so and points at the hub's own `/`, which is the same console.-### 2.6 The rendezvous (built)+### 2.6 The rendezvous (built; nothing runs)-pyrtan.com runs one public federated hub, `hub.pyrtan.com`, from the-same image as everything else (`deploy/pyrtan.com/rendezvous/`). It holds-the example trees so a new hub has something to federate with on day-one, and it keeps the opt-in directory from [`federation.md`](federation.md)-§7, which is how operators find each other. It follows nobody, so pyrtan.com-never routes a stranger's sentence to a stranger's hub. The site links-to it; the console on pyrtan.com can be pointed at it like at any hub.+pyrtan.com runs nothing. Hubs find each other through the directory+every federated hub keeps and gossips ([`federation.md`](federation.md)+§7); the site publishes `rendezvous/seeds.json`, a static list of hubs+to follow first, and its page reads the live directory from those seeds+in the visitor's browser. The first seed is the author's own NAS,+reachable through a tunnel. A hosted hub, when there is one, is just+another seed.### 2.7 Hosting (designed) A tenant gets a hub and one or more services as containers from this repository's image, with a token, `--cors https://pyrtan.com`, `--gaps` on-a volume, `--egress public`, `--federate --advertise https://<tenant>.pyrtan.com`+a volume, `--egress public`, `--federate --advertise https://<tenant>.pyrtan.com --listed`with `--state` on the volume, and TLS from the ingress. A hosted hub and a self-hosted one follow each other like any two hubs ([`federation.md`](federation.md)): a customer's NAS can answer for their diff --git a/deploy/README.md b/deploy/README.md index eb8ddf3..03184dc 100644 --- a/deploy/README.md +++ b/deploy/README.md@@ -62,13 +62,17 @@ restart; the container runs as uid 10001, so `chown 10001 trees` (or`chmod o+w trees`) on the NAS, else the gaps stay in memory and `/health` reports `gaps_file_error`.-To let this swarm answer for another (a friend's hub, or a hosted one),-add `--federate --advertise https://<your-public-url> --state-/trees/.federation.json` to the hub's command once it is behind TLS, then-`pyrtan follow https://other-hub --at https://<your-public-url> --token …`.-Following is one-directional trust: your hub routes to the hubs you-follow, and hubs that follow yours may ask it. See-[`../_meta/federation.md`](../_meta/federation.md).+To let this swarm answer for another (a friend's hub, or any hub in the+directory), give the hub a public address, with a tunnel that opens no+port (`cloudflared tunnel --url http://127.0.0.1:8700`, or `tailscale+funnel 8700`) or a port forward behind TLS, then set in the compose+file's environment `PYRTAN_ADVERTISE` (that address), `PYRTAN_FOLLOW:+seed` and `PYRTAN_LISTED: "true"`, with `PYRTAN_TOKEN` set. The hub then+follows the first seed from pyrtan.com, routes to it, and is listed in+the directory every hub gossips; `pyrtan follow URL --at <your hub>+--token …` follows any other. Following is one-directional trust: your+hub routes to the hubs you follow, and hubs that follow yours may ask it.+See [`../_meta/federation.md`](../_meta/federation.md).### Without the plugin diff --git a/deploy/openmediavault/docker-compose.yml b/deploy/openmediavault/docker-compose.yml index c36020e..3c003a4 100644 --- a/deploy/openmediavault/docker-compose.yml +++ b/deploy/openmediavault/docker-compose.yml@@ -33,6 +33,13 @@ x-pyrtan: &pyrtan# this before the hub is reachable from outside your LAN. # PYRTAN_TOKEN: "change-me-to-something-long" PYRTAN_EGRESS: any+ # Federation: once the hub is reachable from outside (a tunnel or a+ # port-forward with TLS), give it its public URL and it follows the+ # seeds at pyrtan.com and lists itself in the directory. Needs the token.+ # PYRTAN_ADVERTISE: "https://hub.example.net"+ # PYRTAN_FOLLOW: "seed"+ # PYRTAN_LISTED: "true"+ PYRTAN_STATE: /state/federation.jsonservices: hub:@@ -44,6 +51,8 @@ services:PYRTAN_PORT: "8700" ports: - "8700:8700"+ volumes:+ - hub-state:/statenews: <<: *pyrtan@@ -66,3 +75,6 @@ services:volumes: - /srv/dev-disk-by-uuid-CHANGE-ME/pyrtan/trees:/trees depends_on: [hub]++volumes:+ hub-state:diff --git a/deploy/pyrtan.com/README.md b/deploy/pyrtan.com/README.md index 9b8c83f..487344c 100644 --- a/deploy/pyrtan.com/README.md +++ b/deploy/pyrtan.com/README.md@@ -12,7 +12,7 @@ What to create, in order:| :-- | :-- | :-- | :-- | | 1 | `b-i-t-8/pyrtan-trees` | `pyrtan.com/catalog/` | its own `trees/` folder, checked and indexed by CI | | 2 | `b-i-t-8/pyrtan-site` | `pyrtan.com/` and `pyrtan.com/console/` | `index.html` here, and `pyrtan console --out` |-| 3 | a small box with Docker | `hub.pyrtan.com`, the rendezvous | [`rendezvous/`](rendezvous/) here |+| 3 | nothing | the rendezvous | `rendezvous/seeds.json` here, served by the site; your NAS is seed #1 |Both can be one repository if you prefer; two keeps "I wrote a tree" pull requests apart from "I changed the site" ones, and lets the catalog's@@ -141,49 +141,46 @@ a LAN over plain `http://` cannot be reached from an `https://` page (thebrowser blocks it); the console says so and points at the hub's own `/`, which is the same console served locally.-## 3. The rendezvous (`hub.pyrtan.com`)--The one piece of pyrtan.com that runs: a public federated hub with the-example trees, that anyone may follow and that lists the hubs which ask-to be found (`_meta/federation.md` §7). Any small VPS does; it holds no-data but the follow list.--1. **DNS:** an `A` (and `AAAA`) record for `hub.pyrtan.com` to the box.-2. **On the box**, with Docker installed:-- ```- mkdir -p /opt/rendezvous && cd /opt/rendezvous- curl -fsSLO https://raw.githubusercontent.com/b-i-t-8/pyrtan/main/deploy/pyrtan.com/rendezvous/docker-compose.yml- curl -fsSLO https://raw.githubusercontent.com/b-i-t-8/pyrtan/main/deploy/pyrtan.com/rendezvous/Caddyfile- echo "PYRTAN_TOKEN=$(openssl rand -hex 24)" > .env- docker compose up -d- ```-- Caddy fetches the certificate; the hub comes up as- `https://hub.pyrtan.com` with its one service of examples. The image- is published by this repository's `image` workflow on every tag- (`ghcr.io/b-i-t-8/pyrtan:0.3.0`); the first time, make the package- public under the repository's *Packages* settings, or `docker login- ghcr.io` on the box.-3. **Check:** `curl https://hub.pyrtan.com/actor` answers with- `"type": "Hub"`; `pyrtan directory` lists nobody yet;- `pyrtan ask "what's the weather in Oslo" --at https://hub.pyrtan.com- --token …` answers.-4. **From any other hub:** `pyrtan follow --at https://your-hub --token …`- follows it (the default URL is the rendezvous); add `--listed` to- appear in `pyrtan directory`.--To update: edit the image tag in the compose file, `docker compose pull-&& docker compose up -d`. The follow list is in the `state` volume.+## 3. The rendezvous: seeds, and your NAS++Nothing runs for this. Hubs keep and gossip the directory among+themselves (`_meta/federation.md` §7); the site only publishes a list+of hubs to follow first.++1. **Seeds.** Copy `rendezvous/seeds.json` into the site repository at+ `rendezvous/seeds.json` and put your hub's public URL in `hubs`.+ `site.yml` serves it at `https://pyrtan.com/rendezvous/seeds.json`,+ which is what `pyrtan follow` reads when given no URL. Others add+ theirs by pull request.+2. **Your NAS as seed #1.** Its hub needs a public address. Two ways+ that open no port and cost nothing:++ - *Cloudflare Tunnel:* `cloudflared tunnel --url http://127.0.0.1:8700`+ gives a `https://….trycloudflare.com` address at once; a named+ tunnel on a domain you own (`hub.example.net`) is a few more+ commands in Cloudflare's dashboard and survives restarts.+ - *Tailscale Funnel:* `tailscale funnel 8700` publishes the hub at+ `https://<machine>.<tailnet>.ts.net`.++ Then, in the compose file's environment: `PYRTAN_TOKEN` set,+ `PYRTAN_ADVERTISE` the public URL, `PYRTAN_FOLLOW: seed` (or another+ hub), `PYRTAN_LISTED: "true"`. The hub keeps its follows and the+ directory in the `hub-state` volume. `pyrtan directory --at+ https://<your hub>` shows what it knows.+3. **The page.** `index.html` fetches `rendezvous/seeds.json` and asks+ each seed's `/directory` from the visitor's browser (seeds run with+ `--cors https://pyrtan.com`, which the compose file sets), so+ pyrtan.com shows the live directory with no backend. The console link+ on each row opens `/console/?hub=<url>`.## 4. Later: hosting Nothing here needs to change. A tenant's hub is this repository's image started with `--token`, `--cors https://pyrtan.com`, `--gaps` on a volume, `--egress public` and `--federate --advertise https://<tenant>.pyrtan.com ---state <volume>/federation.json`, behind TLS; the console at pyrtan.com-connects to it like to any other, and a self-hosted hub can follow it (or-be followed by it) like any other hub. The control plane that provisions it is a separate,+--listed --state <volume>/federation.json`, behind TLS; the console at+pyrtan.com connects to it like to any other, a self-hosted hub can follow+it (or be followed by it) like any other hub, and it is one more seed. The control plane that provisions it is a separate,private repository, per `_meta/platform.md` §2.6. ## Checklist@@ -198,3 +195,6 @@ private repository, per `_meta/platform.md` §2.6.writes `trees/weather.toml`. - [ ] A hub started with `--token … --cors https://pyrtan.com` connects from `https://pyrtan.com/console/`.+- [ ] `rendezvous/seeds.json` in the site repo names your hub; `pyrtan+ follow --at <another hub>` with no URL follows it; `pyrtan+ directory` lists it; the page shows it.diff --git a/deploy/pyrtan.com/catalog.yml b/deploy/pyrtan.com/catalog.yml index 4d0dba4..4ca7228 100644 --- a/deploy/pyrtan.com/catalog.yml +++ b/deploy/pyrtan.com/catalog.yml@@ -15,7 +15,7 @@ on:pull_request: env:- PYRTAN_REF: v0.3.0+ PYRTAN_REF: v0.4.0permissions: contents: read diff --git a/deploy/pyrtan.com/index.html b/deploy/pyrtan.com/index.html index 6c85b52..5062d6b 100644 --- a/deploy/pyrtan.com/index.html +++ b/deploy/pyrtan.com/index.html@@ -25,6 +25,10 @@.card h3 { margin:0 0 6px; font-size:16px } .card p { margin:0; color:var(--mute); font-size:15px } a { color:var(--accent) } footer { color:var(--mute); font-size:14px; margin-top:48px; border-top:1px solid var(--line); padding-top:16px }+ table { width:100%; border-collapse:collapse; font-size:15px } td,th { text-align:left; padding:6px 8px; border-top:1px solid var(--line); vertical-align:top } th { color:var(--mute); font-weight:600; border-top:0 }+ .link { display:inline-block; width:9px; height:9px; border-radius:50%; margin-right:6px; vertical-align:middle; background:var(--mute) }+ .link.up { background:#1f7a4d } .link.stale { background:#8a6400 } .link.down { background:#a3382b }+ .mute { color:var(--mute) }</style> </head> <body>@@ -35,7 +39,7 @@<div class="actions"> <a class="primary" href="/console/">Open the console</a> <a href="/catalog/">Browse the catalog</a>- <a href="https://hub.pyrtan.com/#directory">The rendezvous</a>+ <a href="#directory">The directory</a><a href="https://github.com/b-i-t-8/pyrtan">Source on GitHub</a> </div>@@ -67,16 +71,47 @@ pyrtan serve trees --port 8701 # then open http://127<h2>The console</h2> <p>Every hub and service serves the console at <code>/</code>. The same console is at <a href="/console/">pyrtan.com/console</a>: point it at your own hub, started with <code>--token</code> and <code>--cors https://pyrtan.com</code>, and ask it from anywhere. Nothing about your swarm passes through this site; the page talks to your hub directly.</p>- <h2>The rendezvous</h2>- <p>A public hub at <code>hub.pyrtan.com</code> holds the example trees and takes follows from anyone. <code>pyrtan follow --at https://your-hub</code> makes your swarm answer through it; <code>--listed</code> puts your hub in its <a href="https://hub.pyrtan.com/#directory">directory</a> so others can find and follow yours. Following is one-way trust and one hop: the rendezvous routes to nobody, and nothing crosses more than one hub.</p>-<pre>pyrtan hub --federate --advertise https://your-hub --token … # your hub, behind TLS-pyrtan follow --at https://your-hub --token … --listed # follow the rendezvous, be listed-pyrtan directory # who else is there</pre>+ <h2 id="directory">The directory</h2>+ <p>Hubs find each other without a server. A hub started <code>--listed</code> announces itself to the hubs it follows and that follow it; every hub keeps what it hears and passes it on. This page asks the <a href="/rendezvous/seeds.json">seed hubs</a> from your browser and shows what they know. An entry is a hint, not a route: to use a hub, follow it from your own.</p>+ <div class="card"><table id="hubs"><thead><tr><th>Heard</th><th>Hub</th><th>Holds</th><th></th></tr></thead><tbody><tr><td colspan="4" class="mute" id="hubs-note">Asking the seeds…</td></tr></tbody></table></div>+<pre>pyrtan hub --federate --advertise https://your-hub --listed --token … # your hub, behind a tunnel or TLS+pyrtan follow --at https://your-hub --token … # follow the first seed; route to it+pyrtan directory # who the swarm knows</pre>+ <p>To become a seed, add your hub's URL to <code>rendezvous/seeds.json</code> by pull request.</p><h2>Hosting</h2> <p>Self-hosting is the product. A hosted option, running the same open-source hub per customer with secrets, retention, scheduled re-runs and alerts around it, is planned; it will speak the same API, so a swarm can move either way without changing a caller.</p> <footer>Pyrtan is free software under the AGPL-3.0-or-later, a product of <a href="https://hillvon.io">Hillvon.io</a>. Trees you write are your own.</footer> </main>+<script>+const esc = (s) => String(s).replace(/[&<>"]/g, c => ({"&":"&","<":"<",">":">",'"':"""}[c]));+(async () => {+ const note = document.getElementById("hubs-note");+ let seeds = [];+ try { seeds = (await (await fetch("rendezvous/seeds.json")).json()).hubs || []; } catch (e) {}+ if (!seeds.length) { note.textContent = "No seeds listed yet."; return; }+ const seen = new Map();+ let answered = 0;+ for (const seed of seeds) {+ try {+ const r = await fetch(seed.replace(/\/+$/, "") + "/directory", {headers: {"Accept": "application/json"}});+ if (!r.ok) continue;+ const d = await r.json();+ answered++;+ for (const h of d.hubs || []) if (!seen.has(h.id) || seen.get(h.id).seen < h.seen) seen.set(h.id, h);+ } catch (e) {}+ }+ if (!answered) { note.textContent = "No seed answered from this browser (a seed must run with --cors https://pyrtan.com)."; return; }+ const body = document.querySelector("#hubs tbody"); body.innerHTML = "";+ const hubs = [...seen.values()].sort((a, b) => ({up:0, stale:1, down:2}[a.link] - {up:0, stale:1, down:2}[b.link]) || a.id.localeCompare(b.id));+ for (const h of hubs) {+ const tr = document.createElement("tr");+ tr.innerHTML = `<td><span class="link ${esc(h.link)}"></span>${esc(h.link)}</td><td><code>${esc(h.name||"")}</code><div class="mute">${esc(h.id)}</div></td><td>${(h.trees||[]).map(n => `<code>${esc(n)}</code>`).join(" ")}</td><td><a href="/console/?hub=${encodeURIComponent(h.id)}">open in console</a></td></tr>`;+ body.appendChild(tr);+ }+ if (!hubs.length) body.innerHTML = `<tr><td colspan="4" class="mute">The seeds know of nobody yet.</td></tr>`;+})();+</script></body> </html> diff --git a/deploy/pyrtan.com/rendezvous/Caddyfile b/deploy/pyrtan.com/rendezvous/Caddyfile deleted file mode 100644 index 0f0f945..0000000 --- a/deploy/pyrtan.com/rendezvous/Caddyfile +++ /dev/null@@ -1,7 +0,0 @@-# TLS from Let's Encrypt, everything to the hub. Caddy sets the usual-# proxy headers; the hub does not read them, it only needs --advertise to-# know its public name.-hub.pyrtan.com {- reverse_proxy hub:8700- encode gzip-}diff --git a/deploy/pyrtan.com/rendezvous/docker-compose.yml b/deploy/pyrtan.com/rendezvous/docker-compose.yml deleted file mode 100644 index 3d532f5..0000000 --- a/deploy/pyrtan.com/rendezvous/docker-compose.yml +++ /dev/null@@ -1,75 +0,0 @@-# The rendezvous: the public hub at hub.pyrtan.com that anyone may follow-# to try federation, and that lists the hubs which asked to be found.-#-# One box with Docker, DNS for hub.pyrtan.com pointing at it, ports 80-# and 443 open. Caddy gets the certificate; the hub and one service with-# the example trees sit behind it on the compose network. Put the token-# in .env (PYRTAN_TOKEN=...) and `docker compose up -d`.-#-# The hub's own paths (/follow, /peers, /outbox, /unanswered, /gaps) need-# the token; /actor, /inbox, /directory, /health and the console are-# public, which is what a rendezvous is for. It follows nobody, so it-# routes to nobody but its own service.--name: rendezvous--services:- caddy:- image: caddy:2- restart: unless-stopped- ports: ["80:80", "443:443"]- volumes:- - ./Caddyfile:/etc/caddy/Caddyfile:ro- - caddy-data:/data- - caddy-config:/config- depends_on: [hub]-- hub:- image: ghcr.io/b-i-t-8/pyrtan:0.3.0 # or build: ../../.. from a checkout- restart: unless-stopped- read_only: true- security_opt: [no-new-privileges:true]- cap_drop: [ALL]- tmpfs: [/tmp]- environment:- PYRTAN_TOKEN: ${PYRTAN_TOKEN:?set PYRTAN_TOKEN in .env}- PYRTAN_PORT: "8700"- command:- - hub- - --host- - 0.0.0.0- - --port- - "8700"- - --name- - rendezvous- - --federate- - --advertise- - https://hub.pyrtan.com- - --state- - /state/federation.json- - --cors- - https://pyrtan.com- - --poll- - "15"- volumes:- - state:/state-- examples:- image: ghcr.io/b-i-t-8/pyrtan:0.3.0- restart: unless-stopped- read_only: true- security_opt: [no-new-privileges:true]- cap_drop: [ALL]- tmpfs: [/tmp]- environment:- PYRTAN_TOKEN: ${PYRTAN_TOKEN}- PYRTAN_EGRESS: public # the example trees reach only public hosts- command: ["serve", "/examples", "--host", "0.0.0.0", "--port", "8701", "--name", "examples", "--hub", "http://hub:8700", "--advertise", "http://examples:8701", "--gaps", "/state/gaps.jsonl", "--egress", "public"]- volumes:- - state:/state- depends_on: [hub]--volumes:- state:- caddy-data:- caddy-config:diff --git a/deploy/pyrtan.com/rendezvous/seeds.json b/deploy/pyrtan.com/rendezvous/seeds.json new file mode 100644 index 0000000..0b17941 --- /dev/null +++ b/deploy/pyrtan.com/rendezvous/seeds.json@@ -0,0 +1,5 @@+{+ "pyrtan-seeds": 1,+ "about": "Hubs to follow first. `pyrtan follow` with no URL takes the first that answers /actor. Add yours by pull request to the site repository; a seed is any federated hub whose operator keeps it up and started it with --listed and --cors https://pyrtan.com.",+ "hubs": []+}diff --git a/deploy/pyrtan.com/site.yml b/deploy/pyrtan.com/site.yml index 2a19cc3..2b034bb 100644 --- a/deploy/pyrtan.com/site.yml +++ b/deploy/pyrtan.com/site.yml@@ -12,7 +12,7 @@ on:workflow_dispatch: env:- PYRTAN_REF: v0.3.0+ PYRTAN_REF: v0.4.0permissions: contents: read@@ -36,6 +36,8 @@ jobs:mkdir -p site/console cp index.html CNAME site/ pyrtan console --out site/console/index.html+ mkdir -p site/rendezvous+ cp rendezvous/seeds.json site/rendezvous/seeds.jsontouch site/.nojekyll # To serve the catalog from this same deployment instead of proxying diff --git a/pyproject.toml b/pyproject.toml index 5d2224a..7f1eef9 100644 --- a/pyproject.toml +++ b/pyproject.toml@@ -1,6 +1,6 @@[project] name = "pyrtan"-version = "0.3.0"+version = "0.4.0"description = "Open-source total automation: a core set of expandable services that act on all forms of I/O through hardcoded, inspectable decision trees." readme = "README.md" license = "AGPL-3.0-or-later" diff --git a/src/pyrtan/__init__.py b/src/pyrtan/__init__.py index 3a0c2e0..6bb53ba 100644 --- a/src/pyrtan/__init__.py +++ b/src/pyrtan/__init__.py@@ -33,4 +33,4 @@ __all__ = ["Response", ]-__version__ = "0.3.0"+__version__ = "0.4.0"diff --git a/src/pyrtan/catalog.py b/src/pyrtan/catalog.py index b474fd0..3285d89 100644 --- a/src/pyrtan/catalog.py +++ b/src/pyrtan/catalog.py@@ -148,7 +148,7 @@ def _fetch(url: str, timeout: float = 20.0) -> bytes:raise CatalogError(f"{url}: {e}") from e if parts.scheme not in ("http", "https"): raise CatalogError(f"{url}: a catalog is reached over https (or a file: path)")- req = urllib.request.Request(url, headers={"User-Agent": "pyrtan-catalog/0.3", "Accept": "application/json, application/toml, text/plain, */*"})+ req = urllib.request.Request(url, headers={"User-Agent": "pyrtan-catalog/0.4", "Accept": "application/json, application/toml, text/plain, */*"})try: with urllib.request.urlopen(req, timeout=timeout) as resp: return resp.read() diff --git a/src/pyrtan/cli.py b/src/pyrtan/cli.py index cf0a47a..716e529 100644 --- a/src/pyrtan/cli.py +++ b/src/pyrtan/cli.py@@ -8,11 +8,12 @@pyrtan ask TEXT --at URL answer a sentence through a service or hub pyrtan serve [PATH ...] [--port N] host trees as a service pyrtan hub [--peer URL ...] route between services- pyrtan hub --federate --advertise URL [--follow URL ...] and between hubs- pyrtan follow [URL] --at HUB [--listed] follow another hub (and route to it); default: the rendezvous+ pyrtan hub --trees DIR and hold trees itself: a swarm in one process+ pyrtan hub --federate --advertise URL [--follow URL ...] [--listed] and between hubs+ pyrtan follow [URL] --at HUB [--listed] follow another hub (and route to it); default: a seedpyrtan unfollow URL --at HUB pyrtan peers --at HUB services, and hubs followed and following- pyrtan directory [--at RENDEZVOUS] hubs that asked to be found+ pyrtan directory [--at HUB] every hub that asked to be found, as the swarm knows itpyrtan gaps --at URL branches waiting to be written pyrtan unanswered --at URL [--scaffold DIR] trees waiting to be written pyrtan new [NAME] [--from TEXT] [--dir PATH] scaffold a tree file@@ -38,7 +39,7 @@ from typing import Anyfrom . import __version__ from .catalog import Catalog, CatalogError, build_index, catalog_url-from .federation import rendezvous_url+from .federation import first_live_seed, seed_urlsfrom .grow import Proposal, propose from .page import PAGE from .registry import Registry, default_paths@@ -106,22 +107,26 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--host", default="127.0.0.1") p.add_argument("--port", type=int, default=8700) p.add_argument("--name", default="hub", help="what this hub calls itself, to the console and to hubs that follow it")+ p.add_argument("--trees", action="append", default=[], metavar="PATH", help="trees this hub holds itself, in-process (repeatable); a swarm in one process")+ p.add_argument("--env", action="append", default=[], metavar="K=V", help="a value the hub's own trees may read as {{ env.K }}")+ p.add_argument("--gaps", metavar="FILE", help="keep the hub's own trees' gaps in this file")+ p.add_argument("--egress", choices=EGRESS_POLICIES, default=os.environ.get("PYRTAN_EGRESS", "any"), help="for the hub's own trees: 'public' refuses private addresses")p.add_argument("--peer", action="append", default=[], help="a service URL to start with (repeatable; services may also register themselves)") p.add_argument("--poll", type=float, default=10.0, help="seconds between health polls") _access_args(p) p.add_argument("--federate", action="store_true", help="answer /actor and /inbox so other hubs can follow this one, and follow others")- p.add_argument("--advertise", help="the URL other hubs reach this hub at (implies --federate; default: this host and port)")- p.add_argument("--follow", action="append", default=[], metavar="URL", help="a hub to follow on start (repeatable)")+ p.add_argument("--advertise", default=os.environ.get("PYRTAN_ADVERTISE"), help="the URL other hubs reach this hub at (implies --federate; default $PYRTAN_ADVERTISE, else this host and port)")+ p.add_argument("--follow", action="append", default=[h for h in os.environ.get("PYRTAN_FOLLOW", "").replace(",", " ").split() if h], metavar="URL", help="a hub to follow on start (repeatable; default $PYRTAN_FOLLOW; 'seed' means the first live seed)")p.add_argument("--federation-secret", default=os.environ.get("PYRTAN_FEDERATION_SECRET"), help="only hubs that sign their Follow with this secret may follow (default $PYRTAN_FEDERATION_SECRET; without it, follows are open)") p.add_argument("--closed", action="store_true", help="take no follows at all (this hub can still follow others)")- p.add_argument("--state", metavar="FILE", help="keep who this hub follows and is followed by in this file, across restarts")- p.add_argument("--listed", action="store_true", help="with --follow: ask those hubs to show this one in their directory")+ p.add_argument("--state", metavar="FILE", default=os.environ.get("PYRTAN_STATE"), help="keep who this hub follows and is followed by, and the directory, in this file across restarts (default $PYRTAN_STATE)")+ p.add_argument("--listed", action="store_true", default=os.environ.get("PYRTAN_LISTED", "").lower() in ("1", "true", "yes"), help="put this hub in the directory it gossips, so others can find it (default $PYRTAN_LISTED)")p.add_argument("--quiet", action="store_true") p = sub.add_parser("follow", help="follow another hub, through a running federated hub of yours")- p.add_argument("url", nargs="?", default=None, help=f"the other hub's URL (default $PYRTAN_RENDEZVOUS or {rendezvous_url()})")+ p.add_argument("url", nargs="?", default=None, help="the other hub's URL (default: the first live seed from $PYRTAN_RENDEZVOUS, $PYRTAN_SEEDS or pyrtan.com)")p.add_argument("--at", default=os.environ.get("PYRTAN_AT", "http://127.0.0.1:8700"), help="your hub (default $PYRTAN_AT or the local hub)")- p.add_argument("--listed", action="store_true", help="ask that hub to show yours in its directory, with the trees it holds")+ p.add_argument("--listed", action="store_true", help="also put your hub in the directory it gossips, so others can find it")_token_arg(p, "your hub's token, if any") p = sub.add_parser("unfollow", help="stop following a hub")@@ -129,8 +134,8 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--at", default=os.environ.get("PYRTAN_AT", "http://127.0.0.1:8700"), help="your hub (default $PYRTAN_AT or the local hub)") _token_arg(p, "your hub's token, if any")- p = sub.add_parser("directory", help="hubs that asked a rendezvous to list them")- p.add_argument("--at", default=None, help=f"the rendezvous (default $PYRTAN_RENDEZVOUS or {rendezvous_url()})")+ p = sub.add_parser("directory", help="every hub that asked to be found, as a hub knows it")+ p.add_argument("--at", default=None, help="a federated hub to ask (default $PYRTAN_AT, else the local hub, else the first live seed)")p.add_argument("--json", action="store_true") p = sub.add_parser("peers", help="a hub's services, and the hubs it follows and is followed by")@@ -401,13 +406,24 @@ def cmd_serve(args: argparse.Namespace) -> int:def cmd_hub(args: argparse.Namespace) -> int: from .hub import serve_hub+ follow = []+ for url in args.follow:+ if url == "seed":+ seed = first_live_seed()+ if seed is None:+ print("pyrtan hub: no seed answers; nothing followed", file=sys.stderr)+ continue+ url = seed+ follow.append(url)hub, server = serve_hub( args.host, args.port, name=args.name, peers=args.peer, poll_every=args.poll, quiet=args.quiet, token=args.token, cors=args.cors,- federate=args.federate or bool(args.advertise) or bool(args.follow), advertise=args.advertise, federation_secret=args.federation_secret,- state=args.state, open_follows=not args.closed, follow=args.follow, listed=args.listed,+ federate=args.federate or bool(args.advertise) or bool(follow) or args.listed, advertise=args.advertise, federation_secret=args.federation_secret,+ state=args.state, open_follows=not args.closed, follow=follow, listed=args.listed,+ trees=[Path(t) for t in args.trees], env=_kv(args.env), gaps=args.gaps, egress=args.egress,)- fed = f", federated as {hub.federation.id}" + (f" following {len(hub.federation.following)}" if hub.federation.following else "") if hub.federation else ""- print(f"pyrtan hub at {server.url}: {len(hub.peers)} peer(s){fed}" + _access_note(args), flush=True)+ fed = f", federated as {hub.federation.id}" + (f" following {len(hub.federation.following)}" if hub.federation.following else "") + (", listed" if hub.federation.listed else "") if hub.federation else ""+ local = f", holding {len(hub.local.trees)} tree(s)" if hub.local else ""+ print(f"pyrtan hub at {server.url}: {len(hub.peers)} peer(s){local}{fed}" + _access_note(args), flush=True)try: server.wait() except KeyboardInterrupt:@@ -445,32 +461,43 @@ def cmd_gaps(args: argparse.Namespace) -> int:def cmd_follow(args: argparse.Namespace) -> int:- url = args.url or rendezvous_url()- return _hub_post(args, "/follow", {"url": url, "listed": bool(args.listed)}, lambda b: f"{args.at} now follows {b['following']['url']} ({b['following']['state']}); it routes there once accepted" + (", and asked to be listed" if args.listed else ""))+ url = args.url+ if not url:+ url = first_live_seed()+ if url is None:+ seeds = seed_urls()+ print("pyrtan follow: no seed answers" + (f" (tried {', '.join(seeds)})" if seeds else " (no seeds listed; set $PYRTAN_SEEDS or $PYRTAN_RENDEZVOUS)") + "; give a hub URL", file=sys.stderr)+ return 3+ return _hub_post(args, "/follow", {"url": url, "listed": bool(args.listed)}, lambda b: f"{args.at} now follows {b['following']['url']} ({b['following']['state']}); it routes there once accepted" + (" and is listed in the directory" if b.get("listed") else ""))def cmd_directory(args: argparse.Namespace) -> int:- at = (args.at or rendezvous_url()).rstrip("/")- try:- status, body = http_json("GET", f"{at}/directory")- except Exception as e:- print(f"pyrtan: {at}: {e}", file=sys.stderr)- return 3- if status == 404:- print(f"pyrtan: {at} is not a federated hub, so it has no directory", file=sys.stderr)- return 3- if status != 200 or not isinstance(body, dict):- print(f"pyrtan: {at} answered {status}", file=sys.stderr)+ tried: list[str] = []+ candidates = [args.at] if args.at else [os.environ.get("PYRTAN_AT") or "http://127.0.0.1:8700", *seed_urls()]+ body: Any = None+ at = ""+ for at in candidates:+ at = at.rstrip("/")+ tried.append(at)+ try:+ status, body = http_json("GET", f"{at}/directory", timeout=5.0)+ except Exception:+ body = None+ continue+ if status == 200 and isinstance(body, dict) and "hubs" in body:+ break+ body = None+ if body is None:+ print(f"pyrtan directory: no federated hub answered (tried {', '.join(tried)})", file=sys.stderr)return 3 if args.json: print(json.dumps(body, indent=2, ensure_ascii=False)) return 0 hubs = body.get("hubs", [])- print(f"{body.get('name') or at} ({body.get('rendezvous', at)}): {len(hubs)} listed hub(s)")+ print(f"directory as {body.get('name') or at} ({body.get('hub', at)}) knows it: {len(hubs)} hub(s)")for h in hubs: print(f"{h['link']:8} {h['name'] or '-':16} {h['id']} {', '.join(h.get('trees', []))}")- if hubs:- print(f"to follow one: pyrtan follow URL --at YOUR-HUB; to be listed here: pyrtan follow {at} --at YOUR-HUB --listed")+ print("to follow one: pyrtan follow URL --at YOUR-HUB; to be in it: pyrtan follow --listed, or pyrtan hub --listed")return 0 diff --git a/src/pyrtan/federation.py b/src/pyrtan/federation.py index 23338b8..8c89bf9 100644 --- a/src/pyrtan/federation.py +++ b/src/pyrtan/federation.py@@ -1,4 +1,4 @@-"""Federation: hubs that follow each other.+"""Federation: hubs that follow each other, and a directory they all keep.A hub on its own routes sentences to the services that registered with it. A federated hub also *follows* other hubs, the way one account follows@@ -37,12 +37,16 @@ and ``X-Pyrtan-Signature`` is HMAC-SHA256 over the method, path, date andthe SHA-256 of the canonical JSON body. The body sent *is* that canonical JSON, so the receiver can verify from the body it parsed.-A hub may also be a *rendezvous*: a public hub others follow to try-federation and, if they ask to be ``listed``, to be found. ``GET-/directory`` names the listed followers, with the trees they hold (the-rendezvous reads a listed follower's ``/capabilities``, signed, because-listing is asking to be seen). The directory is discovery only: a-rendezvous routes to nobody it did not follow, like any hub.+**The directory** is how hubs find each other without a server. A hub+started ``--listed`` puts an entry for itself (id, name, the trees it+holds, when) in every ``Announce`` it sends; every hub keeps the entries+it hears and passes them on in its own announces, in both directions of+every follow. Nothing central holds the directory: every federated hub+does, and pyrtan.com publishes only a static list of *seeds*, hubs to+follow first. An entry is a hint (a hub two follows away cannot verify+who said it), kept until its hub has not been heard of for a week, and+routing never uses it: finding a hub in the directory and following it is+still the one act of trust.What this module refuses: open relaying (no hop beyond one), asymmetric secrets (there is no key infrastructure to lean on in the standard@@ -55,6 +59,7 @@ from __future__ import annotationsimport hashlib import hmac import json+import osimport secrets as _secrets import threading import time@@ -65,23 +70,57 @@ from pathlib import Pathfrom typing import Any, Callable from . import __version__-from .service import bearer_ok, http_json+from .service import http_json-DEFAULT_RENDEZVOUS = "https://hub.pyrtan.com"+DEFAULT_SEEDS = "https://pyrtan.com/rendezvous/seeds.json"SIGNATURE_WINDOW = 300.0 # seconds a signed message stays valid MAX_HOPS = 1 # federation is one hop; raise with care STALE_AFTER = 30.0 DOWN_AFTER = 90.0+ANNOUNCE_EVERY = 60.0 # seconds between announces when nothing changed: the directory's heartbeat+DIRECTORY_TTL = 7 * 86400 # an entry unheard of for this long is dropped+DIRECTORY_MAX = 500+HEARD_UP = 600.0 # a directory entry seen this recently reads "up"+HEARD_STALE = 86400.0ACTIVITY_TYPES = ("Follow", "Accept", "Reject", "Undo", "Announce") SIGNED_PATHS = ("/inbox", "/run", "/intent", "/resume", "/capabilities")-def rendezvous_url() -> str:- """The rendezvous `pyrtan follow` and `pyrtan directory` use when given- no URL: ``$PYRTAN_RENDEZVOUS`` or the one at pyrtan.com."""- import os+# ---------------------------------------------------------------- seeds- return os.environ.get("PYRTAN_RENDEZVOUS") or DEFAULT_RENDEZVOUS++def seed_urls() -> list[str]:+ """Hubs to follow first, from ``$PYRTAN_RENDEZVOUS`` (one hub),+ ``$PYRTAN_SEEDS`` (a URL or file holding ``{"hubs": [...]}``), or the+ list at pyrtan.com. Never raises: no seeds is an empty list."""+ single = os.environ.get("PYRTAN_RENDEZVOUS")+ if single:+ return [single.rstrip("/")]+ source = os.environ.get("PYRTAN_SEEDS") or DEFAULT_SEEDS+ try:+ if source.startswith(("http://", "https://")):+ status, body = http_json("GET", source, timeout=10.0)+ if status != 200:+ return []+ else:+ body = json.loads(Path(source).read_text(encoding="utf-8"))+ except Exception:+ return []+ if not isinstance(body, dict) or not isinstance(body.get("hubs"), list):+ return []+ return [h.rstrip("/") for h in body["hubs"] if isinstance(h, str) and h.startswith(("http://", "https://"))]+++def first_live_seed(seeds: list[str] | None = None) -> str | None:+ """The first seed whose ``/actor`` answers as a hub, or None."""+ for url in seeds if seeds is not None else seed_urls():+ try:+ status, actor = http_json("GET", f"{url}/actor", timeout=5.0)+ except Exception:+ continue+ if status == 200 and isinstance(actor, dict) and actor.get("type") == "Hub":+ return url+ return None# ---------------------------------------------------------------- signing@@ -137,7 +176,6 @@ class Remote:last_error: str | None = None since: float = field(default_factory=time.time) kind: str = "hub"- listed: bool = False # a follower that asked to appear in our directory@property def link(self) -> str:@@ -153,32 +191,43 @@ class Remote:return "down" def to_dict(self) -> dict[str, Any]:- return {"url": self.url, "name": self.name, "kind": "hub", "state": self.state, "link": self.link, "trees": [t["name"] for t in self.trees], "last_ok": self.last_ok, "last_error": self.last_error, "listed": self.listed}+ return {"url": self.url, "name": self.name, "kind": "hub", "state": self.state, "link": self.link, "trees": [t["name"] for t in self.trees], "last_ok": self.last_ok, "last_error": self.last_error}def to_state(self) -> dict[str, Any]:- return {"url": self.url, "inbox": self.inbox, "secret": self.secret, "name": self.name, "state": self.state, "since": self.since, "listed": self.listed}+ return {"url": self.url, "inbox": self.inbox, "secret": self.secret, "name": self.name, "state": self.state, "since": self.since}@classmethod def from_state(cls, d: dict[str, Any]) -> "Remote":- return cls(d["url"], d["inbox"], d["secret"], d.get("name", ""), d.get("state", "pending"), since=float(d.get("since", time.time())), listed=bool(d.get("listed", False)))+ return cls(d["url"], d["inbox"], d["secret"], d.get("name", ""), d.get("state", "pending"), since=float(d.get("since", time.time())))+++def heard(seen: float, now: float | None = None) -> str:+ """How recently a directory entry's hub announced, as gossip carried it."""+ age = (now if now is not None else time.time()) - seen+ return "up" if age < HEARD_UP else "stale" if age < HEARD_STALE else "down"class Federation:- def __init__(self, hub_id: str, *, name: str = "hub", secret: str | None = None, state: str | Path | None = None, open_follows: bool = True, capabilities: Callable[[], list[dict[str, Any]]] | None = None) -> None:+ def __init__(self, hub_id: str, *, name: str = "hub", secret: str | None = None, state: str | Path | None = None, open_follows: bool = True, capabilities: Callable[[], list[dict[str, Any]]] | None = None, listed: bool = False) -> None:self.id = hub_id.rstrip("/") self.name = name self.secret = secret or None # a shared federation secret: Follows must be signed with it self.open_follows = open_follows # without a secret, whether strangers may follow self.state_path = Path(state) if state else None self.capabilities = capabilities or (lambda: [])+ self.listed = listed # put ourselves in the directory we gossipself.following: dict[str, Remote] = {} self.followers: dict[str, Remote] = {}+ self.directory_entries: dict[str, dict[str, Any]] = {}self.outbox: deque[dict[str, Any]] = deque(maxlen=50) self.inbox_log: deque[dict[str, Any]] = deque(maxlen=50)- self._announced: str | None = None+ self._announced: tuple[str, float] = ("", 0.0) # digest and time of the last downward announce+ self._announced_up: tuple[str, float] = ("", 0.0) # likewise upwardself._lock = threading.RLock() if self.state_path and self.state_path.exists(): self._load()+ if listed:+ self.listed = True# ------------------------------------------------------------ actor@@ -187,14 +236,20 @@ class Federation:return f"{self.id}/inbox" def actor(self) -> dict[str, Any]:- return {"type": "Hub", "id": self.id, "name": self.name, "inbox": self.inbox, "capabilities": f"{self.id}/capabilities", "pyrtan": __version__, "follows": "open" if (self.open_follows and not self.secret) else "secret"}+ return {"type": "Hub", "id": self.id, "name": self.name, "inbox": self.inbox, "capabilities": f"{self.id}/capabilities", "directory": f"{self.id}/directory", "pyrtan": __version__, "follows": "open" if (self.open_follows and not self.secret) else "secret", "listed": self.listed}# ------------------------------------------------------------ state def _save(self) -> None: if not self.state_path: return- data = {"following": [r.to_state() for r in self.following.values()], "followers": [r.to_state() for r in self.followers.values()]}+ with self._lock:+ data = {+ "listed": self.listed,+ "following": [r.to_state() for r in self.following.values()],+ "followers": [r.to_state() for r in self.followers.values()],+ "directory": list(self.directory_entries.values()),+ }self.state_path.parent.mkdir(parents=True, exist_ok=True) self.state_path.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8")@@ -204,16 +259,94 @@ class Federation:data = json.loads(self.state_path.read_text(encoding="utf-8")) except (OSError, json.JSONDecodeError): return+ self.listed = bool(data.get("listed", False))for d in data.get("following", []): self.following[d["url"]] = Remote.from_state(d) for d in data.get("followers", []): self.followers[d["url"]] = Remote.from_state(d)+ self.merge_directory(data.get("directory", []), via="state")++ def set_listed(self, listed: bool) -> None:+ with self._lock:+ self.listed = listed+ if not listed:+ self.directory_entries.pop(self.id, None)+ self._save()++ # ------------------------------------------------------------ directory++ def own_entry(self) -> dict[str, Any] | None:+ if not self.listed:+ return None+ return {"id": self.id, "name": self.name, "trees": sorted({t.get("name") for t in self.capabilities() if isinstance(t, dict) and t.get("name")}), "seen": time.time()}++ def gossip(self) -> list[dict[str, Any]]:+ """What this hub tells others about the directory: itself, if+ listed, and every live entry it has heard."""+ now = time.time()+ with self._lock:+ entries = [e for e in self.directory_entries.values() if e["id"] != self.id and now - e["seen"] < DIRECTORY_TTL]+ own = self.own_entry()+ return ([own] if own else []) + sorted(entries, key=lambda e: -e["seen"])++ def merge_directory(self, entries: Any, via: str) -> int:+ """Take entries heard from ``via``: a newer ``seen`` replaces an+ older one, our own entry is never overwritten, nothing from the+ future is believed, and the table stays bounded. Returns how many+ entries changed."""+ if not isinstance(entries, list):+ return 0+ now = time.time()+ changed = 0+ with self._lock:+ for e in entries:+ if not isinstance(e, dict) or not isinstance(e.get("id"), str) or not e["id"].startswith(("http://", "https://")):+ continue+ hub_id = e["id"].rstrip("/")+ if hub_id == self.id:+ continue+ try:+ seen = min(float(e.get("seen", 0)), now)+ except (TypeError, ValueError):+ continue+ if now - seen >= DIRECTORY_TTL:+ continue+ current = self.directory_entries.get(hub_id)+ if current is not None and current["seen"] >= seen:+ continue+ self.directory_entries[hub_id] = {+ "id": hub_id,+ "name": str(e.get("name") or "")[:80],+ "trees": [str(t)[:80] for t in e.get("trees", []) if isinstance(t, (str, int))][:200],+ "seen": seen,+ "via": via,+ }+ changed += 1+ if len(self.directory_entries) > DIRECTORY_MAX:+ for hub_id in sorted(self.directory_entries, key=lambda k: self.directory_entries[k]["seen"])[: len(self.directory_entries) - DIRECTORY_MAX]:+ del self.directory_entries[hub_id]+ if changed:+ self._save()+ return changed++ def directory(self) -> dict[str, Any]:+ """Every hub this one has heard of that asked to be found, this one+ included if listed. ``link`` is how recently each announced."""+ now = time.time()+ hubs = []+ for e in self.gossip():+ hubs.append({"id": e["id"], "name": e["name"], "trees": e["trees"], "seen": e["seen"], "link": heard(e["seen"], now), "via": e.get("via", "self" if e["id"] == self.id else "")})+ hubs.sort(key=lambda h: (h["id"] != self.id, {"up": 0, "stale": 1, "down": 2}[h["link"]], h["name"], h["id"]))+ return {"hub": self.id, "name": self.name, "listed": self.listed, "count": len(hubs), "hubs": hubs}# ------------------------------------------------------------ outgoing def _activity(self, type_: str, object_: Any, **extra: Any) -> dict[str, Any]: return {"type": type_, "id": f"{self.id}/activities/{uuid.uuid4()}", "actor": self.id, "published": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "object": object_, **extra}+ def _announcement(self) -> dict[str, Any]:+ return {"type": "Capabilities", "name": self.name, "trees": self.capabilities(), "directory": self.gossip()}+def deliver(self, remote: Remote, activity: dict[str, Any], secret: str | None = None) -> tuple[bool, str | None]: """Post an activity to a remote's inbox, signed with the pair secret (or the one given). Returns (delivered, error)."""@@ -226,12 +359,14 @@ class Federation:return False, str(e) ok = status in (200, 202) self.outbox.append({**activity, "to": remote.url, "delivered": ok, "error": None if ok else f"{status}: {answer}"})+ if ok and isinstance(answer, dict) and activity.get("type") == "Announce":+ # An inbox answers an Announce with its own gossip: one round trip, both directions.+ self.merge_directory(answer.get("directory"), via=remote.url)return ok, None if ok else f"inbox answered {status}: {answer}"- def follow(self, url: str, listed: bool = False) -> Remote:+ def follow(self, url: str) -> Remote:"""Follow the hub at ``url``: fetch its actor, make a pair secret,- send the Follow. The remote is pending until it Accepts. ``listed``- asks the hub to show this one in its directory."""+ send the Follow. The remote is pending until it Accepts."""url = url.rstrip("/") if url == self.id: raise ValueError("a hub cannot follow itself")@@ -251,13 +386,15 @@ class Federation:remote.name = actor.get("name", remote.name) remote.state = "pending" self._save()- activity = self._activity("Follow", url, secret=remote.secret, inbox=self.inbox, name=self.name, listed=bool(listed))+ activity = self._activity("Follow", url, secret=remote.secret, inbox=self.inbox, name=self.name)# Under a shared federation secret the Follow is signed with it; in # the open case with the pair secret, which proves nothing yet but # keeps every message signed. ok, error = self.deliver(remote, activity, self.secret or remote.secret) if not ok: remote.last_error = error+ elif remote.state == "accepted":+ self.announce_up(force=True) # tell it about us, and hear the directory, nowreturn remote def unfollow(self, url: str) -> bool:@@ -270,15 +407,21 @@ class Federation:self.deliver(remote, self._activity("Undo", {"type": "Follow", "actor": self.id, "object": url})) return True+ def _due(self, last: tuple[str, float], announcement: dict[str, Any], force: bool) -> tuple[bool, str]:+ digest = hashlib.sha256(canonical({"trees": [{k: t.get(k) for k in ("name", "version", "triggers", "inputs", "language", "hosts")} for t in announcement["trees"]], "directory": [(e["id"], e["trees"]) for e in announcement["directory"]]})).hexdigest()+ due = force or digest != last[0] or time.time() - last[1] >= ANNOUNCE_EVERY+ return due, digest+def announce(self, force: bool = False) -> int:- """Tell every follower what this hub's own services hold, when it- changed (or ``force``). Returns how many deliveries succeeded."""- caps = self.capabilities()- digest = hashlib.sha256(canonical([{k: t.get(k) for k in ("name", "version", "triggers", "inputs", "language", "hosts")} for t in caps])).hexdigest()- if digest == self._announced and not force:+ """Tell every follower what this hub's own services hold and what+ the directory says: when either changed, on a heartbeat, or+ ``force``. Returns how many deliveries succeeded."""+ announcement = self._announcement()+ due, digest = self._due(self._announced, announcement, force)+ if not due or not self.followers:return 0- self._announced = digest- activity = self._activity("Announce", {"type": "Capabilities", "name": self.name, "trees": caps})+ self._announced = (digest, time.time())+ activity = self._activity("Announce", announcement)delivered = 0 for remote in list(self.followers.values()): ok, error = self.deliver(remote, activity)@@ -290,6 +433,24 @@ class Federation:remote.last_error = error return delivered+ def announce_up(self, force: bool = False) -> int:+ """Tell every hub this one follows what it holds and what the+ directory says. The followed hub routes none of it (being followed+ grants nothing); it keeps the directory and, if this hub is+ listed, finds it there."""+ announcement = self._announcement()+ due, digest = self._due(self._announced_up, announcement, force)+ accepted = [r for r in self.following.values() if r.state == "accepted"]+ if not due or not accepted:+ return 0+ self._announced_up = (digest, time.time())+ activity = self._activity("Announce", announcement)+ delivered = 0+ for remote in accepted:+ ok, _ = self.deliver(remote, activity)+ delivered += ok+ return delivered+def refresh(self, remote: Remote) -> None: """Poll a hub we follow, signed: the fallback when its announces do not arrive, and what keeps its link honest."""@@ -306,38 +467,11 @@ class Federation:else: remote.last_error = f"capabilities answered {status}"- def refresh_listed(self, remote: Remote) -> None:- """Read what a listed follower holds, for the directory. Signed with- the pair secret: the follower accepts a request from the hub it- follows. Nothing is routed there; this is what it asked to show."""- if not remote.listed or remote.state != "accepted":- return- path = "/capabilities"- try:- status, caps = http_json("GET", f"{remote.url}{path}", timeout=5.0, headers=sign(remote.secret, self.id, "GET", path, None))- except Exception as e:- remote.last_error = str(e)- return- if status == 200 and isinstance(caps, dict):- self._take_capabilities(remote, caps)- else:- remote.last_error = f"capabilities answered {status}"-def refresh_all(self) -> None: for remote in list(self.following.values()): self.refresh(remote)- for remote in list(self.followers.values()):- self.refresh_listed(remote)-- def directory(self) -> dict[str, Any]:- """The listed followers: where to find hubs to follow."""- with self._lock:- hubs = [- {"id": r.url, "name": r.name, "since": r.since, "link": r.link, "trees": [t.get("name") for t in r.trees if isinstance(t, dict) and t.get("name")]}- for r in self.followers.values()- if r.listed and r.state == "accepted"- ]- return {"rendezvous": self.id, "name": self.name, "hubs": sorted(hubs, key=lambda h: (h["link"] != "up", h["since"]))}+ self.announce_up()+ self.announce()def _take_capabilities(self, remote: Remote, caps: dict[str, Any]) -> None: with self._lock:@@ -380,10 +514,21 @@ class Federation:self.inbox_log.append({**activity, "received": time.time()}) if kind == "Follow": return self._on_follow(actor, activity, h)- remote = self.following.get(actor) if kind in ("Accept", "Reject", "Announce") else self.followers.get(actor)+ if kind == "Announce":+ # From a hub we follow: its trees are ours to route to. From a+ # follower: only the directory it carries.+ followed = self.following.get(actor)+ follower = self.followers.get(actor)+ remote = followed or follower+ else:+ remote = self.following.get(actor) if kind in ("Accept", "Reject") else self.followers.get(actor)if remote is None: return 403, {"status": "error", "reason": "unknown-actor", "say": f"{actor} is not a hub this one follows or is followed by."} problem = verify(remote.secret, "POST", "/inbox", activity, h)+ if problem and kind == "Announce" and followed and follower and followed is not follower:+ # Mutual follow: the announce may be signed with the other pair.+ remote = follower if remote is followed else followed+ problem = verify(remote.secret, "POST", "/inbox", activity, h)if problem: return 401, {"status": "error", "reason": "unsigned", "say": problem} if kind == "Accept":@@ -403,8 +548,15 @@ class Federation:obj = activity.get("object") if not isinstance(obj, dict) or not isinstance(obj.get("trees"), list): return 400, {"status": "error", "reason": "bad-activity", "say": "An Announce carries {\"trees\": [...]}."}- self._take_capabilities(remote, obj)- return 200, {"status": "ok", "trees": len(remote.trees)}+ if remote is followed:+ self._take_capabilities(remote, obj)+ else:+ with self._lock:+ remote.name = obj.get("name") or remote.name+ remote.last_ok = time.time()+ remote.last_error = None+ taken = self.merge_directory(obj.get("directory", []), via=actor)+ return 200, {"status": "ok", "trees": len(remote.trees) if remote is followed else 0, "merged": taken, "directory": self.gossip()}if kind == "Undo": with self._lock: self.followers.pop(actor, None)@@ -435,15 +587,15 @@ class Federation:if status != 200 or not isinstance(doc, dict) or doc.get("id", "").rstrip("/") != actor or doc.get("inbox") != inbox: return 400, {"status": "error", "reason": "no-actor", "say": f"{actor}/actor does not describe this follower."} with self._lock:- remote = Remote(actor, inbox, pair, str(activity.get("name") or doc.get("name") or ""), state="accepted", last_ok=time.time(), listed=activity.get("listed") is True)+ remote = Remote(actor, inbox, pair, str(activity.get("name") or doc.get("name") or ""), state="accepted", last_ok=time.time())self.followers[actor] = remote self._save() self.deliver(remote, self._activity("Accept", {"type": "Follow", "actor": actor, "object": self.id}))- self.deliver(remote, self._activity("Announce", {"type": "Capabilities", "name": self.name, "trees": self.capabilities()}))+ self.deliver(remote, self._activity("Announce", self._announcement()))return 202, {"status": "ok", "say": f"{self.id} accepts {actor}."} # ------------------------------------------------------------ views def to_dict(self) -> dict[str, Any]: with self._lock:- return {"id": self.id, "following": [r.to_dict() for r in self.following.values()], "followers": [r.to_dict() for r in self.followers.values()]}+ return {"id": self.id, "listed": self.listed, "following": [r.to_dict() for r in self.following.values()], "followers": [r.to_dict() for r in self.followers.values()], "directory": len(self.directory_entries) + (1 if self.listed else 0)}diff --git a/src/pyrtan/hub.py b/src/pyrtan/hub.py index 3a9042f..0508fe7 100644 --- a/src/pyrtan/hub.py +++ b/src/pyrtan/hub.py@@ -35,6 +35,7 @@ from __future__ import annotationsimport datetime as _dt import threading+from pathlib import Pathimport time from dataclasses import dataclass, field from typing import Any@@ -44,7 +45,9 @@ from .federation import MAX_HOPS, Federation, Remote, signfrom .grow import Unanswered, no_match_outcome from .intent import Trigger, read from .page import PAGE-from .service import Server, bearer_ok, http_json+from .registry import Registry+from .runtime import Outcome+from .service import Server, Service, bearer_ok, http_json, outcome_from_dictSTALE_AFTER = 30.0 DOWN_AFTER = 90.0@@ -75,12 +78,52 @@ class Peer:return {"url": self.url, "name": self.name, "link": self.link, "trees": [t["name"] for t in self.trees], "last_ok": self.last_ok, "last_error": self.last_error}+class Local:+ """Trees the hub holds itself (``pyrtan hub --trees DIR``): a service in+ the same process, reached without HTTP. A swarm that is one process."""++ url = "local"+ kind = "local"+ link = "up"+ last_ok = 0.0+ last_error: str | None = None++ def __init__(self, service: Service) -> None:+ self.service = service+ self.name = service.name++ @property+ def trees(self) -> list[dict[str, Any]]:+ return self.service.capabilities()["trees"]++ def to_dict(self) -> dict[str, Any]:+ return {"url": self.url, "name": self.name, "kind": "local", "link": "up", "trees": [t["name"] for t in self.trees], "last_ok": time.time(), "last_error": None}+++class _HubResolver:+ """What a local tree's ``delegate`` reaches: the hub itself, in-process."""++ def __init__(self, hub: "Hub") -> None:+ self.hub = hub++ def run_tree(self, name: str, inputs: dict[str, Any], depth: int) -> Outcome:+ _, body = self.hub.run({"tree": name, "inputs": inputs, "depth": depth})+ return outcome_from_dict(body, name)++ def run_intent(self, text: str, depth: int) -> Outcome:+ _, body = self.hub.intent({"text": text, "depth": depth})+ return outcome_from_dict(body, "")++class Hub:- def __init__(self, *, name: str = "hub", poll_every: float = 10.0, token: str | None = None) -> None:+ def __init__(self, *, name: str = "hub", poll_every: float = 10.0, token: str | None = None, local: Service | None = None) -> None:self.name = name self.poll_every = poll_every self.token = token or None self.peers: dict[str, Peer] = {}+ self.local: Local | None = None+ if local is not None:+ self.hold(local)self.federation: Federation | None = None self.unanswered = Unanswered() self.started = time.time()@@ -90,15 +133,25 @@ class Hub:# ------------------------------------------------------------ federation- def federate(self, advertise: str, *, secret: str | None = None, state: str | None = None, open_follows: bool = True) -> Federation:+ def federate(self, advertise: str, *, secret: str | None = None, state: str | None = None, open_follows: bool = True, listed: bool = False) -> Federation:"""Give this hub an identity other hubs can follow. ``advertise`` is the URL they reach it at."""- self.federation = Federation(advertise, name=self.name, secret=secret, state=state, open_follows=open_follows, capabilities=self.local_capabilities)+ self.federation = Federation(advertise, name=self.name, secret=secret, state=state, open_follows=open_follows, capabilities=self.local_capabilities, listed=listed)return self.federation+ def hold(self, service: Service) -> Local:+ """Hold trees in this process. The service's delegations reach the+ hub directly, so a local tree can call anything the hub routes to."""+ service.runner.resolver = _HubResolver(self)+ self.local = Local(service)+ return self.local+def local_capabilities(self) -> list[dict[str, Any]]:- """What this hub's own services hold: what it announces."""+ """What this hub holds itself and through its own services: what it announces."""trees: list[dict[str, Any]] = []+ if self.local is not None:+ for t in self.local.trees:+ trees.append({**t, "peer": "local", "link": "up"})with self._lock: for peer in self.peers.values(): for t in peer.trees:@@ -154,10 +207,11 @@ class Hub:# ------------------------------------------------------------ lookup- def holders(self) -> list[Peer | Remote]:- """Everyone this hub may route to: its services, then the hubs it follows."""+ def holders(self) -> list[Local | Peer | Remote]:+ """Everyone this hub may route to: its own trees, its services, then the hubs it follows."""with self._lock:- out: list[Peer | Remote] = list(self.peers.values())+ out: list[Local | Peer | Remote] = [self.local] if self.local is not None else []+ out.extend(self.peers.values())if self.federation is not None: out.extend(self.federation.following.values()) return out@@ -175,10 +229,10 @@ class Hub:continue return out- def holder(self, tree_name: str) -> Peer | Remote | None:- """Who to send a tree to: a service of ours before a hub we follow,- and the healthiest of either."""- best: Peer | Remote | None = None+ def holder(self, tree_name: str) -> Local | Peer | Remote | None:+ """Who to send a tree to: our own trees, then a service of ours, then+ a hub we follow, and the healthiest of each."""+ best: Local | Peer | Remote | None = Nonefor h in self.holders(): if any(t.get("name") == tree_name for t in h.trees): if best is None or _rank(h) < _rank(best):@@ -197,7 +251,8 @@ class Hub:"up": sum(1 for p in peers if p.link == "up"), "stale": sum(1 for p in peers if p.link == "stale"), "down": sum(1 for p in peers if p.link == "down"),- "trees": sum(len(p.trees) for p in peers),+ "trees": sum(len(p.trees) for p in peers) + (len(self.local.trees) if self.local else 0),+ "local": len(self.local.trees) if self.local else 0,"unanswered": len(self.unanswered), "uptime_s": int(time.time() - self.started), }@@ -252,9 +307,14 @@ class Hub:return 200, {"status": "error", "reason": "no-such-tree", "say": "No service holds the tree this snapshot came from."} return self.forward(peer, "/resume", body)- def forward(self, peer: Peer | Remote | None, path: str, body: dict[str, Any]) -> tuple[int, dict[str, Any]]:+ def forward(self, peer: Local | Peer | Remote | None, path: str, body: dict[str, Any]) -> tuple[int, dict[str, Any]]:if peer is None: return 200, {"status": "error", "reason": "no-peer"}+ if isinstance(peer, Local):+ handler = {"/run": peer.service.run, "/intent": peer.service.intent, "/resume": peer.service.resume}[path]+ answer = handler({k: v for k, v in body.items() if k != "hops"})+ answer.setdefault("peer", "local")+ return 200, answerhops = list(body.get("hops") or []) if isinstance(peer, Remote): # Over federation: one hop, signed, and the request says where it has been.@@ -281,6 +341,8 @@ class Hub:def gaps(self) -> dict[str, Any]: out: list[dict[str, Any]] = []+ if self.local is not None:+ out.extend({**g, "peer": "local"} for g in self.local.service.gaps.all())for peer in list(self.peers.values()): try: status, body = http_json("GET", f"{peer.url}/gaps", timeout=5.0, token=self.token)@@ -294,6 +356,8 @@ class Hub:"""This hub's unanswered sentences, then each peer's own (a peer without a hub keeps its own record; one with a hub leaves it here).""" out = self.unanswered.to_list()+ if self.local is not None:+ out.extend({**u, "peer": "local"} for u in self.local.service.unanswered.to_list())for peer in list(self.peers.values()): if peer.link == "down": continue@@ -321,7 +385,7 @@ class Hub:return 200, self.health() if method == "GET" and path == "/peers": with self._lock:- out: dict[str, Any] = {"peers": [p.to_dict() for p in self.peers.values()]}+ out: dict[str, Any] = {"peers": ([self.local.to_dict()] if self.local else []) + [p.to_dict() for p in self.peers.values()]}if fed is not None: out.update(fed.to_dict()) return 200, out@@ -340,11 +404,19 @@ class Hub:url = body.get("url") if not isinstance(url, str) or not url.startswith(("http://", "https://")): return 400, {"status": "error", "reason": "bad-request", "say": "Send {\"url\": \"https://hub.example\"}."}+ if body.get("listed") is True and not fed.listed:+ fed.set_listed(True)try:- remote = fed.follow(url, listed=body.get("listed") is True)+ remote = fed.follow(url)except ValueError as e: return 400, {"status": "error", "reason": "cannot-follow", "say": str(e)}- return 200, {"status": "ok", "following": remote.to_dict()}+ return 200, {"status": "ok", "following": remote.to_dict(), "listed": fed.listed}+ if method == "POST" and path == "/listed":+ fed.set_listed(body.get("listed") is not False)+ if fed.listed:+ fed.announce_up(force=True)+ fed.announce(force=True)+ return 200, {"status": "ok", "listed": fed.listed}if method == "POST" and path == "/unfollow": url = body.get("url") if not isinstance(url, str):@@ -361,7 +433,7 @@ class Hub:if method == "GET" and path == "/api": endpoints = ["/health", "/peers", "/capabilities", "/intent", "/run", "/resume", "/gaps", "/unanswered", "/register"] if fed is not None:- endpoints += ["/actor", "/inbox", "/directory", "/follow", "/unfollow", "/following", "/followers", "/outbox"]+ endpoints += ["/actor", "/inbox", "/directory", "/follow", "/unfollow", "/listed", "/following", "/followers", "/outbox"]return 200, {"pyrtan": "hub", "name": self.name, "version": __version__, "auth": "bearer" if self.token else "none", "federation": fed.id if fed else None, "endpoints": endpoints} if method == "POST" and path == "/register": url = body.get("url")@@ -378,9 +450,9 @@ class Hub:return 404, {"status": "error", "reason": "not-found", "say": f"No {method} {path} here."}-def _rank(peer: Peer | Remote) -> int:- # A service of ours before a hub we follow; the healthier of either.- return {"up": 0, "pending": 1, "stale": 2, "down": 3}[peer.link] + (10 if isinstance(peer, Remote) else 0)+def _rank(peer: Local | Peer | Remote) -> int:+ # Our own trees, then a service of ours, then a hub we follow; the healthier of each.+ return {"up": 0, "pending": 1, "stale": 2, "down": 3}[peer.link] + (10 if isinstance(peer, Remote) else 0) - (1 if isinstance(peer, Local) else 0)def serve_hub(@@ -400,16 +472,21 @@ def serve_hub(open_follows: bool = True, follow: list[str] | None = None, listed: bool = False,+ trees: list[str | Path] | None = None,+ env: dict[str, Any] | None = None,+ gaps: str | None = None,+ egress: str = "any",) -> tuple[Hub, Server]:- hub = Hub(name=name, poll_every=poll_every, token=token)+ local = Service(Registry.from_paths(trees), name=f"{name}", env=env, gaps=gaps, egress=egress) if trees else None+ hub = Hub(name=name, poll_every=poll_every, token=token, local=local)server = Server(hub.route, host, port, quiet, token=token, cors=cors, federated=federate or bool(advertise)).start() if federate or advertise:- hub.federate(advertise or server.url, secret=federation_secret, state=state, open_follows=open_follows)+ hub.federate(advertise or server.url, secret=federation_secret, state=state, open_follows=open_follows, listed=listed)for url in peers or []: hub.add_peer(url) for url in follow or []: try:- hub.federation.follow(url, listed=listed) # type: ignore[union-attr]+ hub.federation.follow(url) # type: ignore[union-attr]except ValueError as e: print(f"pyrtan hub: cannot follow {url}: {e}", flush=True) hub.start_polling() diff --git a/src/pyrtan/page.py b/src/pyrtan/page.py index ebbf33b..9f0b273 100644 --- a/src/pyrtan/page.py +++ b/src/pyrtan/page.py@@ -141,8 +141,8 @@ PAGE = r"""<!doctype html><section id="tab-directory" hidden> <h2>Directory</h2>- <p class="mute" style="font-size:14px;margin-top:0">Hubs that follow this one and asked to be found. This hub routes to none of them; each operator follows whom they choose: <code>pyrtan follow URL --at YOUR-HUB</code>. To be listed here: <code>pyrtan follow <span class="at">URL</span> --at YOUR-HUB --listed</code>.</p>- <div class="card"><table id="directory"><thead><tr><th>Link</th><th>Hub</th><th>Holds</th><th>Since</th></tr></thead><tbody></tbody></table></div>+ <p class="mute" style="font-size:14px;margin-top:0">Every hub that asked to be found, as this hub has heard it through the swarm. Entries are hints carried hub to hub; <em>link</em> is how recently each announced. Nothing here is routed to: to use one, follow it from your own hub, <code>pyrtan follow URL --at YOUR-HUB</code>. To be in it: <code>pyrtan hub --listed</code>.</p>+ <div class="card"><table id="directory"><thead><tr><th>Heard</th><th>Hub</th><th>Holds</th><th>Last announced</th></tr></thead><tbody></tbody></table></div></section> <section id="tab-api" hidden>@@ -209,10 +209,10 @@ async function loadDirectory() {const body = $("#directory tbody"); body.innerHTML = ""; for (const h of d.hubs || []) { const tr = document.createElement("tr");- tr.innerHTML = `<td><span class="link ${esc(h.link)}"></span>${esc(h.link)}</td><td><code>${esc(h.name||"")}</code><div class="mute">${esc(h.id)}</div></td><td>${(h.trees||[]).map(n => `<code>${esc(n)}</code>`).join(" ")}</td><td class="mute">${h.since ? new Date(h.since * 1000).toLocaleDateString() : ""}</td>`;+ tr.innerHTML = `<td><span class="link ${esc(h.link)}"></span>${esc(h.link)}</td><td><code>${esc(h.name||"")}</code>${h.via === "self" ? ' <span class="mute">(this hub)</span>' : ""}<div class="mute">${esc(h.id)}</div></td><td>${(h.trees||[]).map(n => `<code>${esc(n)}</code>`).join(" ")}</td><td class="mute">${h.seen ? new Date(h.seen * 1000).toLocaleString() : ""}</td>`;body.appendChild(tr); }- if (!(d.hubs || []).length) body.innerHTML = `<tr><td colspan="4" class="mute">Nobody listed yet.</td></tr>`;+ if (!(d.hubs || []).length) body.innerHTML = `<tr><td colspan="4" class="mute">Nobody heard of yet. Follow a seed, or start this hub with <code>--listed</code>.</td></tr>`;} window.addEventListener("hashchange", () => showTab((location.hash || "#ask").slice(1)));@@ -244,7 +244,8 @@ async function detect() {} catch (e) {} remote = true; const saved = store.get();- base = saved.url || ""; token = saved.token || "";+ const asked = new URLSearchParams(location.search).get("hub");+ base = asked || saved.url || ""; token = saved.token || "";$("#connect").hidden = false; $("#origin").textContent = location.origin; $("#hub-url").value = base; $("#hub-token").value = token; diff --git a/src/pyrtan/transport.py b/src/pyrtan/transport.py index 1768f44..febef0c 100644 --- a/src/pyrtan/transport.py +++ b/src/pyrtan/transport.py@@ -129,7 +129,7 @@ class UrllibTransport:``public`` (refuse private, loopback, link-local and metadata addresses, on the first URL and on every redirect)."""- def __init__(self, user_agent: str = "pyrtan/0.3 (+https://github.com/b-i-t-8/pyrtan)", egress: str = "any") -> None:+ def __init__(self, user_agent: str = "pyrtan/0.4 (+https://github.com/b-i-t-8/pyrtan)", egress: str = "any") -> None:if egress not in EGRESS_POLICIES: raise ValueError(f"egress policy must be one of {EGRESS_POLICIES}, got {egress!r}") self.user_agent = user_agent diff --git a/tests/test_federation.py b/tests/test_federation.py index d5c0126..716e955 100644 --- a/tests/test_federation.py +++ b/tests/test_federation.py@@ -111,7 +111,7 @@ def test_follow_accept_announce_and_route(two):assert out["reason"] == "no-match" _, box = a.get("/outbox")- assert [x["type"] for x in box["outbox"]] == ["Follow"] and box["outbox"][0]["delivered"]+ assert [x["type"] for x in box["outbox"]] == ["Follow", "Announce"] and all(x["delivered"] for x in box["outbox"]) # the Follow, then our announce upwardassert [x["type"] for x in box["inbox"]] == ["Accept", "Announce"]@@ -135,7 +135,12 @@ def test_signed_requests_replace_the_token_only_for_hubs_we_know(two):# The inbox checks the activity's actor against the signature's. announce = {"type": "Announce", "actor": a.url, "object": {"type": "Capabilities", "trees": []}} status, out = http_json("POST", f"{b.url}/inbox", announce, headers=sign(secret, a.url, "POST", "/inbox", announce))- assert status == 403 and out["reason"] == "unknown-actor" # B does not follow A, so A cannot announce to it+ # A follows B, so B takes A's announce for the directory it carries, and nothing else:+ # being followed grants nothing, so B still routes only to its own trees.+ assert status == 200 and out["status"] == "ok" and out["trees"] == 0+ assert [t["name"] for t in b.get("/capabilities")[1]["trees"]] == ["beta"]+ stranger = {"type": "Announce", "actor": "http://stranger", "object": {"type": "Capabilities", "trees": []}}+ assert http_json("POST", f"{b.url}/inbox", stranger, headers=sign(secret, "http://stranger", "POST", "/inbox", stranger))[0] == 403bad = {"type": "Announce", "actor": b.url, "object": {"type": "Capabilities", "trees": []}} assert http_json("POST", f"{a.url}/inbox", bad, headers=sign("nope", b.url, "POST", "/inbox", bad))[0] == 401 assert http_json("POST", f"{a.url}/inbox", {"type": "Dance", "actor": b.url}, headers=sign(secret, b.url, "POST", "/inbox", {"type": "Dance", "actor": b.url}))[0] == 400@@ -216,7 +221,8 @@ def test_unfollow_announce_on_change_and_state(two, tmp_path):assert b.url not in a.fed.following and a.url not in b.fed.followers assert a.post("/intent", {"text": "beta hello"})[1]["reason"] == "no-match" assert a.post("/unfollow", {"url": b.url})[1]["removed"] is False- assert json.loads(state.read_text()) == {"following": [], "followers": []}+ saved = json.loads(state.read_text())+ assert saved["following"] == [] and saved["followers"] == [] and saved["listed"] is Falsedef test_unfederated_hub_has_no_inbox(tmp_path):@@ -265,53 +271,131 @@ def test_signing_primitives():assert Remote("http://b", "http://b/inbox", "s", state="rejected").link == "down"-def test_rendezvous_directory_lists_only_who_asked(two, tmp_path):+def test_directory_is_gossiped_in_both_directions_and_routes_to_nobody(two, tmp_path):a, b = two (tmp_path / "c").mkdir() (tmp_path / "c" / "gamma.toml").write_text(ECHO.format(name="gamma", word="gamma"))- c = Swarm("c", tmp_path / "c")+ c = Swarm("c", tmp_path / "c", listed=True)try:- # A is the rendezvous. B follows and asks to be listed; C follows quietly.- b.post("/follow", {"url": a.url, "listed": True})- c.post("/follow", {"url": a.url})- assert a.fed.followers[b.url].listed and not a.fed.followers[c.url].listed+ # B lists itself; A follows B and hears about B in the Accept's Announce.+ b.fed.set_listed(True)+ a.post("/follow", {"url": b.url})status, d = http_json("GET", f"{a.url}/directory") # public- assert status == 200 and d["rendezvous"] == a.url and [h["id"] for h in d["hubs"]] == [b.url]- assert d["hubs"][0]["trees"] == [] # not read yet- a.hub.refresh_all() # the rendezvous reads what a listed follower holds- d = http_json("GET", f"{a.url}/directory")[1]- assert d["hubs"][0]["trees"] == ["beta"] and d["hubs"][0]["link"] == "up"- assert a.fed.followers[c.url].trees == [] # a quiet follower is not read- # Listing grants no routing: the rendezvous still answers only for itself.- assert a.post("/intent", {"text": "beta hello"})[1]["reason"] == "no-match"- assert [t["name"] for t in a.get("/capabilities")[1]["trees"]] == ["alpha"]- # Listed survives the state file.+ assert status == 200 and d["hub"] == a.url and [h["id"] for h in d["hubs"]] == [b.url]+ assert d["hubs"][0]["trees"] == ["beta"] and d["hubs"][0]["link"] == "up" and d["hubs"][0]["via"] == b.url+ # C (listed) follows B: it announces upward, so B hears of C; A hears of C from B on B's next announce.+ c.post("/follow", {"url": b.url})+ assert {h["id"] for h in b.fed.directory()["hubs"]} == {b.url, c.url}+ assert c.url not in {h["id"] for h in a.fed.directory()["hubs"]}+ b.hub.refresh_all() # B announces downward (the directory changed) and A merges+ assert {h["id"] for h in a.fed.directory()["hubs"]} == {b.url, c.url}+ assert next(h for h in a.fed.directory()["hubs"] if h["id"] == c.url)["via"] == b.url+ # And C hears of nobody but B until B announces to it; after that, the same two.+ assert {h["id"] for h in c.fed.directory()["hubs"]} >= {b.url, c.url}+ # A is not listed, so it appears nowhere; listing itself puts it in its own gossip and up to B.+ assert a.url not in {h["id"] for h in b.fed.directory()["hubs"]}+ a.post("/listed", {"listed": True})+ assert a.url in {h["id"] for h in b.fed.directory()["hubs"]}+ assert a.fed.directory()["hubs"][0]["id"] == a.url and a.fed.directory()["hubs"][0]["via"] == "self"+ # Nothing in the directory is routed to: B still answers only for itself.+ assert b.post("/intent", {"text": "alpha hello"})[1]["reason"] == "no-match"+ assert [t["name"] for t in b.get("/capabilities")[1]["trees"]] == ["beta"]+ # The directory survives a restart through the state file.state = tmp_path / "state.json" a.fed.state_path = state a.fed._save()- assert Federation(a.url, state=state).followers[b.url].listed is True- # Unfollowing removes the listing.- b.post("/unfollow", {"url": a.url})- assert http_json("GET", f"{a.url}/directory")[1]["hubs"] == []+ again = Federation(a.url, state=state)+ assert again.listed and {h["id"] for h in again.directory()["hubs"]} == {a.url, b.url, c.url}+ # Merge rules: our own entry is never overwritten, the future is not believed, old news is dropped.+ a.fed.merge_directory([{"id": a.url, "name": "impostor", "trees": ["x"], "seen": time.time() + 1e6}], via="x")+ assert a.fed.directory()["hubs"][0]["name"] == "a"+ a.fed.merge_directory([{"id": "http://future", "trees": [], "seen": time.time() + 1e6}], via="x")+ assert abs(a.fed.directory_entries["http://future"]["seen"] - time.time()) < 5+ assert a.fed.merge_directory([{"id": "http://old", "trees": [], "seen": time.time() - 30 * 86400}], via="x") == 0+ assert a.fed.merge_directory([{"id": "not a url"}, "junk", {"id": "http://ok", "seen": "soon"}], via="x") == 0assert "/directory" in http_json("GET", f"{a.url}/api")[1]["endpoints"] finally: c.stop()-def test_cli_directory_and_follow_default(two, capsys, monkeypatch):+def test_seeds_and_cli_defaults(two, capsys, monkeypatch, tmp_path):a, b = two+ from pyrtan.federation import first_live_seed, seed_urlsdef run(*argv): code = main(list(argv)) out = capsys.readouterr() return code, out.out, out.err+ seeds = tmp_path / "seeds.json"+ seeds.write_text(json.dumps({"pyrtan-seeds": 1, "hubs": ["http://127.0.0.1:9", b.url, a.url]}))+ monkeypatch.setenv("PYRTAN_SEEDS", str(seeds))+ monkeypatch.delenv("PYRTAN_RENDEZVOUS", raising=False)+ assert seed_urls() == ["http://127.0.0.1:9", b.url, a.url]+ assert first_live_seed() == b.url # the dead one is skippedmonkeypatch.setenv("PYRTAN_RENDEZVOUS", a.url)- code, out, _ = run("follow", "--at", b.url, "--token", "token-b", "--listed") # no URL: the rendezvous- assert code == 0 and f"now follows {a.url}" in out and "asked to be listed" in out+ assert seed_urls() == [a.url]+ monkeypatch.delenv("PYRTAN_RENDEZVOUS")+ b.fed.set_listed(True)+ code, out, _ = run("follow", "--at", a.url, "--token", "token-a", "--listed") # no URL: the first live seed+ assert code == 0 and f"now follows {b.url}" in out and "listed in the directory" in out+ assert a.fed.listed and a.url in {h["id"] for h in b.fed.directory()["hubs"]}+ monkeypatch.setenv("PYRTAN_AT", "http://127.0.0.1:9") # the local default is dead: fall through to the seedscode, out, _ = run("directory")- assert code == 0 and "1 listed hub(s)" in out and b.url in out and "--listed" in out- code, out, _ = run("directory", "--json")- assert json.loads(out)["hubs"][0]["id"] == b.url- code, _, err = run("directory", "--at", b.svc_srv.url)- assert code == 3 and "no directory" in err+ assert code == 0 and f"as b ({b.url})" in out and a.url in out and b.url in out+ code, out, _ = run("directory", "--at", a.url, "--json")+ assert {h["id"] for h in json.loads(out)["hubs"]} == {a.url, b.url}+ assert run("directory", "--at", b.svc_srv.url)[0] == 3+ monkeypatch.setenv("PYRTAN_SEEDS", str(tmp_path / "missing.json"))+ assert seed_urls() == []+ code, _, err = run("follow", "--at", a.url, "--token", "token-a")+ assert code == 3 and "no seed answers" in err+++def test_hub_holds_trees_itself(tmp_path):+ (tmp_path / "t").mkdir()+ (tmp_path / "t" / "alpha.toml").write_text(ECHO.format(name="alpha", word="alpha"))+ (tmp_path / "t" / "picnic.toml").write_text("""+name = "picnic"+description = "delegates to the hub it lives in"+start = "ask"++[[triggers]]+pattern = "picnic {text}"++[inputs.text]+type = "str"++[nodes.ask]+kind = "delegate"+intent = "alpha {{ inputs.text }}"+into = "a"+next = "say"++[nodes.say]+kind = "done"+say = "picnic says {{ state.a.say }}"+""")+ from pyrtan.hub import serve_hub++ hub, srv = serve_hub(port=0, quiet=True, name="solo", trees=[tmp_path / "t"], federate=True)+ try:+ _, health = http_json("GET", f"{srv.url}/health")+ assert health["local"] == 2 and health["trees"] == 2 and health["peers"] == 0+ _, caps = http_json("GET", f"{srv.url}/capabilities")+ assert {(t["name"], t["peer"]) for t in caps["trees"]} == {("alpha", "local"), ("picnic", "local")}+ _, out = http_json("POST", f"{srv.url}/intent", {"text": "alpha hi"})+ assert out["status"] == "done" and out["say"] == "alpha: hi" and out["peer"] == "local"+ assert out["trace"]["steps"][0]["kind"] == "intent"+ # A local tree's delegate goes back through the hub, in-process.+ _, out = http_json("POST", f"{srv.url}/run", {"tree": "picnic", "inputs": {"text": "now"}})+ assert out["status"] == "done" and out["say"] == "picnic says alpha: now"+ _, peers = http_json("GET", f"{srv.url}/peers")+ assert peers["peers"][0]["kind"] == "local" and peers["peers"][0]["trees"] == ["alpha", "picnic"]+ assert http_json("GET", f"{srv.url}/gaps")[1]["gaps"] == []+ # Hot reload holds for local trees too, and the announce digest follows.+ (tmp_path / "t" / "beta.toml").write_text(ECHO.format(name="beta", word="beta"))+ assert "beta" in {t["name"] for t in hub.local_capabilities()}+ finally:+ hub.stop()+ srv.stop()diff --git a/uv.lock b/uv.lock index 6c6c2e8..724197d 100644 --- a/uv.lock +++ b/uv.lock@@ -147,7 +147,7 @@ wheels = [[[package]] name = "pyrtan"-version = "0.3.0"+version = "0.4.0"source = { editable = "." } [package.optional-dependencies]