Read sentences with rules: rewrites, typed slots, synonyms, typos, pyrtan match
Claude · 2026-10-25 · 848aca0eea498e8f75774e08760612317a02b07b · parents c1bfd35
A sentence now goes through a fixed pipeline before any trigger sees it, and the trace's first step (intent) names every rule that fired: unicode folded; greetings, politeness and request wrappers dropped from the edges (kept when a trigger in the swarm starts or ends with the word); contractions expanded; number words turned to digits. Patterns get the same folding and contraction rules at load time, so "[what's the]" and "what is the" are one trigger. A literal word matches its plural or singular through a conservative stemmer with an exception list.
Slots gain kinds with fixed parsers: date (tomorrow, next friday, in 3 days, oct 25, 2026-10-25; read against the runner's clock, the same clock {{ now }} reads), time, duration, number (digits or words), host, url, email. A value a parser cannot read, or could read two ways (3/10/2026), makes the trigger refuse the sentence rather than guess, and the reading says which slot refused. The checker refuses a slot kind whose value cannot bind to its input's type.
A tree's [language] table adds synonyms (weather = ["forecast"]) and an opt-in typos rule: a word of five letters or more that is one edit from exactly one pattern word is read as that word; close to two, it is left alone. /capabilities carries the table, so a hub applies it when it routes, and the hub puts the reading at the top of the forwarded trace.
`pyrtan match "sentence" --trees DIR` prints the whole reading: the rewrites, every trigger's verdict with its notes, and the winner or the tie. The console shows the reading on a no-match answer. The weather example gains synonyms; site-status reads {host:host}; a proposed tree for a host-shaped word uses the host kind.
No model anywhere: nlp.py is tables and rules, intent.py applies them, and the same sentence with the same clock always reads the same way.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YChr78vDHtQJ15HNbsaRLZ
Changes
README.md | 47 ++++- _meta/design.md | 33 +++- _meta/tree-format.md | 67 ++++++- examples/site-status/site-status.toml | 6 +- examples/weather/weather.toml | 4 + src/pyrtan/cli.py | 55 +++++- src/pyrtan/grow.py | 4 +- src/pyrtan/hub.py | 21 ++- src/pyrtan/intent.py | 278 +++++++++++++++++++++++++--- src/pyrtan/nlp.py | 540 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/pyrtan/page.py | 1 + src/pyrtan/registry.py | 13 +- src/pyrtan/runtime.py | 35 ++-- src/pyrtan/tree.py | 31 +++- tests/test_cli.py | 15 +- tests/test_examples.py | 7 +- tests/test_grow.py | 2 +- tests/test_intent.py | 88 ++++++++- tests/test_nlp.py | 132 +++++++++++++ tests/test_runtime.py | 43 +++++ tests/test_service_hub.py | 28 ++- tests/test_tree.py | 13 ++ 22 files changed, 1390 insertions(+), 73 deletions(-)
diff --git a/README.md b/README.md index 110c9f1..40ca301 100644 --- a/README.md +++ b/README.md@@ -122,6 +122,7 @@ whole format is in [`_meta/tree-format.md`](_meta/tree-format.md).| `pyrtan list DIR` | What is loaded and the sentences it answers to. | | `pyrtan run "…" --trees DIR` | Answers a sentence in this process, printing the trace. `--replay FIXTURE` answers HTTP from a recording; `--record FIXTURE` makes one. | | `pyrtan run --tree NAME --input k=v` | Runs a named tree. |+| `pyrtan match "…" --trees DIR` | How a sentence reads, without running anything: the rewrites, every trigger's verdict, the winner. The tool for writing triggers. || `pyrtan serve DIR --port 8701` | Hosts the trees as a service: `/health`, `/capabilities`, `/intent`, `/run`, `/resume`, `/gaps`, `/unanswered`. Reloads trees when their files change. `--gaps FILE` keeps gaps across restarts; `--egress public` refuses private addresses. | | `pyrtan hub --port 8700` | Routes between services. Services register with `--hub URL`; peers carry a `link` of `up`, `stale` or `down`. | | `pyrtan ask "…" --at URL` | Answers a sentence through a running service or hub. |@@ -223,11 +224,49 @@ directory picks the file up the moment it is saved. No model draftsanything: the proposal is derived from the words by rules, and the demo page shows the same proposal and the same command whenever nothing answers.+## Sentences++Pyrtan reads a sentence with rules, not a model, and the trace's first+step shows exactly what the rules did:++```+$ pyrtan match "Hey Pyrtan, what's the forecast in Oslo please?" --trees examples+sentence: "Hey Pyrtan, what's the forecast in Oslo please?"+read as: 'what is the forecast in Oslo'+ fillers: 'Hey Pyrtan', 'please'+ contractions: what's->what is+triggers:+ weather: "[what's the] weather in {place}" 18 literal chars -> match+ inputs: {"place": "Oslo"}+winner: weather+```++The fixed rules: unicode folded; greetings, politeness and request+wrappers dropped from the edges; contractions expanded; number words+turned to digits (`top three stories` → `top 3 stories`); a literal word+matching its plural or singular. A tree adds its own in a `[language]`+table: `synonyms` (`weather = ["forecast", "temperature"]` is how the+example above matched) and an opt-in `typos` rule that reads a word one+edit from exactly one pattern word as that word, and never chooses between+two.++Slots can be typed, each with a fixed parser: `{when:date}` reads+`tomorrow`, `next friday`, `in 3 days` or `oct 25` as `YYYY-MM-DD` against+the service's clock; `{at:time}` reads `3pm` or `15:30`; `{for:duration}`+reads `an hour and a half` as seconds; `{n:number}` reads `twenty-one`;+`{site:host}`, `{u:url}` and `{to:email}` read what they say. A value a+parser cannot read, or could read two ways (`3/10/2026`), makes the+trigger refuse the sentence rather than guess, and the reading says which+slot refused. The whole set of rules is in+[`_meta/tree-format.md`](_meta/tree-format.md); the tables are in+`src/pyrtan/nlp.py`.+## What Pyrtan refuses-- **No model in the loop.** Nothing in this repository calls an LLM. A model- may one day sit *beside* the swarm, proposing branches for recorded gaps;- it will not sit inside a run deciding anything.+- **No model in the loop.** Nothing in this repository calls an LLM. The+ reading of a sentence is tables and rules, each named in the trace when+ it fires. A model may one day sit *beside* the swarm, proposing branches+ for recorded gaps; it will not sit inside a run deciding anything.- **No endpoint a tree did not name.** Every HTTP node's host is checked against the tree's `hosts` at load time where it is literal and at run time where it is templated. Every further form of I/O arrives with an@@ -240,7 +279,7 @@ shows the same proposal and the same command whenever nothing answers.## Layout ```-src/pyrtan/ the package: tree, intent, template, conditions, extract,+src/pyrtan/ the package: tree, nlp, intent, template, conditions, extract,html, transport, runtime, trace, registry, gaps, grow, service, hub, page (the console), catalog, cli examples/ trees that run against real public APIs, and offline against diff --git a/_meta/design.md b/_meta/design.md index 6e97cad..1ad84e6 100644 --- a/_meta/design.md +++ b/_meta/design.md@@ -106,8 +106,8 @@ traded away by a convenient feature.## 3. Architecture as built ```-sentence ──► intent.py ──► registry.py ──► runtime.py ──► Outcome + Trace- (triggers) (trees, reload) │+sentence ──► nlp.py ──► intent.py ──► registry.py ──► runtime.py ──► Outcome + Trace+ (rules) (triggers) (trees, reload) │├─ template.py {{ paths | filters }} ├─ conditions.py when = { eq, between, … } ├─ extract.py json / regex / css@@ -118,9 +118,25 @@ grow.py an unanswered sentence -> a proposed tree, a scaffold, a logservice.py one process, one registry, HTTP endpoints, hot reload, gaps, unanswered hub.py peers, link up/stale/down, routing by capability, forwarding-cli.py check · list · run · ask · serve · hub · gaps · unanswered · new+cli.py check · list · run · match · ask · serve · hub · gaps · unanswered · new```+**Reading a sentence** (`nlp.py`, `intent.py`) is rules all the way down,+and the trace's first step (`intent`) names each rule that fired. The+rules: fold unicode; drop greetings, politeness and request wrappers from+the edges (kept when a trigger in the swarm starts or ends with the word);+expand contractions; turn number words into digits; match a literal word+to its plural or singular with a conservative stemmer and an exception+list. A tree adds `synonyms` and may opt into a `typos` rule (one edit from+exactly one pattern word; a word close to two is left alone). Typed slots+(`date`, `time`, `duration`, `number`, `host`, `url`, `email`) each have a+fixed parser; a value a parser cannot read, or could read two ways, makes+the trigger refuse the sentence. A `date` is read against the runner's+clock, which is the same clock `{{ now }}` reads, so invariant 1 holds:+same tree, same inputs, same clock, same path. There is no scoring, no+weighting, no fallback; the most literal characters still win and a tie is+still reported. `pyrtan match` prints the whole reading.+**The tree** (`tree.py`) is TOML, loaded into dataclasses and checked on load. `check()` returns problems at three levels; an error refuses the load. It verifies references, conditions, regexes, selectors, input@@ -186,6 +202,17 @@ A browser adapter will be an optional extra, not a requirement.## 4. Paths considered and not taken+**A model, or a classifier, for intent.** The obvious way to make "could+you tell me how warm it is in Oslo" find the weather tree is to embed+sentences and triggers and pick the nearest. Rejected, for the same reason+a model is rejected inside a run: the choice would be a score nobody can+read, and the trace would begin with a guess. What was built instead is a+short list of rewrites and a short list of parsers, each of which either+fires and says so or does nothing. The cost is that a sentence outside+the rules is not read, and is recorded as unanswered with a proposed+trigger; the benefit is that `pyrtan match` can show, line by line, why a+sentence went where it went, and that the same sentence always goes there.+**Rust.** The earlier product of this name chose Rust for a 6 MB resident footprint on site boxes. Pyrtan's services run wherever an operator already has Python, and its contributors are people who write trees, not people diff --git a/_meta/tree-format.md b/_meta/tree-format.md index 18fa28c..c0bfe78 100644 --- a/_meta/tree-format.md +++ b/_meta/tree-format.md@@ -31,16 +31,73 @@ pattern = "top {count:int} stories on hacker news"inputs = { count = 1 } # fixed inputs this sentence implies ```-A slot is `{name}` (any text), `{name:int}`, `{name:float}` or `{name:word}`-(no spaces). Every slot must be a declared input. `[words]` are optional;-`(a|b)` is a required choice; `|` separates alternatives inside either.-Matching ignores case, surrounding punctuation and repeated spaces; a slot's-text keeps its case. A pattern needs at least one literal word.+A slot is `{name}` (any text) or `{name:kind}`. Every slot must be a+declared input whose type fits the kind. `[words]` are optional; `(a|b)`+is a required choice; `|` separates alternatives inside either. A pattern+needs at least one literal word.++| Kind | Reads | Produces | Input type |+| :-- | :-- | :-- | :-- |+| `text` (the default) | any text | as written | `str` |+| `word` | one word, no spaces | as written | `str` |+| `int`, `float` | digits | a number | `int` / `float` |+| `number` | digits or words: `3`, `twenty-one`, `a dozen`, `1,000`, `2.5` | a number | `int` or `float` |+| `date` | `tomorrow`, `friday`, `next friday`, `in 3 days`, `2 weeks ago`, `oct 25`, `25 october 2026`, `2026-10-25`, `13/10` (day first), `end of month` | `YYYY-MM-DD` | `str` |+| `time` | `3pm`, `15:30`, `3:05 pm`, `noon`, `midnight`, `9 o'clock` | `HH:MM` | `str` |+| `duration` | `2 hours`, `30 min`, `1h30m`, `an hour and a half` | seconds | `int` |+| `host` | a host name or a URL | the host, lowercase | `str` |+| `url` | `http(s)://…` | as written | `str` |+| `email` | an address | lowercase | `str` |++A slot's text keeps its case. A `date` is read against the service's clock+(`{{ now }}` in templates is the same clock), so a run is deterministic+given its observations; the trace records what the words were read as.+A value the kind cannot read (`3/10/2026`, which could be either order;+`someday`) makes the trigger refuse the sentence, and the reading says so.++### How a sentence is read++Before any trigger sees a sentence, fixed rules rewrite it, and the trace's+first step (`intent`) names each rule that fired:++1. **Fold:** unicode normalised, curly quotes straightened, spaces collapsed,+ punctuation around the sentence dropped.+2. **Fillers:** greetings, politeness and request wrappers dropped from the+ edges (`hey pyrtan`, `please`, `can you`, `could you`, `thanks`…), unless+ a trigger in the swarm starts or ends with that word.+3. **Contractions:** `what's` → `what is`, `isn't` → `is not`, `gimme` →+ `give me`…+4. **Numbers:** number words become digits: `three` → `3`, `twenty-one` →+ `21`, `two hundred and five` → `205`.++Patterns get the first and third rule at load time, so `[what's the]` and+`what is the` are the same trigger. Matching ignores case, and a literal+word matches its plural or singular (`story` / `stories`; `news` stays+`news`). None of this is a model: every rule is a table in+`src/pyrtan/nlp.py`, and `pyrtan match "sentence"` shows what each one did.When several trees match a sentence, the trigger with the most literal characters wins. An exact tie is answered as `ambiguous`, naming the candidates, rather than resolved by load order.+### `[language]`++What a tree adds to the fixed rules:++```toml+[language]+synonyms = { weather = ["forecast", "temperature"], hacker = ["hn", "y combinator"] }+typos = true+```++`synonyms` maps a literal word in this tree's patterns to other words (or+phrases) that read the same; the plural rule applies to them too. `typos`+(default `false`) lets a sentence word of five letters or more that is one+edit away from exactly one word in this tree's patterns be read as that+word; a word close to two is left alone, because choosing would be a guess.+The trace shows every correction (`typo: wether->weather`). A hub applies a+tree's `[language]` when it routes, because `/capabilities` carries it.+## Inputs ```toml diff --git a/examples/site-status/site-status.toml b/examples/site-status/site-status.toml index 52a57f7..3d016af 100644 --- a/examples/site-status/site-status.toml +++ b/examples/site-status/site-status.toml@@ -11,11 +11,11 @@ hosts = ["*"]start = "probe" [[triggers]]-pattern = "is {host:word} up"+pattern = "is {host:host} up"[[triggers]]-pattern = "is {host:word} down"+pattern = "is {host:host} down"[[triggers]]-pattern = "[check] status of {host:word}"+pattern = "[check] status of {host:host}"[inputs.host] type = "str" diff --git a/examples/weather/weather.toml b/examples/weather/weather.toml index 4c9ccd6..3e65419 100644 --- a/examples/weather/weather.toml +++ b/examples/weather/weather.toml@@ -10,6 +10,10 @@ license = "MIT"hosts = ["geocoding-api.open-meteo.com", "api.open-meteo.com"] start = "geocode"+# "forecast in Oslo" and "temperature in Oslo" read as "weather in Oslo".+[language]+synonyms = { weather = ["forecast", "temperature"] }+[[triggers]] pattern = "[what's the] weather in {place}" [[triggers]] diff --git a/src/pyrtan/cli.py b/src/pyrtan/cli.py index 76aacd1..43cb98d 100644 --- a/src/pyrtan/cli.py +++ b/src/pyrtan/cli.py@@ -4,6 +4,7 @@pyrtan list [PATH ...] what is loaded and what it answers to pyrtan run TEXT [--trees PATH] answer a sentence, in this process pyrtan run --tree NAME --input k=v run a named tree+ pyrtan match TEXT [--trees PATH] how a sentence reads: the rewrites and every trigger's verdictpyrtan ask TEXT --at URL answer a sentence through a service or hub pyrtan serve [PATH ...] [--port N] host trees as a service pyrtan hub [--peer URL ...] route between services@@ -69,6 +70,12 @@ def main(argv: list[str] | None = None) -> int:p.add_argument("--egress", choices=EGRESS_POLICIES, default="any", help="'public' refuses private, loopback and link-local addresses") _token_arg(p, "the token the hub (--hub) requires, if any")+ p = sub.add_parser("match", help="how a sentence reads against the trees: the rewrites, every trigger's verdict, the winner")+ p.add_argument("text")+ p.add_argument("--trees", action="append", default=[], help="tree file or directory (repeatable; default $PYRTAN_TREES or ./trees)")+ p.add_argument("--now", help="read dates against this day (YYYY-MM-DD) instead of today")+ p.add_argument("--json", action="store_true")+p = sub.add_parser("ask", help="answer a sentence through a running service or hub") p.add_argument("text") p.add_argument("--at", default=os.environ.get("PYRTAN_AT", "http://127.0.0.1:8700"), help="service or hub URL (default $PYRTAN_AT or the local hub)")@@ -134,7 +141,7 @@ def main(argv: list[str] | None = None) -> int:args = parser.parse_args(argv) commands = {- "check": cmd_check, "list": cmd_list, "run": cmd_run, "ask": cmd_ask, "serve": cmd_serve, "hub": cmd_hub, "gaps": cmd_gaps, "unanswered": cmd_unanswered, "new": cmd_new,+ "check": cmd_check, "list": cmd_list, "run": cmd_run, "match": cmd_match, "ask": cmd_ask, "serve": cmd_serve, "hub": cmd_hub, "gaps": cmd_gaps, "unanswered": cmd_unanswered, "new": cmd_new,"search": cmd_search, "pull": cmd_pull, "index": cmd_index, "console": cmd_console, } return commands[args.command](args)@@ -243,6 +250,52 @@ def cmd_run(args: argparse.Namespace) -> int:return _report(outcome, args.json, not args.no_trace)+def cmd_match(args: argparse.Namespace) -> int:+ """Show how a sentence is read, without running anything: the rewrites+ the language rules made, each trigger that matched or refused, and the+ winner. The tool for someone writing a trigger."""+ import datetime as _dt++ reg = Registry.from_paths(_paths(args.trees))+ for path, err in reg.errors.items():+ print(f"pyrtan: {path}: {err}", file=sys.stderr)+ now: Any = _dt.datetime.now(_dt.timezone.utc)+ if args.now:+ try:+ now = _dt.date.fromisoformat(args.now)+ except ValueError:+ print(f"pyrtan match: --now must be YYYY-MM-DD, got {args.now!r}", file=sys.stderr)+ return 2+ reading = reg.read(args.text, now)+ if args.json:+ print(json.dumps(reading.to_dict(), indent=2, ensure_ascii=False, default=str))+ return 0 if reading.winner else 1+ print(f"sentence: {reading.text!r}")+ if reading.cleaned != reading.text:+ print(f"read as: {reading.cleaned!r}")+ for step in reading.steps:+ print(f" {step}")+ if not reading.considered:+ print("no trigger matched" + (f" ({len(reg.candidates())} tried)" if reg.candidates() else " (no trees loaded)"))+ return 1+ print("triggers:")+ for m in sorted(reading.considered, key=lambda m: (-m.specificity, m.tree)):+ verdict = "refused" if m.rejected else ("match" if m in reading.matches else "match, less specific")+ print(f" {m.tree}: {m.trigger.pattern!r} {m.specificity} literal chars -> {verdict}")+ if m.inputs and not m.rejected:+ print(f" inputs: {json.dumps(m.inputs, ensure_ascii=False, default=str)}")+ for note in m.notes:+ print(f" {note}")+ if reading.winner:+ print(f"winner: {reading.winner.tree}")+ return 0+ if reading.matches:+ print(f"tie: {sorted({m.tree for m in reading.matches})}; make one trigger more specific")+ else:+ print("no match: every trigger that fit refused a slot")+ return 1++def _answer_questions(outcome: Outcome, resume) -> Outcome: while outcome.status == "asked": if not sys.stdin.isatty(): diff --git a/src/pyrtan/grow.py b/src/pyrtan/grow.py index 71e03bb..0dafaa2 100644 --- a/src/pyrtan/grow.py +++ b/src/pyrtan/grow.py@@ -122,7 +122,7 @@ def propose(text: str, name: str | None = None) -> Proposal:"""A proposal for the sentence ``text``. Numbers in the sentence become ``{count:int}`` / ``{value:float}``- slots, host names ``{host:word}``, ``owner/repo``-shaped words+ slots, host names ``{host:host}``, ``owner/repo``-shaped words``{target:word}``; every other word is literal. The name is the literal words with filler dropped, or ``name`` if given. """@@ -163,7 +163,7 @@ def _slot_for(tok: str, inputs: dict[str, dict[str, Any]]) -> str | None:if HOST_RE.match(bare) and not bare.endswith(("...", ".")): slot = _unique("host", inputs) inputs[slot] = {"type": "str", "description": f"a host name, as in: {bare}", "role": "host"}- return f"{{{slot}:word}}"+ return f"{{{slot}:host}}"if PATH_RE.match(bare): slot = _unique("target", inputs) inputs[slot] = {"type": "str", "description": f"as in: {bare}"} diff --git a/src/pyrtan/hub.py b/src/pyrtan/hub.py index b9f5ae9..1e50d08 100644 --- a/src/pyrtan/hub.py +++ b/src/pyrtan/hub.py@@ -35,7 +35,9 @@ from typing import Anyfrom . import __version__ from .grow import Unanswered, no_match_outcome-from .intent import Trigger, best_matches+import datetime as _dt++from .intent import Trigger, readfrom .page import PAGE from .service import Server, http_json@@ -134,7 +136,7 @@ class Hub:for tree in peer.trees: for pattern in tree.get("triggers", []): try:- out.append(((peer.url, tree["name"]), Trigger(pattern)))+ out.append(((peer.url, tree["name"]), Trigger(pattern, language=tree.get("language") or {})))except Exception: continue return out@@ -177,18 +179,27 @@ class Hub:text = body.get("text") if not isinstance(text, str) or not text.strip(): return 200, {"status": "error", "reason": "bad-request", "say": "Send {\"text\": \"...\"}."}- matches = best_matches(text, self.candidates()) # type: ignore[arg-type]+ reading = read(text, self.candidates(), _dt.datetime.now(_dt.timezone.utc))+ matches = reading.matchesif not matches: entry = self.unanswered.record(text) answer = no_match_outcome(text, "No service in the swarm") answer["output"]["seen"] = entry["count"]+ answer["output"]["reading"] = {"cleaned": reading.cleaned, "steps": reading.steps}return 200, {"status": "error", "reason": "no-match", **answer} tree_names = {m.tree[1] for m in matches} # type: ignore[index] if len(tree_names) > 1:- return 200, {"status": "error", "reason": "ambiguous", "say": f"{text!r} matches more than one tree: {sorted(tree_names)}", "output": {"candidates": sorted(tree_names)}}+ return 200, {"status": "error", "reason": "ambiguous", "say": f"{text!r} matches more than one tree: {sorted(tree_names)}", "output": {"candidates": sorted(tree_names), "reading": {"cleaned": reading.cleaned, "steps": reading.steps}}}peer_url, tree_name = matches[0].tree # type: ignore[misc]+ for m in matches:+ m.tree = tree_name # the step names the tree, not the (peer, tree) pairpeer = self.holder(tree_name) or self.peers.get(peer_url)- return self.forward(peer, "/run", {"tree": tree_name, "inputs": matches[0].inputs, "depth": body.get("depth", 0), "trace": body.get("trace", True)})+ status, answer = self.forward(peer, "/run", {"tree": tree_name, "inputs": matches[0].inputs, "depth": body.get("depth", 0), "trace": body.get("trace", True)})+ if isinstance(answer.get("trace"), dict) and isinstance(answer["trace"].get("steps"), list):+ # The hub read the sentence; the peer only ran the tree. Put the+ # reading where a caller expects it, at the top of the trace.+ answer["trace"]["steps"].insert(0, {**reading.step(), "next": answer["trace"]["steps"][0]["node"] if answer["trace"]["steps"] else None, "elapsed_ms": 0, "depth": 0})+ return status, answerdef run(self, body: dict[str, Any]) -> tuple[int, dict[str, Any]]: tree = body.get("tree") diff --git a/src/pyrtan/intent.py b/src/pyrtan/intent.py index 5684231..66aa7ae 100644 --- a/src/pyrtan/intent.py +++ b/src/pyrtan/intent.py@@ -4,14 +4,24 @@ A trigger is a pattern with slots::"weather in {city}" "top {count:int} stories on hacker news"- "is {host:word} up"+ "is {host:host} up""what's [the] latest release of {repo}"+ "remind me {when:date} at {at:time}"-A slot is ``{name}`` (any text), ``{name:int}``, ``{name:float}`` or-``{name:word}`` (no spaces). Square brackets mark optional words, round-brackets a required choice, and ``|`` separates alternatives inside either:-``"how (hot|cold) is it in {place}"``. Matching is-case-insensitive, ignores surrounding punctuation, and collapses whitespace.+A slot is ``{name}`` (any text) or ``{name:kind}``. The kinds: ``int``,+``float``, ``word`` (no spaces), ``text``; and the read kinds, each with a+fixed parser in :mod:`pyrtan.nlp`: ``number`` (digits or words),+``date``, ``time``, ``duration``, ``host``, ``url``, ``email``. Square+brackets mark optional words, round brackets a required choice, and ``|``+separates alternatives inside either: ``"how (hot|cold) is it in {place}"``.++Before matching, the sentence and the pattern both go through+:func:`pyrtan.nlp.clean`: unicode folded, greetings and politeness dropped+from the edges, contractions expanded, number words turned to digits.+Matching is case-insensitive and a literal word matches its plural or+singular. A tree may add ``synonyms`` and turn on ``typos`` in its+``[language]`` table; both are applied here, and every rule that fires is+reported in the :class:`Match` so the trace can show it.When several triggers match, the most specific wins: the one with the most literal text. A tie is reported as ambiguity rather than resolved by luck.@@ -19,19 +29,32 @@ literal text. A tie is reported as ambiguity rather than resolved by luck.from __future__ import annotations+import datetime as _dtimport re from dataclasses import dataclass, field from typing import Any-SLOT = re.compile(r"\{([A-Za-z_][A-Za-z0-9_]*)(?::(int|float|word|text))?\}")+from . import nlp++KINDS = ("int", "float", "word", "text", "number", "date", "time", "duration", "host", "url", "email")+SLOT = re.compile(r"\{([A-Za-z_][A-Za-z0-9_]*)(?::([a-z]+))?\}")SLOT_RE = { "int": r"-?\d+", "float": r"-?\d+(?:\.\d+)?",+ "number": r"-?\d+(?:[.,]\d+)*","word": r"\S+", "text": r".+?",+ "date": r".+?",+ "time": r"\S+(?:\s*[ap]m)?",+ "duration": r".+?",+ "host": r"\S+",+ "url": r"\S+",+ "email": r"\S+",None: r".+?", }-CAST = {"int": int, "float": float, "word": str, "text": str, None: str}+# The input type each slot kind produces, for the checker.+PRODUCES = {"int": "int", "float": "float", "number": "number", "word": "str", "text": "str", "date": "str", "time": "str", "duration": "int", "host": "str", "url": "str", "email": "str", None: "str"}+TYPO_MIN_LENGTH = 5class TriggerError(ValueError):@@ -39,25 +62,90 @@ class TriggerError(ValueError):def normalise(text: str) -> str:+ """The light form: trimmed, one space between words. Used where a+ sentence is kept as a person wrote it (the unanswered log)."""text = text.strip() text = re.sub(r"^[\s\W_]+|[\s?!.,;:]+$", "", text) text = re.sub(r"\s+", " ", text) return text+@dataclass+class Language:+ """A tree's ``[language]`` table: what it adds to the fixed rules."""++ synonyms: dict[str, list[str]] = field(default_factory=dict)+ typos: bool = False++ @classmethod+ def from_raw(cls, raw: Any, where: str = "language") -> "Language":+ if raw is None:+ return cls()+ if not isinstance(raw, dict):+ raise TriggerError(f"{where} must be a table")+ syn = raw.get("synonyms", {})+ if not isinstance(syn, dict):+ raise TriggerError(f"{where}.synonyms must be a table of word = [words]")+ synonyms: dict[str, list[str]] = {}+ for word, alts in syn.items():+ if isinstance(alts, str):+ alts = [alts]+ if not isinstance(alts, list) or not all(isinstance(a, str) and a.strip() for a in alts):+ raise TriggerError(f"{where}.synonyms.{word} must be a list of words")+ if not re.fullmatch(r"[A-Za-z][A-Za-z'-]*", word):+ raise TriggerError(f"{where}.synonyms: {word!r} is not a word")+ synonyms[word.lower()] = [a.strip().lower() for a in alts]+ typos = raw.get("typos", False)+ if not isinstance(typos, bool):+ raise TriggerError(f"{where}.typos must be true or false")+ for key in raw:+ if key not in ("synonyms", "typos"):+ raise TriggerError(f"{where}: unknown key {key!r}; one of synonyms, typos")+ return cls(synonyms, typos)++ def to_dict(self) -> dict[str, Any]:+ out: dict[str, Any] = {}+ if self.synonyms:+ out["synonyms"] = self.synonyms+ if self.typos:+ out["typos"] = True+ return out++@dataclass class Trigger: pattern: str fixed_inputs: dict[str, Any] = field(default_factory=dict)+ language: Language = field(default_factory=Language)regex: re.Pattern[str] = field(init=False, repr=False) slots: dict[str, str | None] = field(init=False, default_factory=dict) specificity: int = field(init=False, default=0)+ cleaned: str = field(init=False, default="")+ vocabulary: set[str] = field(init=False, default_factory=set, repr=False)+ edges: tuple[str, str] = field(init=False, default=("", ""), repr=False) # first and last literal worddef __post_init__(self) -> None:- self.regex, self.slots, self.specificity = compile_pattern(self.pattern)+ if isinstance(self.language, dict):+ self.language = Language.from_raw(self.language)+ # The pattern gets the rewrites a sentence gets, minus the ones that+ # only make sense for a sentence (fillers, number words): folded,+ # contractions expanded, so "[what's the]" and "what is the" agree.+ self.cleaned = "".join(part if part.startswith("{") else nlp.expand_contractions(part)[0] for part in re.split(r"(\{[^}]*\})", nlp.fold(self.pattern)))+ self.regex, self.slots, self.specificity, self.vocabulary, self.edges = compile_pattern(self.cleaned, self.language)++ def match(self, text: str, *, now: _dt.datetime | _dt.date | None = None, cleaned: bool = False) -> dict[str, Any] | None:+ """The inputs this trigger reads from ``text``, or None. ``cleaned``+ says the text already went through :func:`nlp.clean`."""+ got = self.match_detail(text, now=now, cleaned=cleaned)+ return got.inputs if got is not None else None- def match(self, text: str) -> dict[str, Any] | None:- m = self.regex.fullmatch(normalise(text))+ def match_detail(self, text: str, *, now: _dt.datetime | _dt.date | None = None, cleaned: bool = False) -> "Match | None":+ sentence = text if cleaned else nlp.clean(text)[0]+ notes: list[str] = []+ if self.language.typos:+ sentence, fixed = correct_typos(sentence, self.vocabulary)+ notes.extend(f"typo: {a}->{b}" for a, b in fixed)+ m = self.regex.fullmatch(sentence)if not m: return None inputs: dict[str, Any] = dict(self.fixed_inputs)@@ -66,28 +154,101 @@ class Trigger:if raw is None: continue try:- inputs[name] = CAST[kind](raw.strip())- except ValueError:- return None- return inputs+ inputs[name] = read_slot(kind, raw.strip(), now)+ except (ValueError, nlp.ParseError) as e:+ notes.append(f"slot {name}: {e}")+ return Match("", self, {}, self.specificity, notes, rejected=True)+ if kind in nlp.PARSERS and str(inputs[name]) != raw.strip():+ notes.append(f"{name}: {raw.strip()!r} read as {inputs[name]!r}")+ return Match("", self, inputs, self.specificity, notes)+++def read_slot(kind: str | None, raw: str, now: _dt.datetime | _dt.date | None) -> Any:+ if kind == "int":+ return int(raw)+ if kind == "float":+ return float(raw)+ if kind in (None, "word", "text"):+ return raw+ parser = nlp.PARSERS[kind]+ return parser(raw, now) if kind == "date" else parser(raw)+++def correct_typos(sentence: str, vocabulary: set[str]) -> tuple[str, list[tuple[str, str]]]:+ """Replace a word that is one edit from exactly one vocabulary word+ (and is not itself one), for words of :data:`TYPO_MIN_LENGTH` letters+ or more. Returns the sentence and what changed."""+ if not vocabulary:+ return sentence, []+ fixed: list[tuple[str, str]] = []++ def one(m: re.Match[str]) -> str:+ word = m.group(0)+ low = word.lower()+ if len(low) < TYPO_MIN_LENGTH or low in vocabulary or any(low in nlp.forms(v) for v in vocabulary):+ return word+ close = [v for v in vocabulary if len(v) >= TYPO_MIN_LENGTH and nlp.distance(low, v, 1) <= 1]+ if len(close) != 1:+ return word+ fixed.append((word, close[0]))+ return close[0]++ return re.sub(r"[A-Za-z]+", one, sentence), fixed+++def _word_regex(word: str, language: Language) -> str:+ """A literal word matches itself, its singular or plural, and the same+ for each of its synonyms."""+ spellings = set(nlp.forms(word))+ phrases: list[str] = []+ for syn in language.synonyms.get(word.lower(), []):+ if " " in syn:+ phrases.append(r"\s+".join(re.escape(p) for p in syn.split()))+ else:+ spellings.update(nlp.forms(syn))+ alternatives = [re.escape(s) for s in sorted(spellings, key=lambda s: (-len(s), s))] + phrases+ return "(?:" + "|".join(alternatives) + ")"-def compile_pattern(pattern: str) -> tuple[re.Pattern[str], dict[str, str | None], int]:+def compile_pattern(pattern: str, language: Language | None = None) -> tuple[re.Pattern[str], dict[str, str | None], int, set[str], tuple[str, str]]:+ language = language or Language()if not pattern or not pattern.strip(): raise TriggerError("a trigger pattern cannot be empty") slots: dict[str, str | None] = {} out: list[str] = []+ vocabulary: set[str] = set()+ ordered: list[str] = []literal_chars = 0 pos = 0 depth = 0 text = pattern.strip().lower()+ word: list[str] = []++ def flush_word() -> None:+ if word:+ w = "".join(word)+ out.append(_word_regex(w, language))+ vocabulary.add(w)+ ordered.append(w)+ for syn in language.synonyms.get(w, []):+ vocabulary.update(syn.split())+ word.clear()+while pos < len(text): ch = text[pos]+ if ch.isalpha() or (word and ch in "'-" and pos + 1 < len(text) and text[pos + 1].isalpha()):+ word.append(ch)+ literal_chars += 1+ pos += 1+ continue+ flush_word()if ch == "{": m = SLOT.match(text, pos) if not m: raise TriggerError(f"bad slot in trigger {pattern!r} at {pos}") name, kind = m.group(1), m.group(2)+ if kind is not None and kind not in KINDS:+ raise TriggerError(f"slot {{{name}:{kind}}} in {pattern!r}: kind must be one of {', '.join(KINDS)}")if name in slots: raise TriggerError(f"slot {name!r} appears twice in {pattern!r}") slots[name] = kind@@ -131,6 +292,7 @@ def compile_pattern(pattern: str) -> tuple[re.Pattern[str], dict[str, str | Noneout.append(re.escape(ch)) literal_chars += 1 pos += 1+ flush_word()if depth != 0: raise TriggerError(f"unbalanced [ in trigger {pattern!r}") if literal_chars == 0:@@ -139,7 +301,7 @@ def compile_pattern(pattern: str) -> tuple[re.Pattern[str], dict[str, str | Noneregex = re.compile("".join(out), re.IGNORECASE) except re.error as e: raise TriggerError(f"trigger {pattern!r} does not compile: {e}") from e- return regex, slots, literal_chars+ return regex, slots, literal_chars, vocabulary, (ordered[0], ordered[-1]) if ordered else ("", "")@dataclass@@ -148,17 +310,79 @@ class Match:trigger: Trigger inputs: dict[str, Any] specificity: int+ notes: list[str] = field(default_factory=list)+ rejected: bool = False+++@dataclass+class Reading:+ """How a sentence was read: the rewrites, and every trigger's verdict."""++ text: str+ cleaned: str+ steps: list[str]+ matches: list[Match] # the winners: one, or several at a tie+ considered: list[Match] # every trigger that matched, or matched and then refused a slot+ now: str | None = None++ @property+ def winner(self) -> Match | None:+ return self.matches[0] if len(self.matches) == 1 else None+ def to_dict(self) -> dict[str, Any]:+ return {+ "text": self.text,+ "cleaned": self.cleaned,+ "steps": self.steps,+ "now": self.now,+ "matches": [_match_dict(m) for m in self.matches],+ "considered": [_match_dict(m) for m in self.considered],+ }-def best_matches(text: str, candidates: list[tuple[str, Trigger]]) -> list[Match]:+ def step(self) -> dict[str, Any]:+ """The ``intent`` step a trace starts with: what the sentence became+ and which trigger took it."""+ rewrite = f"{self.text!r}" + (f" -> {self.cleaned!r}" if self.cleaned != self.text else "")+ if self.winner:+ m = self.winner+ summary = f"{rewrite}; {m.tree} via {m.trigger.pattern!r} ({m.specificity} literal chars)"+ if m.notes:+ summary += "; " + "; ".join(m.notes)+ elif self.matches:+ summary = f"{rewrite}; tie between {sorted({m.tree for m in self.matches})}"+ else:+ summary = f"{rewrite}; no trigger matched"+ observed = {"cleaned": self.cleaned, "steps": self.steps}+ if self.winner:+ observed["trigger"] = self.winner.trigger.pattern+ observed["inputs"] = self.winner.inputs+ if self.winner.notes:+ observed["notes"] = self.winner.notes+ return {"node": "intent", "kind": "intent", "summary": summary, "observed": observed}+++def _match_dict(m: Match) -> dict[str, Any]:+ return {"tree": m.tree, "trigger": m.trigger.pattern, "specificity": m.specificity, "inputs": m.inputs, "notes": m.notes, "rejected": m.rejected}+++def read(text: str, candidates: list[tuple[Any, Trigger]], now: _dt.datetime | _dt.date | None = None) -> Reading:+ """Read ``text`` against every candidate and say what happened."""+ keep = frozenset(w for _, trig in candidates for w in trig.edges if w)+ cleaned, steps = nlp.clean(text, keep=keep)+ considered: list[Match] = []+ for tree, trig in candidates:+ got = trig.match_detail(cleaned, now=now, cleaned=True)+ if got is not None:+ got.tree = tree+ considered.append(got)+ live = [m for m in considered if not m.rejected]+ top = max((m.specificity for m in live), default=0)+ matches = [m for m in live if m.specificity == top]+ now_text = now.isoformat() if isinstance(now, (_dt.date, _dt.datetime)) else None+ return Reading(text, cleaned, steps, matches, considered, now_text)+++def best_matches(text: str, candidates: list[tuple[Any, Trigger]], now: _dt.datetime | _dt.date | None = None) -> list[Match]:"""All matches at the highest specificity. One item is a clear winner; several is ambiguity; none is no match."""- matches: list[Match] = []- for tree, trig in candidates:- inputs = trig.match(text)- if inputs is not None:- matches.append(Match(tree, trig, inputs, trig.specificity))- if not matches:- return []- top = max(m.specificity for m in matches)- return [m for m in matches if m.specificity == top]+ return read(text, candidates, now).matchesdiff --git a/src/pyrtan/nlp.py b/src/pyrtan/nlp.py new file mode 100644 index 0000000..9e3844a --- /dev/null +++ b/src/pyrtan/nlp.py@@ -0,0 +1,540 @@+"""Language rules: how a sentence is read before a trigger sees it.++This is the whole of Pyrtan's "natural language processing", and none of+it is a model. Every function here is a fixed table or a fixed rule: a+sentence goes in, a rewritten sentence (or a value) comes out, and the same+input always gives the same output. Each rewrite that fires is named, so+the trace can show exactly how a sentence was read.++What the rules do:++- :func:`clean` rewrites a sentence step by step: fold unicode, drop+ greetings and politeness at the edges, expand contractions, turn number+ words into digits. The same rules are applied to trigger patterns at+ load time, so a pattern written as ``[what's the] weather`` still matches+ a sentence written either way.+- :func:`parse_date`, :func:`parse_time`, :func:`parse_duration`,+ :func:`parse_number`, :func:`parse_host`, :func:`parse_url` and+ :func:`parse_email` read one value each, for the typed slots a trigger+ may declare (``{when:date}``, ``{for:duration}``, ``{site:host}``...).+ Dates are relative to a ``now`` the caller passes in; the runtime passes+ its clock, so a run is deterministic given its observations.+- :func:`singular` is a conservative stemmer used so that a literal word in+ a pattern matches its plural (``story``/``stories``), and nothing else.+- :func:`distance` is edit distance, for the opt-in typo rule.++What the rules refuse: anything that would need a guess. A date that could+be read two ways is not read at all. A word not in any table is left alone.+"""++from __future__ import annotations++import datetime as _dt+import re+import unicodedata+from typing import Any, Callable++# ---------------------------------------------------------------- tables++CONTRACTIONS = {+ "what's": "what is", "whats": "what is", "where's": "where is", "who's": "who is", "how's": "how is",+ "when's": "when is", "why's": "why is", "it's": "it is", "that's": "that is", "there's": "there is",+ "here's": "here is", "let's": "let us", "isn't": "is not", "aren't": "are not", "wasn't": "was not",+ "weren't": "were not", "don't": "do not", "doesn't": "does not", "didn't": "did not", "can't": "can not",+ "cannot": "can not", "couldn't": "could not", "won't": "will not", "wouldn't": "would not",+ "shouldn't": "should not", "haven't": "have not", "hasn't": "has not", "hadn't": "had not",+ "i'm": "i am", "i've": "i have", "i'd": "i would", "i'll": "i will", "you're": "you are",+ "you've": "you have", "you'd": "you would", "you'll": "you will", "we're": "we are", "we've": "we have",+ "they're": "they are", "they've": "they have", "gimme": "give me", "lemme": "let me", "wanna": "want to",+ "gonna": "going to", "u": "you", "pls": "please", "plz": "please", "thx": "thanks",+}++# Said at the start of a sentence and meaning nothing to a tree.+LEADING_FILLERS = [+ "hey pyrtan", "hi pyrtan", "hello pyrtan", "ok pyrtan", "okay pyrtan", "pyrtan",+ "hey", "hi", "hello", "yo", "ok", "okay", "so", "um", "uh", "well", "please", "kindly",+ "can you", "could you", "would you", "will you", "can you please", "could you please", "would you please",+ "i want you to", "i need you to", "i would like you to", "i'd like you to",+ "quick question", "question",+]+# Said at the end, likewise.+TRAILING_FILLERS = ["please", "thanks", "thank you", "for me", "if you can", "if you could", "if possible", "pyrtan"]++UNITS = {+ "zero": 0, "one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9,+ "ten": 10, "eleven": 11, "twelve": 12, "thirteen": 13, "fourteen": 14, "fifteen": 15, "sixteen": 16,+ "seventeen": 17, "eighteen": 18, "nineteen": 19,+}+TENS = {"twenty": 20, "thirty": 30, "forty": 40, "fourty": 40, "fifty": 50, "sixty": 60, "seventy": 70, "eighty": 80, "ninety": 90}+SCALES = {"hundred": 100, "thousand": 1_000, "million": 1_000_000, "billion": 1_000_000_000}+NUMBER_WORDS = set(UNITS) | set(TENS) | set(SCALES) | {"and", "a", "dozen"}++# Words whose trailing s is not a plural, so :func:`singular` leaves them.+NOT_PLURALS = {+ "news", "is", "as", "us", "his", "this", "thus", "yes", "plus", "bus", "gas", "was", "has", "its",+ "status", "bonus", "always", "perhaps", "physics", "series", "species", "means", "analysis", "basis",+ "lens", "chess", "address", "process", "access", "class", "pass", "less", "unless", "across", "ssl",+ "https", "dns", "tls", "os", "ios", "chaos", "canvas", "atlas", "virus", "campus", "census", "focus",+ "iris", "oasis", "crisis", "thesis", "diagnosis", "axis", "texas", "paris", "mars", "venus",+}++WEEKDAYS = ["monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday"]+WEEKDAY_ABBR = {d[:3]: i for i, d in enumerate(WEEKDAYS)}+MONTHS = ["january", "february", "march", "april", "may", "june", "july", "august", "september", "october", "november", "december"]+MONTH_ABBR = {m[:3]: i + 1 for i, m in enumerate(MONTHS)}+MONTH_ABBR["sept"] = 9++DURATION_UNITS = {+ "s": 1, "sec": 1, "secs": 1, "second": 1, "seconds": 1,+ "m": 60, "min": 60, "mins": 60, "minute": 60, "minutes": 60,+ "h": 3600, "hr": 3600, "hrs": 3600, "hour": 3600, "hours": 3600,+ "d": 86400, "day": 86400, "days": 86400,+ "w": 604800, "wk": 604800, "wks": 604800, "week": 604800, "weeks": 604800,+}++WORD = re.compile(r"[A-Za-z][A-Za-z'’]*|\d+(?:[.,]\d+)*|\S")+HOST_RE = re.compile(r"^(?:https?://)?((?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}|localhost|(?:\d{1,3}\.){3}\d{1,3})(?::\d{1,5})?(?:/\S*)?$", re.IGNORECASE)+URL_RE = re.compile(r"^https?://[^\s<>\"']+$", re.IGNORECASE)+EMAIL_RE = re.compile(r"^[A-Za-z0-9._%+-]+@(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,63}$")+++# ---------------------------------------------------------------- cleaning+++def fold(text: str) -> str:+ """Unicode to its plain form: curly quotes straight, full-width ASCII,+ dashes to hyphens, whitespace to one space."""+ text = unicodedata.normalize("NFKC", text)+ text = text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')+ text = re.sub(r"[‐-―−]", "-", text)+ text = re.sub(r"\s+", " ", text).strip()+ return text+++def trim(text: str) -> str:+ """Drop punctuation that carries nothing: around the whole sentence, and+ a comma or full stop between words."""+ text = re.sub(r"^[\s\W_]+|[\s?!.,;:]+$", "", text)+ text = re.sub(r"\s*[,;!?]+\s*", " ", text)+ text = re.sub(r"(?<=[A-Za-z])\.(?=\s|$)", "", text)+ return re.sub(r"\s+", " ", text).strip()+++def strip_fillers(text: str, keep: frozenset[str] = frozenset()) -> tuple[str, list[str]]:+ """Greetings, politeness and request wrappers at either edge, repeatedly.+ A filler whose first word is in ``keep`` stays: a trigger somewhere+ starts (or ends) with that word, so to that swarm it is not filler."""+ dropped: list[str] = []+ changed = True+ while changed and text:+ changed = False+ low = text.lower()+ for f in sorted(LEADING_FILLERS, key=len, reverse=True):+ if (low == f or low.startswith(f + " ") or low.startswith(f + ",")) and f.split()[0] not in keep:+ dropped.append(text[: len(f)])+ text = text[len(f):].lstrip(" ,")+ changed = True+ break+ low = text.lower()+ for f in sorted(TRAILING_FILLERS, key=len, reverse=True):+ if (low == f or low.endswith(" " + f) or low.endswith("," + f)) and f.split()[-1] not in keep:+ dropped.append(text[len(text) - len(f):])+ text = text[: len(text) - len(f)].rstrip(" ,")+ changed = True+ break+ return text, dropped+++def expand_contractions(text: str) -> tuple[str, list[str]]:+ expanded: list[str] = []++ def one(m: re.Match[str]) -> str:+ word = m.group(0)+ full = CONTRACTIONS.get(word.lower().replace("’", "'"))+ if full is None:+ return word+ expanded.append(f"{word}->{full}")+ return full if word.islower() or not word[0].isupper() else full[0].upper() + full[1:]++ return re.sub(r"[A-Za-z]+(?:'[A-Za-z]+)?", one, text), expanded+++def words_to_digits(text: str) -> tuple[str, list[str]]:+ """``twenty-one`` -> ``21``, ``two hundred and five`` -> ``205``, ``a+ dozen`` -> ``12``. Only runs of number words are touched."""+ words = text.split(" ")+ out: list[str] = []+ changed: list[str] = []+ i = 0+ while i < len(words):+ run_end = _number_run(words, i)+ if run_end > i:+ run = words[i:run_end]+ value = _run_value(run)+ if value is not None:+ out.append(str(value))+ changed.append(f"{' '.join(run)}->{value}")+ i = run_end+ continue+ out.append(words[i])+ i += 1+ return " ".join(out), changed+++def _is_number_word(word: str) -> bool:+ parts = word.lower().split("-")+ return all(p in UNITS or p in TENS or p in SCALES or p == "dozen" for p in parts)+++def _number_run(words: list[str], start: int) -> int:+ """The end (exclusive) of the longest run of number words at ``start``:+ ``two hundred and five`` or ``a dozen``; ``and``/``a`` only inside or,+ for ``a``, before a scale word. Returns ``start`` when there is none."""+ i = start+ first = words[i].lower()+ if first == "a" and i + 1 < len(words) and words[i + 1].lower() in ("dozen", "hundred", "thousand", "million", "billion"):+ i += 1+ elif not _is_number_word(first):+ return start+ end = i+ while i < len(words):+ w = words[i].lower()+ if _is_number_word(w):+ i += 1+ end = i+ elif w in ("and", "a") and i + 1 < len(words) and _is_number_word(words[i + 1].lower()):+ i += 1+ else:+ break+ return end+++def _run_value(run: list[str]) -> int | None:+ words: list[str] = []+ for tok in run:+ words.extend(tok.lower().split("-"))+ total = 0+ current = 0+ seen_number = False+ for w in words:+ if w in ("and", "a"):+ if w == "a":+ current = 1+ continue+ if w in UNITS:+ current += UNITS[w]+ seen_number = True+ elif w in TENS:+ current += TENS[w]+ seen_number = True+ elif w == "dozen":+ current = (current or 1) * 12+ seen_number = True+ elif w == "hundred":+ current = (current or 1) * 100+ seen_number = True+ elif w in SCALES:+ total += (current or 1) * SCALES[w]+ current = 0+ seen_number = True+ else:+ return None+ return total + current if seen_number else None+++def clean(text: str, *, fillers: bool = True, numbers: bool = True, keep: frozenset[str] = frozenset()) -> tuple[str, list[str]]:+ """The whole pipeline. Returns the rewritten sentence and the list of+ rules that fired, each as ``rule: detail``, in order. ``keep`` names+ words that are not filler here (see :func:`strip_fillers`)."""+ steps: list[str] = []+ folded = fold(text)+ if folded != text.strip():+ steps.append("fold")+ text = trim(folded)+ if fillers:+ text, dropped = strip_fillers(text, keep)+ if dropped:+ steps.append("fillers: " + ", ".join(repr(d) for d in dropped))+ text = trim(text)+ text, expanded = expand_contractions(text)+ if expanded:+ steps.append("contractions: " + ", ".join(expanded))+ if numbers:+ text, changed = words_to_digits(text)+ if changed:+ steps.append("numbers: " + ", ".join(changed))+ return re.sub(r"\s+", " ", text).strip(), steps+++# ---------------------------------------------------------------- words+++def singular(word: str) -> str:+ """A conservative singular: ``stories`` -> ``story``, ``boxes`` ->+ ``box``, ``items`` -> ``item``; ``news`` stays ``news``. Short words and+ words ending in ``ss`` are left alone."""+ w = word.lower()+ if len(w) < 4 or w in NOT_PLURALS or w.endswith("ss") or not w.isalpha():+ return w+ if w.endswith("ies") and len(w) > 4:+ return w[:-3] + "y"+ if w.endswith(("ches", "shes", "sses", "xes", "zes")):+ return w[:-2]+ if w.endswith("s"):+ return w[:-1]+ return w+++def plural(word: str) -> str:+ w = word.lower()+ if not w.isalpha() or w in NOT_PLURALS:+ return w+ if w.endswith("y") and len(w) > 2 and w[-2] not in "aeiou":+ return w[:-1] + "ies"+ if w.endswith(("ch", "sh", "ss", "x", "z", "s")):+ return w + "es"+ return w + "s"+++def forms(word: str) -> set[str]:+ """Every spelling a literal pattern word may be matched by: itself, its+ singular and its plural."""+ w = word.lower()+ out = {w}+ if w.isalpha() and len(w) >= 3:+ base = singular(w)+ out.add(base)+ out.add(plural(base))+ return out+++def distance(a: str, b: str, limit: int = 2) -> int:+ """Edit distance with transpositions, stopping early past ``limit``."""+ a, b = a.lower(), b.lower()+ if abs(len(a) - len(b)) > limit:+ return limit + 1+ prev_prev: list[int] | None = None+ prev = list(range(len(b) + 1))+ for i, ca in enumerate(a, 1):+ cur = [i] + [0] * len(b)+ for j, cb in enumerate(b, 1):+ cost = 0 if ca == cb else 1+ cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + cost)+ if prev_prev is not None and i > 1 and j > 1 and ca == b[j - 2] and a[i - 2] == cb:+ cur[j] = min(cur[j], prev_prev[j - 2] + 1)+ if min(cur) > limit:+ return limit + 1+ prev_prev, prev = prev, cur+ return prev[-1]+++# ---------------------------------------------------------------- values+++class ParseError(ValueError):+ pass+++def parse_number(text: str) -> int | float:+ t, _ = words_to_digits(text.strip())+ t = t.replace(",", "")+ if re.fullmatch(r"-?\d+", t):+ return int(t)+ if re.fullmatch(r"-?\d+\.\d+", t):+ return float(t)+ raise ParseError(f"{text!r} is not a number")+++def parse_duration(text: str) -> int:+ """``2 hours``, ``30 min``, ``1h30m``, ``1 hour and a half``, ``an hour`` -> seconds."""+ t, _ = words_to_digits(text.strip().lower())+ t = re.sub(r"\bhalf\s+an?\s+([a-z]+)", r"0.5 \1", t)+ t = re.sub(r"\b(an?)\s+(?=(?:hour|minute|second|day|week)s?\b)", "1 ", t)+ t = re.sub(r"(\d+(?:\.\d+)?)\s*([a-z]+)\s+and\s+a\s+half\b", lambda m: f"{float(m.group(1)) + 0.5} {m.group(2)}", t)+ t = re.sub(r"\s+and\s+", " ", t)+ total = 0.0+ found = False+ pos = 0+ for m in re.finditer(r"(\d+(?:\.\d+)?)\s*([a-z]+)", t):+ if t[pos:m.start()].strip():+ raise ParseError(f"{text!r} is not a duration")+ unit = m.group(2)+ if unit not in DURATION_UNITS:+ raise ParseError(f"{text!r}: unknown unit {unit!r}")+ total += float(m.group(1)) * DURATION_UNITS[unit]+ found = True+ pos = m.end()+ if not found or t[pos:].strip():+ raise ParseError(f"{text!r} is not a duration")+ return int(total)+++def parse_time(text: str) -> str:+ """``3pm``, ``15:30``, ``3:05 pm``, ``noon``, ``midnight``, ``9 o'clock`` -> ``HH:MM``."""+ t, _ = words_to_digits(text.strip().lower())+ t = t.replace(".", ":").replace(" ", "")+ if t in ("noon", "midday"):+ return "12:00"+ if t == "midnight":+ return "00:00"+ m = re.fullmatch(r"(\d{1,2})(?::(\d{2}))?(?:o'?clock)?(am|pm)?", t)+ if not m:+ raise ParseError(f"{text!r} is not a time")+ hour, minute, half = int(m.group(1)), int(m.group(2) or 0), m.group(3)+ if half:+ if not 1 <= hour <= 12:+ raise ParseError(f"{text!r}: {hour} with {half}")+ hour = hour % 12 + (12 if half == "pm" else 0)+ elif m.group(2) is None and not m.group(0).endswith("clock"):+ raise ParseError(f"{text!r}: a bare number is not a time; say 15:00 or 3pm")+ if not (0 <= hour <= 23 and 0 <= minute <= 59):+ raise ParseError(f"{text!r} is not a time")+ return f"{hour:02d}:{minute:02d}"+++def parse_date(text: str, now: _dt.date | _dt.datetime | None = None) -> str:+ """A date, as ``YYYY-MM-DD``, relative to ``now`` where it has to be.++ Reads ISO dates, ``25 oct [2026]``, ``oct 25[, 2026]``, ``25/10/2026``+ (day first; a slash date that could be read either way is refused),+ ``today``, ``tomorrow``, ``yesterday``, ``day after tomorrow``,+ weekdays (``friday`` is the next one, today included; ``next friday`` is+ strictly after today; ``last friday`` strictly before), ``in N days``,+ ``N days ago``, ``next week``, ``last month``, ``end of month``.+ """+ today = (now or _dt.datetime.now(_dt.timezone.utc)).date() if isinstance(now, _dt.datetime) or now is None else now+ t, _ = words_to_digits(text.strip().lower())+ t = re.sub(r"^(on|at|for|the|this coming)\s+", "", t).strip()+ t = re.sub(r"(\d)(st|nd|rd|th)\b", r"\1", t)+ t = t.replace(",", " ").replace(" ", " ").strip()++ m = re.fullmatch(r"(\d{4})-(\d{2})-(\d{2})", t)+ if m:+ return _date(int(m.group(1)), int(m.group(2)), int(m.group(3)), text).isoformat()+ m = re.fullmatch(r"(\d{1,2})[/.](\d{1,2})(?:[/.](\d{2,4}))?", t)+ if m:+ d, mo = int(m.group(1)), int(m.group(2))+ if d <= 12 and mo <= 12 and d != mo:+ raise ParseError(f"{text!r} could be day/month or month/day; write it as {today.year}-{mo:02d}-{d:02d} or '{d} {MONTHS[mo - 1]}'")+ year = int(m.group(3)) if m.group(3) else today.year+ if year < 100:+ year += 2000+ return _date(year, mo, d, text).isoformat()+ m = re.fullmatch(r"(\d{1,2})(?:\s+of)?\s+([a-z]+)(?:\s+(\d{4}))?", t) or re.fullmatch(r"([a-z]+)\s+(\d{1,2})(?:\s+(\d{4}))?", t)+ if m and (_month(m.group(1)) or _month(m.group(2))):+ if m.group(1).isdigit():+ day, month = int(m.group(1)), _month(m.group(2))+ else:+ month, day = _month(m.group(1)), int(m.group(2))+ if not month:+ raise ParseError(f"{text!r} is not a date")+ year = int(m.group(3)) if m.group(3) else today.year+ return _date(year, month, day, text).isoformat()++ if t in ("today", "now"):+ return today.isoformat()+ if t in ("tomorrow", "tmrw", "tmr"):+ return (today + _dt.timedelta(days=1)).isoformat()+ if t in ("yesterday",):+ return (today - _dt.timedelta(days=1)).isoformat()+ if t in ("day after tomorrow", "the day after tomorrow"):+ return (today + _dt.timedelta(days=2)).isoformat()+ if t in ("day before yesterday", "the day before yesterday"):+ return (today - _dt.timedelta(days=2)).isoformat()++ m = re.fullmatch(r"(?:(this|next|last)\s+)?([a-z]+)", t)+ if m and _weekday(m.group(2)) is not None:+ target = _weekday(m.group(2))+ which = m.group(1)+ ahead = (target - today.weekday()) % 7+ if which == "next":+ ahead = ahead or 7+ elif which == "last":+ ahead = ahead - 7 if ahead else -7+ return (today + _dt.timedelta(days=ahead)).isoformat()++ m = re.fullmatch(r"in\s+(\d+)\s+(day|days|week|weeks|month|months|year|years)", t)+ if m:+ return _shift(today, int(m.group(1)), m.group(2)).isoformat()+ m = re.fullmatch(r"(\d+)\s+(day|days|week|weeks|month|months|year|years)\s+ago", t)+ if m:+ return _shift(today, -int(m.group(1)), m.group(2)).isoformat()+ m = re.fullmatch(r"(next|last)\s+(week|month|year)", t)+ if m:+ return _shift(today, 1 if m.group(1) == "next" else -1, m.group(2)).isoformat()+ m = re.fullmatch(r"end of (?:the |this )?(month|year)", t)+ if m:+ if m.group(1) == "year":+ return _dt.date(today.year, 12, 31).isoformat()+ nxt = _shift(_dt.date(today.year, today.month, 1), 1, "month")+ return (nxt - _dt.timedelta(days=1)).isoformat()+ if t in ("start of month", "start of the month", "beginning of month", "beginning of the month"):+ return today.replace(day=1).isoformat()+ raise ParseError(f"{text!r} is not a date this reads")+++def _month(word: str | None) -> int | None:+ if not word or word.isdigit():+ return None+ if word in MONTHS:+ return MONTHS.index(word) + 1+ return MONTH_ABBR.get(word[:4] if word.startswith("sept") else word[:3]) if len(word) >= 3 and word[:3] in MONTH_ABBR and MONTHS[MONTH_ABBR[word[:3]] - 1].startswith(word) else None+++def _weekday(word: str) -> int | None:+ if word in WEEKDAYS:+ return WEEKDAYS.index(word)+ if len(word) >= 3 and word[:3] in WEEKDAY_ABBR and WEEKDAYS[WEEKDAY_ABBR[word[:3]]].startswith(word):+ return WEEKDAY_ABBR[word[:3]]+ return None+++def _date(year: int, month: int, day: int, text: str) -> _dt.date:+ try:+ return _dt.date(year, month, day)+ except ValueError as e:+ raise ParseError(f"{text!r}: {e}") from e+++def _shift(day: _dt.date, n: int, unit: str) -> _dt.date:+ unit = unit.rstrip("s")+ if unit == "day":+ return day + _dt.timedelta(days=n)+ if unit == "week":+ return day + _dt.timedelta(weeks=n)+ months = day.month - 1 + (n if unit == "month" else 12 * n)+ year, month = day.year + months // 12, months % 12 + 1+ last = [31, 29 if year % 4 == 0 and (year % 100 != 0 or year % 400 == 0) else 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31][month - 1]+ return _dt.date(year, month, min(day.day, last))+++def parse_host(text: str) -> str:+ m = HOST_RE.fullmatch(text.strip())+ if not m:+ raise ParseError(f"{text!r} is not a host name")+ return m.group(1).lower()+++def parse_url(text: str) -> str:+ t = text.strip()+ if not URL_RE.fullmatch(t):+ raise ParseError(f"{text!r} is not a URL")+ return t+++def parse_email(text: str) -> str:+ t = text.strip()+ if not EMAIL_RE.fullmatch(t):+ raise ParseError(f"{text!r} is not an email address")+ return t.lower()+++PARSERS: dict[str, Callable[..., Any]] = {+ "number": parse_number,+ "duration": parse_duration,+ "time": parse_time,+ "date": parse_date,+ "host": parse_host,+ "url": parse_url,+ "email": parse_email,+}diff --git a/src/pyrtan/page.py b/src/pyrtan/page.py index ae57c0f..1ce1563 100644 --- a/src/pyrtan/page.py +++ b/src/pyrtan/page.py@@ -317,6 +317,7 @@ async function ask(snapshot, answer) {const card = document.createElement("div"); card.className = "card"; let html = `<div><span class="status ${esc(r.status)}">${esc(r.status)}</span>${r.reason ? ` <span class="mute">(${esc(r.reason)})</span>` : ""} <span class="mute">· ${esc(r.tree||"")} · ${ms} ms${r.peer ? " · " + esc(r.peer) : ""}</span></div>`; if (r.say) html += `<div class="say">${esc(r.say)}</div>`;+ if (r.output && r.output.reading && r.output.reading.steps && r.output.reading.steps.length) html += `<div class="mute" style="font-size:14px">Read as <code>${esc(r.output.reading.cleaned)}</code>: ${r.output.reading.steps.map(esc).join("; ")}</div>`;if (r.reason === "no-match" && r.output && r.output.proposal) { const p = r.output.proposal; html += `<div class="gap">No tree answers this yet. The one that would: <code>${esc(p.name)}</code>, answering <code>${esc(p.trigger)}</code>. It is listed under <a href="#unanswered">Unanswered</a>.</div><pre>${esc(p.scaffold)}</pre>`; diff --git a/src/pyrtan/registry.py b/src/pyrtan/registry.py index 7c3c56f..ee56557 100644 --- a/src/pyrtan/registry.py +++ b/src/pyrtan/registry.py@@ -14,7 +14,9 @@ from dataclasses import dataclass, fieldfrom pathlib import Path from typing import Any, Iterable-from .intent import Match, Trigger, best_matches+import datetime as _dt++from .intent import Match, Reading, Trigger, readfrom .tree import Tree, TreeError, load_tree@@ -124,10 +126,15 @@ class Registry:def candidates(self) -> list[tuple[str, Trigger]]: return [(t.name, trig) for t in self.trees() for trig in t.triggers]- def match(self, text: str) -> Match:+ def read(self, text: str, now: _dt.datetime | _dt.date | None = None) -> Reading:+ """How ``text`` reads against every trigger here: the rewrites, the+ verdicts, the winner or the tie."""+ return read(text, self.candidates(), now)++ def match(self, text: str, now: _dt.datetime | _dt.date | None = None) -> Match:"""The one tree that answers ``text``, or :class:`NoMatch` / :class:`Ambiguous`."""- matches = best_matches(text, self.candidates())+ matches = self.read(text, now).matchesif not matches: raise NoMatch(f"no tree answers {text!r}") if len(matches) > 1: diff --git a/src/pyrtan/runtime.py b/src/pyrtan/runtime.py index 6b1058e..6c50ba7 100644 --- a/src/pyrtan/runtime.py +++ b/src/pyrtan/runtime.py@@ -29,7 +29,7 @@ from __future__ import annotationsimport datetime as _dt import time from dataclasses import dataclass, field-from typing import Any, Protocol+from typing import Any, Callable, Protocolfrom . import conditions from .extract import extract@@ -98,6 +98,7 @@ class Runner:transport: Transport = field(default_factory=UrllibTransport) resolver: Resolver | None = None env: dict[str, Any] = field(default_factory=dict)+ clock: Callable[[], _dt.datetime] = field(default=lambda: _dt.datetime.now(_dt.timezone.utc))max_steps: int = MAX_STEPS max_depth: int = MAX_DEPTH emitted: list[dict[str, Any]] = field(default_factory=list)@@ -105,15 +106,27 @@ class Runner:# ------------------------------------------------------------ entry points def run_intent(self, text: str, depth: int = 0) -> Outcome:- try:- match = self.registry.match(text)- except NoMatch:+ """Read a sentence and run the tree it names. The trace begins with an+ ``intent`` step: what the sentence was rewritten to, which rules+ fired, and which trigger took it."""+ reading = self.registry.read(text, self.clock())+ if not reading.matches:if self.resolver is not None: return self.resolver.run_intent(text, depth)- return Outcome("error", "", reason="no-match", **no_match_outcome(text, "Nothing here"))- except Ambiguous as e:- return Outcome("error", "", reason="ambiguous", say=str(e), output={"candidates": [m.tree for m in e.matches]})- return self.run_tree(match.tree, match.inputs, depth)+ out = Outcome("error", "", reason="no-match", **no_match_outcome(text, "Nothing here"))+ out.output = {**(out.output or {}), "reading": {"cleaned": reading.cleaned, "steps": reading.steps, "considered": [m.trigger.pattern for m in reading.considered]}}+ return out+ if len({m.tree for m in reading.matches}) > 1:+ trees = sorted({m.tree for m in reading.matches})+ return Outcome("error", "", reason="ambiguous", say=f"{text!r} matches more than one tree equally well: {', '.join(trees)}", output={"candidates": trees, "reading": {"cleaned": reading.cleaned, "steps": reading.steps}})+ match = reading.matches[0]+ tree = self.registry.get(match.tree)+ if tree is None:+ return self.run_tree(match.tree, match.inputs, depth)+ trace = Trace(tree.name)+ step = reading.step()+ trace.add(Step(step["node"], step["kind"], step["summary"], observed=step["observed"], next=tree.start, depth=depth))+ return self.run(tree, match.inputs, depth=depth, trace=trace)def run_tree(self, name: str, inputs: dict[str, Any] | None = None, depth: int = 0) -> Outcome: tree = self.registry.get(name)@@ -132,7 +145,7 @@ class Runner:except TreeError as e: return Outcome("error", tree.name, reason="bad-input", say=str(e), trace=trace) state = dict(state) if state is not None else {}- scope: dict[str, Any] = {"inputs": bound, "state": state, "tree": {"name": tree.name, "version": tree.version}, "now": _now(), "env": self.env}+ scope: dict[str, Any] = {"inputs": bound, "state": state, "tree": {"name": tree.name, "version": tree.version}, "now": _now(self.clock()), "env": self.env}if scope_extra: scope.update(scope_extra) try:@@ -410,6 +423,6 @@ def _cond_text(cond: dict[str, Any]) -> str:return " and ".join(f"{k} {v!r}" for k, v in cond.items())-def _now() -> dict[str, Any]:- now = _dt.datetime.now(_dt.timezone.utc)+def _now(now: _dt.datetime | None = None) -> dict[str, Any]:+ now = now or _dt.datetime.now(_dt.timezone.utc)return {"iso": now.isoformat(timespec="seconds"), "date": now.date().isoformat(), "epoch": int(now.timestamp()), "year": now.year} diff --git a/src/pyrtan/tree.py b/src/pyrtan/tree.py index b4850e6..a2afd1b 100644 --- a/src/pyrtan/tree.py +++ b/src/pyrtan/tree.py@@ -43,7 +43,7 @@ from pathlib import Pathfrom typing import Any from . import conditions, extract as extractmod-from .intent import Trigger, TriggerError+from .intent import KINDS as SLOT_KINDS, PRODUCES, Language, Trigger, TriggerErrorfrom .template import PLACEHOLDER, placeholders NAME = re.compile(r"^[a-z0-9][a-z0-9\-]*$")@@ -162,6 +162,7 @@ class Tree:source: str | None = None raw: dict[str, Any] = field(default_factory=dict, repr=False) license: str = ""+ language: Language = field(default_factory=Language)def node(self, node_id: str) -> Node: try:@@ -211,6 +212,7 @@ class Tree:"tags": self.tags, "hosts": self.hosts, "license": self.license,+ "language": self.language.to_dict(),"triggers": [t.pattern for t in self.triggers], "inputs": { n: {k: v for k, v in {"type": s.type, "default": s.default, "required": s.required and s.default is None, "description": s.description}.items() if v not in (None, "", False)}@@ -256,6 +258,11 @@ def build(raw: dict[str, Any], source: str | None = None) -> Tree:if not isinstance(license_, str): raise TreeError(f"{name}: 'license' must be a string such as \"MIT\" (an SPDX identifier)")+ try:+ language = Language.from_raw(raw.get("language"), "language")+ except TriggerError as e:+ raise TreeError(f"{name}: {e}") from e+triggers: list[Trigger] = [] for i, t in enumerate(raw.get("triggers", [])): if isinstance(t, str):@@ -263,7 +270,7 @@ def build(raw: dict[str, Any], source: str | None = None) -> Tree:if not isinstance(t, dict) or "pattern" not in t: raise TreeError(f"{name}: triggers[{i}] needs a 'pattern'") try:- triggers.append(Trigger(t["pattern"], dict(t.get("inputs", {}))))+ triggers.append(Trigger(t["pattern"], dict(t.get("inputs", {})), language))except TriggerError as e: raise TreeError(f"{name}: triggers[{i}]: {e}") from e@@ -310,7 +317,7 @@ def build(raw: dict[str, Any], source: str | None = None) -> Tree:hint = " (top-level keys such as 'start' must come before the first [table] in TOML)" if _buried(raw, "start") else "" raise TreeError(f"{name}: 'start' must name a node{hint}")- tree = Tree(name, description, version, hosts, triggers, inputs, start, nodes, tags, source, raw, license_.strip())+ tree = Tree(name, description, version, hosts, triggers, inputs, start, nodes, tags, source, raw, license_.strip(), language)errors = [p for p in check(tree) if p.level == "error"] if errors: raise TreeError(f"{name}: " + "; ".join(f"{p.where}: {p.message}" for p in errors))@@ -341,6 +348,18 @@ def _infer_type(value: Any) -> str:# ---------------------------------------------------------------- checking+def _slot_fits(kind: str | None, input_type: str) -> bool:+ """Whether what a slot kind reads can be bound to an input of this type."""+ produced = PRODUCES[kind]+ if produced == "int":+ return input_type in ("int", "float", "str")+ if produced == "float":+ return input_type in ("float", "str")+ if produced == "number":+ return input_type in ("int", "float", "str")+ return input_type == "str"++def check(tree: Tree) -> list[Problem]: """Validate structure, references, templates, conditions, selectors, hosts and reachability. Never raises; returns what it found."""@@ -355,9 +374,13 @@ def check(tree: Tree) -> list[Problem]:err("start", f"start node {tree.start!r} does not exist") for i, t in enumerate(tree.triggers):- for slot in t.slots:+ for slot, kind in t.slots.items():if slot not in tree.inputs: err(f"triggers[{i}]", f"slot {{{slot}}} is not a declared input")+ continue+ want = tree.inputs[slot].type+ if not _slot_fits(kind, want):+ err(f"triggers[{i}]", f"slot {{{slot}:{kind}}} produces a {PRODUCES[kind]}, but input {slot!r} is a {want}")for fixed in t.fixed_inputs: if fixed not in tree.inputs: err(f"triggers[{i}]", f"fixed input {fixed!r} is not a declared input") diff --git a/tests/test_cli.py b/tests/test_cli.py index f990965..685bb11 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py@@ -98,7 +98,7 @@ def test_unanswered_lists_and_scaffolds(tmp_path, capsys):http_json("POST", f"{server.url}/intent", {"text": "ping example.org"}) http_json("POST", f"{server.url}/intent", {"text": "tell me a joke"}) code, out, _ = run(capsys, "unanswered", "--at", server.url)- assert code == 0 and out.startswith("'ping example.org' x2") and "would be: host-ping answering ping {host:word}" in out and "pyrtan new joke --from" in out+ assert code == 0 and out.startswith("'ping example.org' x2") and "would be: host-ping answering ping {host:host}" in out and "pyrtan new joke --from" in outcode, out, _ = run(capsys, "unanswered", "--at", server.url, "--json") assert [e["text"] for e in json.loads(out)["unanswered"]] == ["ping example.org", "tell me a joke"] code, out, _ = run(capsys, "unanswered", "--at", server.url, "--scaffold", str(tmp_path))@@ -108,3 +108,16 @@ def test_unanswered_lists_and_scaffolds(tmp_path, capsys):assert "kept" in out and "wrote" not in out finally: server.stop()+++def test_match_explains_a_sentence(capsys):+ code, out, _ = run(capsys, "match", "Hey Pyrtan, what's the forecast in Oslo please?", "--trees", str(EXAMPLES))+ assert code == 0+ assert "read as: 'what is the forecast in Oslo'" in out+ assert "fillers: 'Hey Pyrtan', 'please'" in out and "contractions: what's->what is" in out+ assert "winner: weather" in out and '"place": "Oslo"' in out+ code, out, _ = run(capsys, "match", "is example dot com up", "--trees", str(EXAMPLES))+ assert code == 1 and "no trigger matched" in out+ code, out, _ = run(capsys, "match", "weather in Oslo", "--trees", str(EXAMPLES), "--json")+ assert code == 0 and json.loads(out)["matches"][0]["tree"] == "weather"+ assert run(capsys, "match", "x", "--trees", str(EXAMPLES), "--now", "yesterday")[0] == 2diff --git a/tests/test_examples.py b/tests/test_examples.py index e9a85ca..1e02dd4 100644 --- a/tests/test_examples.py +++ b/tests/test_examples.py@@ -65,8 +65,13 @@ def test_site_status_branches(runner):def test_site_status_rejects_bad_host(runner):+ # The {host:host} slot refuses what is not a host name, so the trigger+ # does not fire at all, and the reading says which trigger refused.out = runner.run_intent("is http://x up")- assert out.status == "error" and out.reason == "bad-input"+ assert out.status == "error" and out.reason == "no-match"+ assert out.output["reading"]["considered"] == ["is {host:host} up"]+ out = runner.run_tree("site-status", {"host": "http://x"})+ assert out.status == "error" and out.reason == "bad-input" # the input's own pattern still guards a direct rundef test_github_release(runner): diff --git a/tests/test_grow.py b/tests/test_grow.py index 098da22..166fb18 100644 --- a/tests/test_grow.py +++ b/tests/test_grow.py@@ -12,7 +12,7 @@ from pyrtan.tree import check, load_tree_text"text, name, trigger, inputs", [ ("top 3 stories on hacker news", "top-stories-hacker-news", "top {count:int} stories on hacker news", {"count": 3}),- ("is example.com up", "host-up", "is {host:word} up", {"host": "example.com"}),+ ("is example.com up", "host-up", "is {host:host} up", {"host": "example.com"}),("what's the latest release of astral-sh/uv", "latest-release", "what's the latest release of {target:word}", {"target": "astral-sh/uv"}), ("convert 12.5 km to miles", "convert-km-miles", "convert {value:float} km to miles", {"value": 12.5}), ("turn on the lights", "turn-lights", "turn on the lights", {}), diff --git a/tests/test_intent.py b/tests/test_intent.py index 0476e34..d517cc9 100644 --- a/tests/test_intent.py +++ b/tests/test_intent.py@@ -11,7 +11,8 @@ def test_normalise():def test_slots_and_types(): t = Trigger("top {count:int} stories on hacker news") assert t.match("Top 5 stories on Hacker News") == {"count": 5}- assert t.match("top five stories on hacker news") is None+ assert t.match("top five stories on hacker news") == {"count": 5} # number words are read+ assert t.match("top many stories on hacker news") is Noneassert Trigger("is {host:word} up").match("is example.com up") == {"host": "example.com"} assert Trigger("is {host:word} up").match("is example dot com up") is None assert Trigger("set {x:float}").match("set 1.5") == {"x": 1.5}@@ -57,3 +58,88 @@ def test_ties_are_reported_not_resolved():a, b = Trigger("ping {x}"), Trigger("ping {y}") matches = best_matches("ping me", [("a", a), ("b", b)]) assert {m.tree for m in matches} == {"a", "b"}+++# ---------------------------------------------------------------- reading++import datetime as dt++from pyrtan.intent import Language, read+++def test_patterns_and_sentences_get_the_same_rewrites():+ t = Trigger("[what's the] weather in {place}")+ assert t.match("What is the weather in Oslo") == {"place": "Oslo"}+ assert t.match("Hey Pyrtan, what's the weather in Oslo, please?") == {"place": "Oslo"}+ assert t.match("could you tell me the weather in Oslo") is None # "tell me" is not filler+++def test_literal_words_match_their_plural_or_singular():+ t = Trigger("top {count:int} stories on hacker news")+ assert t.match("top 1 story on hacker news") == {"count": 1}+ assert Trigger("list the item").match("list the items") == {}+ assert Trigger("latest news").match("latest new") is None # "news" is not a plural+++def test_synonyms_from_the_language_table():+ lang = Language.from_raw({"synonyms": {"weather": ["forecast", "temperature"], "stories": "posts"}})+ t = Trigger("weather in {place}", language=lang)+ assert t.match("forecast in Oslo") == {"place": "Oslo"}+ assert t.match("temperature in Oslo") == {"place": "Oslo"}+ assert t.match("climate in Oslo") is None+ assert Trigger("top {n:int} stories", language=lang).match("top 3 posts") == {"n": 3}+ assert Trigger("hacker news", language=Language.from_raw({"synonyms": {"hacker": ["hn", "y combinator"]}})).match("y combinator news") == {}+ for bad in [{"synonyms": "weather"}, {"synonyms": {"weather": [1]}}, {"typos": "yes"}, {"other": 1}, {"synonyms": {"two words": ["x"]}}]:+ with pytest.raises(TriggerError):+ Language.from_raw(bad)+++def test_typos_are_opt_in_and_only_when_one_word_is_close():+ strict = Trigger("weather in {place}")+ loose = Trigger("weather in {place}", language=Language(typos=True))+ assert strict.match("wether in Oslo") is None+ got = loose.match_detail("wether in Oslo")+ assert got.inputs == {"place": "Oslo"} and got.notes == ["typo: wether->weather"]+ assert loose.match("weathr in Oslo") == {"place": "Oslo"}+ assert loose.match("wather in Oslo") == {"place": "Oslo"}+ assert loose.match("water in Oslo") is None # two edits away: not corrected+ assert loose.match("in in Oslo") is None # short words are never corrected+ both = Trigger("(hotel|model) in {place}", language=Language(typos=True))+ assert both.match("hodel in Oslo") is None # one edit from both: a guess, so refused+ assert both.match("hostel in Oslo") == {"place": "Oslo"} # one edit from hotel only+++def test_typed_slots_read_values():+ sunday = dt.date(2026, 10, 25)+ t = Trigger("remind me {when:date} at {at:time}")+ assert t.match("remind me tomorrow at 3pm", now=sunday) == {"when": "2026-10-26", "at": "15:00"}+ assert t.match("remind me next friday at 15:30", now=sunday) == {"when": "2026-10-30", "at": "15:30"}+ got = t.match_detail("remind me someday at 3pm", now=sunday)+ assert got.rejected and got.notes == ["slot when: 'someday' is not a date this reads"]+ assert Trigger("wait {for:duration}").match("wait an hour and a half") == {"for": 5400}+ assert Trigger("is {site:host} up").match("is https://Example.com/ up") == {"site": "example.com"}+ assert Trigger("is {site:host} up").match("is example dot com up") is None+ assert Trigger("open {u:url}").match("open https://example.com/a") == {"u": "https://example.com/a"}+ assert Trigger("mail {to:email}").match("mail Someone@Example.com") == {"to": "someone@example.com"}+ assert Trigger("top {n:number} stories").match("top twenty-one stories") == {"n": 21}+ with pytest.raises(TriggerError):+ Trigger("{x:money}")+++def test_read_reports_the_rewrites_and_every_verdict():+ sunday = dt.date(2026, 10, 25)+ candidates = [("w", Trigger("[what's the] weather in {place}")), ("r", Trigger("remind me {when:date}")), ("g", Trigger("hi {name}"))]+ r = read("Hey, what's the weather in Oslo please", candidates, sunday)+ assert r.cleaned == "what is the weather in Oslo"+ assert r.steps == ["fillers: 'Hey', 'please'", "contractions: what's->what is"]+ assert r.winner.tree == "w" and r.winner.inputs == {"place": "Oslo"}+ step = r.step()+ assert step["kind"] == "intent" and "w via" in step["summary"] and step["observed"]["inputs"] == {"place": "Oslo"}+ r = read("remind me someday", candidates, sunday)+ assert r.matches == [] and [m.rejected for m in r.considered] == [True]+ assert "no trigger matched" in r.step()["summary"]+ # "hi" is kept because a trigger starts with it.+ r = read("hi Bob", candidates, sunday)+ assert r.winner.tree == "g" and r.winner.inputs == {"name": "Bob"}+ r = read("hi", [("w", candidates[0][1])], sunday)+ assert r.cleaned == "" and r.matches == []diff --git a/tests/test_nlp.py b/tests/test_nlp.py new file mode 100644 index 0000000..2af8210 --- /dev/null +++ b/tests/test_nlp.py@@ -0,0 +1,132 @@+"""The language rules: fixed tables and fixed rules, the same answer every+time, and a refusal where a guess would be needed."""++import datetime as dt++import pytest++from pyrtan import nlp++SUNDAY = dt.date(2026, 10, 25)+++def test_clean_names_every_rule_that_fired():+ text, steps = nlp.clean("Hey Pyrtan, what's the weather in Oslo please?")+ assert text == "what is the weather in Oslo" # the place keeps its case+ assert steps == ["fillers: 'Hey Pyrtan', 'please'", "contractions: what's->what is"]+ text, steps = nlp.clean("Could you tell me twenty-one things")+ assert text == "tell me 21 things" and steps == ["fillers: 'Could you'", "numbers: twenty-one->21"]+ assert nlp.clean("is example.com up") == ("is example.com up", [])+ assert nlp.clean("What’s New York like")[0] == "What is New York like" # curly quote folded+++def test_fillers_are_kept_when_a_trigger_starts_with_them():+ assert nlp.clean("hi there")[0] == "there"+ assert nlp.clean("hi there", keep=frozenset({"hi"}))[0] == "hi there"+ assert nlp.clean("hey pyrtan, hi there", keep=frozenset({"hi"}))[0] == "hi there"+ assert nlp.clean("say thanks", keep=frozenset({"thanks"}))[0] == "say thanks"+ assert nlp.clean("say thanks")[0] == "say"+++@pytest.mark.parametrize("text,value", [+ ("three", 3), ("twenty-one", 21), ("twenty one", 21), ("a dozen", 12), ("two hundred and five", 205),+ ("one hundred thousand", 100000), ("1,000", 1000), ("2.5", 2.5), ("-4", -4), ("ninety nine", 99),+])+def test_numbers(text, value):+ assert nlp.parse_number(text) == value+++def test_number_words_only_touch_number_runs():+ assert nlp.words_to_digits("a dozen eggs and two hundred and five items") == ("12 eggs and 205 items", ["a dozen->12", "two hundred and five->205"])+ assert nlp.words_to_digits("set a reminder and one more")[0] == "set a reminder and 1 more"+ assert nlp.words_to_digits("one of a kind")[0] == "1 of a kind"+ with pytest.raises(nlp.ParseError):+ nlp.parse_number("many")+++@pytest.mark.parametrize("text,expected", [+ ("today", "2026-10-25"), ("tomorrow", "2026-10-26"), ("yesterday", "2026-10-24"),+ ("the day after tomorrow", "2026-10-27"), ("friday", "2026-10-30"), ("next friday", "2026-10-30"),+ ("last friday", "2026-10-23"), ("sunday", "2026-10-25"), ("this sunday", "2026-10-25"), ("next sunday", "2026-11-01"),+ ("mon", "2026-10-26"), ("in 3 days", "2026-10-28"), ("in two weeks", "2026-11-08"), ("2 weeks ago", "2026-10-11"),+ ("oct 25", "2026-10-25"), ("25 october 2026", "2026-10-25"), ("October 3rd", "2026-10-03"), ("3rd of march", "2026-03-03"),+ ("13/10", "2026-10-13"), ("25.10.2026", "2026-10-25"), ("2026-02-28", "2026-02-28"), ("next month", "2026-11-25"),+ ("last year", "2025-10-25"), ("end of month", "2026-10-31"), ("end of year", "2026-12-31"), ("on tuesday", "2026-10-27"),+])+def test_dates_relative_to_a_fixed_now(text, expected):+ assert nlp.parse_date(text, SUNDAY) == expected+++def test_dates_refuse_a_guess():+ with pytest.raises(nlp.ParseError, match="could be day/month or month/day"):+ nlp.parse_date("3/10/2026", SUNDAY)+ with pytest.raises(nlp.ParseError):+ nlp.parse_date("2026-02-30", SUNDAY)+ with pytest.raises(nlp.ParseError):+ nlp.parse_date("soonish", SUNDAY)+ assert nlp.parse_date("2024-01-31", SUNDAY) == "2024-01-31"+ assert nlp.parse_date("in 1 month", dt.date(2026, 1, 31)) == "2026-02-28" # clamped, not rolled over+++@pytest.mark.parametrize("text,expected", [+ ("3pm", "15:00"), ("15:30", "15:30"), ("3:05 pm", "15:05"), ("noon", "12:00"), ("midnight", "00:00"),+ ("9 o'clock", "09:00"), ("12am", "00:00"), ("12 pm", "12:00"), ("7.45am", "07:45"),+])+def test_times(text, expected):+ assert nlp.parse_time(text) == expected+++def test_times_refuse():+ for bad in ["25:00", "7", "13pm", "half past"]:+ with pytest.raises(nlp.ParseError):+ nlp.parse_time(bad)+++@pytest.mark.parametrize("text,seconds", [+ ("2 hours", 7200), ("30 min", 1800), ("1h30m", 5400), ("an hour", 3600), ("two days", 172800), ("90s", 90),+ ("1 hour 15 minutes", 4500), ("1 hour and a half", 5400), ("half an hour", 1800), ("a week", 604800),+])+def test_durations(text, seconds):+ assert nlp.parse_duration(text) == seconds+++def test_durations_refuse():+ for bad in ["3 fortnights", "soon", "2", "hours"]:+ with pytest.raises(nlp.ParseError):+ nlp.parse_duration(bad)+++def test_hosts_urls_emails():+ assert nlp.parse_host("https://Example.com/x") == "example.com"+ assert nlp.parse_host("api.github.com") == "api.github.com"+ assert nlp.parse_host("localhost:8700") == "localhost"+ assert nlp.parse_url("https://example.com/a?b=1") == "https://example.com/a?b=1"+ assert nlp.parse_email("Someone@Example.COM") == "someone@example.com"+ for fn, bad in [(nlp.parse_host, "example dot com"), (nlp.parse_host, "not a host"), (nlp.parse_url, "example.com"), (nlp.parse_email, "nope@")]:+ with pytest.raises(nlp.ParseError):+ fn(bad)+++@pytest.mark.parametrize("word,base", [+ ("stories", "story"), ("news", "news"), ("boxes", "box"), ("items", "item"), ("status", "status"),+ ("cities", "city"), ("bus", "bus"), ("this", "this"), ("series", "series"), ("addresses", "address"),+ ("analysis", "analysis"), ("is", "is"), ("releases", "release"), ("up", "up"),+])+def test_singular_is_conservative(word, base):+ assert nlp.singular(word) == base+++def test_forms_cover_singular_and_plural_only():+ assert nlp.forms("story") == {"story", "stories"}+ assert nlp.forms("stories") == {"story", "stories"}+ assert nlp.forms("news") == {"news"}+ assert nlp.forms("up") == {"up"}+++def test_distance():+ assert nlp.distance("weather", "weather") == 0+ assert nlp.distance("wether", "weather") == 1+ assert nlp.distance("waether", "weather") == 1 # a transposition is one edit+ assert nlp.distance("wather", "weather") == 1+ assert nlp.distance("abc", "xyz") == 3+ assert nlp.distance("a", "abcdef", 1) == 2 # stops early past the limitdiff --git a/tests/test_runtime.py b/tests/test_runtime.py index f369a07..6ebfc4f 100644 --- a/tests/test_runtime.py +++ b/tests/test_runtime.py@@ -255,3 +255,46 @@ say = "{{ state.r.reason }}"""" out = Runner(reg_with(text), RecordedTransport()).run_tree("recursive") assert out.status == "done" and "too-deep" in out.trace.render()+++REMINDER = """+name = "reminder"+description = "reads a date and a time"+start = "say"++[language]+synonyms = { remind = ["ping", "nudge"] }++[[triggers]]+pattern = "remind me {when:date} at {at:time}"++[inputs.when]+type = "str"+[inputs.at]+type = "str"++[nodes.say]+kind = "done"+say = "{{ inputs.when }} {{ inputs.at }}"+"""+++def test_trace_starts_with_how_the_sentence_was_read():+ import datetime as dt++ runner = Runner(reg_with(REMINDER), RecordedTransport(), clock=lambda: dt.datetime(2026, 10, 25, 12, tzinfo=dt.timezone.utc))+ out = runner.run_intent("Hey Pyrtan, could you nudge me next friday at 3pm please")+ assert out.status == "done" and out.say == "2026-10-30 15:00"+ first = out.trace.steps[0]+ assert first.kind == "intent" and first.node == "intent" and first.next == "say"+ assert first.observed["cleaned"] == "nudge me next friday at 3pm"+ assert first.observed["steps"] == ["fillers: 'Hey Pyrtan', 'please', 'could you'"] # in the order they were dropped+ assert first.observed["inputs"] == {"when": "2026-10-30", "at": "15:00"}+ assert "when: 'next friday' read as '2026-10-30'" in first.observed["notes"]+ assert "reminder via 'remind me {when:date} at {at:time}'" in out.trace.render()+ # The same clock drives {{ now }}: a run is deterministic given its observations.+ out2 = runner.run_intent("remind me tomorrow at noon")+ assert out2.say == "2026-10-26 12:00"+ # A sentence that fits the shape but not the value is refused, and says why.+ out3 = runner.run_intent("remind me someday at noon")+ assert out3.reason == "no-match" and out3.output["reading"]["considered"] == ["remind me {when:date} at {at:time}"]diff --git a/tests/test_service_hub.py b/tests/test_service_hub.py index 4ab46ef..f6387ef 100644 --- a/tests/test_service_hub.py +++ b/tests/test_service_hub.py@@ -89,7 +89,7 @@ def test_service_endpoints(example_service):assert {t["name"] for t in caps["trees"]} >= {"weather", "hacker-news"} _, out = http_json("POST", f"{server.url}/intent", {"text": "weather in Oslo"}) assert out["status"] == "done" and out["output"]["sky"] == "raining"- assert out["trace"]["steps"][0]["node"] == "geocode"+ assert out["trace"]["steps"][0]["kind"] == "intent" and out["trace"]["steps"][1]["node"] == "geocode"_, out = http_json("POST", f"{server.url}/run", {"tree": "hacker-news-item", "inputs": {"id": 101}, "trace": False}) assert out["status"] == "done" and "trace" not in out _, out = http_json("POST", f"{server.url}/intent", {"text": "nothing matches this"})@@ -230,3 +230,29 @@ def test_demo_page_is_served_by_service_and_hub(example_service):assert http_json("GET", f"{hub_srv.url}/api")[1]["pyrtan"] == "hub" finally: hub_srv.stop()+++def test_hub_reads_sentences_with_the_tree_language_and_puts_the_reading_in_the_trace():+ reg = Registry.from_paths([EXAMPLES])+ svc = Service(reg, name="examples", transport=RecordedTransport.from_file(FIXTURES / "examples.json"))+ srv = Server(svc.route, quiet=True).start()+ hub = Hub(poll_every=60)+ hub_srv = Server(hub.route, quiet=True).start()+ try:+ hub.add_peer(srv.url)+ _, caps = http_json("GET", f"{hub_srv.url}/capabilities")+ weather = next(t for t in caps["trees"] if t["name"] == "weather")+ assert weather["language"] == {"synonyms": {"weather": ["forecast", "temperature"]}}+ # The synonym lives in the tree; the hub applies it when it routes.+ _, out = http_json("POST", f"{hub_srv.url}/intent", {"text": "Hey, what's the forecast in Oslo?"})+ assert out["status"] == "done" and out["output"]["sky"] == "raining"+ first = out["trace"]["steps"][0]+ assert first["kind"] == "intent" and first["next"] == "geocode"+ assert first["observed"]["cleaned"] == "what is the forecast in Oslo"+ assert out["trace"]["steps"][1]["node"] == "geocode"+ _, out = http_json("POST", f"{hub_srv.url}/intent", {"text": "please do nothing"})+ assert out["reason"] == "no-match" and out["output"]["reading"]["cleaned"] == "do nothing"+ finally:+ hub.stop()+ hub_srv.stop()+ srv.stop()diff --git a/tests/test_tree.py b/tests/test_tree.py index d7139a8..0daf108 100644 --- a/tests/test_tree.py +++ b/tests/test_tree.py@@ -146,3 +146,16 @@ next = "ok"load_tree_text(base.replace('over = "r.json"', 'over = "r.json"\ntree = "a"\nstart = "ok"')) notes = problems(base.replace('over = "r.json"', 'over = "r.json"\ntree = "a"')) assert any("no 'limit'" in n for n in notes)+++def test_slot_kinds_must_fit_their_inputs():+ base = 'name = "t"\nstart = "s"\n[[triggers]]\npattern = "{pat}"\n[inputs.x]\ntype = "{typ}"\n[nodes.s]\nkind = "done"\nsay = "x"\n'+ for pat, typ in [("at {x:date}", "int"), ("wait {x:duration}", "bool"), ("n {x:float}", "int"), ("on {x:host}", "list")]:+ with pytest.raises(TreeError, match="produces a"):+ load_tree_text(base.format(pat=pat, typ=typ))+ for pat, typ in [("at {x:date}", "str"), ("wait {x:duration}", "int"), ("wait {x:duration}", "str"), ("n {x:number}", "int"), ("n {x:number}", "float"), ("n {x:int}", "float")]:+ load_tree_text(base.format(pat=pat, typ=typ))+ with pytest.raises(TreeError, match="language.synonyms"):+ load_tree_text('name = "t"\nstart = "s"\n[language]\nsynonyms = "no"\n[nodes.s]\nkind = "done"\nsay = "x"\n')+ with pytest.raises(TreeError, match="unknown key"):+ load_tree_text('name = "t"\nstart = "s"\n[language]\nstemming = true\n[nodes.s]\nkind = "done"\nsay = "x"\n')