Document the reputation query API
docker-build-push / build-push (push) Successful in 5s

README gets the full contract (request/response shape, the cost ceiling and
why it exists, the block-width cap and expiry, and the two new tables).
CLAUDE.md gets the two rules that are easy to undo by accident: the endpoint
must never serve unauthenticated, and the width cap and expiry are policy
decisions rather than tuning knobs.
This commit is contained in:
pmb committed 2026-09-13 16:24:48 -07:00
1 parent 6952fbadf6
commit 47e92ca10f
2 files changed
+135 -1

No files matched your search

+27
View File
@@ -19,6 +19,18 @@ A FastAPI webhook server. One Mastodon admin webhook delivers `account.created`,
and auto-**silence** young/dormant accounts past the threshold — lowered for and auto-**silence** young/dormant accounts past the threshold — lowered for
accounts with a flagged signup IP (see README). accounts with a flagged signup IP (see README).
Alongside the webhook it serves a **reputation query API** — `POST
/api/reputation` — which is the pre-emptive counterpart to all of the above.
Mastodon calls it during signup, immediately before its own IpBlock /
EmailDomainBlock lookups; it classifies the IP/email domain, records it,
registers the block, and only then returns, so the lookup that follows
rejects the signup outright. It **writes**, so it is authenticated
(`X-Reputation-Key`) and 404s unless `REPUTATION_API_ENABLED` *and*
`REPUTATION_API_KEY` are both set. The caller is the
`call-welcomebot-reputation-api` patch on the Mastodon fork — see
`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`. README.md's
"Reputation query API" section has the full contract.
Alongside the webhook it serves an optional read-only **blocklist API** — Alongside the webhook it serves an optional read-only **blocklist API** —
`GET /api/blocks`, `/api/blocks/ip-ranges`, `/api/blocks/email-domains` — `GET /api/blocks`, `/api/blocks/ip-ranges`, `/api/blocks/email-domains` —
publishing the IP ranges and email domains the bot has auto-blocked. It is publishing the IP ranges and email domains the bot has auto-blocked. It is
@@ -62,6 +74,17 @@ user-memory note `project_welcomebot` has the current state.
check-mail.org's disposable/risk fields back into the response — that would check-mail.org's disposable/risk fields back into the response — that would
be redistributing a paid reputation feed's data. Account identifiers be redistributing a paid reputation feed's data. Account identifiers
(handles, ids, emails, raw signup IPs) are out for the usual reasons. (handles, ids, emails, raw signup IPs) are out for the usual reasons.
- **The reputation API must never serve unauthenticated.** It installs IP
blocks; an open endpoint that does that is a denial-of-service lever. An
empty `REPUTATION_API_KEY` disables the route rather than opening it, and
that behaviour is asserted in `test_local.py` — don't "fix" it into a
permissive default.
- **Don't remove the block-width cap or the block expiry.** `IP_SCRUTINY_MAX_
BLOCK_PREFIX_V4`/`_V6` (24/48) and `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` (24h)
exist because ipapi.is reports whole ASN routes: without the cap a single
`abuser` flag on a consumer address blocks its entire ISP route (routes of
/8–/11 are routine, and `197.211.32.0/19` was already registered against a
Benin mobile carrier). Raising them is a policy decision, not a tuning knob.
- **`admin.yttrx.com`'s `/root/yttrx-welcomebot/` is not a git repo any more** - **`admin.yttrx.com`'s `/root/yttrx-welcomebot/` is not a git repo any more**
(cleaned up 2026-07-06). Don't `git init`/clone it back into place — the (cleaned up 2026-07-06). Don't `git init`/clone it back into place — the
registry image + `docker-compose.yml`/`.env` is the whole deploy surface now. registry image + `docker-compose.yml`/`.env` is the whole deploy surface now.
@@ -217,6 +240,10 @@ needed. Without these scopes the reconcile 403s and exits having changed
nothing, which is the deliberate failure mode: never read an outage as "every nothing, which is the deliberate failure mode: never read an outage as "every
block was removed". block was removed".
The reputation API needs no scopes beyond those above — it registers the same
`Admin::IpBlock` / `EmailDomainBlock` records as the webhook path, so
`admin:write:ip_blocks` + `admin:write:email_domain_blocks` cover it.
**The suspicious-signup sweep (`app/suspicious_sweep.py`) needs another scope, **The suspicious-signup sweep (`app/suspicious_sweep.py`) needs another scope,
`admin:read:accounts`**, to `GET /api/v1/admin/accounts/:id` (fetches current `admin:read:accounts`**, to `GET /api/v1/admin/accounts/:id` (fetches current
post/follow counts + role/suspended state to decide whether to suspend) — post/follow counts + role/suspended state to decide whether to suspend) —
+108 -1
View File
@@ -21,6 +21,12 @@ A small FastAPI webhook server that receives Mastodon **admin webhooks** for
- Signatures are HMAC-verified; both flows dedupe in a sqlite store so nobody - Signatures are HMAC-verified; both flows dedupe in a sqlite store so nobody
is welcomed twice and no account is auto-actioned twice. is welcomed twice and no account is auto-actioned twice.
Both of those are **reactive** — they run after Mastodon has created the
account. The server also exposes a **pre-emptive** path: Mastodon can ask it
about an IP or email domain *during* signup, before consulting its own block
tables, and the answer arrives with the block already installed. See
[Reputation query API](#reputation-query-api-apireputation).
``` ```
new signup / new report on yttrx new signup / new report on yttrx
└─> Mastodon (mammut) fires admin webhook account.created|approved | report.created └─> Mastodon (mammut) fires admin webhook account.created|approved | report.created
@@ -332,6 +338,107 @@ blocks a moderator added by hand in the admin UI are not in it. Entries appear
only once the block actually landed on the Mastodon side (a failed only once the block actually landed on the Mastodon side (a failed
`POST /api/v1/admin/ip_blocks` is logged and *not* recorded as blocked). `POST /api/v1/admin/ip_blocks` is logged and *not* recorded as blocked).
## Reputation query API (`/api/reputation`)
The inversion of everything above. `POST /api/reputation` classifies an IP
and/or email domain, records it, registers the matching `Admin::IpBlock` /
`EmailDomainBlock`, and **only then returns** — so a caller that consults its
own block tables immediately afterwards finds a block that did not exist when
its request began. A bad network gets turned away on first contact instead of
cleaned up after the first account out of it.
```
signup begins on yttrx
└─> Mastodon (mammut), about to run its own IpBlock lookup
└─> POST https://hooks.yttrx.com/api/reputation (X-Reputation-Key: …)
└─> classify (ipapi.is / check-mail.org, cached)
record in reputation_query
register Admin::IpBlock / EmailDomainBlock
return
└─> IpBlock.severity_sign_up_block.containing(ip).exists? ← now true
```
Unlike the blocklist API this one **writes**, so it is authenticated and
refuses to serve without a key: an unauthenticated endpoint that installs IP
blocks is a denial-of-service lever, and there is deliberately no open mode.
With `REPUTATION_API_ENABLED` false or `REPUTATION_API_KEY` empty the path
404s as if it didn't exist.
```bash
curl -s -X POST https://hooks.yttrx.com/api/reputation \
-H "Content-Type: application/json" \
-H "X-Reputation-Key: $REPUTATION_API_KEY" \
-d '{"ip": "198.51.100.20", "email": "[email protected]", "source": "sign_up"}'
```
```json
{
"ip": {"ip": "198.51.100.20", "classification": "datacenter+abuser",
"org": "Example Cloud Inc", "flagged": true, "cidr": "198.51.100.0/24",
"blocked": true, "cached": false, "deferred": false},
"email": {"domain": "example.com", "disposable": false, "risk": 12,
"flagged": false, "blocked": false, "cached": true, "deferred": false},
"checked_at": "2026-09-13T23:11:58+00:00",
"flagged": true,
"blocked": true
}
```
Both request fields are optional (the caller's IP hook has no email address to
offer yet); at least one is required, or the call 400s. Either side may be
`null` if that signal is switched off or had nothing to look at. `blocked`
means "something is blocked on the Mastodon side as a result of this call" —
true both for a block installed just now and for one installed earlier that is
still in force.
**The caller** is the `call-welcomebot-reputation-api` patch on yttrx's
Mastodon fork (`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`).
It is best-effort on that side: if this endpoint is slow, down or disabled,
Mastodon logs a warning and proceeds exactly as it would without the patch.
### Cost control
The caller hits this when the signup form is merely *rendered*, not only on
submission, so the endpoint is reachable by anyone crawling that page.
`REPUTATION_API_MAX_LOOKUPS_PER_HOUR` (default 500) caps **fresh** provider
lookups per rolling hour; cache hits are free and never counted. Over budget,
an uncached value is answered `"classification": "unknown", "deferred": true`
and nothing is installed — never `flagged`, since the point of the ceiling is
to stop paying, and guessing would either block innocents or waste the guard.
The webhook path deliberately never spends from this budget: `account.created`
is low-volume and authoritative, and must not be starved by crawler traffic.
Measured against the live service: ~70ms for a cache hit answered locally,
~700ms from mammut (dominated by TLS setup across the WAN), ~1.3-1.8s on the
first sighting of a bad range or domain, where it also registers the block.
### Block width and expiry
Two limits apply to every block this bot registers, from the query API and the
reactive webhook path alike:
| Setting | Default | Why |
|---|---|---|
| `IP_SCRUTINY_MAX_BLOCK_PREFIX_V4` / `_V6` | `24` / `48` | ipapi.is reports the flagged address's whole ASN route. Right for hosting space, wrong for an ISP aggregate — routes of `/8`–`/11` turn up routinely, and `197.211.32.0/19` was registered against a Benin **mobile carrier** off one `abuser` flag. Anything wider falls back to the single address. |
| `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` | `86400` (24h) | An IP block claims who is behind an address *now*, and VPS addresses are recycled constantly. A day breaks a signup run — which is the shape this abuse takes — without the blocklist accumulating forever. Email domain blocks have no expiry in Mastodon and are unaffected. |
The cap applies to what gets **blocked**, not what gets **cached**: the wide
route is still the right cache key, since it genuinely belongs to one operator
and one lookup answers for the whole of it.
### Records
| Table | Holds |
|---|---|
| `reputation_query` | Audit log of every answer: what was asked, what the providers said, whether it led to a block. Domains only — never a full address. |
| `reputation_block` | One row per block installed here, keyed by the CIDR/domain (there is no account to key it to). Lets a repeat query skip the Mastodon round trip, and carries `expires_at` so the dedup record lapses exactly when the block does. |
Both feed `/api/blocks` and the reconcile pass alongside the signup-derived
rows, so neither under-reports as blocking shifts to the pre-signup path. The
published entry shape is unchanged; `signups` still counts signups, so a range
blocked before anyone signed up from it reads `"signups": 0`.
## Layout ## Layout
| Path | Purpose | | Path | Purpose |
@@ -341,7 +448,7 @@ only once the block actually landed on the Mastodon side (a failed
| `app/reconcile_blocks.py` | Scheduled pass: marks blocks that no longer exist in Mastodon inactive (cron) | | `app/reconcile_blocks.py` | Scheduled pass: marks blocks that no longer exist in Mastodon inactive (cron) |
| `Dockerfile`, `docker-compose.yml` | Container build + run | | `Dockerfile`, `docker-compose.yml` | Container build + run |
| `.env.example` | Config template (copy to `.env`, never commit) | | `.env.example` | Config template (copy to `.env`, never commit) |
| `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API) | | `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API, reputation API) |
| `bin/welcomebot-logs`, `bin/welcomebot-signups`, `bin/welcomebot-suspicious` | Ops CLIs installed to `/root/bin` on admin.yttrx.com | | `bin/welcomebot-logs`, `bin/welcomebot-signups`, `bin/welcomebot-suspicious` | Ops CLIs installed to `/root/bin` on admin.yttrx.com |
| `bin/yttrx-blocklist-import` | Importer other admins can run to pull this list into their own instance | | `bin/yttrx-blocklist-import` | Importer other admins can run to pull this list into their own instance |
| `bin/backfill-block-cidr.py` | One-shot: reconstruct `block_cidr` for rows predating it (run 2026-09-13) | | `bin/backfill-block-cidr.py` | One-shot: reconstruct `block_cidr` for rows predating it (run 2026-09-13) |