README gets the full contract (request/response shape, the cost ceiling and why it exists, the block-width cap and expiry, and the two new tables). CLAUDE.md gets the two rules that are easy to undo by accident: the endpoint must never serve unauthenticated, and the width cap and expiry are policy decisions rather than tuning knobs.
This commit is contained in:
1 parent
6952fbadf6
commit
47e92ca10f
2 files changed
+135
-1
No files matched your search
@@ -19,6 +19,18 @@ A FastAPI webhook server. One Mastodon admin webhook delivers `account.created`,
|
|||||||
and auto-**silence** young/dormant accounts past the threshold — lowered for
|
and auto-**silence** young/dormant accounts past the threshold — lowered for
|
||||||
accounts with a flagged signup IP (see README).
|
accounts with a flagged signup IP (see README).
|
||||||
|
|
||||||
|
Alongside the webhook it serves a **reputation query API** — `POST
|
||||||
|
/api/reputation` — which is the pre-emptive counterpart to all of the above.
|
||||||
|
Mastodon calls it during signup, immediately before its own IpBlock /
|
||||||
|
EmailDomainBlock lookups; it classifies the IP/email domain, records it,
|
||||||
|
registers the block, and only then returns, so the lookup that follows
|
||||||
|
rejects the signup outright. It **writes**, so it is authenticated
|
||||||
|
(`X-Reputation-Key`) and 404s unless `REPUTATION_API_ENABLED` *and*
|
||||||
|
`REPUTATION_API_KEY` are both set. The caller is the
|
||||||
|
`call-welcomebot-reputation-api` patch on the Mastodon fork — see
|
||||||
|
`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`. README.md's
|
||||||
|
"Reputation query API" section has the full contract.
|
||||||
|
|
||||||
Alongside the webhook it serves an optional read-only **blocklist API** —
|
Alongside the webhook it serves an optional read-only **blocklist API** —
|
||||||
`GET /api/blocks`, `/api/blocks/ip-ranges`, `/api/blocks/email-domains` —
|
`GET /api/blocks`, `/api/blocks/ip-ranges`, `/api/blocks/email-domains` —
|
||||||
publishing the IP ranges and email domains the bot has auto-blocked. It is
|
publishing the IP ranges and email domains the bot has auto-blocked. It is
|
||||||
@@ -62,6 +74,17 @@ user-memory note `project_welcomebot` has the current state.
|
|||||||
check-mail.org's disposable/risk fields back into the response — that would
|
check-mail.org's disposable/risk fields back into the response — that would
|
||||||
be redistributing a paid reputation feed's data. Account identifiers
|
be redistributing a paid reputation feed's data. Account identifiers
|
||||||
(handles, ids, emails, raw signup IPs) are out for the usual reasons.
|
(handles, ids, emails, raw signup IPs) are out for the usual reasons.
|
||||||
|
- **The reputation API must never serve unauthenticated.** It installs IP
|
||||||
|
blocks; an open endpoint that does that is a denial-of-service lever. An
|
||||||
|
empty `REPUTATION_API_KEY` disables the route rather than opening it, and
|
||||||
|
that behaviour is asserted in `test_local.py` — don't "fix" it into a
|
||||||
|
permissive default.
|
||||||
|
- **Don't remove the block-width cap or the block expiry.** `IP_SCRUTINY_MAX_
|
||||||
|
BLOCK_PREFIX_V4`/`_V6` (24/48) and `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` (24h)
|
||||||
|
exist because ipapi.is reports whole ASN routes: without the cap a single
|
||||||
|
`abuser` flag on a consumer address blocks its entire ISP route (routes of
|
||||||
|
/8–/11 are routine, and `197.211.32.0/19` was already registered against a
|
||||||
|
Benin mobile carrier). Raising them is a policy decision, not a tuning knob.
|
||||||
- **`admin.yttrx.com`'s `/root/yttrx-welcomebot/` is not a git repo any more**
|
- **`admin.yttrx.com`'s `/root/yttrx-welcomebot/` is not a git repo any more**
|
||||||
(cleaned up 2026-07-06). Don't `git init`/clone it back into place — the
|
(cleaned up 2026-07-06). Don't `git init`/clone it back into place — the
|
||||||
registry image + `docker-compose.yml`/`.env` is the whole deploy surface now.
|
registry image + `docker-compose.yml`/`.env` is the whole deploy surface now.
|
||||||
@@ -217,6 +240,10 @@ needed. Without these scopes the reconcile 403s and exits having changed
|
|||||||
nothing, which is the deliberate failure mode: never read an outage as "every
|
nothing, which is the deliberate failure mode: never read an outage as "every
|
||||||
block was removed".
|
block was removed".
|
||||||
|
|
||||||
|
The reputation API needs no scopes beyond those above — it registers the same
|
||||||
|
`Admin::IpBlock` / `EmailDomainBlock` records as the webhook path, so
|
||||||
|
`admin:write:ip_blocks` + `admin:write:email_domain_blocks` cover it.
|
||||||
|
|
||||||
**The suspicious-signup sweep (`app/suspicious_sweep.py`) needs another scope,
|
**The suspicious-signup sweep (`app/suspicious_sweep.py`) needs another scope,
|
||||||
`admin:read:accounts`**, to `GET /api/v1/admin/accounts/:id` (fetches current
|
`admin:read:accounts`**, to `GET /api/v1/admin/accounts/:id` (fetches current
|
||||||
post/follow counts + role/suspended state to decide whether to suspend) —
|
post/follow counts + role/suspended state to decide whether to suspend) —
|
||||||
|
|||||||
@@ -21,6 +21,12 @@ A small FastAPI webhook server that receives Mastodon **admin webhooks** for
|
|||||||
- Signatures are HMAC-verified; both flows dedupe in a sqlite store so nobody
|
- Signatures are HMAC-verified; both flows dedupe in a sqlite store so nobody
|
||||||
is welcomed twice and no account is auto-actioned twice.
|
is welcomed twice and no account is auto-actioned twice.
|
||||||
|
|
||||||
|
Both of those are **reactive** — they run after Mastodon has created the
|
||||||
|
account. The server also exposes a **pre-emptive** path: Mastodon can ask it
|
||||||
|
about an IP or email domain *during* signup, before consulting its own block
|
||||||
|
tables, and the answer arrives with the block already installed. See
|
||||||
|
[Reputation query API](#reputation-query-api-apireputation).
|
||||||
|
|
||||||
```
|
```
|
||||||
new signup / new report on yttrx
|
new signup / new report on yttrx
|
||||||
└─> Mastodon (mammut) fires admin webhook account.created|approved | report.created
|
└─> Mastodon (mammut) fires admin webhook account.created|approved | report.created
|
||||||
@@ -332,6 +338,107 @@ blocks a moderator added by hand in the admin UI are not in it. Entries appear
|
|||||||
only once the block actually landed on the Mastodon side (a failed
|
only once the block actually landed on the Mastodon side (a failed
|
||||||
`POST /api/v1/admin/ip_blocks` is logged and *not* recorded as blocked).
|
`POST /api/v1/admin/ip_blocks` is logged and *not* recorded as blocked).
|
||||||
|
|
||||||
|
## Reputation query API (`/api/reputation`)
|
||||||
|
|
||||||
|
The inversion of everything above. `POST /api/reputation` classifies an IP
|
||||||
|
and/or email domain, records it, registers the matching `Admin::IpBlock` /
|
||||||
|
`EmailDomainBlock`, and **only then returns** — so a caller that consults its
|
||||||
|
own block tables immediately afterwards finds a block that did not exist when
|
||||||
|
its request began. A bad network gets turned away on first contact instead of
|
||||||
|
cleaned up after the first account out of it.
|
||||||
|
|
||||||
|
```
|
||||||
|
signup begins on yttrx
|
||||||
|
└─> Mastodon (mammut), about to run its own IpBlock lookup
|
||||||
|
└─> POST https://hooks.yttrx.com/api/reputation (X-Reputation-Key: …)
|
||||||
|
└─> classify (ipapi.is / check-mail.org, cached)
|
||||||
|
record in reputation_query
|
||||||
|
register Admin::IpBlock / EmailDomainBlock
|
||||||
|
return
|
||||||
|
└─> IpBlock.severity_sign_up_block.containing(ip).exists? ← now true
|
||||||
|
```
|
||||||
|
|
||||||
|
Unlike the blocklist API this one **writes**, so it is authenticated and
|
||||||
|
refuses to serve without a key: an unauthenticated endpoint that installs IP
|
||||||
|
blocks is a denial-of-service lever, and there is deliberately no open mode.
|
||||||
|
With `REPUTATION_API_ENABLED` false or `REPUTATION_API_KEY` empty the path
|
||||||
|
404s as if it didn't exist.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST https://hooks.yttrx.com/api/reputation \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "X-Reputation-Key: $REPUTATION_API_KEY" \
|
||||||
|
-d '{"ip": "198.51.100.20", "email": "[email protected]", "source": "sign_up"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"ip": {"ip": "198.51.100.20", "classification": "datacenter+abuser",
|
||||||
|
"org": "Example Cloud Inc", "flagged": true, "cidr": "198.51.100.0/24",
|
||||||
|
"blocked": true, "cached": false, "deferred": false},
|
||||||
|
"email": {"domain": "example.com", "disposable": false, "risk": 12,
|
||||||
|
"flagged": false, "blocked": false, "cached": true, "deferred": false},
|
||||||
|
"checked_at": "2026-09-13T23:11:58+00:00",
|
||||||
|
"flagged": true,
|
||||||
|
"blocked": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Both request fields are optional (the caller's IP hook has no email address to
|
||||||
|
offer yet); at least one is required, or the call 400s. Either side may be
|
||||||
|
`null` if that signal is switched off or had nothing to look at. `blocked`
|
||||||
|
means "something is blocked on the Mastodon side as a result of this call" —
|
||||||
|
true both for a block installed just now and for one installed earlier that is
|
||||||
|
still in force.
|
||||||
|
|
||||||
|
**The caller** is the `call-welcomebot-reputation-api` patch on yttrx's
|
||||||
|
Mastodon fork (`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`).
|
||||||
|
It is best-effort on that side: if this endpoint is slow, down or disabled,
|
||||||
|
Mastodon logs a warning and proceeds exactly as it would without the patch.
|
||||||
|
|
||||||
|
### Cost control
|
||||||
|
|
||||||
|
The caller hits this when the signup form is merely *rendered*, not only on
|
||||||
|
submission, so the endpoint is reachable by anyone crawling that page.
|
||||||
|
`REPUTATION_API_MAX_LOOKUPS_PER_HOUR` (default 500) caps **fresh** provider
|
||||||
|
lookups per rolling hour; cache hits are free and never counted. Over budget,
|
||||||
|
an uncached value is answered `"classification": "unknown", "deferred": true`
|
||||||
|
and nothing is installed — never `flagged`, since the point of the ceiling is
|
||||||
|
to stop paying, and guessing would either block innocents or waste the guard.
|
||||||
|
|
||||||
|
The webhook path deliberately never spends from this budget: `account.created`
|
||||||
|
is low-volume and authoritative, and must not be starved by crawler traffic.
|
||||||
|
|
||||||
|
Measured against the live service: ~70ms for a cache hit answered locally,
|
||||||
|
~700ms from mammut (dominated by TLS setup across the WAN), ~1.3-1.8s on the
|
||||||
|
first sighting of a bad range or domain, where it also registers the block.
|
||||||
|
|
||||||
|
### Block width and expiry
|
||||||
|
|
||||||
|
Two limits apply to every block this bot registers, from the query API and the
|
||||||
|
reactive webhook path alike:
|
||||||
|
|
||||||
|
| Setting | Default | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| `IP_SCRUTINY_MAX_BLOCK_PREFIX_V4` / `_V6` | `24` / `48` | ipapi.is reports the flagged address's whole ASN route. Right for hosting space, wrong for an ISP aggregate — routes of `/8`–`/11` turn up routinely, and `197.211.32.0/19` was registered against a Benin **mobile carrier** off one `abuser` flag. Anything wider falls back to the single address. |
|
||||||
|
| `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` | `86400` (24h) | An IP block claims who is behind an address *now*, and VPS addresses are recycled constantly. A day breaks a signup run — which is the shape this abuse takes — without the blocklist accumulating forever. Email domain blocks have no expiry in Mastodon and are unaffected. |
|
||||||
|
|
||||||
|
The cap applies to what gets **blocked**, not what gets **cached**: the wide
|
||||||
|
route is still the right cache key, since it genuinely belongs to one operator
|
||||||
|
and one lookup answers for the whole of it.
|
||||||
|
|
||||||
|
### Records
|
||||||
|
|
||||||
|
| Table | Holds |
|
||||||
|
|---|---|
|
||||||
|
| `reputation_query` | Audit log of every answer: what was asked, what the providers said, whether it led to a block. Domains only — never a full address. |
|
||||||
|
| `reputation_block` | One row per block installed here, keyed by the CIDR/domain (there is no account to key it to). Lets a repeat query skip the Mastodon round trip, and carries `expires_at` so the dedup record lapses exactly when the block does. |
|
||||||
|
|
||||||
|
Both feed `/api/blocks` and the reconcile pass alongside the signup-derived
|
||||||
|
rows, so neither under-reports as blocking shifts to the pre-signup path. The
|
||||||
|
published entry shape is unchanged; `signups` still counts signups, so a range
|
||||||
|
blocked before anyone signed up from it reads `"signups": 0`.
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
| Path | Purpose |
|
| Path | Purpose |
|
||||||
@@ -341,7 +448,7 @@ only once the block actually landed on the Mastodon side (a failed
|
|||||||
| `app/reconcile_blocks.py` | Scheduled pass: marks blocks that no longer exist in Mastodon inactive (cron) |
|
| `app/reconcile_blocks.py` | Scheduled pass: marks blocks that no longer exist in Mastodon inactive (cron) |
|
||||||
| `Dockerfile`, `docker-compose.yml` | Container build + run |
|
| `Dockerfile`, `docker-compose.yml` | Container build + run |
|
||||||
| `.env.example` | Config template (copy to `.env`, never commit) |
|
| `.env.example` | Config template (copy to `.env`, never commit) |
|
||||||
| `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API) |
|
| `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API, reputation API) |
|
||||||
| `bin/welcomebot-logs`, `bin/welcomebot-signups`, `bin/welcomebot-suspicious` | Ops CLIs installed to `/root/bin` on admin.yttrx.com |
|
| `bin/welcomebot-logs`, `bin/welcomebot-signups`, `bin/welcomebot-suspicious` | Ops CLIs installed to `/root/bin` on admin.yttrx.com |
|
||||||
| `bin/yttrx-blocklist-import` | Importer other admins can run to pull this list into their own instance |
|
| `bin/yttrx-blocklist-import` | Importer other admins can run to pull this list into their own instance |
|
||||||
| `bin/backfill-block-cidr.py` | One-shot: reconstruct `block_cidr` for rows predating it (run 2026-09-13) |
|
| `bin/backfill-block-cidr.py` | One-shot: reconstruct `block_cidr` for rows predating it (run 2026-09-13) |
|
||||||
|
|||||||
Reference in new issue
Block a user