DD 76: Paivana - Fighting AI Bots with GNU Taler

23.77. DD 76: Paivana - Fighting AI Bots with GNU Taler#

Design status:

Accepted

Implementation status:

Implemented

DD shepherd:

TBD

Historical contributors:

Florian Dold, Christian Grothoff

First published:

2025-11-26

Last substantive change:

2026-08-07

Implementation evidence:

merchant (2026-01-20; 2026-04-25; 2026-04-27; 2026-08-04); paivana (2026-04-19)

Normative references:

taler-paivana-manual.rst, frags/paivana-httpd-manual.rst, and core/api-merchant.rst

23.77.1. Summary#

This design document describes the architecture of an AI Web firewall using GNU Taler, as well as new features that are required for the implementation.

23.77.2. Motivation#

AI bots are causing enormous amounts of traffic by scraping sites like git forges. They neither respect robots.txt nor 5xx HTTP responses. Solutions like Anubis and IP-based blocking do not work anymore at this point.

23.77.3. Requirements#

  • Must withstand high traffic from bots, requests before a payment happened must be very cheap, both in terms of response generation and database interaction. This includes good support for caching.

  • Should work not just for our paivana-httpd but also for Turnstile-style paywalls that need to work with purely static paywall pages without PHP sessions.

23.77.4. Proposed Solution#

23.77.4.1. Architecture#

  • paivana-httpd is a reverse proxy that sits between ingress HTTP(S) traffic and the protected upstream service.

  • paivana-httpd is configured with a particular merchant backend.

  • A payment template must be set up in the merchant backend (called {template_id} from here on).

Steps:

  • Browser visits {website} (for example, https://git.taler.net) where {domain} is the domain name of {website}.

  • paivana-httpd working as a reverse-proxy for {website}. Whenever called for a non-whitelisted URL, it checks for a the presence of a Paivana cookie valid for this client IP address and {website} at this time. The Paivana Cookie is computed as:

    expiration || '-' || crock32(HKDF(salt=expiration, ikm=paivana_server_secret, info=website || '\0' || client_ip)).

    where expiration in the prefix is the expiration time for the cookie (and thus the access to the article) in seconds (to keep it short) while in the salt it is the binary GNUnet absolute time (microseconds) in network byte order. Note that this value is the end of the access being sold, chosen by the client and capped by the contract; it is not a statement about when anything happened, and in particular it is not the client’s idea of the current time. (It was called cur_time in earlier drafts of this document, which invited exactly that misreading.) HKDF is GNUnet’s HKDF (GNUNET_CRYPTO_hkdf_gnunet(), which extracts with HMAC-SHA-512 and expands with HMAC-SHA-256), and the output is 512 bits. Using a keyed PRF instead of a plain hash over the concatenation ensures that the cookie cannot be forged without the server secret and that the inputs are unambiguously separated: website is terminated by a zero byte before client_ip is appended, so different (website, client_ip) pairs can never yield the same info string. crock32 is GNUnet’s Crockford-inspired base32 encoding.

    The cookie is computed and verified exclusively by paivana-httpd; the browser only stores and returns it and thus never has to reconstruct this value.

    • If such a cookie is set and valid, the request is reverse-proxied to upstream. Stop.

    • Otherwise, an HTTP 303 See Other to /.well-known/paivana/templates/$ID#$WEBSITE is returned. Here, $ID is the template ID and $WEBSITE is base64url-encoding of the full URL of the website currently being visited. This way, the template page can be fully static and cached, and the JavaScript logic on that page can learn which website to pay for (and after payment redirect the browser there).

  • When the browser requests /.well-known/paivana/templates/$ID

    a static cachable paywall page is returned, including a machine-readable Paivana HTTP header with the taler://pay-template/ URL minus the client-computed {paivana_id} and fullfillment URL (see below).

  • The browser (rendering the paywall page) generates a random paivana ID via JS using the end of the access it intends to buy (expiration) in seconds since the Epoch and the current URL ({website}) plus some freshly generated entropy ({nonce}):

    paivana_id := expiration || '-' || b64url(SHA256(nonce || website || '\0' || expiration)).

    The exact byte string that is hashed is the concatenation of:

    • the 16-byte (128-bit) binary nonce;

    • the UTF-8 encoding of website, including its terminating zero byte (which separates it unambiguously from the timestamp);

    • expiration as an 8-byte big-endian (network byte order) number of microseconds since the Epoch, that is, the value of the seconds-based expiration multiplied by 1000000.

    Note that expiration thus appears twice in two different encodings: the paivana_id prefix is the timestamp in seconds (as decimal ASCII, to keep the identifier short), while the hashed value is the same instant in microseconds in network byte order.

    The client is free to pick this value — it is asking for access until a particular moment, and it is the contract’s max_pickup_time that decides whether it may have it. Since the same value goes into the session ID the order is created under, it cannot be revised after the fact.

    Here b64url is the RFC 7515 base64 URL encoder without padding, used to keep the result short (same reason for the use of SHA-256). The same computation could also easily be done by a non-JS client that processes the Paivana HTTP header (or a GNU Taler wallet running as a Web extension).

  • Based on this paivana ID, a taler://pay-template/{merchant_backend}/{template_id}?session_id={paivana_id}&fulfillment_url={website} URI is generated and rendered as a QR code and link, prompting the user to pay for access to the {website} using GNU Taler.

  • The JavaScript in the paywall page running in the browser (or the non-JS client) long-polls on a new https://{merchant_backend}/sessions/{paivana_id} endpoint that returns when an order with the given session ID has been paid for (regardless of the order ID, which is not known to the browser).

  • A wallet now needs to instantiate the pay template, passing the session_id and the fulfillment_url as an additional inputs to the order creation (the session ID here will work just like existing use of session_ids in session-bound payments). Similarly, the {website} works as the fulfillment URL as usual.

  • The wallet then must pay for the resulting order by talking to the Merchant backend.

  • When the long-poller returns and the payment has succeeded, the browser (still rendering the paywall page) also learns the order ID.

  • The JavaScript of the paywall page (or the non-JS client processing the Paivana HTTP header) then POSTs the order ID, nonce, expiration and website to {domain}/.well-known/paivana. In this JSON request, the nonce is crock32-encoded and expiration is a normal GNU Taler timestamp object ({"t_s": ...}, in seconds); the server re-derives the binary inputs given above from these values.

    Note that by the time this POST is made, the client already has the merchant backend’s word that the order was paid: that is precisely what its long poll on /sessions/{paivana_id} returned, and it is where the order ID being posted came from. The step below is therefore a confirmation of something the client has been told, and not an open-ended wait for a payment that may still be in progress.

  • paivana-httpd re-computes the paivana ID from nonce, website and expiration, and asks the merchant backend, over its own authenticated connection, whether the posted order ID was paid under exactly that session ID. Recomputing rather than accepting the ID is what binds the answer to this request: a client cannot post an order it paid for one article and be let into another, because a different website yields a different paivana ID and the order is then not found under it.

    The reply is accepted only if all of the following hold:

    • the order status is paid, and the order has neither been refunded nor has a refund pending — otherwise a client could take its money back and keep the cookie;

    • the contract’s fulfillment_url, if it has one, equals the posted website; if it has none, the website must lie under paivana-httpd’s own configured base URL, so that the client cannot choose which site it is admitted to;

    • expiration is not later than the contract’s max_pickup_time, which is what stops a client from buying five minutes of access and minting itself a cookie valid for a year.

    If so, paivana-httpd issues the Paivana cookie described above, with Max-Age derived from expiration, and redirects to the {website}.

    This query is made as a long poll with a short, fixed bound (5 seconds in the current implementation). Both halves matter:

    • Long poll, because the client’s confirmation and the backend’s own view of the order can be a moment apart, and because paivana-httpd and the client may be talking to different backend processes. An honest client that is merely early is waited for rather than turned away, which is the difference between a working paywall and one that intermittently refuses people who have paid.

    • Short and bounded, because this endpoint is unauthenticated and reachable before any payment has been shown to exist. The wait is the interval for which an attacker can pin a connection by posting a random order ID, so it is a cost that is deliberately kept small. The same bound is applied client-side, so a merchant backend that stops answering cannot pin connections either.

    Where the order genuinely was not paid, the client is told so (HTTP 409 Conflict) after that bound has elapsed; where the backend did not answer at all, it gets 504 Gateway Timeout, and where the backend answered something unusable, 502 Bad Gateway. Distinguishing these matters operationally: only the first is the client’s fault.

  • The browser reloads the page with the correct Paivana cookie (see first step).

23.77.4.2. Problems:#

  • A smart attacker might still create a lot of orders via the pay-template.

    • Solution A: Don’t care, unlikely to happen in the first place.

    • Solution B: Rate-limit template instantiation on a per-IP basis.

23.77.4.3. Accepted risks:#

Four properties of this design were examined and deliberately kept as they are. Each is recorded here together with the assumption that makes it acceptable, because a deployment that does not satisfy the assumption does not get the property.

23.77.4.3.1. Only one price per URL#

paivana-httpd quotes a price by taking the first configured template whose anchored website_regex matches the requested URL; a template configured without a regex matches every URL. At redemption, the checks listed above are all that can be made: the contract carries a fulfillment URL and a max_pickup_time, and it does not carry the identity of the template it was instantiated from. paivana-httpd therefore cannot tell an order created under one template from an order created under another, and in particular cannot check that the amount paid is the amount its own template search would have quoted for the URL being unlocked.

Where an instance carries more than one paivana template — or one template without a website_regex, which matches everything — this is exploitable in the obvious way. A client that wants an expensive URL instantiates the cheap template with that URL as its fulfillment URL, pays the cheap price, and posts the result for redemption; both orders name the same fulfillment URL, which is all the redemption check inspects. The exposure is wider than the configured regular expressions suggest, because the merchant backend matches website_regex unanchored where paivana-httpd anchors it: the set of URLs the backend will sell a template for is a superset of the set paivana-httpd paywalls with it.

The mitigation is a property of the configuration rather than of the code. A merchant instance used by paivana-httpd carries exactly one paivana template, so that every URL it paywalls has exactly one price and there is nothing to substitute; differentiated pricing across a site is then a matter of separate instances, each with its own template and its own paivana-httpd. The risk is accepted on the assumption that a deployment presents a single price for every URL it paywalls. A deployment that puts two paivana templates on one instance is selling its expensive articles at the cheaper price.

Closing the gap properly requires the merchant backend to record the instantiating template_id in the contract terms and to report it with the order status; paivana-httpd could then re-run its own template search for the posted website and require the two to agree. That is a merchant-side change in a separate upstream, and this document does not assume it.

23.77.4.3.2. Payment buys access, not a seat#

The redemption POST is idempotent and unmetered, and nothing records that an order or a paivana ID has already been redeemed. The cookie it returns is bound to the address of whoever posted the redemption, not to the address that paid. Anyone holding the four posted fields — order ID, nonce, expiration and website — can therefore obtain their own cookie, for their own address, at any point until expiration, and a buyer who publishes those four fields has given the article to everyone who reads them.

This is intended behaviour and not a defect. What is sold is access to one resource until one moment, and the buyer may pass that on, in the same way and for the same reasons that the buyer of a newspaper may hand it to the next reader. The purchase stays bounded by what was bought: sharing extends a payment to more readers, never to more URLs and never past expiration, so a client that wants the whole site still pays for the whole site. That bound is what lets the design’s actual goal — making bulk automated retrieval expensive — survive the sharing. The risk is accepted on the assumption that deployments price access per resource and per unit of time, and that none of them requires per-seat licensing, which this design cannot provide and must not be configured as though it could.

One consequence has to be stated plainly, because the construction of the cookie invites the opposite reading: binding the cookie to the client address is a cookie-theft mitigation and nothing else. It ensures that a cookie which leaks — from a log, a shared machine, a proxy — is useless to whoever picks it up. It provides no anti-sharing property whatsoever, since the redemption that mints cookies is open to every address.

The redemption endpoint is thus unmetered by intent. Metering it would not restore any property this design claims; the rate-limiting question raised above for template instantiation is a question about load, it applies to this endpoint in the same form, and it is open in the same way.

23.77.4.3.3. Entropy of the server secret#

paivana_server_secret is derived from the configured secret by a single unsalted SHA-512. There is no stretching and no salt, so the cost of guessing that secret offline from one observed cookie is one hash per candidate: the strength of every cookie the deployment will ever issue is the entropy of the configured string, and nothing more.

The requirement that follows is placed on the operator. The configured secret must carry at least 128 bits drawn from a cryptographic random source, and must never be a passphrase, a hostname, a token reused from elsewhere, or the placeholder that ships in the sample configuration — which is an example rather than a secret, and leaves a deployment that keeps it with no secret at all. Where no secret is configured, paivana-httpd uses a fresh random value per process, which is safe but invalidates every outstanding cookie whenever the service restarts.

Absorbing the requirement into the construction was considered and rejected. A memory-hard KDF exists to make human-chosen, low-entropy secrets expensive to guess; it buys a fixed factor, no fixed factor rescues a guessable phrase, and against 128 genuine bits it buys nothing that is needed. paivana_server_secret is a machine-generated configuration value that nobody has to remember or type, so the situation a KDF defends against is one the deployment can simply not be in. The risk is accepted on the assumption that the secret is produced by a random generator and never chosen by a person; where a person chooses it, the cookies are forgeable and the paywall is decorative. How to generate such a secret is operator guidance and belongs with the manual rather than here.

23.77.4.3.4. Truncation is unreportable to an HTTP/1.0 client#

paivana-httpd relays bodies as they arrive rather than assembling them first, which is what lets it serve content larger than memory and lets the client start receiving before the upstream has finished. The cost is that the upstream’s status line and headers reach the client long before the body is complete, so an upstream that fails mid-body cannot be reported as 502 — that status has already been spent.

What is left is to break the framing, which for almost every client is enough: a declared Content-Length is left unmet, or a chunked response is closed without its terminating chunk, and RFC 9112 section 8.1.2 requires a recipient to treat either as a failed message. The exception is an HTTP/1.0 client receiving a response whose length the upstream never declared. Such a client cannot be sent chunks, so the close of the connection is the end-of-body marker, and a truncated body is byte-for-byte indistinguishable from a complete one.

The alternative would be to buffer each response until it is known to be complete, which is exactly the property being given up, and which bounds every response by memory to buy correct reporting for one obsolete client version. The risk is accepted on the assumption that clients speak HTTP/1.1, which every browser and every HTTP library in current use has done since well before this design; where a genuine HTTP/1.0 client is expected, the upstream should be configured to declare a Content-Length, which restores detection for it too.

23.77.4.4. Implementation#

  • [x] Merchant backend can look up order IDs under a Paivana session ID.

  • [x] Merchant backend can instantiate Paivana templates with paivana_id and the target website.

  • [x] Paivana component implemented.

  • [x] Wallet/Web utility support implemented.

23.77.5. Test Plan#

  • Deploy it for git.taler.net

23.77.6. Definition of Done#

  • [x] Merchant, Paivana, and wallet-side protocol support implemented.

  • [x] Protocol and operator documentation published.

  • [ ] Production deployment and end-to-end QC recorded.

23.77.7. Alternatives#

  • Do not re-use the session ID mechanism but introduce some new concept. This has the drawback of us needing additional tables and indicies, and also the existing use of the session ID is very parallel to this one.

  • Instead of doing a 303 See Other, cache control could have been achieved by specifying a “Vary: Cookie” HTTP header. We may combine these and use that to additionally enable caching of the 303 See Other. The 303 solution has the advantage that there is only one page to cache per template, and the disadvantage of an additional redirect. Note that this is purely a frontend design choice, wallets and merchant backends work nicely with either approach.

23.77.8. Drawbacks#

  • This exposes an order ID to anyone who knows the session ID. This is clearly not an issue in this context, and for the existing uses of the session ID it also seems clear that knowledge of the session ID requires an attacker to have access that would easily also already give them any order ID, so this seems harmless.

23.77.9. Discussion / Q&A#