12.50. DD 50: Libeufin-Nexus¶

12.50.1. Summary¶

This document proposes a new design for the libeufin Nexus component, focusing on the user-interaction and where what state is to be stored.

12.50.2. Motivation¶

The existing Nexus design is overly complex to configure, develop and maintain. It supports EBICS features we do not need, and lacks key features (like long-polling) that are absolutely needed.

We also have several implementations with Nexus, Bank and Depolymerization subsystems, and it would be good to combine some of them.

12.50.3. Requirements¶

Easy to use, high-level abstraction over EBICS details

Reduce complexity, no multi-account, multi-connection support

No general EBICS client, Taler-specific logic

Support for Taler facade, including bouncing of transactions with malformed subject

Clear separation between configuration and runtime, minimal runtime footprint

No built-in cron-jobs, background tasks runnable via systemd (one-shot and persistent mode)

Configuration style same as other GNUnet/Taler components

No private keys in database, as in other Taler components

Enable future unified implementation with Depolymerization to share database and REST API logic

12.50.4. Proposed Solution¶

Split up Nexus into four components:

nexus-ebics-setup: register account with EBICS server and perform key generation and exchange

nexus-ebics-fetch: obtain wire transfers and payment status from EBICS server

nexus-ebics-submit: send payment initiation messages to EBICS server

nexus-httpd: serve Taler REST APIs (wire gateway API, revenue API) to Taler clients and implement facade logic

All four components should read a simple INI-style configuration file, possibly with component-specific sections.

12.50.4.1. Configuration file¶

[nexus-ebics]
CURRENCY = EUR
HOST_BASE_URL = http://ebics.bank.com/
HOST_ID = mybank
USER_ID = myuser
PARTNER_ID = myorg
IBAN = MY-IBAN
BIC = MY-BIC
NAME = MY NAME
BANK_PUBLIC_KEYS_FILE = enc-auth-keys.json
CLIENT_PRIVATE_KEYS_FILE = my-private-keys.json
BANK_DIALECT = postfinance # EBICS+ISO20022 style used by the bank.

[nexus-postgres]
CONFIG = postgres:///libeufin-nexus

[nexus-ebics-fetch]
FREQUENCY = 30s # used when long-polling is not supported
STATEMENT_LOG_DIRECTORY = /tmp/ebics-messages/

[nexus-ebics-submit]
FREQUENCY = 30s # use 0 to always submit immediately (via LISTEN trigger)

[nexus-httpd]
PORT = 8080
UNIXPATH =
SERVE = tcp | unix

[nexus-httpd-wire-gateway-facade]
ENABLED = YES
AUTH_METHOD = token
AUTH_TOKEN = "secret-token:foo"

[nexus-httpd-revenue-facade]
ENABLED = YES
AUTH_METHOD = token
AUTH_TOKEN = "secret-token:foo"

12.50.4.1.1. File contents: BANK_PUBLIC_KEYS_FILE¶

JSON with 3 fields:

bank_encryption_public_key (base32)

bank_authentication_public_key (base32)

accepted (boolean)

12.50.4.1.2. File contents: CLIENT_PRIVATE_KEYS_FILE¶

JSON with:

signature_private_key (base32)

encryption_private_key (base32)

authentication_private_key (base32)

submitted_ini (boolean)

submitted_hia (boolean)

12.50.4.2. Database schema¶

CREATE TABLE incoming_transactions
  (incoming_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY
  ,amount taler_amount NOT NULL
  ,wire_transfer_subject TEXT
  ,execution_time INT8 NOT NULL
  ,debit_payto_uri TEXT NOT NULL
  ,bank_transfer_id TEXT NOT NULL -- EBICS or Depolymerizer (generic)
  ,bounced BOOL DEFAULT FALSE -- to track if we bounced it
  );

CREATE TABLE outgoing_transactions
  (outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY
  ,amount taler_amount NOT NULL
  ,wire_transfer_subject TEXT
  ,execution_time INT8 NOT NULL
  ,credit_payto_uri TEXT NOT NULL
  ,bank_transfer_id TEXT NOT NULL
  );

CREATE TABLE initiated_outgoing_transactions
  (initiated_outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY -- used as our ID in PAIN
  ,amount taler_amount NOT NULL
  ,wire_transfer_subject TEXT
  ,execution_time INT8 NOT NULL
  ,credit_payto_uri TEXT NOT NULL
  ,out_transaction_id INT8 REFERENCES outgoing_transactions (out_transaction_id)
  ,submitted BOOL DEFAULT FALSE
  ,hidden BOOL DEFAULT FALSE -- FIXME: exaplain this.
  ,client_request_uuid TEXT NOT NULL UNIQUE
  ,failure_message TEXT -- NOTE: that may mix soon failures (those found at initiation time), or late failures (those found out along a fetch operation)
  );

COMMENT ON COLUMN initiated_outgoing_transactions.out_transaction_id
  IS 'Points to the bank transaction that was found via nexus-fetch.  If "submitted" is false or nexus-fetch could not download this initiation, this column is expected to be NULL.'

12.50.4.3. nexus-ebics-setup¶

The ebics-setup tool performs the following:

Checks if the require configuration options are present and well-formed (like the file names), if not exits with an error message. Given –check-full-config, also sanity-check the configuration options of the other subsystems.

Checks if the private keys file exists, if not creates new private keys with flags “not submitted”.

If any private key flags are set to “not submitted” or a command-line override is given (–force-keys-resubmission), attempt to submit the corresponding client public keys to the bank. If the bank accepts the client public key, update the flags to “submitted”.

If a public key was submitted or if a command-line override (–generate-registration-pdf) is given, generate a PDF with the public key for the user to print and send to the bank.

Checks if the public keys of the bank already exist on disk, if not try to download them with a flag “not accepted”. If downloading fails, display a message asking the user to register the private keys (and give override options for re-submission of private keys or re-generation of the registration PDF).

If we just downloaded public keys, display the corresponding public keys (or fingerprints) to the user and ask the user to interactively confirm to accept them (or auto-accept via –auto-accept-keys). If the user accepted the public key, update the flag to “accepted”.

12.50.4.4. nexus-ebics-fetch¶

Fetches by default all incoming and outgoing bank transactions and error messages and inserts them into the Postgres database tables (including updating the initiated outgoing transaction table). First, considers the last transactions in the database and fetches statements from that day forward (inclusive). Afterwards, fetches reports and when the day rolls over (before committing any transactions with the next day!) also fetches a final statement of the previous day, thus ensuring we get a statement every day plus intra-day reports.

Note

(1) “from that day forward (inclusive)” must not rely on EBICS returning the unseen messages: that’s because they might already be downloaded but never made it to the database.

(2) “and when the day rolls over”. When does a day roll over? => A day rolls over when the current time is at least on the day after the transaction with the most recent timestamp that’s stored in the database.

(3) “Afterwards, fetches reports”. This must happen only after any possible previous statement got downloaded.

To summarize: at any point in time the database must contain (the content of) any possible statement up to the current time, plus any possible report up to the current time (in case that’s not covered by any statement so far).

Bounces transactions with mal-formed wire transfer subjects.

Optionally logs EBICS messages to disk, one per file, based on configuration. Filenames must include the timestamp of the download. The date must be in the path and the time of day at the beginning of the filename. This will facilitate easy deletion of logs.

Optionally only fetches reports (–only-reports) or statements (–only-statements) or only error messages (–only-failures).

Optionally terminates after one fetch (–transient) or re-fetches based on the configured frequency.

Terminates hard (with error code) if incoming transactions are not in the expected (configured) currency.

12.50.4.5. nexus-ebics-submit¶

Generates a payment initiation message for all client-initiated outgoing transactions that have not yet been initiated. If the server accepts the message, sets the initiated flag in the table to true. The EBICS order ID is set to the lowest initiated_outgoing_transaction_id in the transaction set modulo 2^20 encoded in BASE36. The payment information ID is set to the initiated_outgoing_transaction_id of each transaction as a text string. The message identification is set to the lowest initiated_outgoing_transaction_id plus (“-”) the highest initiated_outgoing_transaction_id as a text string.

Optionally terminates after one fetch (–transient) or re-submits based on the configured frequency.

If configured frequency is zero (default), listens to notifications from nexus-httpd for insertions of outgoing payment initiation records.

12.50.4.6. nexus-httpd¶

Offers REST APIs as per configuration.

Listens to notifications from nexus-ebics-fetch to run facade-logic and wake-up long pollers.

Offers a new REST API to list failed (initiated outgoing) transactions and allows the user to re-initiate those transactions (by creating new records in the initiated outgoing transactions table). Also allows the user to set the “hidden” flag on failed transactions to not show them anymore.

12.50.5. Definition of Done¶

Code implemented

Testcases migrated (including exchange, merchant, etc.)

Man pages updated

Manual updated

Tested with actual banks (especially error handling and idempotency)

Tested against server-side EBICS mock (to be resurrected)

Tested against various ISO 20022 messages

12.50.6. Alternatives¶

Only run Taler on top of Bitcoin.

12.50.7. Drawbacks¶

Uses EBICS.

12.50.8. Discussion / Q&A¶

(This should be filled in with results from discussions on mailing lists / personal communication.)

From private discussion: bouncing goes inside nexus-fetch as it saves one database event, makes the HTTPd simpler, and lets the bouncing happen even when no HTTPd runs.
Sign-up PDF is ever only generated if both INI & HIA have the “submitted” state.
What is ‘override option for re-submission of private keys?’. –force-keys-submission already re-submits the keys but it does not override them. If the user wants new keys, they can easily remove the keys file on disk. That makes the CLI shorter.
Implementation sticks to the IBAN found in the configuration, if the bank does not show any IBAN related to the EBICS subscriber.
from nexus-ebics-submit: “if the server accepts the request, sets the initiated flag in the table to true”. May there be a case where the server accepted the request, but the client never got any response (some network issue..), and therefore didn’t set the submitted flag, ending up in submitting the payment twice? Also: flagging the payment _after_ the bank response, may lead double-submission even if the HTTP talk ended well: it suffices to crash after having received a “200 OK” response but before setting the submitted flag to the database.
the ebics-submit section mentions the EBICS order ID. The following excerpt was found however at page 88 of the EBICS 3 specifications:

OrderID is only present if a file is transmitted to the bank relating to an order with an already existing order number (only allowed for AdminOrderType = HVE or HVS)

Nexus does not support HVE or HVS.
As of private communication, the responsibility of submitting idempotent payments relies on the use of request_uid (a database column of the initiated payment) as the MsgId value of the corresponding pain.001 document.
submitted column of an initiated payment evolved into the following enum:
```
CREATE TYPE submission_state AS ENUM (
  'unsubmitted'
  ,'transient_failure'
  ,'permanent_failure'
  ,'success'
  ,'never_heard_back'
);
```
- unsubmitted: default state when a payment is initiated
- transient_failure: submission failed but can be retried, for example after a network issue.
- permanent_failure: EBICS- or bank-technical error codes were not EBICS_OK (nor any tolerated EBICS code like EBICS_NO_DOWNLOAD_DATA_AVAILABLE), never retry.
- never_heard_back: the payment initiation submission has been success but it was never confirmed by any outgoing transaction (from a camt.5x document) or any pain.002 report. It is responsability of a garbage collector to set this state after a particular time period.
the initiated_outgoing_transactions table takes two more columns: last_submission_date, a timestamp in microseconds, and a submission_counter. Both of them would serve to decide retry policies.
the failure_text column at the initiated_outgoing_transactions table should contain a JSON object that contains any useful detail about the problem. That could be modeled after the Taler ErrorDetail, where at least the error code and the hint fields are provided.

12.49. DD 49: Authentication

12.51. DD 51: Fractional Digits

GNU Taler