DD 34: Considerations for Wallet Database Migrations

23.35. DD 34: Considerations for Wallet Database Migrations#

Design status:

Accepted

Implementation status:

Implemented

DD shepherd:

TBD

Historical contributors:

Florian Dold

First published:

2023-01-27

Last substantive change:

2023-01-27

Implementation evidence:

taler-typescript-core (2023-07-11, 2026-07-19, 2026-08-06)

Normative references:

Wallet Developer Manual

Note

The wallet now has both IndexedDB and native SQLite database backends. The The wallet developer manual describes the migration contract; current wallet source is authoritative for backend-specific implementation details.

23.35.1. Summary#

This design document discusses considerations for wallet database migrations.

23.35.2. Motivation#

The schema of the GNU Taler Wallet database is evolving over time, either because new features are added or because bugs are fixed.

23.35.3. Requirements#

  • Migrations may not result in data loss and must be automatic.

  • Migrations of the IndexedDB backend must be compatible with how IndexedDB works. This means that we cannot do arbitrary schema migrations at any time, but need to increment the IndexedDB database version every time we add, remove or change an object store or index. The native SQLite backend has a separate versioned migration mechanism.

23.35.4. Proposed Solution#

The IndexedDB schema of the wallet database is described in https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/db/indexeddb/schema.ts. The native SQLite schema is described separately in https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/db/sqlite/schema.ts. These schema descriptions are used to initialize and upgrade their respective databases automatically.

In IndexedDB terminology, the wallet has two databases:

  1. The "taler-wallet-meta" stores metadata about the current major-version database

  2. The major-version database (currently "taler-wallet-main-v10") stores the actual data of the wallet.

This indirection allows major database migrations to be safe despite the limitations of IndexedDB. The computation that is allowed during an IndexedDB migration is very limited. By migrating to a completely new database, we can keep around the old database until we’re sure that the migration has succeeded and, if required, push new code to fix migration errors.

The IndexedDB backend has three mechanisms to introduce changes:

  1. Major migrations. These migrations introduce a new major-version database and must manually migrate the data from the previous major-version database. This migration should be added in db/indexeddb/database.ts#openTalerDatabase. Major migrations should be used very seldomly. It can make sense to implement them as a backup cycle, i.e. implement a backup export from the old version, upgrade to the latest backup version and then re-import into the new major-version database.

  2. Minor schema migrations: These migrations add or remove object stores or indexes. They are done by adding new elements to the schema descriptions and specifying the versionAdded attribute. This causes an IndexedDB upgrade transaction to be executed.

  3. Fixups. Fixups change data within or between minor schema versions and contain arbitrary code to make changes to object stores. They are usually used when a new mandatory field is added to an existing object store or some data format changes. Fixups are also useful to retroactively fix bugs introduced by previously deployed wallet versions. They must be added to db/indexeddb/fixups.ts#walletDbFixups.

The native SQLite backend uses ordered schema migrations in db/sqlite/schema.ts. Migration from the IndexedDB emulation to the native schema is implemented separately in db/migration/native.ts.

23.35.5. Alternatives#

  • Per-object versioning instead of using IndexedDB minor versions

  • Always use the backup mechanism to upgrade the database

    • Would be overkill for minor migrations

23.35.6. Drawbacks#

N/A.

23.35.7. Discussion / Q&A#

(This should be filled in with results from discussions on mailing lists / personal communication.)