11.34. DD 034: Considerations for Wallet Database Migrations

11.34.1. Summary

This design document discusses considerations for wallet database migrations.

11.34.2. Motivation

The schema of the GNU Taler Wallet database is evolving over time, either because new features are added or because bugs are fixed.

11.34.3. Requirements

  • Migrations may not result in data loss and must be automatic.
  • Our schema migration must be compatible with how IndexedDB works. This means that we can’t do arbitrary schema migrations at any time, but need to increment the IndexedDB database version every time we add/remove/change an object store or index.

11.34.4. Proposed Solution

The schema of the wallet database is described in code in https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/db.ts#n1959 (walletStoresV1). This schema description is used to initialize and upgrade the database automatically.

In IndexedDB terminology, the wallet has two databases:

  1. The "taler-wallet-meta" stores metadata about the current major-version database
  2. The major-version database (currently "taler-wallet-main-v9" stores the actual data of the wallet.

This indirection allows major database migrations to be safe despite the limitations of IndexedDB. The computation that is allowed during an IndexedDB migration is very limited. By migrating to a completely new database, we can keep around the old database until we’re sure that the migration has succeeded and, if required, push new code to fix migration errors.

We have three different mechanisms to introduce changes to the database:

  1. Major migrations. These migrations introduce a new major-version database and must manually migrate the data from the previons major-version database. This migration should be added in db.ts#openTalerDatabase. Major migrations should be used very seldomly. It can make sense to implement them as a backup cycle, i.e. implement a backup export from the old version, upgrade to the latest backup version and then re-import into the new major-version database.
  2. Minor schema migrations: These migrations add or remove object stores or indexes. They are done by adding new elements to the schema descriptions and specifying the versionAdded attribute. This causes an IndexedDB upgrade transaction to be executed.
  3. Fixups. Fixups change data within or between minor schema versions and contain arbitrary code to make changes to object stores. They are usually used when a new mandatory field is added to an existing object store or some data format changes. Fixups are also useful to retroactively fix bugs introduced by previously deployed wallet versions. They must be added to db.ts#walletDbFixups

11.34.5. Alternatives

  • Per-object versioning instead of using IndexedDB minor versions
  • Always use the backup mechanism to upgrade the database
    • Would be overkill for minor migrations

11.34.6. Drawbacks

N/A.

11.34.7. Discussion / Q&A

(This should be filled in with results from discussions on mailing lists / personal communication.)