12.43. DD 43: Managing Prebuilt Artifacts and Source-Level Dependencies

12.43.1. Summary

This design document defines how the GNU Taler project manages prebuilt artifacts and other source-level dependencies in repositories that are part of project.

12.43.2. Motivation

Some repositories have source-level dependencies on the build results of other repositories. While it is possible to build these dependencies from scratch, we often want to avoid that extra step in order to make building the software easier and faster.

Examples are:

  • man-pages built via Sphinx, produced in docs.git and consumed in exchange.git.
  • SPAs used by the exchange, merchant libeufin.
  • (formerly): The taler-wallet-core-embedded.js file used by the Android repo.

Another type of source-level dependency is on other source files. Examples for this are:

  • The build-common.git repository that contains common build-time helpers
  • The gana.git repository

12.43.3. Requirements

  • We are a free software project, users must always be able to easily re-build the prebuilt artifacts for themselves.
  • We want to reduce reliance on third party infrastructure as much as possible.
  • We want to keep our own infrastructure as slim as possible.
  • Builds must be reproducible and remain reproducible in the future.
  • Offline builds should be supported as well as possible.
  • Ideally, prebuilt artifacts can easily be cached / archived by third parties.

12.43.4. Proposed Solution

Instead of using a full-blown artifact management solution from the start, we make use of Git. Since Git is not very good at managing binary artifacts, prebuilt files are not managed alongside the source code. Instead, we use (orphan-)branches) in a (possibly separate) repository to manage them. This allows us to re-use Git authentication and Git commit signing.

Due to a historical accident, prebuilt files are currently stored in the prebuilt branch of the wallet-core.git repository. This might change in the future.

To enable forwards-compatibility, we are implementing the following approach: The taler.net HTTP server redirects requests in the form of https://artifacts.taler.net/$component/$version/$file to https://git.taler.net/$repo.git/tree/$version/$file?h=$branch, where $component determines $repo and $branch.

We are also introducing the new rule that a prebuilt artifact must be a single archive file (preferable .tar.gz.).

Repositories that produce artifacts MUST have phony Makefile helper targets to (a) create the prebuilt artifact (b) check out the prebuilt branch as a Git worktree, (c) move the artifact to the right place in the prebuilt branch and commit/push.

Repositories that depend on prebuilt components must download them in their ./bootstrap script. After downloading, the bootstrap script SHOULD verify a checksum of the downloaded artifact.

FIXME: Link to an example bootstrap file, once the approach from this document is fully implemented.

12.43.5. Alternatives

  • The prebuilt branch could be integrated into source repositories that depend on it via a git submodule. This has the following disadvantages:
    1. Git submodules are difficult to handle for some developers. We’ve had issues in the past where the prebuilt submodule wasn’t integrated properly and would lead to non-reproducible builds
    2. Relying purely on the Git commit hash of the prebuilt branch makes it very difficult to know which version of the prebuilt artifact is being used. It also makes it very difficult to reliable find older versions of the artifact.
    3. Relying on the commit hash instead of a version and artifact hash couples the artifact to the Git commit. That means that we can never compact the commit history of the prebuilt branch of move artifacts around.
  • Set up a full-blown artifact management solution like JFrog Artifactory. That’s costly and requires a lot of admin work.
  • Never rely on prebuilt files. That slows down the build process and in some cases requires exotic dependencies.

12.43.6. Drawbacks

12.43.7. Discussion / Q&A

(This should be filled in with results from discussions on mailing lists / personal communication.)