DD 43: Managing Prebuilt Artifacts and Source-Level Dependencies

14.43. DD 43: Managing Prebuilt Artifacts and Source-Level Dependencies#

Note

History:

  • 2025-11-26: Updated to reflect current state and future plans

14.43.1. Summary#

This design document defines how the GNU Taler project manages prebuilt artifacts and other source-level dependencies in repositories that are part of project.

14.43.2. Motivation#

Some repositories have source-level dependencies on the build results of other repositories. While it is possible to build these dependencies from scratch, we often want to avoid that extra step in order to make building the software easier and faster.

Examples are:

  • man-pages built via Sphinx, produced in docs.git and consumed in exchange.git.

  • SPAs used by the exchange, merchant libeufin.

  • (formerly): The taler-wallet-core-embedded.js file used by the Android repo.

Another type of source-level dependency is on other source files. Examples for this are:

  • The build-common.git repository that contains common build-time helpers

  • The gana.git repository

14.43.3. Requirements#

  • We are a free software project, users must always be able to easily re-build the prebuilt artifacts for themselves.

  • We want to reduce reliance on third party infrastructure as much as possible.

  • We want to keep our own infrastructure as slim as possible.

  • Builds must be reproducible and remain reproducible in the future.

  • Offline builds should be supported as well as possible.

  • Ideally, prebuilt artifacts can easily be cached / archived by third parties.

14.43.4. Proposed Solution#

Instead of using a full-blown artifact management solution from the start, we make use of Git. Since Git is not very good at managing binary artifacts, prebuilt files are not managed alongside the source code. Instead, we use (orphan-)branches) in a (possibly separate) repository to manage them. This allows us to re-use Git authentication and Git commit signing.

Repositories that produce artifacts MUST have phony Makefile helper targets to (a) create the prebuilt artifact (b) check out the prebuilt branch as a Git worktree, (c) move the artifact to the right place in the prebuilt branch and commit/push.

Prebuilt files are consumed via git submodules. The following rules apply for consumers:

  • The ./bootstrap script of a component must initialize the submodule.

  • The ./bootstrap script SHOULD fall back gracefully when the submodule commit of the prebuilt submodule is not available anymore.

  • The consuming module MUST rely on a particular commit of the prebuilt branch and the commit MUST be tagged to enable sparse checkouts of the submodule.

  • Consumers SHOULD use sparse checkouts for prebuilt submodules.

Layout of the prebuilt branch and prebuilt tags:

  • The prebult files for a component are under {component_name}/{version}/

  • The version MUST correspond both to the current version number of the component and a tag of the source repo of {component_name}.

  • The prebuilt branch MUST have a tag of the form prebuilt-{serial} for every version that may be consumed by another component.

  • The prebuilt branch MAY have additional tags of the form {component_name}/{version} to indicate that a particular version of a particular component was added as a prebuilt version.

14.43.5. Alternatives#

  • Only maintain one version per module.

    • We were doing this (as of 2025-11-26) and it doesn’t work very well, because the same package (say exchange.git) might depend on different versions of prebuilt packages (say aml-spa and kyc-spa). Thus the prebuilt branch would need a commit with exactly the right combination of versions, which is hard to do correctly.

    • This is okay for prebuilt branches that only contain one component, such as taler-docs.git.

  • Set up a full-blown artifact management solution like JFrog Artifactory. That’s costly and requires a lot of admin work.

  • Never rely on prebuilt files. That slows down the build process and in some cases requires exotic dependencies.

14.43.6. Discussion / Q&A#

(This should be filled in with results from discussions on mailing lists / personal communication.)