#3601 new enhancement

CI builds new images for test jobs even when Tahoe's dependencies haven't changed

Reported by: exarkun Owned by:
Priority: normal Milestone: undecided
Component: dev-infrastructure Version: n/a
Keywords: self-contained ci docker Cc:
Launchpad Bug:

Description

Currently on CircleCI we use pre-built Docker images populated with our dependencies in order to make build results more easily reproducible and to speed up individual test jobs.

The Docker images are built and replaced nightly:

https://github.com/tahoe-lafs/tahoe-lafs/blob/94716c078a5c9cbaf468b88206d911146ba6091e/.circleci/config.yml#L97-L129

There are some downsides to replacing the images this way:

  • Often Tahoe's dependencies have not changed. The introduction of a new image only allows for outside factors to lead to changes in behavior. This reduces reproducibility since it means the same Tahoe source tested on two different days might encounter two different environments and end up with two different results.
  • It's wasteful to build images when nothing has changed. It's wasteful of CircleCI's resources (which we currently consume for free). It's wasteful of our time since re-built images are less likely to be cached on build hosts and need to be downloaded which takes time.

For a set of "stable" CI jobs which should only fail if Tahoe introduces a regression, not if the outside world changes in a way that breaks Tahoe, the ideal would be to only rebuild the images when Tahoe's dependencies change, only allow the new image to become the canonical image if the Tahoe works on the image, and only allow Tahoe dependencies to be resolved during image creation, not during test execution.

Change History (3)

comment:1 Changed at 2021-01-27T14:07:26Z by exarkun

  • Keywords self-contained ci added

comment:2 Changed at 2023-02-13T14:27:37Z by exarkun

The CI images aren't build nightly anymore (as of ticket:3958) but it's still possible to use the rebuild-images.sh tool to introduce regressions by pulling in modified dependencies.

comment:3 Changed at 2024-12-19T10:29:05Z by btlogy

  • Component changed from unknown to dev-infrastructure
  • Keywords docker added
  • Type changed from defect to enhancement

Having lately looked into those images, I was thinking of two steps to improve the current situation in regards of this ticket:

  1. limit the Docker context of those images to the strict minimum (e.g.: not the all src/, neither the doc/ folders): this could be achieved w/ .dockerignore and/or restructuring files/directories.
  2. trigger the build-images workflow on any PR that would change this limited context, but publish only when building on the master: this idea was partly discussed in ticket:3484.
Note: See TracTickets for help on using tickets.