Costing a Dependency Bump in CI Minutes

The estimate starts with an honest count. Three ways to get it, in increasing order of accuracy.

Diff the lockfile. This is the most direct method. Generate the lockfile with the bump applied, then count changed entries.

# Node.js: count changed package entries between the branch and the merge base
git diff --numstat origin/main -- package-lock.json

# Count package path entries in the new lockfile that differ
git diff origin/main -- package-lock.json | grep -c '^+.*"node_modules/'

For Python with Poetry, poetry.lock lists every resolved package with its version; for Cargo, Cargo.lock does the same; for Go, go.sum records module hashes the sbomify drift explainer. The principle is identical across ecosystems: the lockfile is the record of what you actually got, so a diff of it is the record of what changed.

Count the installed tree. npm ls --all prints the full logical dependency tree including transitive packages, and npm ls --json --all gives the same thing in a form a script can read the npm ls reference. Run it on the base branch and on the bump branch and diff the package name and version pairs. This catches cases where the lockfile format changed without a version change.

Ask the resolver. npm explain <package> walks the tree upward and tells you why a package is present at all the npm ls reference. That is useful when a single bump drags in a surprising dependency and you want to know which parent pulled it in before you decide whether the cost is acceptable.

A worked number: suppose a scanner flags a transitive JSON parser three levels down. The team bumps its parent, a loader package. The lockfile diff shows 11 packages changed: the parent, the parser, and nine sibling utilities the parent's new range resolves differently. That is a transitive package count of 11, not 1. The manifest diff said one line; the resolved set said eleven.

Two cautions. First, a count of changed entries is not the same as a count of new downloads; a version bump of a package already present is a changed entry but the tree size is unchanged. Second, do not count the whole tree if only a subtree changed. The multiplier that matters is packages the bump added or moved.

Now multiply. The number of jobs a push triggers is not the number of workflow files. It is the number of job instances across every workflow that runs on that branch and every matrix cell inside them. CI systems that support job matrices create one job per combination of the configured dimensions, and all matrix jobs run in parallel if concurrency allows the AppVeyor job workflow documentation. Three operating systems times four language runtimes is 12 jobs from one YAML block.

Pipeline cost is therefore roughly:

pipeline minutes = changed packages × jobs triggered × per package install cost, plus jobs triggered × fixed setup cost per job

The fixed cost per job is the part people forget. Every job checks out the repository, restores or rebuilds a cache, sets up the runtime, and installs dependencies before a single test runs. On a cache miss, the install step dominates. On a cache hit, the checkout and setup still dominate.

A comparison of what drives the number:

Note the interaction between rows one and four. A lockfile change invalidates dependency caches on any CI setup that keys the cache on the lockfile hash, which is the standard technique for keeping cache contents correct. So a bump does not just install more packages; it often converts cache hits into full installs on every job that runs. That is the effect a raw package count understates.

A worked example. Suppose the bump changes 11 lockfile entries. The repository runs a workflow on every push with a 3 by 4 matrix, so 12 jobs. Fixed setup is 90 seconds per job. A warm install is 20 seconds on top; a cold install after the lockfile change is 150 seconds per job. On the first push after the bump: 12 jobs × (90 + 150) = 2,880 seconds, or 48 job-minutes. On a later push with a warm cache: 12 × (90 + 20) = 1,320 seconds, or 22 job-minutes. If reviewers ask for two rounds of changes, the branch accumulates roughly 48 + 22 + 22 = 92 job-minutes before it merges. Compare that with a documentation change that hits a single lint job at 3 minutes. That is the comparison the ledger exists to make visible.

The fixed cost is easy to get wrong in three ways.

Double counting the install step. If the install step is inside the setup measurement, do not also charge per package for it. Pick one model: either a flat per job cost that already covers a warm install, with an adder for cache misses, or a per package cost on the cold path. Mixing them inflates estimates and reviewers stop trusting them.

Ignoring jobs that exit early. Path filters and conditional steps mean not every job in a workflow actually runs the expensive part. Measure the jobs that execute the install and test steps, not every job defined in YAML.

Forgetting fan-in and sequential flows. A workflow where a build job feeds several test jobs, which then feed a deploy job, pays setup for every node the AppVeyor job workflow documentation. Sequential dependencies also mean wall clock time is longer than the sum of job minutes, and reviewers waiting on a merge queue feel the wall clock more than the ledger total will show.

The practical rule: measure once. Take the most recent green run of the candidate branch, read the actual duration of each job from the CI API, and subtract a baseline run of the parent commit. The difference is the bump's real pipeline cost. Use the formula for triage before the run, and the measured delta for the ledger after it.

The ledger is a table with one row per change. For a dependency bump the row has two cost columns: human review minutes and pipeline minutes. Both are estimates before merge and both can be replaced with measured values afterward.

Change: bump loader-utils parent 2.0.3 -> 2.0.4 (transitive security fix)
Lockfile delta: 11 packages changed
Jobs triggered per push: 12 (3 OS x 4 runtimes)
Rerun rounds (predicted): 2
Pipeline minutes (predicted): 92
Human review minutes (predicted): 15 (manifest diff 1 line, lockfile diff 240 lines)

The interesting decision rule falls out of the ratio. When pipeline minutes dominate, the fix is usually to narrow the trigger surface: path filters that skip the full matrix for dependency-only changes, or a dedicated short pipeline for lockfile-only commits. When human minutes dominate, the fix is presentational: summarize the lockfile delta in the pull request description so reviewers do not have to read 240 lines to learn that 11 packages moved.

There is a defensible policy default in this space too. Auto-merging patch-level updates is reasonable for direct, pinned dependencies when a written policy gates it on a trusted test suite, provenance checks, and a rehearsed rollback path the Seal Security policy guide. That guide also notes the limit of the approach: a bot acts only where a newer release exists for a direct dependency, and does nothing for transitive packages buried several levels deep the Seal Security policy guide. Those deep transitive bumps are precisely the ones that change many lockfile entries and therefore cost the most pipeline minutes. The ledger tells you which tier of bump you are looking at.

Be careful with the human half of the ledger. Studies of pull request acceptance have found that code quality signals from static analysis did not by themselves influence whether a maintainer accepted a pull request, which is a warning against assuming reviewers act on automated scores the pull request acceptance study. A ledger works when it informs a decision the reviewer is already making, such as whether to ask for a narrower change or a staged rollout.

Three pieces of plumbing turn the formula into data.

Capture the job list per push. Most CI providers expose a run and its jobs through an API or CLI. Record job name, duration, and conclusion for every run on the branch. The AppVeyor workflow model, where jobs are named and can depend on each other, is a useful mental template even if your provider differs the AppVeyor job workflow documentation.

Fail CI on lockfile drift. If the manifest and lockfile disagree, the resolved set is not what anyone tested. Strict install commands in CI make that a build failure rather than a silent deviation, which keeps the count honest the sbomify drift explainer. The drift explainer lists the specific commands per ecosystem, such as the frozen-lockfile flags for Yarn and pnpm and the locked check for Cargo the sbomify drift explainer. Pants generates and consumes lockfiles for Python with hashes checked at build time, which is the same principle inside a build tool the Pants lockfile documentation.

Store the ledger row with the change. A comment on the pull request, or a field in the change record, with predicted and actual values. Over a quarter, you get a distribution of pipeline minutes per bump, which is the input to a policy decision about whether the full matrix should run on dependency-only branches at all.

If your team runs coding agents that open dependency bumps and iterate on the resulting failures autonomously, serial agent runs are where pipeline minutes quietly multiply. Worktree-per-agent setups give each task its own isolated workspace and branch the worktree workflow write-up, and each branch push pays the full job count again. Quanta Skills is the skill library behind the Quanta experts, installable into a coding agent so the bump, the lockfile regeneration and the cost estimate run the same way every time.