Add code coverage + HTML report machinery by woodruffw · Pull Request #19692 · astral-sh/uv

woodruffw · 2026-06-05T19:10:26Z

Summary

This adds a new cargo dev coverage subcommand, which invokes nextest under our new coverage pesudo-profile. This profile in turn enables -C instrument-coverage.

cargo dev coverage also takes care of merging raw profiling samples into profdata and LCOV reports for us, which our new coverage-html-report.py script can turn into a readable (but very large) HTML report.

The basic flow for this is:

$ # any nextest flags will work
$ cargo dev coverage ...

That will output something like:

Merged profile: .../astral-sh/uv/target/coverage/profdata/b46cc0f3-7cd0-4309-9df2-10ab653e6dd8.profdata
LCOV profile: .../astral-sh/uv/target/coverage/lcov/b46cc0f3-7cd0-4309-9df2-10ab653e6dd8.lcov
Generate an HTML report with: uv run --script scripts/coverage-html-report.py b46cc0f3-7cd0-4309-9df2-10ab653e6dd8

And then you can run:

uv run --script scripts/coverage-html-report.py b46cc0f3-7cd0-4309-9df2-10ab653e6dd8

(or add --open to auto-open in the browser).

Under the hood, this only uses existing dependencies plus Rust's own LLVM toolchain, plus coverage.py for the HTML report generation.

TODOs:

Document this in CONTRIBUTING.md
Figure out if/how best to surface this in CI -- maybe we want to export combined LCOVs for each test job, similarly to what we do for JUnit test summaries?

Test Plan

Internal only.

woodruffw · 2026-06-05T19:11:17Z

Example from running cargo dev coverage -p uv-audit:

codspeed-hq · 2026-06-05T21:31:21Z

Merging this PR will not alter performance

⚡ 1 improved benchmark
✅ 12 untouched benchmarks

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	WallTime	`resolve_warm_airflow`	437.7 ms	424.7 ms	+3.07%

Tip

Curious why this is faster? Use the CodSpeed MCP and ask your agent.

_{Comparing ww/cov (05f8df5) with main (1ed52dc)}

woodruffw · 2026-06-05T21:48:41Z

Noting: the reason I'm currently gating this behind a label is because it makes tests in CI more than twice as slow.

zanieb · 2026-06-12T15:13:58Z

+[build]
+rustflags = ["-C", "instrument-coverage"]
+target-dir = "target/coverage"


Does this have a runtime cost?

xref https://github.com/astral-sh/uv/pull/19692/changes#r3404368541

Enabling instrumentation? Yeah, it's significant, which is why it gets its own target here.

zanieb · 2026-06-12T15:14:42Z


          # Decisions
          [[ ! $has_skip_label && ($any_code_changed || $on_main_branch) ]] && test_code=1
+          [[ $on_main_branch || $has_coverage_label ]] && collect_coverage=1


Why do we only collect coverage on pull requests with a label?

Mostly because it's slow -- instrumentation makes our test runs roughly 50% slower, sadly.

woodruffw · 2026-06-12T17:54:05Z

+use uv_dev::{COVERAGE_RUSTC_WRAPPER_ENV, coverage_rustc_wrapper, run};
 use uv_static::EnvVars;

+fn main() -> ExitCode {


Noting: it feels like a pretty bad hack to reuse uv-dev as a rustc wrapper, but this needs to live somewhere. Maybe I can abstract it as uv-dev rustc ... or similar.

This adds a new `cargo dev coverage` subcommand, which invokes nextest under our new coverage pesudo-profile. This profile in turn enables `-C instrument-coverage`. `cargo dev coverage` also takes care of merging raw profiling samples into profdata and LCOV reports for us, which our new `coverage-html-report.py` script can turn into a readable (but very large) HTML report. The basic flow for this is: ``` $ # any nextest flags will work $ cargo dev coverage ... ``` That will output something like: ``` Merged profile: .../astral-sh/uv/target/coverage/profdata/b46cc0f3-7cd0-4309-9df2-10ab653e6dd8.profdata LCOV profile: .../astral-sh/uv/target/coverage/lcov/b46cc0f3-7cd0-4309-9df2-10ab653e6dd8.lcov Generate an HTML report with: uv run --script scripts/coverage-html-report.py b46cc0f3-7cd0-4309-9df2-10ab653e6dd8 ``` And then you can run: ``` uv run --script scripts/coverage-html-report.py b46cc0f3-7cd0-4309-9df2-10ab653e6dd8 ``` (or add `--open` to auto-open in the browser). Under the hood, this only uses existing dependencies plus Rust's own LLVM toolchain, plus coverage.py for the HTML report generation. Signed-off-by: William Woodruff <william@yossarian.net> MVP coverage workflow Signed-off-by: William Woodruff <william@yossarian.net> MVP coverage workflow, take 2 Signed-off-by: William Woodruff <william@yossarian.net> Use a pool of 16 raw profiles for pooling Signed-off-by: William Woodruff <william@yossarian.net> Avoid some Windows UNC pain Signed-off-by: William Woodruff <william@yossarian.net> Fix rustflags for coverage on Windows Signed-off-by: William Woodruff <william@yossarian.net> Remove coverage ID chum Signed-off-by: William Woodruff <william@yossarian.net> Avoid Windows nonsense Signed-off-by: William Woodruff <william@yossarian.net> Add coverage guide to CONTRIBUTING Signed-off-by: William Woodruff <william@yossarian.net> Be less strict about the ID Signed-off-by: William Woodruff <william@yossarian.net> Prettier Signed-off-by: William Woodruff <william@yossarian.net>

Signed-off-by: William Woodruff <william@yossarian.net>

woodruffw self-assigned this Jun 5, 2026

woodruffw added the internal A refactor or improvement that is not user-facing label Jun 5, 2026

woodruffw requested a review from Gankra June 5, 2026 19:12

woodruffw added the coverage Perform a dedicated coverage-instrumented test run label Jun 5, 2026

woodruffw commented Jun 5, 2026

View reviewed changes

Comment thread .github/workflows/coverage.yml Outdated

woodruffw commented Jun 5, 2026

View reviewed changes

Comment thread crates/uv-dev/src/coverage.rs Outdated

woodruffw force-pushed the ww/cov branch from ce3c11f to d2300fc Compare June 8, 2026 13:26

zanieb reviewed Jun 8, 2026

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

zanieb reviewed Jun 8, 2026

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

woodruffw force-pushed the ww/cov branch 4 times, most recently from f310af7 to 226b0e5 Compare June 12, 2026 14:26

woodruffw marked this pull request as ready for review June 12, 2026 14:42

zanieb reviewed Jun 12, 2026

View reviewed changes

Comment thread .github/workflows/coverage.yml Outdated

zanieb reviewed Jun 12, 2026

View reviewed changes

Comment thread .github/workflows/test.yml

zanieb reviewed Jun 12, 2026

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

zanieb reviewed Jun 12, 2026

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

woodruffw commented Jun 12, 2026

View reviewed changes

woodruffw added 6 commits June 15, 2026 10:26

Use plan job to determine coverage run

c53e99b

Signed-off-by: William Woodruff <william@yossarian.net>

Remove coverage.yml

c69cdd4

Signed-off-by: William Woodruff <william@yossarian.net>

Remove test-only changes

feced85

Signed-off-by: William Woodruff <william@yossarian.net>

Run macos tests if we're collecting coverage

3810caa

Signed-off-by: William Woodruff <william@yossarian.net>

Make pool size a property of concurrency

de6b719

Signed-off-by: William Woodruff <william@yossarian.net>

Only instrument workspace members

9aa7a40

Signed-off-by: William Woodruff <william@yossarian.net>

woodruffw force-pushed the ww/cov branch from 658c8e2 to 9aa7a40 Compare June 15, 2026 14:26

Use single-argument form

05f8df5

Signed-off-by: William Woodruff <william@yossarian.net>

Conversation

woodruffw commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

woodruffw commented Jun 5, 2026

Uh oh!

Uh oh!

Uh oh!

codspeed-hq Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Performance Changes

Uh oh!

woodruffw commented Jun 5, 2026

Uh oh!

Uh oh!

Uh oh!

zanieb Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

zanieb Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

woodruffw Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

zanieb Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

woodruffw Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

woodruffw Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

woodruffw commented Jun 5, 2026 •

edited

Loading

codspeed-hq Bot commented Jun 5, 2026 •

edited

Loading