Recent regression in coverage

[alex]

I apologize in advance, because I lack a clear minimal reproducer for this:

In the last day day or two, for pyca/cryptography we began observing a regression in coverage reports:

We have a fairly involved coverage process where we run many different CI jobs with -Cinstrument-coverage, covert them to LCOV with llvm-profdata + llvm-cov export, and then we import them into Python's coverage.py to get a unified coverage report.

We also mandate 100% line coverage for our Rust code.

Starting in the last few days, these jobs started failing non-deterministically, here is an example failure: https://github.com/pyca/cryptography/actions/runs/16183191041?pr=13155#summary-45684131641 (I say non-deterministically because if we re-run the tests, we'll occasionally get 100% coverage.)

Notably, if you look at the missing coverage lines, they don't particularly makes sense. For example:

https://github.com/pyca/cryptography/blob/main/src/rust/cryptography-openssl/src/fips.rs#L42-L43

There is no way for these two lines to be missing coverage while the rest of the function is covered.

https://github.com/pyca/cryptography/blob/main/src/rust/src/backend/keys.rs#L138-L139

Same thing, if 137 is covered then 138/139 logically must be.

I recognize that this is barely a reproducer, and not particularly convenient. I'm hoping that if someone can identify recent commits that may have impacted coverage metrics, we can use that to figure out how to build a reproducer.

[alex]

Looks like July 7th 8:30PM EDT was the first time this failed, CI was green up July 7 9:40_AM_ EDT. Its non-deterministic, so that's not 100% certain to be when the regression occurred, but it's something.

[alex]

Since I didn't write this: we include nightly in our coverage-producing CI, so I'm inferring that the proximate cause is that nightly started producing lcov files with these lines.

[alex]

Ok, this looks like it was a change between 1.87 and 1.88, where in 1.87 this line was covereged and in 1.88 it's not considered a line at all.

Unfortunately in older rustc, it was considered a line but with 0 coverage. I no longer think this is a rust bug, this is just a mess of how we merge coverage.

Sincerest apologies for the noise!