Skip to content

Filter PR coverage comments to relevant projects - #7932

Open
svick wants to merge 3 commits into
dotnet:mainfrom
svick:filter-pr-coverage-comments
Open

svick wants to merge 3 commits into
dotnet:mainfrom
svick:filter-pr-coverage-comments

Conversation

@svick

@svick svick commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Why

The coverage bot currently reports findings for every project in the repository on each PR and each push, including coverage shortfalls and threshold-increase suggestions in projects the PR did not touch. This buries relevant feedback in unrelated findings and can make existing coverage gaps look like regressions introduced by the PR.

Filter the GitHub comment to PR-relevant projects so authors can focus on actionable feedback, while keeping repository-wide validation and the complete Azure DevOps coverage report unchanged.

Changes

  • Include projects containing changed source files and source projects directly referenced by changed tests. Shared code or build/configuration changes retain the full comment; no relevant findings means no comment.
  • Preserve repository-wide coverage validation and the full saved coverage report.
  • Run the coverage-script tests on Windows and Ubuntu with pinned Pester 6.2.0. Let subsequent build tasks run after a test failure, then publish NUnit results and fail the job at the end if tests failed or results are missing.

If you think running script tests on CI or even having script tests in the first place is unnecessary, I can remove those.

Validation

  • All 12 script tests pass on Windows and Ubuntu with Pester 6.2.0.
  • An intentionally failing test returns exit code 1 and produces NUnit failure results.
  • Dry-ran against Add HybridCache.GetOrCreateAsync overloads for dynamic setting of options #7568 using its exact build revision and original merged coverage artifact: the comment shrinks from 33 rows to one HybridCache row, with the full saved report and validation exit code unchanged. No comment was posted.

Note

This PR was generated with GitHub Copilot.

Microsoft Reviewers: Open in CodeFlow

svick and others added 2 commits November 9, 2026 18:11
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4d46afcf-3229-49c2-83ca-e59c727e3fe8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4d46afcf-3229-49c2-83ca-e59c727e3fe8
@github-actions github-actions Bot added the area-ai-eval Microsoft.Extensions.AI.Evaluation and related label Nov 9, 2026
@svick svick added area-infrastructure and removed area-ai-eval Microsoft.Extensions.AI.Evaluation and related labels Nov 9, 2026
@svick
svick requested a balanced review from Copilot November 9, 2026 19:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Coverage filtering can suppress relevant findings by missing projects that link changed source files and omitting shared tooling and build paths from the full-comment fallback.

2 open issues
What changed in this PR

This PR focuses coverage-bot comments on projects relevant to a pull request while preserving repository-wide validation and the complete saved coverage report.

Changes:

  • Filters comments by changed-file ownership and direct source references from changed tests, with shared-change fallbacks.
  • Adds coverage-script tests and CI result publishing, deferring test failure until after other build steps.
  • Fetches the PR merge commit’s parent for changed-file comparisons.
File Description
eng/​scripts/​ValidateProjectCoverage.Tests.ps1 Tests filtering, full-report preservation, and shallow-checkout comparisons.
eng/​scripts/​ValidateProjectCoverage.ps1 Separates filtered comments from repository-wide coverage reporting.
eng/​pipelines/​templates/​VerifyCoverageReport.yml Supplies the comparison revision for public PR builds.
eng/​pipelines/​templates/​BuildAndTest.yml Runs pinned Pester tests and publishes results with failure checks.
azure-pipelines.yml Increases coverage checkout depth to include the parent commit.
azure-pipelines-unofficial.yml Increases coverage checkout depth to include the parent commit.
azure-pipelines-public.yml Increases coverage checkout depth to include the parent commit.

Review effort: Balanced

Comment on lines +96 to +98
$owners = @($projects | Where-Object {
$fullPath.StartsWith($_.DirectoryName + [System.IO.Path]::DirectorySeparatorChar, [System.StringComparison]::Ordinal)
} | Sort-Object { $_.DirectoryName.Length } -Descending)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think doing this is worth it, especially since it requires attempting to interpret MSBuild wildcards.

Comment thread eng/scripts/ValidateProjectCoverage.ps1 Resolved
@dotnet-comment-bot

Copy link
Copy Markdown
Collaborator

‼️ Found issues ‼️

Project Coverage Type Expected Actual
Microsoft.Extensions.AI Line 89 87.81 🔻
Microsoft.Extensions.AI Branch 89 79.23 🔻
Microsoft.Extensions.AI.OpenAI Line 75 68.8 🔻
Microsoft.Extensions.AI.OpenAI Branch 75 60.3 🔻
Microsoft.Extensions.DataIngestion.MarkItDown Line 75 4.46 🔻
Microsoft.Extensions.DataIngestion.MarkItDown Branch 75 0.75 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring Line 99 96.06 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring Branch 99 92.28 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes Line 99 97.73 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes Branch 99 92.86 🔻
Microsoft.Extensions.Diagnostics.Testing Line 99 98.22 🔻
Microsoft.Extensions.Diagnostics.Testing Branch 99 93.66 🔻
Microsoft.Extensions.ServiceDiscovery Line 75 58.1 🔻
Microsoft.Extensions.ServiceDiscovery Branch 75 64.41 🔻
Microsoft.Extensions.ServiceDiscovery.Abstractions Line 75 41.14 🔻
Microsoft.Extensions.ServiceDiscovery.Abstractions Branch 75 45.1 🔻
Microsoft.Extensions.ServiceDiscovery.Dns Line 75 60.22 🔻
Microsoft.Extensions.ServiceDiscovery.Dns Branch 75 74.08 🔻
Microsoft.Extensions.ServiceDiscovery.Yarp Line 75 73.85 🔻
Microsoft.Extensions.ServiceDiscovery.Yarp Branch 75 60 🔻
Microsoft.Extensions.Telemetry Line 93 90.47 🔻
Microsoft.Extensions.Telemetry Branch 93 86.83 🔻
Microsoft.Extensions.VectorData.Abstractions Line 75 32.15 🔻
Microsoft.Extensions.VectorData.Abstractions Branch 75 26.31 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project Expected Actual
Microsoft.Extensions.AI.Evaluation.NLP 0 74
Microsoft.Extensions.Caching.Hybrid 81 83
Microsoft.Extensions.DataIngestion 75 77
Microsoft.Extensions.DataIngestion.Markdig 75 88
Microsoft.Extensions.DocumentExtraction 80 94
Microsoft.Extensions.DocumentExtraction.Abstractions 80 100
Microsoft.Extensions.Documents.Abstractions 80 98
Microsoft.Gen.MetadataExtractor 57 72
Microsoft.Gen.MetricsReports 67 69

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1671414&view=codecoverage-tab

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4d46afcf-3229-49c2-83ca-e59c727e3fe8
@svick
svick marked this pull request as ready for review November 10, 2026 16:24
@svick
svick requested a review from a team as a code owner November 10, 2026 16:24
@svick
svick requested a review from evgenyfedorov2 November 10, 2026 16:25
@dotnet-comment-bot

Copy link
Copy Markdown
Collaborator

‼️ Found issues ‼️

Project Coverage Type Expected Actual
Microsoft.Extensions.AI Line 89 87.81 🔻
Microsoft.Extensions.AI Branch 89 79.23 🔻
Microsoft.Extensions.AI.OpenAI Line 75 68.8 🔻
Microsoft.Extensions.AI.OpenAI Branch 75 60.3 🔻
Microsoft.Extensions.DataIngestion.MarkItDown Line 75 4.46 🔻
Microsoft.Extensions.DataIngestion.MarkItDown Branch 75 0.75 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring Line 99 96.06 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring Branch 99 92.28 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes Line 99 97.73 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes Branch 99 92.86 🔻
Microsoft.Extensions.Diagnostics.Testing Line 99 98.22 🔻
Microsoft.Extensions.Diagnostics.Testing Branch 99 93.66 🔻
Microsoft.Extensions.ServiceDiscovery Line 75 59.27 🔻
Microsoft.Extensions.ServiceDiscovery Branch 75 64.41 🔻
Microsoft.Extensions.ServiceDiscovery.Abstractions Line 75 41.14 🔻
Microsoft.Extensions.ServiceDiscovery.Abstractions Branch 75 45.1 🔻
Microsoft.Extensions.ServiceDiscovery.Dns Line 75 62.53 🔻
Microsoft.Extensions.ServiceDiscovery.Dns Branch 75 74.08 🔻
Microsoft.Extensions.ServiceDiscovery.Yarp Line 75 73.85 🔻
Microsoft.Extensions.ServiceDiscovery.Yarp Branch 75 60 🔻
Microsoft.Extensions.Telemetry Line 93 90.47 🔻
Microsoft.Extensions.Telemetry Branch 93 86.83 🔻
Microsoft.Extensions.VectorData.Abstractions Line 75 32.15 🔻
Microsoft.Extensions.VectorData.Abstractions Branch 75 26.31 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project Expected Actual
Microsoft.Extensions.AI.Evaluation.NLP 0 74
Microsoft.Extensions.Caching.Hybrid 81 83
Microsoft.Extensions.DataIngestion 75 77
Microsoft.Extensions.DataIngestion.Markdig 75 88
Microsoft.Extensions.DocumentExtraction 80 94
Microsoft.Extensions.DocumentExtraction.Abstractions 80 100
Microsoft.Extensions.Documents.Abstractions 80 98
Microsoft.Gen.MetadataExtractor 57 72
Microsoft.Gen.MetricsReports 67 69

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1673141&view=codecoverage-tab

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants