Post Rollup Warnings

[rylev]

When creating rollups some PRs are naturally riskier than others. If the rollup introduces bugs or performance issues it can be helpful to get a hint at what PRs might be causing an issue and which PRs can most likely be ignored.

With this in mind, I propose adding support to triagebot for listening for rollup PRs and posting a message with the results of inspecting how risking the rollup is. It could use simple heuristics to begin with.

Configuration Changes

The following would be added to the triagebot.toml:

[rollup-warnings]
trigger_files = [
  "compiler/**/*.rs" # warn on any Rust files in the compiler directory
  # ... more patterns here
]

Warning

For any rolled-up PR included in a rollup that has a file change identified by the patterns in trigger_files, a warning would be posted that looks like the following:

⚠Vulnerable PRs

This rollup has 2 vulnerable PRs.

The following PRs may be particularly vulnerable to breakage:

• #NNN (10 vulnerable files changed)
• #NNN (7 vulnerable files changed)

If this rollup introduces any type of regression, you may wish to investigate these PRs first.

Trigger Rate

Looking at some recent rollups, I expect this warning to show the following results:

• Rollup of 5 pull requests rust#102509 (3 of 5 with vulnerable changes)
• Rollup of 7 pull requests rust#102482 (4 of 7 with vulnerable changes)
• Rollup of 7 pull requests rust#102471 (6 of 7 with vulnerable changes)

Future extensions and open questions

• The mechanism proposed here is very naive. There are many changes that we probably shouldn't consider as vulnerable (e.g., changes to comments). Should we include that in an initial version?
• Some changes are likely a bad idea to have in a rollup. For example, in rust-lang/rust, a PR that changes over 50 files in compiler should probably not have been rolled up (or at the very least, a rollup should not contain two such PRs). Perhaps a strong waning in such cases would be useful?

[jyn514]

• Some changes are likely a bad idea to have in a rollup. For example, in rust-lang/rust, a PR that changes over 50 files in compiler should probably not have been rolled up (or at the very least, a rollup should not contain two such PRs). Perhaps a strong waning in such cases would be useful?

I'm not sure this is a useful metric ... a PR that changes that many files is probably mass-renaming a common function, which is actually quite safe. Lines of code is a little better, but can be fooled by the same sort of change - lines of code within a file is more reliable, but can be tricked by moving code from one file to another. I think this is going to be really hard to automate in any useful way.

[jyn514]

I don't think we should add any automation here until we agree what would actually be useful; or at least, agree that it's ok to turn it off if we're not finding it useful.