non-lexical lifetimes (NLL) tracking issue

[nikomatsakis]

This issue tracks the status of the transition to non-lexical lifetimes and a MIR-based borrow-checker. Both of these are jargon-y terms for compiler authors: the effect of these features on end-users is, simply put, that the compiler accepts a wider range of code with fewer bugs (and, hopefully, better error messages). We will refer to the combination of the above as NLL.

Current status

Most of NLL is already enabled in current versions of Rust. However, 'migration mode' is still enabled (#58781). This causes us to reject some code that we would eventually like to accept, and to use different code paths for emitting some diagnostics. The progress towards disabling 'migration mode' is tracked in #58781

Key facts for getting involved

• Weekly triage meeting Wednesdays at 3:30pm Boston time (see in your timezone)
  • Join the chat on Zulip
  • Notes in this dropbox paper document
  • If you are unable to attend, minutes and further discussion takes place on this internals thread.
  • Ask @nikomatsakis (on Zulip, ideally) to be added to the calendar invite.
• You can also find a YouTube playlist of pair programming videos; these are sessions where we are working through how to solve a particular problem, often explaining big parts of the system on the way.

Implementation plan

• Initial prototype: features available on nightly builds with #![feature(nll)] for experimentation.
• Valid code works: (issues labeled with NLL-complete)
  • all run-pass tests pass
  • bootstrap builds with nll enabled: see [nll] bootstrap rustc with NLL enabled #51823.
    • summary: We have bootstrapped at least once. We need to integrate such bootstrapping into the CI to ensure it stops breaking.
  • crater run with nll enabled passes without regressions
    • see NLL ICEs from nll-3 crater #52217 for the set of ICEs from recent crater run.
• Invalid code gets errors: (issues labeled with NLL-sound)
  • compile-fail/ui tests generate errors in the right places (possibly with bad diagnostics)
    • see NLL: need a (new) round of review of deltas between .stderr and .nll.stderr output #52663 as evidence that we generate errors in all cases where expected for the ui test suite. (Current plan for compile-fail suite is to port majority of those tests to ui)
• Performance is good (issues labeled with NLL-performant)
  • see this dropbox paper which tracks profiling information
  • see also the NLL perf dashboard which compares running cargo check without and with NLL turned on, so that we can see a summary of the current performance impact on the compiler's static analyses in isolation.
• Future compatibility warnings on stable: Once crater runs are clean, we can enable the future-compatibility warnings on stable: see Want -Z borrowck=migrate to test planning migration to non-lexical lifetimes #46908 and Add -Z borrowck=migrate #52681
  • currently, due to Add -Z borrowck=migrate #52681, we are issuing compatibility warnings on the 2018 edition.
  • next step is to migrate the 2015 edition in the same manner (NLL: turn on borrowck=migrate by default on 2015 edition #57804)
• Diagnostic parity: (issues labeled with NLL-diagnostics) Ensure that all diagnostics are "as good or better" than before.
  • We will primarily validate this from our test base, but also consider errors that are reported "from the wild".
  • round one: NLL: must review all .nll.stderr files #49862
  • round two: NLL: need a (new) round of review of deltas between .stderr and .nll.stderr output #52663
  • round three: NLL: need 3rd round of review comparing .stderr and .nll.stderr output #54528; remaining work items are cards on https://github.com/rust-lang/rust/projects/10
• transition from migration mode to full NLL
  • NLL warnings (downgraded from errors) become errors again
  • as an intermediate step, potentially use a deny-by-default lint before making these hard errors
• Feature stabilized: NLL and MIR-based borrow checker enabled by default,
• AST-based borrow checker removed: Rest In Peace, AST borrowck (2012-2019) #64790.

Many of the work items for unchecked bullets above are gathered into one place on #57895

How to join the working group

If you'd like to help, please join the NLL working group -- just leave a comment below or ping @nikomatsakis on gitter and you will be added to the @rust-lang/wg-compiler-nll team. You will then get occasional pings (e.g., when new mentoring instructions are available or when looking for help), as well as being eligible to be assigned to issues and so forth. We discuss things on the WG-compiler-nll channel on Gitter.

How to find an issue to work on

To find important issues, use one of the following queries:

• Top priority right now are issues tagged with NLL-Complete, corresponding to code that should be accepted but isn't right now.
• Next priority are issues tagged with NLL-sound, corresponding to code that should get errors, but isn't right now.
• Next priority are issues tagged with NLL-diagnostics, corresponding to low quality error messages.
• Issues tagged with NLL-performant, corresponding to cases where NLL analysis takes too long.

You may also wish to look for E-mentor issues, which means that they have mentoring instructions. Also, if an issue is assigned, then someone is supposed to be working on it, but it's worth checking if they have made progress and pinging them -- people often get busy with other things.

Issues tagged with NLL-deferred are low priority right now, but if one strikes your fancy, feel free to tackle it!

How are issues are organized

All issues related to NLL are tagged with WG-compiler-nll. They are further tagged with a NLL-foo label to indicate a subcategory. Issues that have no NLL-label are considered "untriaged" and need to be sorted. Issues tagged with NLL-deferred are low priority right now.

Finally, you can always take a look at the full list of NLL-related issues.

In particular, issues tagged with E-mentor are those that contain mentoring instructions that can help you get started.

Issues (and pull requests) tagged with I-nominated are meant to be reviewed by the WG-compiler-nll at each weekly meeting. Here's the current nominated list.

If you can't find anything, reach out to @nikomatsakis on gitter.

Other issues

This section tracks related issues and notes.

Bugs in AST borrow check fixed in MIR borrowck

#47366 tracks known bugs from the AST borrow checker that were fixed in the MIR borrow checker. For each such bug, we have added a test to the repo with #![feature(nll)] and then closed the relevant issue (obviously, though, the bug will still be reproducable until the MIR-based borrow checker is stabilized, presuming one uses the AST-based borrow checker). You can also search for things tagged with NLL-fixed-by-NLL.

Questions to be resolved before stabilization

• [NLL] false edges on infinite loops #46036: infinite loop false edges
• NLL should identify and respect the lifetime annotations that the user wrote NLL should identify and respect the lifetime annotations that the user wrote #47184

Possible extensions

• refine liveness with maybe-initialized at greater resolution than a single variable

[nikomatsakis]

I've written up a gist with the initial draft of a plan. It lays out the first few steps anyway and some of the initial vision. I'll just copy-and-paste the end here, which lays out some of the PRs I think would make sense:

(list moved to header)

[nikomatsakis]

@Nashenas88 has expressed some interest in doing this! It occurs to me that this project may be big enough for multiple people, too.

cc @rust-lang/compiler