Make closure capturing have consistent and correct behaviour around patterns #138961

meithecatte · 2025-03-26T02:59:42Z

This PR has two goals:

firstly, it fixes internal compiler error: two identical projections #137467. In order to do so, it needs to introduce a small breaking change surrounding the interaction of closure captures with matching against enums with uninhabited variants. Yes – to fix an ICE!
- this also fixes ICE: called Option::unwrap() on a None value with refutable patterns #138973, a slightly different case with the same root cause.
secondly, it fixes Closure captures are inconsistent between x and x @ _ irrefutable patterns #137553, making the closure capturing rules consistent between let patterns and match patterns. This is new insta-stable behavior.

Background

This change concerns how precise closure captures interact with patterns. As a little known feature, patterns that require inspecting only part of a value will only cause that part of the value to get captured:

fn main() {
    let mut a = (21, 37);
    // only captures a.0, writing to a.1 does not invalidate the closure
    let mut f = || {
        let (ref mut x, _) = a;
        *x = 42;
    };
    a.1 = 69;
    f();
}

I was not able to find any discussion of this behavior being introduced, or discussion of its edge-cases, but it is documented in the Rust reference.

The currently stable behavior is as follows:

if any pattern contains a binding, the place it binds gets captured (implemented in current walk_pat)
patterns in refutable positions (match, if let, let ... else, but not destructuring let or destructuring function parameters) get processed as follows (maybe_read_scrutinee):
- if matching against the pattern will at any point require inspecting a discriminant, or it includes a variable binding not followed by an @-pattern, capture the entire scrutinee by reference

You will note that this behavior is quite weird and it's hard to imagine a sensible rationale for at least some of its aspects. It has the following issues:

firstly, it assumes that matching against an irrefutable pattern cannot possibly require inspecting any discriminants. With or-patterns, this isn't true, and it is the cause of the internal compiler error: two identical projections #137467 ICE.
secondly, the presence of an @-pattern doesn't really have any semantics by itself. This is the weird behavior tracked as Closure captures are inconsistent between x and x @ _ irrefutable patterns #137553.
thirdly, the behavior is different between pattern-matching done through let and pattern-matching done through match – which is a superficial syntactic difference

This PR aims to address all of the above issues. The new behavior is as follows:

like before, if a pattern contains a binding, the place it binds gets captured as required by the binding mode
if matching against the pattern requires inspecting a disciminant, the place whose discriminant needs to be inspected gets captured by reference

"requires inspecting a discriminant" is also used here to mean "compare something with a constant" and other such decisions. For types other than ADTs, the details are not interesting and aren't changing.

The breaking change

During closure capture analysis, matching an enum against a constructor is considered to require inspecting a discriminant if the enum has more than one variant. Notably, this is the case even if all the other variants happen to be uninhabited. This is motivated by implementation difficulties involved in querying whether types are inhabited before we're done with type inference – without moving mountains to make it happen, you hit this assert:

rust/compiler/rustc_middle/src/ty/inhabitedness/mod.rs

Line 121 in 43f0014

debug_assert!(!self.has_infer());

Now, because the previous implementation did not concern itself with capturing the discriminants for irrefutable patterns at all, this is a breaking change – the following example, adapted from the testsuite, compiles on current stable, but will not compile with this PR:

#[derive(Clone, Copy, PartialEq, Eq, Debug)]
enum Void {}

pub fn main() {
    let mut r = Result::<Void, (u32, u32)>::Err((0, 0));
    let mut f = || {
        let Err((ref mut a, _)) = r;
        *a = 1;
    };
    let mut g = || {
    //~^ ERROR: cannot borrow `r` as mutable more than once at a time
        let Err((_, ref mut b)) = r;
        *b = 2;
    };
    f();
    g();
    assert_eq!(r, Err((1, 2)));
}

Is the breaking change necessary?

One other option would be to double down, and introduce a set of syntactic rules for determining whether a sub-pattern is in an irrefutable position, instead of querying the types and checking how many variants there are.

This would not eliminate the breaking change, but it would limit it to more contrived examples, such as

let ((true, Err((ref mut a, _, _))) | (false, Err((_, ref mut a, _)))) = x;

In this example, the Errs would not be considered in an irrefutable position, because they are part of an or-pattern. However, current stable would treat this just like a tuple (bool, (T, U, _)).

While introducing such a distinction would limit the impact, I would say that the added complexity would not be commensurate with the benefit it introduces.

The new insta-stable behavior

If a pattern in a match expression or similar has parts it will never read, this part will not be captured anymore:

fn main() {
    let mut a = (21, 37);
    // now only captures a.0, instead of the whole a
    let mut f = || {
        match a {
            (ref mut x, _) => *x = 42,
        }
    };
    a.1 = 69;
    f();
}

Note that this behavior was pretty much already present, but only accessible with this One Weird Trick™:

fn main() {
    let mut a = (21, 37);
    // both stable and this PR only capture a.0, because of the no-op @-pattern
    let mut f = || {
        match a {
            (ref mut x @ _, _) => *x = 42,
        }
    };
    a.1 = 69;
    f();
}

Implementation notes

The PR has two main commits:

"ExprUseVisitor: properly report discriminant reads" makes walk_pat perform all necessary capturing. This is the part that fixes internal compiler error: two identical projections #137467.
"ExprUseVisitor: remove maybe_read_scrutinee" removes the unnecessary "capture the entire scrutinee" behavior, fixing Closure captures are inconsistent between x and x @ _ irrefutable patterns #137553.

The new logic stops making the distinction between one particular example that used to work, and another ICE, tracked as #119786. As this requires an unstable feature, I am leaving this as future work.

rustbot · 2025-03-26T17:28:34Z

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

meithecatte · 2025-03-26T17:32:16Z

Nadrieril suggested that this should be resolved through a breaking change – updated the PR description accordingly.

@rustbot label +needs-crater

r? @Nadrieril

rustbot · 2025-03-26T17:32:19Z

Error: Label needs-crater can only be set by Rust team members

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

meithecatte · 2025-03-26T18:32:29Z

@compiler-errors You've requested that the fix for #137553 land in a separate PR. However, ironically, the breaking changes are actually required by #137467 and not #137553. Do you think the removal of the now-obsolete maybe_read_scrutinee should happen in a separate PR, or should I do it here so that it also benefits from the crater run?

compiler-errors · 2025-03-26T18:37:30Z

We can crater both together if you think they're not worth separating. I was just trying to accelerate landing the parts that are obviously-not-breaking but it's up to you if you think that effort is worth it or if you're willing to be patient about waiting for the breaking parts (and FCP, etc).

@bors try

ExprUseVisitor: properly report discriminant reads This PR fixes rust-lang#137467. In order to do so, it needs to introduce a small breaking change surrounding the interaction of closure captures with matching against enums with uninhabited variants. Yes – to fix an ICE! ## Background The current upvar inference code handles patterns in two parts: - `ExprUseVisitor::walk_pat` finds the *bindings* being done by the pattern and captures the relevant parts - `ExprUseVisitor::maybe_read_scrutinee` determines whether matching against the pattern will at any point require inspecting a discriminant, and if so, captures *the entire scrutinee*. It also has some weird logic around bindings, deciding to also capture the entire scrutinee if *pretty much any binding exists in the pattern*, with some weird behavior like rust-lang#137553. Nevertheless, something like `|| let (a, _) = x;` will only capture `x.0`, because `maybe_read_scrutinee` does not run for irrefutable patterns at all. This causes issues like rust-lang#137467, where the closure wouldn't be capturing enough, because an irrefutable or-pattern can still require inspecting a discriminant, and the match lowering would then panic, because it couldn't find an appropriate upvar in the closure. My thesis is that this is not a reasonable implementation. To that end, I intend to merge the functionality of both these parts into `walk_pat`, which will bring upvar inference closer to what the MIR lowering actually needs – both in making sure that necessary variables get captured, fixing rust-lang#137467, and in reducing the cases where redundant variables do – fixing rust-lang#137553. This PR introduces the necessary logic into `walk_pat`, fixing rust-lang#137467. A subsequent PR will remove `maybe_read_scrutinee` entirely, which should now be redundant, fixing rust-lang#137553. The latter is still pending, as my current revision doesn't handle opaque types correctly for some reason I haven't looked into yet. ## The breaking change The following example, adapted from the testsuite, compiles on current stable, but will not compile with this PR: ```rust #[derive(Clone, Copy, PartialEq, Eq, Debug)] enum Void {} pub fn main() { let mut r = Result::<Void, (u32, u32)>::Err((0, 0)); let mut f = || { let Err((ref mut a, _)) = r; *a = 1; }; let mut g = || { //~^ ERROR: cannot borrow `r` as mutable more than once at a time let Err((_, ref mut b)) = r; *b = 2; }; f(); g(); assert_eq!(r, Err((1, 2))); } ``` The issue is that, to determine that matching against `Err` here doesn't require inspecting the discriminant, we need to query the `InhabitedPredicate` of the types involved. However, as upvar inference is done during typechecking, the relevant type might not yet be fully inferred. Because of this, performing such a check hits this assertion: https://github.com/rust-lang/rust/blob/43f0014ef0f242418674f49052ed39b70f73bc1c/compiler/rustc_middle/src/ty/inhabitedness/mod.rs#L121 The code used to compile fine, but only because the compiler incorrectly assumed that patterns used within a `let` cannot possibly be inspecting any discriminants. ## Is the breaking change necessary? One other option would be to double down, and introduce a deliberate semantics difference between `let $pat = $expr;` and `match $expr { $pat => ... }`, that syntactically determines whether the pattern is in an irrefutable position, instead of querying the types. **This would not eliminate the breaking change,** but it would limit it to more contrived examples, such as ```rust let ((true, Err((ref mut a, _, _))) | (false, Err((_, ref mut a, _)))) = x; ``` The cost here, would be the complexity added with very little benefit. ## Other notes - I performed various cleanups while working on this. The last commit of the PR is the interesting one. - Due to the temporary duplication of logic between `maybe_read_scrutinee` and `walk_pat`, some of the `#[rustc_capture_analysis]` tests report duplicate messages before deduplication. This is harmless.

bors · 2025-03-26T18:38:42Z

⌛ Trying commit 8ed61e4 with merge 3b30da3...

meithecatte · 2025-03-26T19:42:57Z

We can crater both together if you think they're not worth separating. I was just trying to accelerate landing the parts that are obviously-not-breaking but it's up to you if you think that effort is worth it or if you're willing to be patient about waiting for the breaking parts (and FCP, etc).

That's the thing – one part is a breaking change, the other introduces insta-stable new behavior. There's no easily mergeable part to this.

compiler-errors · 2025-03-26T20:03:52Z

could we give this a less weird pr title pls 💀

@bors try

bors · 2025-03-26T20:05:05Z

⌛ Trying commit 7d5a892 with merge 630b4e8...

ExprUseVisitor: murder maybe_read_scrutinee in cold blood This PR fixes rust-lang#137467. In order to do so, it needs to introduce a small breaking change surrounding the interaction of closure captures with matching against enums with uninhabited variants. Yes – to fix an ICE! ## Background The current upvar inference code handles patterns in two parts: - `ExprUseVisitor::walk_pat` finds the *bindings* being done by the pattern and captures the relevant parts - `ExprUseVisitor::maybe_read_scrutinee` determines whether matching against the pattern will at any point require inspecting a discriminant, and if so, captures *the entire scrutinee*. It also has some weird logic around bindings, deciding to also capture the entire scrutinee if *pretty much any binding exists in the pattern*, with some weird behavior like rust-lang#137553. Nevertheless, something like `|| let (a, _) = x;` will only capture `x.0`, because `maybe_read_scrutinee` does not run for irrefutable patterns at all. This causes issues like rust-lang#137467, where the closure wouldn't be capturing enough, because an irrefutable or-pattern can still require inspecting a discriminant, and the match lowering would then panic, because it couldn't find an appropriate upvar in the closure. My thesis is that this is not a reasonable implementation. To that end, I intend to merge the functionality of both these parts into `walk_pat`, which will bring upvar inference closer to what the MIR lowering actually needs – both in making sure that necessary variables get captured, fixing rust-lang#137467, and in reducing the cases where redundant variables do – fixing rust-lang#137553. This PR introduces the necessary logic into `walk_pat`, fixing rust-lang#137467. A subsequent PR will remove `maybe_read_scrutinee` entirely, which should now be redundant, fixing rust-lang#137553. The latter is still pending, as my current revision doesn't handle opaque types correctly for some reason I haven't looked into yet. ## The breaking change The following example, adapted from the testsuite, compiles on current stable, but will not compile with this PR: ```rust #[derive(Clone, Copy, PartialEq, Eq, Debug)] enum Void {} pub fn main() { let mut r = Result::<Void, (u32, u32)>::Err((0, 0)); let mut f = || { let Err((ref mut a, _)) = r; *a = 1; }; let mut g = || { //~^ ERROR: cannot borrow `r` as mutable more than once at a time let Err((_, ref mut b)) = r; *b = 2; }; f(); g(); assert_eq!(r, Err((1, 2))); } ``` The issue is that, to determine that matching against `Err` here doesn't require inspecting the discriminant, we need to query the `InhabitedPredicate` of the types involved. However, as upvar inference is done during typechecking, the relevant type might not yet be fully inferred. Because of this, performing such a check hits this assertion: https://github.com/rust-lang/rust/blob/43f0014ef0f242418674f49052ed39b70f73bc1c/compiler/rustc_middle/src/ty/inhabitedness/mod.rs#L121 The code used to compile fine, but only because the compiler incorrectly assumed that patterns used within a `let` cannot possibly be inspecting any discriminants. ## Is the breaking change necessary? One other option would be to double down, and introduce a deliberate semantics difference between `let $pat = $expr;` and `match $expr { $pat => ... }`, that syntactically determines whether the pattern is in an irrefutable position, instead of querying the types. **This would not eliminate the breaking change,** but it would limit it to more contrived examples, such as ```rust let ((true, Err((ref mut a, _, _))) | (false, Err((_, ref mut a, _)))) = x; ``` The cost here, would be the complexity added with very little benefit. ## Other notes - I performed various cleanups while working on this. The last commit of the PR is the interesting one. - Due to the temporary duplication of logic between `maybe_read_scrutinee` and `walk_pat`, some of the `#[rustc_capture_analysis]` tests report duplicate messages before deduplication. This is harmless.

meithecatte · 2025-03-26T20:11:42Z

could we give this a less weird pr title pls 💀

Sure thing. I also updated the PR description to describe both changes. I want to add a section on what exactly the insta-stable behavior will be, but I realized that I haven't added a test for that. Should I hold off on pushing that to not break the bors try and crater?

compiler-errors · 2025-03-26T20:13:04Z

Once bors is done with the try build then you can push, no need to wait until crater is done.

meithecatte · 2025-04-09T16:35:36Z

In fact the other calls to {try_}to_place in match MIR lowering are potential places where we mess things up. E.g. here, we want to add a PlaceMention to the scrutinee which we can't do if it's not captured, could that have weird consequences maybe?

Ask and ye shall receive (playground):

use std::mem::transmute;

fn main() {
    let r: &&u32 = unsafe {
        let x = 42;
        transmute(&&x)
    };

    // no UB detected
    let f = || { let _ = **r; };
    f();
    
    // UB due to the inner deref
    let _ = **r;
}

(behavior is the same between nightly and this PR)

RalfJung · 2025-04-09T16:45:14Z

Nice example. :D Could you file an issue in the Miri repo?

meithecatte · 2025-04-09T16:49:34Z

Filed rust-lang/miri#4258

rfcbot · 2025-04-09T16:59:07Z

🔔 This is now entering its final comment period, as per the review above. 🔔

rfcbot · 2025-04-19T17:04:10Z

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

traviscross · 2025-04-20T00:21:13Z

FCP is complete, and there are conflicts to resolve.

@rustbot author

rustbot · 2025-04-20T00:21:18Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

This solves the "can't find the upvar" ICEs that resulted from `maybe_read_scrutinee` being unfit for purpose.

The split between walk_pat and maybe_read_scrutinee has now become redundant. Due to this change, one testcase within the testsuite has become similar enough to a known ICE to also break. I am leaving this as future work, as it requires feature(type_alias_impl_trait)

As per code review, it is preferred to not use derives in tests that aren't about them.

meithecatte · 2025-04-20T16:02:53Z

I've resolved the conflicts. The Miri tests requested above are still pending.

jackh726 · 2025-04-21T22:40:44Z

cc @rust-lang/types

I'm not sure if we own closure capture analyses or not (one could also argue it's more @rust-lang/opsem), but it's sufficiently grey that I think a ping here is enough.

This aims to make each major part responsible for modifying the precision be visible in the logs.

rustbot · 2025-04-22T00:52:14Z

The Miri subtree was changed

cc @rust-lang/miri

meithecatte · 2025-04-22T01:27:52Z

@RalfJung I've added the tests you suggested, could you take a look?

@rustbot ready

rustbot · 2025-04-22T01:33:22Z

Some changes occurred in match lowering

cc @Nadrieril

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 26, 2025

This comment has been minimized.

Sign in to view

meithecatte force-pushed the expr-use-visitor branch from 6b31250 to 4b7bf58 Compare March 26, 2025 15:33

This comment has been minimized.

Sign in to view

meithecatte force-pushed the expr-use-visitor branch 2 times, most recently from c225f17 to ce47a4c Compare March 26, 2025 16:21

This comment has been minimized.

Sign in to view

meithecatte force-pushed the expr-use-visitor branch from ce47a4c to 75afceb Compare March 26, 2025 16:57

meithecatte changed the title ~~[WIP] ExprUseVisitor: properly report discriminant reads~~ ExprUseVisitor: properly report discriminant reads Mar 26, 2025

meithecatte marked this pull request as ready for review March 26, 2025 17:28

rustbot assigned Nadrieril Mar 26, 2025

meithecatte force-pushed the expr-use-visitor branch from 75afceb to 8ed61e4 Compare March 26, 2025 17:36

jieyouxu added the needs-crater This change needs a crater run to check for possible breakage in the ecosystem. label Mar 26, 2025

meithecatte changed the title ~~ExprUseVisitor: properly report discriminant reads~~ ExprUseVisitor: murder maybe_read_scrutinee in cold blood Mar 26, 2025

meithecatte changed the title ~~ExprUseVisitor: murder maybe_read_scrutinee in cold blood~~ ExprUseVisitor: get rid of maybe_read_scrutinee Mar 26, 2025

This comment has been minimized.

Sign in to view

rfcbot added final-comment-period In the final comment period and will be merged soon unless new substantive objections are raised. and removed proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. labels Apr 9, 2025

rustbot mentioned this pull request Apr 19, 2025

Tracking issue for release notes of #138961: Make closure capturing have consistent and correct behaviour around patterns #140048

Open

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 20, 2025

meithecatte added 4 commits April 20, 2025 04:07

ExprUseVisitor: properly report discriminant reads

2759284

This solves the "can't find the upvar" ICEs that resulted from `maybe_read_scrutinee` being unfit for purpose.

Add test case for issue rust-lang#138973

f4fc4a7

Avoid using #[derive] in test

af3aaba

As per code review, it is preferred to not use derives in tests that aren't about them.

meithecatte force-pushed the expr-use-visitor branch from 7a07089 to af3aaba Compare April 20, 2025 15:58

meithecatte added 2 commits April 22, 2025 02:47

Add debug logging in hir_typeck::upvar

00a0e86

This aims to make each major part responsible for modifying the precision be visible in the logs.

Add miri tests for new closure capture behavior

972b851

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 22, 2025

add a comment: MatchPair and ExprUseVisitor must stay in sync

5f73b21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make closure capturing have consistent and correct behaviour around patterns #138961

Make closure capturing have consistent and correct behaviour around patterns #138961

meithecatte commented Mar 26, 2025 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rustbot commented Mar 26, 2025

meithecatte commented Mar 26, 2025

rustbot commented Mar 26, 2025

meithecatte commented Mar 26, 2025

compiler-errors commented Mar 26, 2025

bors commented Mar 26, 2025

meithecatte commented Mar 26, 2025

compiler-errors commented Mar 26, 2025

bors commented Mar 26, 2025

meithecatte commented Mar 26, 2025

compiler-errors commented Mar 26, 2025

This comment has been minimized.

meithecatte commented Apr 9, 2025

RalfJung commented Apr 9, 2025

meithecatte commented Apr 9, 2025

rfcbot commented Apr 9, 2025

rfcbot commented Apr 19, 2025

traviscross commented Apr 20, 2025

rustbot commented Apr 20, 2025

meithecatte commented Apr 20, 2025

jackh726 commented Apr 21, 2025

rustbot commented Apr 22, 2025

meithecatte commented Apr 22, 2025

rustbot commented Apr 22, 2025

Make closure capturing have consistent and correct behaviour around patterns #138961

Are you sure you want to change the base?

Make closure capturing have consistent and correct behaviour around patterns #138961

Conversation

meithecatte commented Mar 26, 2025 • edited Loading

Background

The breaking change

Is the breaking change necessary?

The new insta-stable behavior

Implementation notes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rustbot commented Mar 26, 2025

meithecatte commented Mar 26, 2025

rustbot commented Mar 26, 2025

meithecatte commented Mar 26, 2025

compiler-errors commented Mar 26, 2025

bors commented Mar 26, 2025

meithecatte commented Mar 26, 2025

compiler-errors commented Mar 26, 2025

bors commented Mar 26, 2025

meithecatte commented Mar 26, 2025

compiler-errors commented Mar 26, 2025

This comment has been minimized.

meithecatte commented Apr 9, 2025

RalfJung commented Apr 9, 2025

meithecatte commented Apr 9, 2025

rfcbot commented Apr 9, 2025

rfcbot commented Apr 19, 2025

traviscross commented Apr 20, 2025

rustbot commented Apr 20, 2025

meithecatte commented Apr 20, 2025

jackh726 commented Apr 21, 2025

rustbot commented Apr 22, 2025

meithecatte commented Apr 22, 2025

rustbot commented Apr 22, 2025

meithecatte commented Mar 26, 2025 •

edited

Loading