Unions with unsized variants

[mahkoh]

I believe it would be technically feasible (easy, even) to have unions with at most one unsized variant.

Unsizing should be safe:

1. T -> dyn U: The vtable contains 2 values relevant for the unsized union:

   1. The alignment: This is stored directly in the vtable of T, accessing it does not access the data pointer
   2. The size: Same.

   Note that drop_in_place is not relevant because unions do not drop their contents.

2. [T; N] -> [T]: The metadata consists of the array length which is statically known and does not require accessing the variant.

@eddyb: In the last paragraph here you wrote that there is a problem with unsized unions but I haven't been able to figure it out. Maybe you were talking about unions with more than one unsized variant?

[RustyYato]

The problem is that the meta-data in the fat pointer won't always be valid, i.e. what should this do?

union Foo<T: ?Sized> {
    value: T,
    uninit: (),
}

let foo = Foo::<i32> { uninit: () };
let foo_ref = &foo as &Foo<dyn Any>; // the fat pointer meta-data isn't valid for the data actually stored

Also, if you ban accessing the data, it makes it harder to implement something like Custom DSTs. Custom DSTs are important because they would allow CStr to be a thin pointer. (Is unsized unions that important? I don't know, maybe, maybe not)

[mahkoh]

The problem is that the meta-data in the fat pointer won't always be valid

Yes, and? Afaik, the only parts of the metadata that MUST be valid are size and alignment so that the layout can be computed at runtime. The other parts (drop_in_place and trait methods) cannot be accessed without first accessing the unsized field. Accessing the wrong field is already undefined behavior so that the metadata being invalid does not add any additional unsafety.

[Lokathor]

I'm not clear on why CStr can't already be a thin pointer.

[burdges]

Any map T -> dyn U for T: !Sized sounds quite extreme for Rust. It'd require Box, et al. magically copy the vtable somewhere, not sure that's zero cost, and really not sure what all that breaks.

I think previous discussions indicate that truly unsized types should panic whenever accessing their size or alignment, so align_of_val and size_of_val should never access their data pointer.

If I understood, Rust could add some TrueCStr (name?) which yields thin pointers, and maybe even makes align_of_val work, but for which size_of_val panics. It'd still have a method like strlen and a conversion into slices.

[kennytm]

but for which size_of_val panics

that would make Box<CStr> crash

[mahkoh]

Please let's stop this academic discussion around CStr. CStr will never be changed simply because it would

1. Completely change the performance characteristics of a core type that is relied upon by lots of production code.
2. Break all code that relies on transmuting CStr to [u8] of which I assume there is much.

People who actually want a thin pointer to null terminated strings have written such a thing themselves years ago and have no need for a thin CStr.

---

Feel free to talk about new forms of metadata in general.

[RalfJung]

I don't know how to even implement this. Imagine

union Example {
  a: str,
  b: String,
}

Now, given &Example (which I suppose is a wide pointer), how do you determine the size?

Note that not even enums can have unsized fields currently, and that seems like a strictly simpler problem since for enums we know which variant they are in. So I think we should specify, implement, and gather experience with unsized enums before venturing into the land of unsized unions.

[mahkoh]

I don't know how to even implement this. Imagine

union Example {
  a: str,
  b: String,
}

Now, given &Example (which I suppose is a wide pointer), how do you determine the size?

How do you construct &Example? Presumably with transmute? In this case you would either transmute it from &str or from [&String, 0]. So the size would be max(metadata, sizeof(String)). Or more generally for [T] fields:

max(metadata * sizeof(T), sizeof(union without the unsized field))

(Rounded up to the correct alignment.)

Automatic unsizing of [T; N] fields would of course set the metadata to the correct value just like unsizing &[T; N] would.

Note that not even enums can have unsized fields currently, and that seems like a strictly simpler problem since for enums we know which variant they are in.

I think enums are significantly more complicated in this case. The &Example above can be constructed via transmute. If it were an enum, this would not be possible.

[RalfJung]

How do you construct &Example?

That doesn't matter for my question. No matter how it is constructed, you have to be able compile code like

fn example1(e: &Example) -> usize { mem::size_of_val(e) }

fn example2(e: Box<Example>) { drop(e) }

in a uniform way, without knowing how the arguments got created.

[RalfJung]

max(metadata * sizeof(T), sizeof(union without the unsized field))

So you are suggesting that the metadata is 0 when the String variant was used to create this enum? Leaving aside that it is completely unclear how to do the unsizing coercion, what do you propose to do if the unsized field is of type dyn Trait? (Note that the vtable metadata field is currently marked nonnull.)

[mahkoh]

So you are suggesting that the metadata is 0 when the String variant was used to create this enum?

We're talking about unions and not enums. The variant does not matter for unsizing and the compiler does not have to know which variant was last written to.

Leaving aside that it is completely unclear how to do the unsizing coercion

Which part is unclear? Given

union X<T> {
	t: ManuallyDrop<T>,
	v: ManuallyDrop<String>,
}

X<[u8; 22]> unsizes to X<[u8]> with metadata 22.

what do you propose to do if the unsized field is of type dyn Trait? (Note that the vtable metadata field is currently marked nonnull.)

X<u8> unsizes to X<dyn Clone> with the metadata of u8 as Clone.

Accessing the wrong field is simply undefined behavior. E.g.

let x: X<u8> = X { v: ManuallyDrop::new(String::new()) };
let x: &X<dyn Clone> = &x;
unsafe {
	x.t.clone(); // UB
}

(The current rules for sized fields are a bit more relaxed but for unsized fields we can simply make up the new rule that accessing an "inactive" unsized field is unconditionally UB.)

[RalfJung]

Btw, since you mentioned that earlier -- note that unsizing-by-transmute is incorrect since the layout of wide pointers is not specified.

X<[u8; 22]> unsizes to X<[u8]> with metadata 22.

Oh, I see. That makes sense. Basically where struct with unsized field adds up all the static field sizes with the one of the unsized field, union takes a max instead. Unsizing will always be able to determine that size without looking at the data stored in the union because pre-unsizing, it is all present in the type. Initializing a X<[u8; 22]> requires specifying the 22 even if we initialize it in the String field -- this is the bit that I missed before.

This also works for dyn Trait.

When nesting union and struct, this can lead to more complicated code for size_of_val (a sequence of additions and max), but it is still possible to implement this in time linear to the depth of the type (so far it's constant, though). Maybe there is a more clever compliation strategy, but that's not something I am fundamentally worried about.

(The current rules for sized fields are a bit more relaxed but for unsized fields we can simply make up the new rule that accessing an "inactive" unsized field is unconditionally UB.)

There is no such thing as an "active" or "inactive" union field, and I think adding that notion would be a mistake. (It would also be quite complicated.) It also seems unnecessary -- your code is equivalent to reading x.t from the original sized x, i.e., it can still be explained by "union field accesses are transmutes".

---

The one issue that remains is the same as what I mentioned in rust-lang/rust#80158: this proposal relies on determining the size of Box<T: !Sized> without accessing the pointee, which is in conflict with very general proposals for custom DST. That conflict could be resolvable in principle, but will lead to a lot of friction nevertheless.

[mahkoh]

rc::Weak already calls align_of_val_raw on invalid pointers.

[RalfJung]

rc::Weak already calls align_of_val_raw on invalid pointers.

The code is careful to spell out that it is valid only when the "unsized tail" of T is a slices or a trait object. AFAIK the public API of Rc makes it impossible to create other Rc<T> for unsized T, even if the respective unsizing coercions were added. Do you see a loophole in that approach?

[mahkoh]

The code I was thinking of is

https://play.rust-lang.org/?version=beta&mode=debug&edition=2018&gist=62b9754a432976a2646cf30812e60913

If I understand you correctly, you're saying that Weak<T> can never be safely converted to Weak<U> where U is not sized but neither a slice or a trait.

The only other unsized types I'm aware of are extern types and I'm not aware of any way to unsize any type to an extern type. Therefore this statement would be trivially true in the current language and also hold for unions.

But even if there were special logic (maybe some marker trait?) that allowed Weak to coerce to slices or traits but not to other unsized types, could the same logic not also be applied to unions without extending the language?