include_dir rustdoc segfault

[Mark-Simulacrum]

https://crater-reports.s3.amazonaws.com/beta-1.32-1/beta-2018-12-05/reg/include_dir-0.2.1/log.txt

Dec 10 14:30:38.105 INFO [stderr] error: process didn't exit successfully: `rustdoc --test /source/src/lib.rs --crate-name include_dir -L dependency=/target/debug/deps -L native=/target/debug/build/backtrace-sys-53f83b3161bfc7b9/out -L dependency=/target/debug/deps --cfg 'feature="default"' --extern glob=/target/debug/deps/libglob-aa59bbe61ff26f4b.rlib --extern include_dir=/target/debug/deps/libinclude_dir-d38de47afd31d3f5.rlib --extern include_dir_impl=/target/debug/deps/libinclude_dir_impl-7bf5a142001021da.so --extern proc_macro_hack=/target/debug/deps/libproc_macro_hack-5640320867fde5d4.rlib` (signal: 11, SIGSEGV: invalid memory reference)

[euclio]

Looks like unbounded memory growth. And it's not just a problem with rustdoc, trying to build the integration test triggers the bug as well.

[alexcrichton]

Agreed, this doesn't reproduce locally for me and is likely within some sort of C library or something like that which may not handle malloc failure well

[euclio]

It reproduces for me. I don't think there's a C library involved, because it's just a proc-macro that uses proc-macro-hack. The failing test attempts to include the directory of the crate itself, and there was a recent change to how include_bytes! deals with inserting its own source file in #54517, so I wouldn't be surprised if this is legit.

[emilyalbini]

Ping @rust-lang/rustdoc, can someone look at this?

[QuietMisdreavus]

This test managed to take down my server! 😆 Based on @euclio's comment, i don't think this is a problem specific to rustdoc, since i can at least see logs up to where it tries to compile the sample. cc @michaelwoerister and @mcr431 since #54517 seems to be related.

[matthew-russo]

I'll give it a look tonight.

[matthew-russo]

@QuietMisdreavus @euclio Sorry if this is ignorant -- I haven't been helping with the compiler for that long -- but how do I reproduce this? I haven't used crater before and when I try to run the rustdoc test manually it won't compile due to unwrapping an error inside the macro but I can't see exactly where things are breaking and it seems my error is different than the one you're talking about.

[QuietMisdreavus]

@mcr431 This is the procedure i used to get the issue i did:

1. With cargo clone, download a copy of the crate include_dir. (You can also clone the repo and check out the tag v0.2.1, i'm assuming that will have the same source as the published crate.)
2. Run cargo test --doc in the downloaded copy, using the latest beta or nightly, or a locally-built compiler/rustdoc using the technique listed in CONTRIBUTING.md. This test hangs for me, consuming more and more memory until i have none left to use my system with.

I'm not totally sure what's going wrong, but i'm assuming it's in the implementation of the macros inside that crate, and an interaction with the new file-inclusion mechanism.

[matthew-russo]

@QuietMisdreavus Thanks, I was trying to manually run rustdoc with the test flags and I don't think i was properly linking all dependencies. I'll look at it today and see if I can find anything

[euclio]

FYI, if you're on Linux or Mac you can use ulimit to limit the memory available to the process so you can see the segfault without bringing down your system. I found that 2GB worked well.

[matthew-russo]

@QuietMisdreavus So I found that it works as expected when cloned from github but not when using cargo clone. The github version includes the include_dir_impl right in the source tree where with cargo clone it is an externally fetched crate. The source of include_dir is the same in both so I'm fairly certain there is a difference in the published include_dir_impl vs the local one. Will look into it a bit more tonight.