trainer.evaluate of metrics is inconsistent due to batch-wise aggregation

[jhnwu3]

See #conformal-prediction channel in discord for the reference.

[Dreamwavedata]

Hi! I'm interested in working on this as a contribution. Is it still available?

[jhnwu3]

Yeah, happy to take any help @Dreamwavedata

[Dreamwavedata]

@jhnwu3
Thank you! Is there more information on this bug?

[jhnwu3]

Hey @Dreamwavedata sorry for the late response, it'd be easier to chat on discord haha about this. We do have some scripts, mostly with our EEG dataset. But, tldr; if you try to benchmark a model and a dataset repeatedly (even a small one), despite setting a seed, sometimes the output metrics come out different. @Jathurshan0330 can probably better describe it.