Dear All,We are working on an OOD-Bible titled "Oodles of Odds".This is a work-in-progress and we plan to disseminate the paper this coming week.Here's the project URL:https://matthew-mcateer.github.io/oodles-of-oods/

Also, should you be using MNIST in your OOD-experiments, we have curated two 'semantically relevant' OOD datasets that you might find useful:- https://www.kaggle.com/c/Kannada-MNIST- https://github.com/Daniel-Wu/AfroMNIST

Follow up question: Do you consider a blank image to be OOD? Almost every generative model will assign extremely high logp to a blank image.

I’m assuming all of the results are asymmetric, right? I.e. training on SVHN won’t lead to high log p(x) on CIFAR, and training on MNIST won’t lead to high log p(x) on Fashion MNIST?

@Andrew - good question - I don't want to stop Balaji now but please hold on to it and ask it

Answered now :)

Yep :)

How much easier does this problem become if you consider the problem of detecting whether a *batch* of samples is iid from the training distribution, rather than an individual example?

(Maybe a question for the end)

(I guess you’re getting to it right now :)

@Andrew you seem to be good at bits/dimension compression of this talk :)

Hehe :)

When you say ensembles consistently perform the best, are all the models in the ensemble trained on the exact same training data, or different data distributions?

Can you clarify what kind of ensembles are you referring to? Are they simple MLPs, ConvNets, other common architectures? Also, when training ensembles is that simple standard training, just SGD and probably some regularisation but nothing else?

Have you tried using Anchored Ensembles (https://arxiv.org/pdf/1810.05546.pdf) in such experiments?

I think he's close to the Q&A part so let's wait till then?

Please continue if possible

Thanks for the great talk!

Thank you for having this!! This was great.