Mini RAG Benchmark

[lea-33]

Hi @haesleinhuepf @kaabl @marabuuu ,

I created some sort of 'mini benchmark' for two different RAG approaches for the Chatbot.
20 queries, relating to our training material, were processed by both approaches in order to find the 10 best matching slides in our training material.
You can find the results PDF here.

It would be really nice, if you could take a look at the 20 examples and decide for each one, which results (i.e. which slides) match best to the corresponding query. So for each query, we should decide if approach A or B led to a better result over all.
By doing that, we can decide which approach (or if any of the two approaches) is best to use for the multimodal RAG Chatbot.

Thanks and have a great day,
Lea

[lea-33]

If you are interested, you also find Notebooks for the two different approaches here and here.

[marabuuu]

Hi @lea-33,

attached you find my answers. Note that I start counting the slides at 1 and I am just writing down the slides, that are suiting in my opinion for each approach just for my thought process and after "→" is then my final decision based on which approach had more fitting slides. As there are duplicate slides I did only count the slide once and I did not count slides that contained only a headline

For the others: maybe don't take a look as I don't want to bias you ;)

Query: How is AI used in bio image analysis?
Approach A: 5, 7
Approach B: 3, 4, 5
→ Approach B

Query: What are the benefits of using machine learning in microscopy?
Approach A: 2
Approach B: 9, 10
→ Approach B

Query: Bioimage Analysis Definition
Approach A: 1
Approach B: 1, 2, 5, 6, 7
→ Approach B

Query: Tools for bioimage analysis
Approach A: 1, 5, 6, 7, 9
Approach B: 1, 2
→ Approach A

Query: Segmentation methods
Approach A: 1, 2, 4, 7, 10
Approach B: 5, 6, 7, 8,
→ Approach A

Query: Bioinfo vs. Bioimage analysis
Approach A: /
Approach B: /
→ None, but I am not sure if we have slides that answer this question

Query: FAIR microscopy data
Approach A: 4, 10
Approach B: /
→ Approach A

Query: FAIR principles in RDM
Approach A: 1, 2, 3, 5, 10
Approach B: 2, 3, 5, 6, 7
→ Both

Query: Fiji
Approach A: 1, 2, 5, 6, 7, 8
Approach B: 1, 2, 3, 4, 5
→ Approach A

Query: Napari
Approach A: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
Approach B: 2, 5, 6, 7, 8, 9, 10
→ Approach A

Query: OMERO data storage/sharing
Approach A: 3, 4, 5, 6
Approach B: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
→ Approach B

Query: License types
Approach A: 1, 2, 5, 6, 7, 8
Approach B: 1, 3, 5, 6
→ Approach A

Query: OS for data sharing
Approach A: 1, 4, 6, 7, 8, 9
Approach B: 1, 3, 4
→ Approach A

Query: OS software in research
Approach A: 1, 2, 4, 5, 6, 9
Approach B: 1, 3, 5, 6, 7, 8
→ Both

Query: Python for bioimage analysis
Approach A: 10
Approach B: /
→ Approach A

Query: Python libraries for image processing
Approach A: 4, 5, 7
Approach B: 1, 5, 7, 8
→ Approach B

Query: Ensuring reproducible image analysis pipeline
Approach A: 2, 3
Approach B: 5, 6, 7
→ Approach B

Query: Common steps in reproducible bioimage analysis workflow
Approach A: /
Approach B: 5, 9
→ Approach B

Query: Key parts of RDM
Approach A: 1, 5, 7, 8, 9, 10
Approach B: 1, 2, 4, 6, 7, 9
→ Both

Query: How to share microscopic data
Approach A: 1, 2, 4, 6, 7, 8, 10
Approach B: /
→ Approach A

[kaabl]

Hi @lea-33,

Here are my answers. I followed the same approach as @marabuuu to keep the results directly comparable for you.

Just one note: In some cases, the text on the slides was difficult to read, so I may have missed some relevant info related to the query. If you'd like, I can take another look in case you have a version with larger slides or higher-resolution images.

Slides marked with "()" didn't fully answer the query in my view, but they could still be helpful (after some adjustments) for an overall slide set on the query topic.

Best,
Lea

[marabuuu]

I now also finished writing down my answers (see above) @lea-33

[lea-33]

Hi @marabuuu @kaabl Thanks for your results! I will evaluate them as soon as I got Roberts answers as well and put it all in a Notebook. I will link you as well, in case you are interested in the results! :)

[haesleinhuepf]

Ok, here are my ratings. I did not read the responses from the others, I just checked the format they used in advance.

Query: How is AI used in bioimage analysis?
A:
B:

Query: What are the benefits of using machine learning in microscopy?
A: 5
B:

Query: What is bioimage analysis?
A: 2, 7
B: 9, 10

Query: What tools are commonly used for image analysis in bioimaging?
A: 1, 2, 4, 8
B: 1, 2, 5, 6, 7, 9

Query: What are different segmentation methods used for microscopy data analysis?
A: 1, 9
B:

Query: What are different segmentation methods used for microscopy data analysis?
A: 1, 3, 4, 5, 7
B: 5, 6, 9,

Query: What s the difference between bioinformatics and bioimage analysis?
A:
B:

Query: How can I make my microscopy data FAIR?
A: 3, 6, 10
B:

Query: What are the FAIR principles in data management?
A: 2, 6, 9
B: 2, 5, 6, 8, 9

Query: What is Fiji and how is it used?
A: 8
B: 4

Query: What is Napari and when should I use it?
A: 1, 2, 3, 4, 5, 6, 8, 9, 10
B: 1, 2, 3, 4, 5, 6, 7, 8, 9

Query: How does OMERO help with data storage and sharing?
A: 3, 6, 8, 9, 10
B: 1, 2, 3, 4, 5, 6, 7, 8, 9

Query: What are different license types that can be used for sharing data?
A: 2, 5, 6, 7, 8, 9, 10
B: 3, 4

Query: What does open science mean for data sharing?
A: 1, 6, 7
B: 3, 7

Query: Why is open source software important in scientific research?
A: 1, 5, 8, 10
B: 1

Query: How can I use Python for bioimage analysis?
A: 4, 6, 7, 10
B: 2, 10

Query: What Python libraries are helpful for image processing?
A: 2, 7,
B: 1, 2, 7

Query: How can I ensure reproducibility in my image analysis pipeline?
A: 2, 7
B:

Query: What are common steps in a reproducible bioimage analysis workflow?
A:
B: 5, 7, 8, 9, 10

Query: What are the key parts of research data management?
A: 1, 2, 6, 7, 9, 10
B: 1, 2, 3, 4, 7, 8

Query: How can I share my microscopy data with others?
A: 1, 2, 3, 5, 7, 8, 9, 10
B:

My general impression is that Approach B sometimes selects Quiz-slides which are similar to the question asked, biased by the fact that quiz slides contain questions.

And minor feedback for the next survey: If you do not tell us which page contains approach A and approach B (and randomize the order) we might be less biased. Or bias can be minimized.

[lea-33]

Hi @haesleinhuepf , @kaabl @marabuuu !
Thanks for your time, that helped a lot!

Here you can see the final result:

Seems like approach A got the highest score overall. Scores were calculated by summing up how many times an approach was chosen to be the better one. You can find all of the code in the Notebook here.

And also thanks to @haesleinhuepf for the feedback!

Best,
Lea