Combining Retrieval Augmented Generation with Image Generation

This repository provides a reference implementation for combining Retrieval Augmented Generation (RAG) with image generation, a pattern we refer to as RAGE. The project demonstrates how retrieved images can be used as in-context visual references to guide image generation.

Getting Started

Clone the Repository

git clone https://github.com/Neurl-LLC/RAGE
cd RAGE

Install Dependencies

Install the required Python dependencies:

pip install -r requirements.txt

Environment Setup

Create a .env file in the root of the project and add your OpenAI API key:

OPENAI_API_KEY=sk-...

Note: Your OpenAI account must be verified to use the image generation model.

Indexing Images

Before using RAGE, images must be indexed into a vector database. This is handled by the index.py script.

The script processes all images located in the images/ directory. The repository includes example images, but you can replace them with your own.

Run the indexing step:

python index.py

After indexing completes, a new chroma_db/ directory will be created. This directory stores the vector embeddings used for retrieval.

Retrieval and Image Generation

Once the images are indexed, you can test the retrieval and generation pipeline using:

python retrieve.py

This script retrieves the most relevant images for a given prompt and uses them as context to generate a new image.

Flask RAGE App

For a more interactive experience, the repository includes a Flask-based RAGE application.

Start the app with:

python app.py

This launches a UI that allows you to enter prompts, retrieve relevant images from the database, and generate new images using those images as references.