GapFinder — ML Research Gap Discovery Engine

An AI-powered engine that discovers unexplored research opportunities in Machine Learning by analyzing academic papers and open-source code.

🚀 Features

• Paper Analysis — Fetches and analyzes thousands of papers from Semantic Scholar
• Repo Analysis — Discovers GitHub implementations with filtering and README analysis
• Topic Extraction — UMAP + HDBSCAN clustering to identify research topics
• Citation Network — NetworkX graph analysis with PageRank and community detection
• Gap Detection — Identifies research, implementation, and cross-domain gaps
• Gap Scoring — Weighted novelty × feasibility × impact × demand formula (0-10)
• AI Questions — LLM-powered research question generation (Claude / OpenAI)
• Premium UI — Dark theme React frontend with D3.js visualizations

📁 Project Structure

backend/           → FastAPI + SQLite + FAISS
├── app/
│   ├── main.py            # FastAPI app
│   ├── config.py          # Settings from .env
│   ├── models/            # SQLAlchemy models + Pydantic schemas
│   ├── api/routes.py      # REST API endpoints
│   ├── services/
│   │   ├── ingestion/     # Semantic Scholar + GitHub connectors
│   │   ├── analysis/      # Embeddings, topics, citations, trends
│   │   ├── gap_detection/ # Detector + scorer
│   │   └── question_gen/  # LLM question generator
│   └── tasks/pipeline.py  # Background pipeline orchestrator
frontend/          → React 18 + TypeScript + Vite
├── src/
│   ├── pages/             # Home, Processing, Results, GapDetail
│   ├── services/api.ts    # Axios API client
│   ├── types/index.ts     # TypeScript interfaces
│   └── styles/index.css   # Design system

⚡ Quick Start

1. Backend

cd backend
python -m venv venv
venv\Scripts\activate        # Windows
pip install -r requirements.txt

# Copy .env.example to .env and add your API keys
copy .env.example .env

# Start the server
uvicorn app.main:app --reload --port 8000

2. Frontend

cd frontend
npm install
npm run dev

Then open http://localhost:5173

3. API Docs

http://localhost:8000/docs (Swagger UI)

🔑 Required API Keys

Key	Purpose	Required?
GITHUB_TOKEN	GitHub repo search (5K req/hr)	Recommended
ANTHROPIC_API_KEY	Claude question generation	One of these
OPENAI_API_KEY	OpenAI question generation	One of these
SEMANTIC_SCHOLAR_API_KEY	Higher rate limits	Optional

🔌 API Endpoints

Method	Endpoint	Description
POST	/api/v1/search	Start gap analysis
GET	/api/v1/status/{id}	Poll progress
GET	/api/v1/results/{id}	Get full results
GET	/api/v1/gaps/{id}	Gap details
POST	/api/v1/export	Export (MD/JSON)

🧪 Tech Stack

• Backend: Python + FastAPI + SQLAlchemy + SQLite
• NLP: sentence-transformers + HDBSCAN + UMAP + FAISS
• Network: NetworkX + python-louvain
• Frontend: React 18 + TypeScript + Vite + D3.js
• LLM: Anthropic Claude / OpenAI GPT