Crowdscope

Turn raw market data into actionable intelligence

---

The Problem

Marketing teams spend weeks manually collecting competitor data from Instagram, TikTok, YouTube, and Reddit. The data ends up in disconnected spreadsheets. By the time insights are ready, the market has moved.

Crowdscope automates the entire pipeline — from scraping to AI-powered insight generation — and delivers a single dashboard with real-time competitive intelligence.

Run It in 30 Seconds

# Clone and start
git clone https://github.com/dsantoreis/crowdscope.git
cd crowdscope
cp .env.example .env

# Option A: Docker (recommended)
docker compose up --build

# Option B: Manual
cd backend && pip install -r requirements.txt && uvicorn app.main:app --port 8000 &
cd ../frontend && npm install && NEXT_PUBLIC_API_URL=http://localhost:8000 npm run dev

Open http://localhost:3000 — the dashboard loads with realistic demo data (no API keys needed).

What You Get

Market Overview Dashboard

Live metrics across your competitive landscape:

• Audience growth: +12.4% (30-day trend)
• Engagement rate: 4.8% (cross-platform average)
• Sentiment split: 68% positive / 21% neutral / 11% negative
• Channel breakdown: Instagram 41% · TikTok 29% · YouTube 18% · Reddit 12%

3-Round Enrichment Pipeline

Each company goes through progressive analysis:

Round	What Happens	Output
1. Collection	Scrape posts + comments across platforms	347 posts, 12.8K comments
2. Profile Enrichment	Deep-dive into commenter profiles	7,456 profiles enriched (90.7% rate)
3. AI Classification	Demographics, segments, competitor overlap	Audience segments + strategic insights

AI-Generated Insights (Claude Opus 4.6)

Real output from the enrichment pipeline:

"Sustainability content drives 3.2x more saves than product posts" "TikTok ROAS potential: high — 42% of audience is 18-24 with strong intent signals" "Recommend: partner with micro-influencers in the 'morning routine' niche (est. 2.1x engagement lift)"

Use Cases

Use Case	What Crowdscope Does
Competitor Analysis	Track 5+ competitors in real-time. See content strategy, audience overlap, share of voice.
Lead Scoring	Score prospects by engagement signals and audience fit across social platforms.
Market Sizing	Quantify total addressable audience by demographics, location, and intent signals.
Influencer Vetting	Analyze actual audience composition — not follower counts. Detect fake engagement.
Content Strategy	Know what resonates: themes, formats, posting times — backed by 107K+ comments analyzed.

Architecture

┌──────────────────────────────────────────────────┐
│                  Data Collection                  │
│  Playwright Worker Pool (8 concurrent sessions)   │
│  Rate-limit aware · Proxy rotation · Anti-detect  │
├──────────────────────────────────────────────────┤
│                 Processing Pipeline               │
│  Round 1: Posts + Comments                        │
│  Round 2: Commenter Profile Deep-Dive             │
│  Round 3: Hyper-Classification (AI)               │
├──────────────────────────────────────────────────┤
│              Supabase (PostgreSQL)                │
│  61K persons · 107K comments · 1.3K posts         │
│  Full-text search · Real-time subscriptions       │
├──────────────────────────────────────────────────┤
│              Intelligence Dashboard               │
│  Next.js 14 + FastAPI · Real-time metrics         │
│  Audience segments · Competitor matrix            │
│  AI-generated insights (Claude Opus 4.6)          │
└──────────────────────────────────────────────────┘

Production Scale

Metric	Numbers
Profiles Analyzed	61,159
Comments Processed	107,443
Posts Tracked	1,349
Data Points	160,000+
Enrichment Pipeline	3 rounds, 90.7% enrichment rate
AI Model	Claude Opus 4.6

Tech Stack

API Endpoints

Method	Path	Description
GET	/health	Service status + mode (demo/live)
GET	/api/market-metrics	Headline KPIs for dashboard
GET	/api/market-overview	Full market snapshot with trends
GET	/api/companies	All tracked companies
GET	/api/companies/{id}	Single company profile
GET	/api/companies/{id}/enrichment	3-round enrichment results + AI insights

Testing

# Backend (coverage gate: 80%+)
PYTHONPATH=backend pytest -q backend/tests

# Frontend build
cd frontend && npm run build

Project Structure

crowdscope/
├── backend/           # FastAPI + Python
│   ├── app/main.py    # API routes
│   ├── app/demo_data.py  # Seed data for demo mode
│   └── tests/         # pytest suite
├── frontend/          # Next.js 14 + TypeScript
├── docs-site/         # Astro Starlight documentation
├── k8s/               # Kubernetes deployment
├── docker-compose.yml # One-command local setup
└── .env.example       # All configuration vars

More

• DEMO.md — Full product walkthrough
• CHANGELOG.md — Release history
• docs-site/ — Astro Starlight documentation

About

Built by Daniel Reis — AI Engineer & Data Architect, Zurich.

License

Proprietary. Architecture and aggregated metrics shared for portfolio purposes. Raw data and scraping logic are private.