SherLock

A Multimodal agentic system that proactively stops financial fraud by analyzing web pages for scam signals and live audio for deepfakes.

What’s included

• fraudshield/ (Chrome Extension: SherLock)
  • Real-time page analysis + risk flags
  • Warning overlays and “input protection” (blur/lock sensitive fields on risky pages)
  • Optional mobile relay alert support (pair-code + polling model)
• frauddetector/ (Python “engine”)
  • Socket.IO server that accepts call transcripts + metadata and returns a strict JSON “CONTINUE / HANG UP” decision
  • Uses a local LLM (Qwen/Qwen2.5-3B-Instruct) via Hugging Face Transformers
• fraudvoice/ (Swift / Xcode)
  • iOS/macOS-side voice components (project scaffolding and Whisper-related code)

Repository structure

GenAIFraud/
  fraudshield/                 Chrome extension (SherLock)
    manifest.json
    background/
    content/
    popup/
    server-mock/               Reference relay server
    icons/
  frauddetector/               Python model server + test client
    engine.py
    client.py
    requirements.txt
  fraudvoice/                  Swift/Xcode voice components
  package.json                 Root Node dependencies (project-wide utilities)

Quick start: Chrome extension (SherLock)

1. Open Chrome and go to:
   • chrome://extensions
2. Enable Developer mode
3. Click Load unpacked
4. Select the fraudshield/ folder

Notes:

• The extension runs on pages matching <all_urls>.
• The popup UI lives in fraudshield/popup/.
• The content script (page overlays, input scanning/locking) is fraudshield/content/content.js.
• The background orchestrator is fraudshield/background/service-worker.js.

Optional: Mobile relay (reference server)

There is a reference relay server in:

• fraudshield/server-mock/relay-server.js

Typical flow:

• A mobile app posts “alert active/inactive” to the relay server keyed by a pair code.
• The extension polls the relay server periodically and shows a banner/alert when active.

If you use the server mock, follow the instructions inside fraudshield/README.md and/or the server file itself.

Quick start: FraudDetector engine (Python)

The frauddetector/engine.py file runs a Socket.IO server that listens for analyze_call events and returns a strict JSON decision.

1) Create and activate a virtual environment (recommended)

From the repo root:

python -m venv .venv
# Windows PowerShell:
.venv\Scripts\Activate.ps1

2) Install dependencies

pip install -r frauddetector/requirements.txt

3) Run the engine

python frauddetector/engine.py

By default it listens on port 3000.

4) Run the test client (optional)

In another terminal:

python frauddetector/client.py

This sends a few example transcripts to the engine and prints the JSON responses.

Fraud detection approach (high level)

SherLock is designed to be “explainable-by-default”:

• It detects suspicious patterns and presents them as user-readable flags.
• When risk is elevated, it can shift from passive warnings to active protection by disabling/locking sensitive inputs (passwords, payment fields, etc.) until the user confirms they understand the risk.

The Python engine focuses on transcript-based fraud risk classification:

• Input: transcript text + simple metadata
• Output: strict JSON containing risk score, a command (HANG UP or CONTINUE), reasoning, and suspicious phrases

Security and privacy notes

• This repo contains components that may transmit or process sensitive data depending on configuration.
• Treat any API keys, pair codes, and transcripts as sensitive.
• Do not hardcode API keys into source files; prefer environment variables or local-only config.

Development notes

• The Chrome extension is MV3 (Manifest V3).
• If you change manifest.json, reload the extension in chrome://extensions.
• If you edit content scripts, refresh the target web page to see updates.

License

ISC (see package.json).