Merged
Conversation
added 12 commits
March 12, 2026 05:52
…ash + GeoIP per day
Schema: visitor_daily (id, day, ip_hash, country_code, route_group, tier,
api_key_prefix, request_count, error_count, updated_at)
Privacy model:
- ip_hash = SHA-256(rawIp + YYYY-MM-DD + IP_HASH_SECRET)
- Daily-rotating salt means same IP on different days → different hash
- No long-term user tracking possible by design
- Raw IP is never persisted
Indexes: IDX_VD_DATE_HASH (day, ip_hash) for upsert lookups
IDX_VD_DATE (day) for daily aggregation queries
…+ sweep
Core logic:
record(): called per-request, increments in-memory bucket keyed by
(day | ip_hash | route_group | tier | api_key_prefix).
IP → SHA-256(ip|YYYY-MM-DD|IP_HASH_SECRET) — raw IP never leaves this method.
Country resolved via geoip-lite.lookup() → ISO 3166-1 alpha-2 or 'XX'.
flush(): runs every 60s + on module destroy. Upserts dirty buckets into
visitor_daily via orUpdate so concurrent instances don't race.
sweep(): runs every 10min. Evicts in-memory entries older than 25h to
prevent unbounded heap growth (mirrors UsageService.sweepStale pattern).
Stats queries (called by upcoming admin endpoints):
getSummary() — today / 7d / 30d unique IPs + request counts + tier breakdown
getDailyUnique()— per-day unique IPs + requests for sparkline charts
getByApi() — route_group breakdown with error rate for today
getByCountry() — country breakdown with unique IPs for today
resolveRouteGroup() exported helper maps /api/<prefix> → logical group name.
…tricsModule
MetricsInterceptor (metrics.interceptor.ts):
- Injects VisitorService alongside existing MetricsService + AnomalyDetector
- After each non-skipped request: calls visitor.record() with rawIp
(resolved from X-Forwarded-For → req.ip → socket.remoteAddress),
route path, HTTP status, tier and first-8-chars API key prefix
- Admin routes are still skipped via SKIP_METRICS decorator as before
MetricsModule (metrics.module.ts):
- VisitorDailyEntity added to TypeOrmModule.forFeature
(TypeORM synchronize=true will auto-create the table on startup)
- VisitorService added to providers and exports
All endpoints: @RequiresAdminKey() + @SkipMetrics() + @SkipAnomalyGuard()
GET /admin/stats/visitors/summary
→ today unique IPs, requests, errors, tier breakdown + 7d + 30d totals
GET /admin/stats/visitors/daily?days=30
→ per-day array { day, uniqueIps, totalRequests } — sparkline source
days param: 1–90, default 30
GET /admin/stats/visitors/by-api?day=YYYY-MM-DD
→ route group breakdown { routeGroup, uniqueIps, totalRequests, errorRate }
day param: defaults to today
GET /admin/stats/visitors/by-country?day=YYYY-MM-DD
→ country breakdown { countryCode, uniqueIps, totalRequests }
day param: defaults to today
286/286 tests pass.
…se comment .env.example: added IP_HASH_SECRET section explaining the daily-salt mechanism and how to generate a secure value (openssl rand -hex 32). Also serves as a reminder that changing the secret invalidates existing ip_hash rows (old data becomes un-relatable — rows are kept as aggregate counts only). docker-compose.yml: added inline comment on the .env volume mount listing DATA_DIR and IP_HASH_SECRET as required env vars that must be present in the server-side .env file.
…line, API/country breakdown
stats.service.ts:
New interfaces: VisitorSummary, VisitorDailyPoint, VisitorByApi,
VisitorByCountry + 4 new methods: getVisitorSummary(), getVisitorDaily(),
getVisitorByApi(), getVisitorByCountry().
visitors.component.ts / .html:
Standalone component with signals — loads all 4 endpoints on init.
Layout:
Row 1: 4 summary cards (unique IPs today/7d/30d, requests today)
Row 2: SVG sparkline (last 14 days) + tier breakdown bar chart
Row 3: By-API table (uniqueIps, requests, error%) + By-Country table
with flag emoji + ISO code (top 10)
Refresh button + privacy note (daily-rotating hash disclaimer).
flagEmoji() converts ISO 3166-1 alpha-2 to regional indicator emoji.
sparklinePoints builds a polyline from daily unique-IP counts.
app.routes.ts: /admin/visitors lazy-loads VisitorsComponent.
stats-dashboard.component.html: 👥 Visitors nav tab added.
styles.scss: .vstat-card, .vstat-panel, .sparkline, .tier-bar, .vstat-table,
.route-badge — all scoped with vstat- prefix, consistent with design tokens.
geoip-lite ships its own MaxMind GeoLite2 database as part of the npm package and requires a postinstall script to update/verify the DB files. Since the Dockerfile uses --ignore-scripts for npm ci, geoip-lite must be installed explicitly via npm install (no --ignore-scripts) in the runtime stage alongside pg and sharp.
Security issues fixed:
- Stats, API Keys and Visitors were all rendering content (or firing API
requests) before an admin key was entered. Now all three pages show a
lock screen until statsService.apiKey() is truthy.
- api-keys.component.ts: ngOnInit no longer calls load() unless a key is
already stored in sessionStorage.
- visitors.component.ts: ngOnInit no longer calls load() without a key.
AdminNavComponent (admin-nav.component.ts) — new shared component:
- Single source of truth for the key input + all three nav tabs
(📊 Stats / 🔑 API Keys / 👥 Visitors).
- Replaces the duplicated nav+key-input markup that existed independently
in stats-dashboard and api-keys templates.
- Calls svc.setApiKey() directly on input — no per-page handler needed.
- Imported by all three admin pages.
Result: entering/clearing the key on any tab immediately reflects across
all tabs via the shared StatsService signal.
286/286 backend tests pass. TypeScript clean.
1. AnomalyDetectorService — 5 Maps now bounded (anomaly-detector.service.ts)
Added OnModuleInit/OnModuleDestroy + setInterval sweep every 10 min.
New lastSeen Map tracks last-observed timestamp per IP.
sweepStale() evicts all 5 counter-Maps for IPs not seen in 20 min.
Returns evicted count (testable). Mirrors the UsageService.sweepStale()
pattern already in use. Prevents unbounded heap growth under sustained
traffic from many distinct IPs.
2. ValidationPipe — forbidNonWhitelisted: true (main.ts)
Was false: unknown query/body params silently passed through.
Now returns 400 Bad Request if a caller sends e.g. ?scalee=2 (typo).
286/286 tests still pass — no existing test sent unknown params.
3. DATA_DIR env var (docker-compose.yml)
Volume mounts /home/simon/simonapi/data → /data but DATA_DIR was never
set, so signpack files were written to /app/data/signpacks (inside the
container layer, lost on rebuild). Fixed by adding:
environment:
- DATA_DIR=/data/signpacks
Signpack files now persist correctly on the host volume.
…ignpack oracle
BUG 1 — AdminKey not enforced (CRITICAL):
@RequiresAdminKey() / @RequiresTier() decorators only set metadata.
Without a guard that reads that metadata, they were complete no-ops.
ApiKeyGuard was never registered as APP_GUARD — only via @UseGuards()
on barcode routes. MetricsController and ApiKeyAdminController were
therefore fully open to anyone.
Fix: register ApiKeyGuard as APP_GUARD in AppModule providers.
The guard reads decorator metadata via Reflector. Routes with no
decorator pass through (public). @RequiresAdminKey() now actually
enforces constant-time ADMIN_KEY comparison. 286/286 tests still pass.
BUG 2 — VisitorService flush crash (QueryFailedError):
orUpdate() tried to update 'updated_at' column, but @UpdateDateColumn
is not included in the INSERT VALUES clause — so excluded.updated_at
does not exist in Postgres ON CONFLICT DO UPDATE.
Fix: remove 'updated_at' from orUpdate column list. TypeORM's
@UpdateDateColumn is handled automatically by the ORM on save(),
not via raw QueryBuilder upserts. The flush now only updates
request_count and error_count on conflict.
BUG 3 — Signpack timing oracle:
assertToken() threw ForbiddenException (403) when token was wrong,
but NotFoundException (404) when pack didn't exist. An attacker could
distinguish the two responses to determine whether a pack ID exists.
Fix: assertToken() now always throws NotFoundException('Signpack not
found') regardless of reason — 404 in both cases.
…rget
Root cause:
ON CONFLICT DO UPDATE requires a UNIQUE or EXCLUSION constraint on the
conflict columns. @Index alone creates an ordinary index — not a unique
constraint — so Postgres rejected the upsert with:
'there is no unique or exclusion constraint matching the ON CONFLICT spec'
Entity (visitor-daily.entity.ts):
Replaced @Index('IDX_VD_DATE_HASH', [...]) with
@unique('UQ_VD_BUCKET', ['day','ipHash','routeGroup','tier','apiKeyPrefix'])
This creates a real UNIQUE constraint that Postgres can use for upserts.
Service (visitor.service.ts):
orUpdate() conflict target changed from column-list to constraint name
'UQ_VD_BUCKET'. TypeORM maps this to:
ON CONFLICT ON CONSTRAINT "UQ_VD_BUCKET" DO UPDATE ...
This is more explicit and avoids column-name vs DB-name mismatches.
Also removed stale .setParameter('now', ...) leftover.
⚠️ DEPLOY NOTE:
TypeORM synchronize=true cannot add a UNIQUE constraint to an existing
table. The visitor_daily table must be dropped once on the server so
TypeORM recreates it with the correct schema:
docker exec simonapi-backend psql $TYPEORM_URL -c 'DROP TABLE IF EXISTS visitor_daily;'
Data loss is acceptable — the table only holds rolling 30-day stats.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.