fix: VRSpy scraper updates for site changes#2137
Open
mvdeckard wants to merge 4 commits intoxbapps:masterfrom
Open
fix: VRSpy scraper updates for site changes#2137mvdeckard wants to merge 4 commits intoxbapps:masterfrom
mvdeckard wants to merge 4 commits intoxbapps:masterfrom
Conversation
- Fix Scrapers page not showing scrapers on slow devices due to race condition with async store loading. Changed showAllScrapers to local component state with default true instead of persisted store state. - Fix Auto-limit scraping being enabled even when scrape failed (e.g., "Studio code not found" errors). Now only enables after successful scrape. - Fix SLR URL generation producing invalid double-dash URLs. Now uses the `label` field from SLR API directly instead of regenerating URL slugs from titles, preserving legitimate double-dash URLs while avoiding invalid ones. - Disable limit scraping when user selects "Force update scenes" to allow full re-scrape, matching the behavior of "Delete scraped scenes".
- Replace N individual COUNT queries with single GROUP BY query - Use map-based lookup for HasScraper check instead of nested loop - Add index on scenes.scraper_id via migration for existing users - Add gorm index annotation for new installs Fixes blank Scrapers page issue affecting users with large scene databases (20k+ scenes) on resource-constrained systems (NAS/Docker).
- Update gallery image URLs to use direct CDN URLs without resize params (more resilient to future CDN changes) - Fix pagination to start from page 2 instead of page -1 - Add homepage and /videos to scene collection URLs - Collect all scene URLs before scraping (proper sequential behavior) - Add auto limit scraping feature
mvdeckard
force-pushed
the
fix/vrspy-scraper-updates
branch
from
b66661d to
8e063b1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary