-
Notifications
You must be signed in to change notification settings - Fork 50
Description
What
Create a dev box that runs RPC backfill over full history from the genesis ledger to the current tip ledger.
Why
This should be done in the wake of the recent RPC SQLite error incident to discover similar issues.
On 02-20-2026, RPC nodes went down due to ledger 61322488 containing over 3900 events, triggering a "too many variables" SQLite error. Part of the difficulty in reproducing this was that one could not reingest the problem ledger through conventional RPC startup since it had already occurred. In remediation of this, RPC backfill was used to locally reproduce the error caused by ingesting ledger 61322488 as it uses the same ingestion flow as live ingestion and was thus able to fault on the same ledger as the one causing the outage.
In investigating this, it was discovered that in 2024, there were transactions containing as many as 11k events (though these were never processed by any RPC). Had these been ingested by an RPC, this would've been discovered before the incident occurred. Therefore, we should test backfill ingestion on full history to discover if any other bugs could cause similar issues.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status