Skip to content

Use backfill feature to test RPC to discover issues similar to the RPC SQLite error incident #613

@cjonas9

Description

@cjonas9

What

Create a dev box that runs RPC backfill over full history from the genesis ledger to the current tip ledger.

Why

This should be done in the wake of the recent RPC SQLite error incident to discover similar issues.

On 02-20-2026, RPC nodes went down due to ledger 61322488 containing over 3900 events, triggering a "too many variables" SQLite error. Part of the difficulty in reproducing this was that one could not reingest the problem ledger through conventional RPC startup since it had already occurred. In remediation of this, RPC backfill was used to locally reproduce the error caused by ingesting ledger 61322488 as it uses the same ingestion flow as live ingestion and was thus able to fault on the same ledger as the one causing the outage.

In investigating this, it was discovered that in 2024, there were transactions containing as many as 11k events (though these were never processed by any RPC). Had these been ingested by an RPC, this would've been discovered before the incident occurred. Therefore, we should test backfill ingestion on full history to discover if any other bugs could cause similar issues.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status

    To Do

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions