Skip to content

feat(ledger): add v3 reconciler with Raft StatefulSet support #434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gfyrag wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feat(ledger): add v3 reconciler with Raft StatefulSet support #434

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9ca91be
feat(ledger): add v3 reconciler with Raft StatefulSet support
gfyrag Mar 25, 2026
dd00278
docs: add ledger v3 settings documentation
gfyrag Mar 25, 2026
dd2cca4
feat(ledger): add v3 integration tests and fix settings prefix
gfyrag Mar 25, 2026
24a53f1
feat(ledger): add preStop hook for proper Raft scale-down
gfyrag Mar 25, 2026
a9652ec
chore: regenerate helm RBAC template with StatefulSet permissions
gfyrag Mar 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions config/rbac/role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,18 @@ rules:
- patch
- update
- watch
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- batch
resources:
Expand Down
115 changes: 115 additions & 0 deletions docs/04-Modules/03-Ledger.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,118 @@ Available fields:
- `push-retry-period`: Retry period for failed pushes
- `sync-period`: Synchronization period
- `logs-page-size`: Number of logs per page

## Ledger v3

Ledger v3 is an architecturally different version that uses Raft consensus with Pebble embedded storage instead of PostgreSQL. When the version is `>= v3.0.0-alpha`, the operator deploys a **StatefulSet** instead of a Deployment.

### Requirements

Ledger v3 does **not** require PostgreSQL or a message broker. Storage is fully embedded (Pebble LSM).
Comment on lines +104 to +110
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Clarify version-scoped requirements to avoid contradictory guidance.

Lines 106 and 110 correctly describe v3, but the page still starts with global requirements that say PostgreSQL is required. Please add a short note in this new v3 section (or adjust headings) that the top requirements apply to v2, so readers don’t misconfigure installs.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/04-Modules/03-Ledger.md` around lines 104 - 110, Update the "Ledger v3"
section to clarify that the global "Requirements" above apply to v2 (not v3) by
adding a short note or adjusting headings; specifically modify the "Ledger v3"
heading/content to include a sentence like "Note: the top-level requirements
apply to Ledger v2 — Ledger v3 uses embedded Pebble storage and does not require
PostgreSQL or a message broker." Ensure you reference the "Ledger v3" section
and the existing "Requirements" wording so readers won't misconfigure installs.


### Architecture

The operator creates the following resources for v3:

| Resource | Purpose |
|----------|---------|
| `StatefulSet/ledger` | Raft cluster nodes with `OrderedReady` pod management |
| `Service/ledger-raft` (headless) | DNS-based peer discovery for Raft consensus |
| `Service/ledger` (ClusterIP) | Gateway-facing service, maps port 8080 to container port 9000 |
| 3 PVCs per pod | `wal`, `data`, `cold-cache` |

### Cluster Settings

```yaml
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-replicas
spec:
stacks: ["*"]
key: module.ledger.v3.replicas
value: "3"
---
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-cluster-id
spec:
stacks: ["*"]
key: module.ledger.v3.cluster-id
value: default
```

- `module.ledger.v3.replicas`: Number of Raft nodes. **Must be odd** for quorum (default: 3).
- `module.ledger.v3.cluster-id`: Raft cluster identifier (default: "default").

### Persistence Settings

Each pod gets three PVCs. Size and storage class are configurable:

```yaml
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-persistence
spec:
stacks: ["*"]
key: module.ledger.v3.persistence.wal.size
value: "5Gi"
---
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-data-size
spec:
stacks: ["*"]
key: module.ledger.v3.persistence.data.size
value: "10Gi"
---
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-cold-cache-size
spec:
stacks: ["*"]
key: module.ledger.v3.persistence.cold-cache.size
value: "10Gi"
```

| Key | Default | Description |
|-----|---------|-------------|
| `module.ledger.v3.persistence.wal.size` | 5Gi | WAL PVC size |
| `module.ledger.v3.persistence.wal.storage-class` | (cluster default) | WAL storage class |
| `module.ledger.v3.persistence.data.size` | 10Gi | Pebble data PVC size |
| `module.ledger.v3.persistence.data.storage-class` | (cluster default) | Data storage class |
| `module.ledger.v3.persistence.cold-cache.size` | 10Gi | Cold cache PVC size |
| `module.ledger.v3.persistence.cold-cache.storage-class` | (cluster default) | Cold cache storage class |

### Pebble Tunables

All Pebble settings are optional. When unset, the ledger binary defaults apply.

| Key | Example | Description |
|-----|---------|-------------|
| `module.ledger.v3.pebble.cache-size` | 1073741824 | Block cache size in bytes |
| `module.ledger.v3.pebble.memtable-size` | 268435456 | Memtable size in bytes |
| `module.ledger.v3.pebble.memtable-stop-writes-threshold` | 2 | Memtable count before stopping writes |
| `module.ledger.v3.pebble.l0-compaction-threshold` | 4 | L0 files to trigger compaction |
| `module.ledger.v3.pebble.l0-stop-writes-threshold` | 12 | L0 files before stopping writes |
| `module.ledger.v3.pebble.lbase-max-bytes` | 67108864 | L1 max size in bytes |
| `module.ledger.v3.pebble.target-file-size` | 67108864 | SST file target size |
| `module.ledger.v3.pebble.max-concurrent-compactions` | 2 | Compaction parallelism |

### Raft Tunables

All Raft settings are optional. When unset, the ledger binary defaults apply.

| Key | Example | Description |
|-----|---------|-------------|
| `module.ledger.v3.raft.snapshot-threshold` | 5000 | Log entries before snapshot |
| `module.ledger.v3.raft.election-tick` | 10 | Election timeout in ticks |
| `module.ledger.v3.raft.heartbeat-tick` | 1 | Heartbeat interval in ticks |
| `module.ledger.v3.raft.tick-interval` | 100ms | Duration of one tick |
| `module.ledger.v3.raft.max-size-per-msg` | 1048576 | Max message size in bytes |
| `module.ledger.v3.raft.max-inflight-msgs` | 256 | Max in-flight messages |
| `module.ledger.v3.raft.compaction-margin` | 1000 | Log retention after snapshot |
Loading