[GRAPH-006] Dynamic Links / Live Views

[jtnelson]

Phase: 6

Summary

Add dynamic links (also called "live links" or "computed relationships") that automatically maintain themselves based on a criteria expression. Unlike resolvable links (which evaluate a criteria once at insert time and then become static), dynamic links continuously reflect the current state of the database: when a record starts matching the criteria it is automatically linked; when it stops matching it is automatically unlinked.

Motivation

• Automatic graph maintenance: Today, maintaining relationships requires explicit link and unlink calls. Dynamic links automate this.
• Live views on graph structure: Materialized views of relationships that stay current without application intervention.
• Documented aspiration: The data types guide already mentions dynamic links as an "Advanced Type" but they are not implemented.
• Use cases: "Link this project to all active engineers", "Link this alert to all servers with CPU > 90%", "Link this dashboard to all records tagged 'featured'".

Concept

// Static link (existing): fixed pointer
Link.to(42)

// Resolvable link (existing): evaluated once at insert
Link.toWhere("department = 'eng'")
// Resolves to {1, 3, 7} at insert time, then static

// Dynamic link (new): continuously evaluated
DynamicLink.where("department = 'eng'")
// Currently resolves to {1, 3, 7}
// If record 9 later gets department='eng', link to 9 auto-added
// If record 3's department changes, link to 3 auto-removed

Storage Model (Recommended: Criteria + Materialized + Trigger)

Store the criteria as a special DYNAMIC_LINK marker value. Maintain materialized static links alongside it. Use write triggers to incrementally update materialized links when data changes.

// What is stored for record 1, key "team":
[DynamicLink("department = 'eng'"), @3, @7, @12]

• Reads see only the materialized links (marker is filtered out)
• Existing navigate/trace/follows work unchanged on materialized links
• Graph index tracks materialized links normally

Write Trigger System

DynamicLinkManager

New component that watches writes and maintains dynamic links:

1. Registry: Maps (record, key) to criteria for all dynamic link definitions
2. Inverted index: Maps criteria-referenced keys to dynamic link definitions (when "department" is written, look up all dynamic links referencing "department")
3. onWrite(): Called after every write, checks whether the write affects any dynamic link criteria, triggers re-evaluation

Incremental Update Logic

When key K is written to record R:
  For each dynamic link DL that references key K in its criteria:
    Re-evaluate DL's criteria
    Add links for new matches
    Remove links for stale matches

Write Amplification Mitigation

• Debounce: Batch triggers within a time window
• Async refresh: Run re-evaluation in background thread (eventually consistent)
• Cycle detection: Prevent dynamic links whose re-evaluation triggers further dynamic link updates

New API Methods

boolean dynamicLink(String key, Criteria criteria, long record);
boolean dynamicLink(String key, String ccl, long record);
boolean removeDynamicLink(String key, long record);
Map<String, String> dynamicLinks(long record);  // {key -> criteria}
void refreshDynamicLinks();  // force re-evaluation

New Value Type

// In shared.thrift Type enum
DYNAMIC_LINK = 12

The marker is invisible in select/get -- only materialized Link values are returned.

CCL Grammar Changes (ccl project)

New Tokens (4)

DYNAMIC_LINK, REMOVE_DYNAMIC_LINK, DYNAMIC_LINKS, REFRESH_DYNAMIC_LINKS

New Command Productions

dynamic_link <key> <record> where <criteria>
remove_dynamic_link <key> <record>
dynamic_links <record>
refresh_dynamic_links

New Symbol Classes (4)

DynamicLinkSymbol, RemoveDynamicLinkSymbol, DynamicLinksSymbol, RefreshDynamicLinksSymbol

CaSH Examples

cash> dynamic_link "team_members" 1 where department = "eng"
true

cash> select "team_members" from 1
{1: {team_members: [@3, @7, @12]}}

cash> add "department", "eng", 15
true

cash> select "team_members" from 1
{1: {team_members: [@3, @7, @12, @15]}}
// @15 automatically appeared!

cash> dynamic_links 1
{team_members: "department = 'eng'"}

cash> remove_dynamic_link "team_members" 1
true

Thrift API Changes

• 5 new method signatures in concourse.thrift
• New DYNAMIC_LINK = 12 in Type enum (shared.thrift)
• 4 new TCommandVerb values in data.thrift

Touch Surfaces

Layer	File(s)	Change
Thrift IDL	interface/shared.thrift	DYNAMIC_LINK type value
Thrift IDL	interface/concourse.thrift	5 new method signatures
Thrift IDL	interface/data.thrift	4 new verb values
Thrift codegen	All stubs	Regenerate
New class	DynamicLink.java (driver)	Dynamic link model
New class	DynamicLinkManager.java (server)	Registry + write trigger + refresh
Server	ConcourseServer.java	5 handler methods + startup init
Server	Engine.java	Hook DynamicLinkManager into write pipeline
Server ops	Operations.java	Atomic wrappers
Server dispatch	TCommandDispatcher.java	4 verb mappings
Server storage	Write pipeline	Call onWrite after link-watched key changes
Server storage	Read pipeline	Filter out DYNAMIC_LINK markers
Driver	Concourse.java	5 new abstract methods
Driver impl	ConcourseThriftDriver.java	Thrift implementations
Driver	Convert.java	Handle DYNAMIC_LINK type
CCL grammar	ccl/grammar/grammar.jjt	4 tokens, 4 command productions
CCL symbols	ccl/src/.../command/	4 new symbol classes
CaSH	ShellEngine.java, ApiMethodCatalog.java, CompletionService.java	Dispatch + completions
Plugin API	StatefulConcourseService.java	New methods
All drivers	Python, PHP, Ruby	New methods
Integration tests	DynamicLinkTest.java
Documentation	docs/guide/src/graph.md, docs/guide/src/data-types.md	Dynamic links section
Changelog	CHANGELOG.md	Entry under 1.0.0 (TBD)

Testing Plan

• testDynamicLinkMaterializesMatchingRecords
• testDynamicLinkAutoAddsNewMatch
• testDynamicLinkAutoRemovesStaleMatch
• testDynamicLinkMarkerNotVisibleInSelect
• testRemoveDynamicLinkClearsMaterializedLinks
• testDynamicLinksReturnsAllDefinitions
• testRefreshDynamicLinksReconciles
• testCycleDetectionPreventsCascade
• Server restart: verify dynamic links survive and re-sync
• Stress: many dynamic links watching same key, high write throughput

Transaction Semantics

Recommended initial approach: Async (non-transactional) refresh. The triggering write commits immediately; link materialization happens in a separate operation. Simpler, with a small consistency window. Transactional dynamic links can be added later.

Dependencies

• Depends on: GRAPH-001 (Follows) -- dynamic links generate regular links
• Benefits from: GRAPH-002 (Graph Index) -- materialized links are tracked by the index
• Independent of: GRAPH-003, GRAPH-004, GRAPH-005

Acceptance Criteria

• dynamicLink(key, criteria, record) creates a dynamic link definition
• Materialized links appear immediately for current matches
• New matching records automatically get linked
• Records that stop matching automatically get unlinked
• Dynamic link markers are invisible in select/get
• removeDynamicLink cleans up definition and materialized links
• dynamicLinks(record) lists all definitions
• refreshDynamicLinks() reconciles all materialized links
• Dynamic links survive server restart
• Cascading cycles are detected and prevented
• CaSH commands work
• All drivers expose the methods
• Changelog and docs updated

Risks

• Write amplification: Single write can trigger re-evaluation of multiple dynamic links
• Consistency window: Async refresh means links may briefly be stale
• Complexity budget: Touches storage engine write pipeline, introduces new system component
• Criteria complexity: Expensive criteria increase re-evaluation cost

Complexity: Very High

Full spec: docs/specs/graph/GRAPH-006-dynamic-links.md