feat: Measure execution status duration

[morgan-wowk]

Changes

• Adds status_updated_at column to ExecutionNode table to track when execution status last changed
• Implements SQLAlchemy event listener to automatically update status_updated_at timestamp when container_execution_status changes
• Creates execution_status_transition_duration histogram metric to measure time spent in each execution status
• Adds _transition_execution_status() helper function to centralize status updates and metric recording across all status transitions
• Implements database migration logic to add the new status_updated_at column to existing tables
• Replaces direct status assignments throughout the orchestrator with calls to the new transition helper function

Show of work

Note: Attribute names have since changed to execution_status_ prefix

Local smoke-test and verification completed ✅

[morgan-wowk]

• feat: Measure execution status duration #174 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[morgan-wowk]

Load Impact: feat: Measure execution status duration

The commit adds zero additional database calls during steady-state operation. status_updated_at is written as part of the existing UPDATE statement that already commits container_execution_status — SQLAlchemy batches both columns into one flush. Reading the previous status and timestamp before each transition is always an in-memory access from the already-loaded SQLAlchemy instance. The only one-time database cost is a single SELECT on every process restart to check for the new column, plus a single ALTER TABLE on first deployment. On the metrics side, record_status_transition appends to an in-memory histogram bucket and returns immediately; the OTel SDK exports observations to the collector in background batches, adding one additional metric stream with no synchronous I/O on the hot path.

Area	Impact
DB calls at startup	+1 SELECT per restart; +1 ALTER TABLE on first deploy only
DB calls per transition	None — status_updated_at piggybacks on the existing UPDATE
DB calls for reading previous status	None — in-memory reads from the live SQLAlchemy instance
CPU per transition	Negligible — one datetime.now() + one histogram bucket increment
Memory	Negligible — one additional datetime field per loaded ExecutionNode
Network (OTel export)	One additional histogram metric stream, exported in background batches

[yuechao-qin]

Add * in the function for keyword-only arguments, good style even with just 1 parameter.

[morgan-wowk]

Done — added * to _add_columns_if_missing.

[yuechao-qin]

Let's derive these instead of hard coding them

_COLUMN_MIGRATIONS = [
    bts.ExecutionNode.__table__.c.statys_updated_at,
]

# Example
col = bts.ExecutionNode.__table__.c.statys_updated_at
table_name = col.table.name          # "execution_node"
column_name = col.name                # "statys_updated_at"
column_type = col.type                # DateTime()

[morgan-wowk]

Done — _COLUMN_MIGRATIONS now holds bts.ExecutionNode.table.c.status_updated_at and derives table name, column name, and type from it.

[yuechao-qin]

Let's use DDL for this and try not to use text if possible. For type safety and portability, which is why we're using SQLAlchemy.

from sqlalchemy import Column, DateTime
from sqlalchemy.schema import AddColumn
col_obj = bts.ExecutionNode.__table__.c.statys_updated_at
with db_engine.connect() as conn:
    conn.execute(AddColumn("execution_node", col_obj.copy()))
    conn.commit()

[morgan-wowk]

sqlalchemy.schema.AddColumn isn't available in our SQLAlchemy version, so I kept sqlalchemy.text but compile the type string from the model via col.type.compile(dialect=db_engine.dialect) to stay dialect-portable

[yuechao-qin]

From my backfill lessons learned, i would suggestion

• add logs to know if the migration is happening or not, and any errors
• a single transaction, even if you only have 1 column, for future proofing. Else remove the for loop for now to make the code align.

Merging all my suggestions into something like this:

    _COLUMN_MIGRATIONS = [
        bts.ExecutionNode.__table__.c.statys_updated_at,
    ]
    inspector = sqlalchemy.inspect(db_engine)
    with db_engine.connect() as conn:
        for col in _COLUMN_MIGRATIONS:
            existing = {c["name"] for c in inspector.get_columns(col.table.name)}
            if col.name not in existing:
                _logger.info(
                    f"Migrating: ALTER TABLE {col.table.name} ADD COLUMN {col.name} ({col.type})"
                )
                try:
                    conn.execute(sqlalchemy.schema.AddColumn(col.table.name, col.copy()))
                except sqlalchemy.exc.OperationalError:
                    _logger.info(
                        f"Column {col.table.name}.{col.name} already exists (concurrent migration)"
                    )
            else:
                _logger.info(
                    f"Column {col.table.name}.{col.name} already exists — skipping"
                )
        conn.commit()

[morgan-wowk]

Done — adopted your suggestion almost verbatim: single with db_engine.connect() wrapping the loop, an info log before each ALTER TABLE, and a catch for OperationalError on concurrent migration

[yuechao-qin]

Curious, for my learning purposes, what does "rollback will expire ..." mean? Can you give me some examples. I'm not following the comment.

[morgan-wowk]

Expanded the docstring with a concrete explanation: rollback() marks every attribute on every loaded instance as "expired", so reading prev_status or prev_status_updated_at after a rollback would silently fire a fresh SELECT instead of returning the in-memory value.

[yuechao-qin]

this is really cool!

Curious, this should handle ALL node status transitions right? Asking because I might want to piggyback off this function in the future for doing status search in the search API.

[morgan-wowk]

Yes!

[yuechao-qin]

Can this be a strenum? so we know all the units that are possible.

[morgan-wowk]

Done — added MetricUnit(str, enum.Enum) with SECONDS = "s" and ERRORS = "{error}", used for both instruments.