lookup service not load balancing when workshops exists in multiple clusters

[djschny]

Describe the bug

Consider the following two scenarios where the same workshop name exists in both clusters and both clusters are ideal with no active workshop sessions:

Equal Capacity

• Cluster A - portal.sessions.maximum: 30
• Cluster B - portal.sessions.maximum: 30

When requesting sessions from lookup service it continually will hand out sessions from the same cluster as opposed to load balancing between them.

Different capacity

• Cluster A - portal.sessions.maximum: 60
• Cluster B - portal.sessions.maximum: 30

When requesting sessions from lookup service it continually will hand out sessions from Cluster A instead of load balancing.

Additional information

No response

[GrahamDumpleton]

This is how the scoring system works as to where a workshop will be allocated.

Here's how the portal scoring/sorting works in sort_workshop_environments:

The code sorts workshop environments (each tied to a portal) using a composite sort key with 4 scoring functions, evaluated as a tuple in priority order (highest priority first). The sort is descending (reverse=True), so higher scores are preferred.

1. Portal availability (binary: 0 or 1)
   Returns 1 if the portal has remaining capacity (or has no capacity limit set)
   Returns 0 if the portal is at its maximum capacity
   This is the highest priority — full portals sink to the bottom
2. Environment availability (binary: 0 or 1)
   Same logic but for the specific workshop environment within the portal
   Returns 1 if the environment has remaining capacity (or no limit), 0 if full
3. Reserved sessions (tuple: (portal_remaining, env_available))
   portal_remaining: actual remaining portal capacity, or 1 if no limit is set (deliberately low to deprioritize unlimited portals)
   env_available: the number of pre-provisioned/reserved sessions ready to go (environment.available)
   Environments with more ready-to-use sessions are preferred
4. Available capacity (tuple: (portal_remaining, env_remaining))
   Same portal remaining logic as above
   env_remaining: actual remaining environment capacity, or 1 if no limit
   Environments with more total remaining capacity are preferred

Key design choices

• Binary checks first: the first two scores act as hard filters — environments/portals at capacity get score 0 and sort to the bottom, regardless of other scores
• Unlimited = low priority: when a portal or environment has no capacity limit, it's treated as having only 1 spot. This deliberately favors portals with defined capacity limits, since unlimited ones are treated as a fallback
• Reserved sessions before raw capacity: pre-provisioned sessions (score 3) are prioritized over just having raw remaining capacity (score 4), meaning faster session startup is preferred
• Sorted list, then first-fit: after sorting, the code iterates through and tries to allocate on each environment in order, returning the first successful allocation

What is the full training portal definition, ie., defaults for reserved capacity, or reserved capacity for specific workshops.

It will favour environments which have reserved sessions available so things start quicker.

So are your training portals setup the same way, or is one perhaps not setting reserved sessions.

Also, it is actual reserved sessions available at the time. This may cause one undesirable behaviour I can think of. If there is a common pool for reserved sessions across multiple workshops on the one training portal and one workshop was very popular such that reserved sessions in the common pool got biased in their behaviour at one point, then the way the algorithm works it may steer allocation to the other cluster because it has more actual reserved sessions. I would have to think about how to address this, maybe by introducing a bit of randomness if can detect this situation so that will try to bleed some allocations to the other cluster, so it can start reclaiming reserved pool slots and so rebalance things over time.

Anyway, need to see how reserved sessions are configured and session maximum isn't enough to understand the reason.

[GrahamDumpleton]

So having a look at this, seems like a number of changes are needed.

• Fair-share stealing - In the training portal, when freeing capacity, steal from the environment with the highest available/reserved fill ratio instead of the oldest session globally. Thus tries to maintain balance between workshops for reserved sessions.

• Age-based rebalancing - In the training portal, new reconciliation step that detects stale sessions (>10 min idle) on over-represented environments and gradually reclaims 1 per 15s cycle. Thus correcting skew on idle portals.

• Log-dampened scoring - Compress reserved counts via log2 in the lookup service sort. Reduces cross-cluster routing skew but attempting balance across cluster earlier.