Skip to content

[BUG]-Thread-safety issue in GetRouteToSource - Collection modified during enumeration #1400

@jkdevito

Description

@jkdevito

Description

When multiple route requests or NVX tie line registrations occur concurrently (e.g., during room combiner scenario activation), a System.InvalidOperationException is thrown because the routing graph's internal collection is modified while being enumerated.

Environment

  • Essentials Version: 2.24.2
  • Platform: Crestron 4-Series
  • Trigger: Room combiner scenario with multiple rooms activating simultaneously while NVX endpoints register tie lines

Stack Trace

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at System.Collections.Generic.List1+Enumerator[T].MoveNextRare()    at System.Collections.Generic.List1+Enumerator[T].MoveNext()
at System.Linq.Enumerable+WhereListIterator1[TSource].MoveNext()    at PepperDash.Essentials.Core.Extensions.GetRouteToSource(        IRoutingInputs destination,         IRoutingOutputs source,         RoutingOutputPort outputPortToUse,         List1 alreadyCheckedDevices,
eRoutingSignalType signalType,
Int32 cycle,
RouteDescriptor routeTable,
RoutingInputPort destinationPort,
RoutingOutputPort sourcePort)
at PepperDash.Essentials.Core.Extensions.GetRouteToSource(...)
at PepperDash.Essentials.Core.Extensions.RunRouteRequest(RouteRequest request)

Log Context

Timestamps show concurrent operations causing the collision:

[14:03:26.189][EROR] Exception Running Route Request Route codec-b:auto to display-b1:hdmiIn1
[14:03:26.222][VERB] Adding rx Tie line: NvxRouter-PrimaryStream:display-d1-nvx-rx-StreamOutput --> display-d1-nvx-rx:Stream AudioVideo
[14:03:26.224][VERB] Generating rx tieLine for codec-a-camera1-nvx-rx

Steps to Reproduce

  1. Configure a room combiner with 3+ rooms (including NVX-based routing)
  2. Configure scenario activationActions that call Activate() on multiple rooms
  3. Trigger a partition state change that activates the scenario
  4. Observe exception when concurrent route requests overlap with NVX tie line registration

Expected Behavior

Route requests should complete without exception regardless of concurrent operations.

Suggested Fix

The collection being enumerated in GetRouteToSource (likely TieLines or OutputPorts) should either:

  1. Use a thread-safe collection (ConcurrentBag<T> or ConcurrentDictionary<K,V>)

  2. Create a snapshot before enumeration:

   // Instead of:
   foreach (var item in collection.Where(x => condition))
   
   // Use:
   foreach (var item in collection.ToList().Where(x => condition))
  1. Add locking around routing operations in RunRouteRequest and tie line registration

Metadata

Metadata

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions