Skip to content

feat(hydro_lang): add m2e/e2m external inputs for clusters in the simulator#2703

Merged
shadaj merged 1 commit intomainfrom
sandbox-6489897c-f46d-4de6-806a-d0a72f26a345
Apr 4, 2026
Merged

feat(hydro_lang): add m2e/e2m external inputs for clusters in the simulator#2703
shadaj merged 1 commit intomainfrom
sandbox-6489897c-f46d-4de6-806a-d0a72f26a345

Conversation

@shadaj
Copy link
Copy Markdown
Member

@shadaj shadaj commented Mar 26, 2026

Add support for external-to-cluster (e2m) and cluster-to-external (m2e) connections in the simulation testing framework.

Changes

  • Add e2m_source, e2m_connect, and m2e_sink methods to the Deploy trait with default todo!() implementations, and implement them for SimDeploy.
  • Update IR compilation in compile_network to handle cluster locations for ExternalInput and SendExternal nodes (previously todo!()).
  • Update the sim builder to handle cluster locations in create_external_source (using a process-level dispatch DFIR graph) and create_external_output (tagging with member ID).
  • Add SimClusterSender and SimClusterReceiver types that use a u32 LE prefix wire format for member ID routing.
  • Add Cluster::sim_input() method that returns a SimClusterSender and a per-member Stream<T>.
  • Add Stream::sim_cluster_output() for cluster streams that returns a SimClusterReceiver.
  • Add test sim_cluster_e2m_m2e verifying end-to-end cluster sim I/O.

Fixes #2690

@shadaj shadaj force-pushed the sandbox-6489897c-f46d-4de6-806a-d0a72f26a345 branch from 501ba65 to 52e4586 Compare March 26, 2026 18:22
@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages bot commented Mar 26, 2026

Deploying hydro with  Cloudflare Pages  Cloudflare Pages

Latest commit: 62bb30f
Status: ✅  Deploy successful!
Preview URL: https://71fcd60e.hydroflow.pages.dev
Branch Preview URL: https://sandbox-6489897c-f46d-4de6-8.hydroflow.pages.dev

View logs

@shadaj shadaj force-pushed the sandbox-6489897c-f46d-4de6-806a-d0a72f26a345 branch 4 times, most recently from 75dda84 to 62bb30f Compare April 2, 2026 22:52
&source_port,
format!("{}_{}", *to_external_key, *to_port_id)
),
parse_quote!(DUMMY),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be better to represent this as None one day?

Comment on lines +334 to +339
type DylibResult = (
Vec<(&'static str, Option<u32>, Dfir<'static>)>,
Vec<(&'static str, Option<u32>, Dfir<'static>)>,
Hooks<&'static str>,
InlineHooks<&'static str>,
);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

worth making an actual struct here? At least add comments per tuple field

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't because it's referenced from the generated code.

@shadaj shadaj force-pushed the sandbox-6489897c-f46d-4de6-806a-d0a72f26a345 branch from 62bb30f to ce701dc Compare April 3, 2026 23:56
@shadaj shadaj enabled auto-merge (squash) April 3, 2026 23:57
@shadaj shadaj merged commit 78e0afa into main Apr 4, 2026
18 of 19 checks passed
@shadaj shadaj deleted the sandbox-6489897c-f46d-4de6-806a-d0a72f26a345 branch April 4, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for m2e / e2m inputs in the simulator

2 participants