RFC Discussion: Subgraph

[webfiltered]

This is the discussion thread for RFC PR #15.

Please provide feedback and discuss the RFC here rather than in the PR comments.

PR Link: #15

[JorgeR81]

Design requirements

• Can open/edit subgraph instance or template in its own tab
• Can also open/edit subgraph in-place on a workflow, by double-click or similar

So will it be possible to "expand" the subgraph directly into the main graph ( like we do with group nodes ) ?

That would be great !

[webfiltered]

Similar - the current graph will remain unchanged. When the subgraph is opened, it will be displayed instead of the parent graph, with a close button to return, and breadcrumb navigation to exit multiple nested subgraphs at once.

[set-soft]

Reading the proposed implementation a couple of details came to my mind:

1. Allow more than one input/output blocks. So you can separate lines and get a clearer graph. All input nodes will be like just one, same for outputs. Just to avoid spaghetti.
2. Allow renaming the exported widgets, i.e. allow mapping the "sigma" in the example to something else, lets say "strength". In this way technical parameters can become user friendly.

[bezo97]

Agree with 2nd point that naming subgraph inputs and outputs would bring value. In Blender this is solved with a sidebar editor like this:

[webfiltered]

1. Allow more than one input/output blocks. So you can separate lines and get a clearer graph. All input nodes will be like just one, same for outputs. Just to avoid spaghetti.

Reducing spaghetti is a noble goal, however I feel that a subgraph-specific workaround is not the right approach. Anything implemented here could not be utilised to clean up spaghetti in a parent graph, for example.

Does it make sense to split regular nodes up in the same way? Could the complexity instead be reduced by using a nested subgraph?

2. Allow renaming the exported widgets, i.e. allow mapping the "sigma" in the example to something else, lets say "strength". In this way technical parameters can become user friendly.

Renaming inputs and outputs will be possible - it will use the same label / i18n API as node inputs and outputs. I'll note it in the RFC.

[set-soft]

I mean this: If we have the left case allow the right case

[webfiltered]

Excellent - that lines up perfectly with what I was imagining.

So at present, you cannot split any normal node into multiple chunks of inputs and outputs. I think that this is expected behaviour, and that normal nodes should not get a feature that breaks them up.

Assuming you agree, does it still make sense to treat the subgraph input nodes as a special case, here? Is this a feature that you think should be in the RFC, or something that could be added later as an optional feature?

[bezo97]

Individual node widgets can be exposed directly as widgets in the parent graph
Highlighted / visually flagged in some fashion when viewed in the subgraph

I suggest not differentiating these apart from simple inputs to the subgraph, especially considering the "Widget Input Socket" RFC. Then the subgraph node can simply display it as a widget or input.

[webfiltered]

If we add no special support for widgets at all, and already have widget sockets merged as-is, this already feels decent:

• Widget sockets work the same everywhere (I think this is important)
• image and sigmas are both subgraph inputs with no special anything, only differences are name and type (IMAGE, FLOAT).
• On the main graph, they are shown as inputs (no widgets)

The issue is the just that no widgets show on main graph.

I actually think I have a way to handle both my requirement for simpler UX / automating, and your requirement to allow customisation. I am not certain it will go into initial launch yet; will find some time to write it up.

[JorgeR81]

Anyway, regarding my original comment I still think that the "highlighted widget" mechanic is complicating things and it would be better to treat them as subgraph inputs like this:

• image and sigmas are both subgraph inputs with no special anything, only differences are name and type (IMAGE, FLOAT).
• On the main graph, they are shown as inputs (no widgets)

If we create this configuration in the subgraph ( e.g. connecting the "sigma" widget to the input node ), it would make more sense if "sigma" shows up as a widget in the main graph ( instead of an input ).

Then, we can use it as an input, if we want to, the same way we would do with normal widgets.
This way we would keep some of the advantages of the "highlighted widgets", without the extra GUI complications. 

Keep in mind that, if we create many inputs, then we need extra nodes to connect to these inputs, and the subgraph would be less useful for reducing complexity in the main graph.

---

We might want to drop the support for group node if we have subgraph planned on the horizon. In case it's too much effort to fix the group node.

Comfy-Org/ComfyUI_frontend#2240 (comment)

Having widgets in the main graph, would also allow replacing group nodes, which may have compatibility issues with new features.

[webfiltered]

You know you quoted two different people and replied to them as a single block, right? I know -I'm- confused by it. 😄

I'm assuming you quoted me because it would show up in the quoted text, where the image would not. If you are actually replying to me, please let me know, and I'll try to clear up the missing context.

[JorgeR81]

OK, let me clarify:

• I like the "highlighted widgets" concept. I would like to have them in the subgraph implementation.
• If this brings too much complexity, and it's not possible to implement, I wouldn't mind the alternative shown in @bezo97's image ( e.g. connecting the "sigma" widget to the input node, in the subgraph ).
• But in this case, I would like the "sigma" to be shown as a widget, by default, in the main graph ( instead of an input ).

[webfiltered]

Thanks - appreciate the clarification, and the feedback!

[enternalsaga]

There're few inspirational custom nodes I feel Subgraph can adapt:

• https://github.com/numz/Comfyui-FlowChain: create your chain of nodes seperately and can call to your main workflow. its downside is that it was based on API so the saved api workflow would be broken when open inside; and it soooo labourious to collect and change every single input/output node if my chain contains many nodes.
• https://github.com/Nuked88/ComfyUI-N-Sidebar?tab=readme-ov-file#-templates-support- : my top QoL I intent to keep using forever. Can save part of my nodes in one click and super handy when I need to call out everywhere because the template was saved in right-click menu.

Subgraphs can adopt these 2 nodes features to be easier to adjust either in or outside of main workflow, even better to turn N-sidebar's template as native feature, given the case the dev has stopped updating the repo for so long.

[webfiltered]

Apologies, but could you please be a little more specific? I am not sure exactly what it is you are asking for.

[enternalsaga]

sorry for my bad English, let me outline what i think can be adopted:

• FlowChain's "Continue" node for conditional workflow execution.
  
  
• N-Sidebar's one-click template saving via right-click context menu
  
• N-Sidebar's quick template instantiation with a single click
• N-Sidebar's pop-up modal box showing "USED NODES"
• N-sidebar's quick load template via right-click context menu
  
• N-Sidebar's robust and user friendly visual library
• Can we design our own subgraph UI to fit our need ( for example: rgthree's power lora loader, combo, slider node,etc.) ?
• Can subgraph nest subgraphs?

[webfiltered]

I would recommend having a look at the full RFC. I am not sure how well it translates, however most of these questions have already been answered.

• FlowChain's Continue: This looks like partial graph execution - e.g. execute only a subgraph, or execute to the end of a subgraph, but nothing after it. This is not in the RFC, but there is nothing that prevents this feature from being added to ComfyUI after subgraphs are finished. Do you believe that this feature is a requirement to include in the first version?
• N-Sidebar's templates: Save Selected as Template and Node Templates in the context menu are actually part of ComfyUI. This is unrelated.
• N-Sidebar's library: The workflow sidebar tab will be improved for use with subgraphs (which are also workflows) - see RFC for details.
• Inputs and outputs can be changed - see RFC for details.
• Subgraphs can be nested - see RFC for details.

[JorgeR81]

N-Sidebar's one-click template saving via right-click context menu

There is an issue to integrate templates in the native sidebar:

• [Feature Request]: Manage node templates in workflow sidebar tab ComfyUI_frontend#1801

[guill]

FlowChain's Continue: This looks like partial graph execution - e.g. execute only a subgraph, or execute to the end of a subgraph, but nothing after it. This is not in the RFC, but there is nothing that prevents this feature from being added to ComfyUI after subgraphs are finished. Do you believe that this feature is a requirement to include in the first version?

I think it's worth calling out that partial graph execution is already possible using custom nodes like ComfyUI-Impact-Pack via the Switch (Any) node.

The addition of subgraphs definitely raises the value of getting a 'Switch' node added to the core ComfyUI, but IMO it shouldn't be considered a necessary prerequisite given the availability of other solutions from custom nodes.

[guill]

Super excited about this proposal! I won't comment on the frontend stuff (since there's already a lot of great conversation on that), but I would urge you to consider an implementation in which the backend resolves subgraphs rather than the frontend attempting to flatten graphs prior to submission. 99% of the functionality necessary for this is already implemented on the backend in the form of node expansion (unless there are existing bugs with it that I haven't heard about), and I'd be happy to implement whatever little might be missing (as long as I'm not signing up to do the frontend work as well).

There are a couple reasons I would urge this direction:

1. The risk of combinatorial explosion during flattening which can be mitigated by lazy evaluation on the backend. Once reusable subgraphs become possible (and especially once they're easy to share), lazy evaluation becomes significantly more useful -- whether that's a subgraph that takes different actions depending on the requested size of the output or a subgraph with a toggle for "upscale using Flux". Frequently, this comes in the form of duplicate branches like this:

flowchart LR
  Input --> TextToImage1
  Input --> TextToImage2
  Input --> Conditional
  TextToImage1[TextToImage]:::em --> Conditional
  TextToImage2[TextToImage]:::em --> Upscale
  Upscale --> ImageToImage
  ImageToImage --> Conditional
  Conditional --> Output
  classDef em font-weight:bold,color:red

Loading

The problem comes from the fact that there are two TextToImage nodes which might themselves be subgraphs (even if only one is ever executed). If ComfyUI ever develops a healthy ecosystem of subgraphs that people are sharing and reusing, it will only take a few levels of subraphs for the 'flattened' graph to contain tens of thousands of nodes that don't actually end up executing.

2. Uses existing support for subgraph caching. With node expansion, we can avoid expanding a subgraph at all if none of its inputs have changed (and it contains no output nodes within) -- this is already implemented. Combined with the combinatorial explosion risk mentioned above, this could be a meaningful performance improvement when iterating on complex workflows.

3. Maintaining full support for alternative frontends. I know this is something I harp on a lot that isn't really important to most people, but I think there's a lot of potential in ComfyUI's future as part of automated pipelines -- and that that potential is greatly diminished the moment we make functionality dependent on a browser running the TypeScript UI. The more logic we make people replicate (and keep up to date) themselves, the more ComfyUI becomes a prototyping and recreational tool with more limited back-end potential.

4. It'll probably be way easier to implement. You'll already be storing the subgraphs as independent graphs for workflow storage purposes -- why not just submit them to the backend like that? Again, I'm pretty sure the backend work for the actual execution would be minimal as long as it leverages the existing node expansion functionality.

[webfiltered]

Amazing; thank you! I have blocked out some time to look into these improvements properly. I had thought about tying lazy eval into the work here, and was hoping you'd pop in at some point. Consider me urged. 😛

Purely by way of explanation: the RFC being designed this way is primarily to try to keep the scope manageable, but equally important, designed in a way that does not block things like your recommendations. I don't want a barely-useable MVP, but it is a significant change no matter how you slice it, so have tried to go lean where possible.

I definitely do not disagree with point 3, but just want to flag that as-written, the subgraph flattening code is fairly straightforward - everything is already designed to work with a single graph. It would really not take much to replicate that part. For all frontends, the vast majority of effort will be in UI and schema.

We can inject the (un)flattening as middleware and almost be done. Overall, it may even be less effort. The hardest part of all this will be extension compatibility (but there's no fix there).

[guill]

That's fair -- saying it would be easier to implement in the back-end is definitely colored by my own relative inexperience in the front-end and how difficult it would be for me to implement 😅
Even if we did flatten the subgraph on the frontend, it also probably wouldn't be difficult to implement the same thing in Python and provide that as an optional endpoint for alternative front-ends if it became a problem. I suppose the biggest risk on that end would be the temptation of custom nodes to modify the graph after flattening on the front-end.
Nevertheless, I still think the first two items are compelling reasons to consider doing subgraph expansion on the back-end. Because the node expansion functionality has already been merged for some time, I don't think making use of it would expand the scope of this RFC or its implementation by any meaningful amount.

[SirVeggie]

A slightly related comment I have is that when expanding the nodes whether in frontend or backend, we should make sure all inputs are treated separately, aka the inputs shouldn't expand into an "input node". The sub-nodes extension also uses an input and output node, and since they are treated as real nodes, any dirty input will cause all other inputs to get dirty.

In the above case both nodes are completely separate, but with sub-nodes, even if only one input was changed, both nodes will get re-processed since they share the same input node, which is "dirty" causing all outputs the also be dirty.

[SirVeggie]

The current Group Nodes handles this correctly, but I wanted to mention it just in case.

[guill]

This is a really good point. It should be fairly easy to do whether expansion is on the frontend or backend though. We'll just want to make sure we have tests for this behavior.

[GeminiSquishGames]

I'd love to have it but I have questions. Wouldn't unifying the functionality of Groups and Group Nodes be better than adding something? Or does this aim to replace Group Nodes and Groups with something better? Can we not just have Group Nodes that work better and consolidate the Group and Group Nodes since they work more accurately to the current workflow and UI in the frontend?

I really like how Blender 3D handles things and this seems basically like that, but I like the idea of refactoring what is there and not bloating things with more. (The group node manager needs QoL attention but probably the only overhaul needed in my mind, but I really don't know if we are adding or replacing, and I need to familiarize myself with the source)

[webfiltered]

Just basically repeating another reply here, but for anyone else arriving here:

Group nodes will eventually be removed. Subgraphs will entirely supersede them. Litegraph groups are staying, and will remain unrelated to this.

[PrometheusDante]

The main QoL issues of group nodes will be remedied by subgraphs not requiring an editor to begin with 😅

If I understand correctly, it is like preparing a group node and instead of converting it—jumping through all the burning hoops within the editor—you instead just compact it away into a pocket dimension, within which it can be edited like any other part of the workflow, while also allowing going multiple levels deep.

And that would really make every aspect of the concept easier, ultimately not adding anything, but rather taking away something no longer needed. I am very excited for this, as all my workflows rely heavily on group nodes to reduce the complexity for new users as much as possible.

[nistvan86]

Assuming you'd like to have some kind of widget editor for the node representing the subgraph, please consider making it possible to add arbitrary separators. The current grouping just throws in everything into a single list. Making things worse you cannot rename neither the input/output pads, nor the widgets. So if you use internally multiple nodes of the same type, it becomes a guessing game what's exposed on the group node. Even better would be some kind of collapsible groups for the widgets, so we can structure them more logically.

[PrometheusDante]

Separators for the subgraph node on the parent workflow would be a nice feature to have.
Something as simple as an empty line or a subtle line like ↓

---

About telling apart multiple of the same node type, there are various ways to rename those already, either by naming the in/output before converting it to a group node or by manually naming the respective field within the group node editor.

How I currently build separators into my group nodes might be a little unorthodox, but allows for using the Impact Pack's "Set Widget Value" to send color values, which allows using it as a highlighter of sorts as well.

Group Node Example

[nistvan86]

Yikes, that hack is really not for the faint heart :) But yeah, something like the right box on the image is what I'm after.

[JorgeR81]

I like how the separator works in the Trung0246 nodes ( https://github.com/Trung0246/ComfyUI-0246 ).

We can add multiple nodes to the Hub node, to duplicate all widgets in one place.

The separator it's added automatically, it shows the node's ID number and it's even color coded.

---

[tresorama]

It may sound stupid but...

If GroupNode (current implementation) could toggle between expanded and collapsed state (ui state) we could have a UX similar to this rfc.

1. Is not practical to implement? It's messy? impossible by design ? breaks every existing workflows if shipped?

(I don't know nothing about the source of comfy, sorry...)

If this is practical, the next think that comes to my mind is that, The user should not be able to make a socket connection between a node that is part of the main workflow and a node that is a child of the GroupNode, while the GroupNode is in expanded state. This can be tempting due to the fact the nodes are near , so maybe a visual indication that the GroupNode "wraps" its own nodes should be visible even in expanded state to give feedback to the user.

[webfiltered]

The total effort required to repurpose GroupNode would be greater than starting from scratch.

[tresorama]

Gotcha! Thanks.

[tresorama]

Other softwares that resolved this problem well:

• Blender , as already mentioned
  • https://docs.blender.org/manual/en/latest/interface/controls/nodes/groups.html
  • https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.youtube.com/watch%3Fv%3D_GVrUtmbZeY&ved=2ahUKEwic-OqA1KCMAxViVeUKHQEKIYMQwqsBegQIDBAG&usg=AOvVaw2aiXVYNn2NH8xFpQYmWXfL
• n8n, it's a worklow automation no-code editor that uses nodes as main ui.
  • https://youtu.be/xr05Ie_Hkyg?si=YgRsC7-kBBmDIfzx&t=246

---

n8n overview

It's used in marketing automation, "when a user fill a form, the frontend of the form call a n8n workflow that is a webhook, then a pipeline of operation is triggered, like saving the form in Google Sheets, send an email, call LLM to produce something based on form data, and so on..."

• What we call workflow in comfy is called workflow even in n8n
• A workfkow consists of a pipeline of nodes, linked by socket connection (the UI for the connection is different but not important for this rfc purpose, but i assume the implemntation of a node differs from comfy)
• nodes are divided in two types: Trigger, and Action
• a workflow requires a trigger node as first node, otherwise it does nothing (in comfy there is no concept of trigger, but the first node of this ubgraph rfc, InputNode can be seen as the trigger of the subgraph)

- after the trigger, you pipe multiple action nodes - a workflow can have multiple triggers (but at least one is required)

n8n implementation of subgraph

in any workflow you can add a trigger node, called When Executed by Another Workflow, that execute the workflow when an other workflow use an action node Execute sub-workflow

• sub-workflow:
  
• parent workflow:
  
• in the sub-workflow, inside When Executed by Another Workflow node (the input node) you define the input api
  
• what the sub-workflow returns at the end is considered the output, that will be available in the parent workflow as the output of Execute sub-workflow

other behavior

This behavior is not correlated (or useful) to this rfc, but it's good to see it, just to be aware of.

• because a workflow can have multiple triggers, a workflow can invoke itself as sub-workflow

[webfiltered]

I would recommend having a look at the full RFC.

[christian-byrne]

Marking this RFC as active, as development is under way. Please see the associated PRs in https://github.com/Comfy-Org/ComfyUI_frontend and https://github.com/Comfy-Org/litegraph.js to keep up with the progress.