Graph creation needs to be more flexible

[tfarago]

In the spirit of this and consequently that I want to start a discussion on the future of dynamic graph creation. With the prospect of adding more and more complexity tofu will get out of hand. That is why we should stop abusing for what it was originally developed (a couple of scripts for executing the most used pipelines), like I just did in the commits above, and start thinking how to make our life easier and the users happier thanks to more freedom over how they make, combine and execute their graphs.
IMHO, a way to go is something like ufo-launch, with these additinal points in mind:

• Arbitrary graph shapes (not only linear) must be possible
• Node properties should be accessible to other nodes in an easy way (e.g. width and height information must be computed by tofu separately now)
• Provide also an API, not just a command line interface, which will be crucial for further GUI development, e.g. integration with vistrails
• Don't restrict ourselves only to available ufo tasks but enable their higher-level combination, e.g. the phase retrieval comes with plenty of boiler plate code which no one cares about if speed is not crucial, so I would like to use some json or however described high level task.

An example of why tofu is not the way to go is given above, in the first commit I had to do a large amount of programming to get phase retrieval to work with laminography. Yes, there is some "quasi" modularity in the preprocess module but in principle one shouldn't need to do any programming to combine tasks (except for their connection). That's why I didn't even put the phase retrieval into its "own" function, I would even discourage merging that branch (but keep it up for the upcoming beam time).

Rough example of how it could look in the future:
ufo-launch read path=projections/ tag=p ! read path=flats/ tag=f ! read path=darks/ tag=d ! flat-field-correct tag=fc ! backproject tag=bp ! write filename=slice-%04.tif tag=w ! connect p-fc f-fc d-fc fc-bp bp-w
We should make sure that whatever the implementation, we provide a Python API, then we could use the same stuff from the command line, Concert and some GUI like vistrails. Maybe we could even re-implement ufo-launch in Python, the command line manipulation and help generation would be easier, and more importantly, programmable! The config options could be used directly from the underlying task definitions, so all the hard-coded options could be gone.

[matze]

Frankly speaking, I would not overload ufo-launch. It is (and should stay) a simple C program to launch simple tasks on the command line. It was never meant for elaborate programmatic access (besides a bit of bash scripting). Though I feel your pain about not being able to specify arbitrary chains, though I have a solution (never implemented though) in my head already.

However, I agree with your concerns about modularity but still think that tofu is the wrong place to fix it and rather divert any such efforts to our higher-level Python wrapper(s) distributed with ufo-core. We could enhance them and make tofu depend on that.