NewBuildSystemDraft

Why?

Because the limitations of the current build system are, everyday, becoming less affordable to an increasing number of users:

1. it's not automatic: the user must explicitly launch a configuration related to the project, with a "REPL" installed
2. it's even less automatic than that: before a first full compilation of the project can occur, the user must edit and save some file in the project (it's a change in one of the project's resources which triggers the Builder) * This particular point is addressed by issue #7 (on Google Code) and is already working in a private branch of mine
3. when the builder is finally called, it does far more than expected: it runs through all the .clj files, and calls 'compile on them. This works, but does more than necessary. * the goal of calling 'compile was to have all those namespaces that embed genclass, etc. calls to be AOT compiled, so that their corresponding classes/interfaces are visible from the Eclipse Java Development Tools, and can then, themselves, compile gracefully! 1. all the regular functions are compiled also. But when the genclass calls overwrite the existing files on disk, recompiling regular functions implies having new classes for the functions on the disk. The disk very quickly becomes full of these artifacts. 1. even script files (without ns calls) will be compiled, which may cause weird problems for the user ! 1. for people with projects consisting of scripts, the only solution, currently, is to disable the "Build automatically" flag. This is a problem, because the flag is not project-wide but workspace-wide, and also because it is not clojure-builder specific (so eclipse will then not compile automatically java classes anymore).
4. To correctly interact with the Eclipse Java Development Tools, we cannot just compile into the output folder defined for the java source paths (that is, we cannot bind compile-path to the output folder). 1. because when we do so, the JDT does not see the AOT compiled classes. Indeed, the JDT does not consider the output folder as a place to look for classes. Because for the JDT, the output folder is just "derived" from the source folder, and it already has in memory a "model" of the classes from the java source files in the source folder.
   • the solution was to make the clojure AOT compilation happen in another folder, currently hardcoded into the clojure default: "classes/". And to indicate, in the project's dependencies, this "classes/" folder as a "folder" library. This way, the folder is taken into account by the JDT to build its "model" in memory, and it will also be included by the launchers in the runtime classpath. 1. because when we do so, it appeared to me (but maybe I'm wrong, say I'm confident 90% on this item) that the JDT considers the directory to be its property. It is capable of deleting from a directory not just the previous java classes, but to just delete everything from the directory. This causes the risk of deleting our clojure artifacts happening to be in the same package as java artifacts. 1. this works "almost correctly", but not with runtime environments which expect things to be placed in well known places. For example there's a problem with webapps when running in "inplace" mode: The Eclipse IDE is configured to compile the classes in the WEB-INF/classes folder (the output folder is set to "WEB-INF/classes"). But then, we cannot make WEB-INF/classes the target of compile-path ! If we did so, there will be the same problem again: our classes will not be seen by the IDE for interactive development, for example.
5. currently, launching a project with a REPL is limited to regular java projects. The "clojure launch configuration" is a highjack of the "java launch configuration". So for example it's not currently possible to use the same technique with "PDE Wlaunch configurations" (which are configurations for launching eclipse plugins for tests). Same for all webapp related stuff (not tested, but I'd be surprised if it worked out-of-the-box with WebDevelopmentTools, GoogleEclipsePlugin, etc.).
6. currently, users wanting to have more than one REPL open on a running project's JVM must launch several separate configurations, thus several separate JVMs. While sometimes it's what users may want, generally, they would rather prefer be able to create just a new REPL for the same JVM.
7. When working with several open projects in the workspace, where the projects depend on each other, we must ensure that saving something in a project which is depended upon by another project triggers a recompilation of the depended upon project. I guess this is not the case currently. 1. but if the depended upon project has no REPL currently open, then what will the recompilation of the dependent projects see ? Still old versions of class files ? And since they may not see the clojure files (are the clojure files in the source folders visible from dependent projects, hmmm, not sure about this). Must take care of that, too.

How to solve these problems?

I already have some thoughts concerning some of these problems. Since it was hard to think about it globally (one day under the shower, I think I've found a solution to problem #1, and then the other day I envision that it may not be so good with regards to problem #2, ..).

So, keeping the whole list of problems and envisioned solutions in one place will help address things "as a whole". The next version may not address all the problems, but at least it will do so in a mindfull way.

BUT BEFORE I PLACE MY IDEAS HERE, let's see if there are other problems (related to this topic) which are not yet listed ...