mjhjmtu

Various tools and mechanisms exist for tracing the execution of a process:

• 
  strace(ptrace)/DTrace
  


• 
  dnotify/inotify/FAM
  


• LD_PRELOAD

Through the use of one or more of these, it should be possible to detect when a shell script starts a process, and it should be possible to record the inputs and outputs of that process on the filesystem.

It should be technically straightforward to instrument a common command shell, such as Bash, so that inputs and outputs of each command are recorded in a database somewhere, and so that command execution can be avoided when it can be ascertained that the inputs have not changed and the outputs are still around somewhere.

A strict implementation of this idea would suffer from making everything out of date whenever there is a system upgrade, for example, that touches Libc - or whenever a commonly-read file like /etc/passwd changes. However, by restricting the set of inputs we care about to the source files in a project tree, we could have something very similar to Make and other build systems. The result, however, would have several advantages over rule-based systems like Make:

• 
  more concise - you only have to name the inputs and outputs of a command once, as part of the command itself, or as part of other input files (as with e.g. C header files)


• 
  more flexible - ability to use the build system as an interactive shell; possibility to track inputs or outputs being renamed


• 
  more correct - files will be rebuilt if the build specification changes; "out-of-date"-ness applies to commands+files rather than just files; not worrying about "recursive Make" issues


• 
  more extensible - this would be a better framework on which to build a caching facility, for example, to share computation when building two different checkouts of the same repository

Does the system I'm looking for exist? What are the obstacles to implementing and promoting such a system?

These are questions that I ask myself when I look at various build system options, particularly for data-processing applications where command execution can be very slow and dependency structures can be ad-hoc. Usually I choose GNU Make, but I wonder why something simpler and better hasn't come around.