FWIW, I run SSDs on all of my systems, and I still have the problems I described in the article.

The general rule of thumb is that productivity is sub-linear in the growth of code base complexity. You could argue that as complexity rises, it pressures the project toward smaller source files on average, but I haven’t witnessed this, except in projects that were out of control.

More accurately, the total number of file tests (in a conventional build system) is O(N^2) over the life of the project, where N is the final project size.

Even this makes a radical assumption that a large project isn’t partitioned into a 100 shared libraries, where each shared library is built independently during active development.

You seem to have a master build fetish. In a distributed project, this is often the first build each new participant performs, which seems to cause heart palpitations in a certain type of engineer who likes to perform wild extrapolations on the basis step using strangely fabricated denominators.

You also neglected to take into account the negative exponential term due to rising system performance. True, this is linked to the nearly constant physical size of the drive read head over the past two decades. However, if you spin around 180 degree and examine the future, we’re on the precipice of the largest negative step function in file IO since the Rubik’s cube was invented: the transition to SSD. What kind of astrobucks does it take to develop a source code tree that won’t comfortably fit on a cheap SSD drive three years from now? At which point we’re back to negative exponential scaling in file stat times.

This will become even more clear as file systems further adapt to exploit SSD (or miracle RAM) performance scaling.

There are many applications that could benefit from a derived authority with different performance characteristics than the authoritative backing store. Amortized O(1) is often possible with space overhead linear in the number of objects tracked (which generally implies a RAM based data structure).

On the flip side, you really have to put a lot of faith in the derived authority layer, it adds complexity, and it’s a bad, bad day when you discover your derived authority hasn’t been telling the truth since the last patch Monday.

Seems like a strange point in history to mount an architectural campaign on the observation that HDD head mechanisms don’t scale.

Brandon: You’re absolutely right. I will discuss that in my next post.

Mike Samuel: Thanks for the heads up! I will take a deeper look at prebake.

Sorry, everyone, about the comments. I wasn’t getting moderation e-mails so I didn’t notice them until the other day. >_<

From http://code.google.com/p/prebake/wiki/Goals

Build Time is Independent of Source Repository Size

When a build system has to stat every file in the repo, build is necessarily O(|project|). By hooking into the filesystem to get updates on files that change, prebake avoids this O(|project|) cost ; if you change one file, your build time depends only on the number of files that depend on that file and the time required to rebuild them.

When svn and make are watching the same directory for update notifications, anytime svn needs to update a large portion of the tree, make is going to be slowing down subversion to handle the incoming file / directory changed requests. The same operation will happen when a compile is writing intermediate files all over the tree (unless you compile to an out of tree location). Thus on the reverse side you are switching O(1) for the amount of time it takes to update a file for O(Tools) watching the tree for updates.

Another downside is that this only works when the daemon is able to stay running from build to build. If you need to reboot between tests, have security policies that log you out of the machine, or are tight on RAM and need to kill the listener daemon, you will quickly be back to the original slow build times.

There are other ways to solve this that could are not quite O(1) on the no-op build and VCS status side and also do not have O(Tools) running in the background consuming memory and CPU with each update notification. One example would be for file modification times to trickle up the directory chain so that individual file stat operations could be culled.

Build system wish list:
* Higher level than Make (I want to say “build this library” or “build this executable” and have it do dependency analysis for me.

* Some story for dependencies. I want to be able to either import third party libraries into my project, so I can tell people something like “check out this single tree and hit make.” At the same time, i want to be able to easily send those patches “upstream” to the original project. This requires some build system integration to handle the dependency analysis and rebuilds correctly. I’m imagining maven, but “better”?

The problem, as I see it, is that there is little commercial value in build systems, so there is little incentive for others to really do a good job of this. Everyone hacks their own system to satisfy their own needs, then moves on.

Evan

That’s the problem.

It’s just as bad as calculating fibbonaci using naive recursive algorithm.

Can it be taken farther? Eclipse auto-builds on save, and JUnit Max auto-runs the tests on save. But in a few years we’ll have twice as many cores at our disposal, and then twice again. What can we do to take advantage of that?

Instead of every save, can we kick off the chain every time the code is plausibly compilable? Can we speculate on what the next few edits will be and have the build running before the coding is done? Can we go beyond automated unit testing to automated load testing, search for refactoring opportunities, impact analysis, profiling, or even UI evaluation?

Make was invented in 1977. Build tools created today will be in their prime when make hits 40. The amount of computing power it is economical to give a developer is absurdly greater. Your analysis here makes me think that ibb is just the start of what we could get if we take a fresh look at our common tools.