by Chase McCarthy on April 8, 2015

Broccoli is the blazingly fast build tool used by Ember.js and Ember CLI. Though as we'll see it has uses in any JavaScript project and maybe beyond that.

If you have created an Ember CLI application you are probably using Broccoli very minimally. Your app is initialized with a Brocfile.js that looks something like this:

// Brocfile.js
var EmberApp = require('ember-cli/lib/broccoli/ember-app');
var app = new EmberApp();
module.exports = app.toTree();

Typically the only interaction you will have with the Brocfile.js of an Ember CLI app is when you need to include a vendored library installed with Bower.

// Brocfile.js
var EmberApp = require('ember-cli/lib/broccoli/ember-app');
var app = new EmberApp();
app.import(app.bowerDirectory + '/moment/moment.js'); // <--- +
module.exports = app.toTree();

There are some more advanced use cases but it's likely that anything you do in your Brocfile.js should be moved to an Ember CLI addon. A great example of the wrong way to do this is the RSS feed generator currently being used by Ember Weekend. If you take a look at the Brocfile.js for that repo, you'll see a whole mess of things going on. This will eventually be moved out into an Ember CLI addon because it's functionality that can be tested and maintained outside of the main repository and would be useful in other applications.

Even the app.import(...) statements can be moved to addons. You can see this in ember-moment.

Addons use Broccoli to a somewhat greater extent than apps. In the setupPreprocessorRegistry hook you add preprocessors to handle transpiling things like CoffeeScript, ECMAScript 2015, or HTMLBars.

The preprocessor object is basically just this:

{ 
  name: '...',
  ext: '...',
  toTree: function(tree, inputPath, outputPath) {
    // build output tree
    return outputTree;
  }
}

This object is simple enough to construct so some addons simply build it in place, such as ember-cli-babel. Others extract this into a separate module, as is the case with ember-cli-coffeescript. In this case, the instance of MyCustomPreprocessor has the same properties and behavior as above:

setupPreprocessorRegistry: function(type, registry) {
  var preprocessor = new MyCustomPreprocessor();
  registry.add(type, preprocessor); // common types are 'js', 'css', and 'template'
},

Now in the toTree function of the preprocessor we see where Broccoli comes into play. For clairity and to comply with the single responsibility principle, the building of the output tree should take place in another object.

toTree: function(tree, inputPath, outputPath, options) {
  return new MyCutomTreeBuilder(tree);
}

Again this can live in the same repo like ember-cli-htmlbars does, or can be extracted to a generic Broccoli library, as broccoli-babel-transpiler does.

All of the preprocessors I've mentioned build upon a simple base prototype called broccoli-filter. As you can see, there is no implementation of read for this tree. You'll see why that is important when we discuss the tree API later. The broccoli-filter plugin handles much of the work for us, accepting simple configuration data and a processString callback to be executed for each file in the input tree.

var Filter = require('broccoli-filter');
function MyCutomTreeBuilder(inputTree, options) {
  this.inputTree = inputTree;
}
MyCutomTreeBuilder.prototype = Object.create(Filter.prototype);
MyCutomTreeBuilder.prototype.extensions = ['js'];
MyCutomTreeBuilder.prototype.targetExtension = 'js';
MyCutomTreeBuilder.prototype.processString = function (string, relativePath) { ... };

We could keep going down this rabbit hole, but I find it is easier to understand Broccoli from the bottom-up at this point. We will come back to visit broccoli-filter later.

It's worth mentioning that many other libraries such as my Haml-like template parser, hbars, use Broccoli to manage transpilation, linting, and building distributions entirely outside of the Ember ecosystem.

The Broccoli tool actually handles quite a few tasks. There are utilities that find/load the Brocfile.js, a file watcher, a server to issue live reload commands, and other related functionality. We will concentrate on just the Builder. But first...

Here is an example of a minimal Broccoli tree according to the Broccoli Plugin API Specification:

var tree = {
  read: function(readTree){
    var tmp = quickTemp.makeOrRemake(this, 'tmpDestDir');
    // or use promise-map-series
    return readTree(subtree).then(function(dir){
      // subtree has finished proccesing now
      // read from subtree if needed and write to tmp files
      return tmp;
    });
  },
  cleanup: function(){
    quickTemp.remove(this, 'tmpDestDir');
  }
};

Note: quickTemp is from node-quick-temp, and is used in various Broccoli plugins to generate temporary directories

As you can see there are only two functions, read and cleanup, defined on this object. This API allows for tremendous flexiblility. Something so simple can be easily implemented by plugin authors. The seemingly complicated broccoli-filter mentioned earlier is just an extension of these two functions.

At the core of Broccoli is the Builder. The builder is constructed with a path string or a tree:

var broccoli = require('broccoli');

var tree = broccoli.loadBrocfile();
var builder = new broccoli.Builder(tree);
// or
var builder = new broccoli.Builder('someDir');

Ember CLI creates a builder just like this.

The build command is called to kick off the process of building the output. Instead of blocking here while the output is built, a promise to an output is returned.

builder.build().then(function(output){
  // finished building
});

This output is an object containing the output directory, the node representation of the tree and the time it took to build.

var output = {
  directory: 'outputDirname'
  graph: node,
  totalTime: timeInMillis
};

The node is the top level tree and its associated metadata:

var node = {
  tree: tree,
  subtrees: [ subtreeNode ],
  selfTime: timeInMillis,
  totalTime: timeInMillis,
  directory: 'outputDirname'
};

Let's step through the process used to build this output.

The builder simply calls readAndReturnNodeFor calls read on the given tree and returns a promise to the node for that tree to be resolved once it's done being built. This is an abbreviated version:

function Builder (tree) {
  this.tree = tree;
}

Builder.prototype.build = function(willReadStringTree) {
  ...
  return RSVP.resolve()
    .then(function() {
      return readAndReturnNodeFor(tree);
    })
    .then(function(node) {
      return { directory: node.directory, graph: node, totalTime: node.totalTime };
    })
    .catch(...);
};

The readAndReturnNodeFor does exactly what is says, calling read on the given tree.

function readAndReturnNodeFor(tree){
  ...
  var node = {
    tree: tree,
    subtrees: [],
    selfTime: 0,
    totalTime: 0,
    directory: null // <-- populated once reolved
  };
  ...
  return RSVP.resolve()
    .then(function() {
      return tree.read(readTree); // <-- call `read` on input tree
    }).then(function (treeDir) {
      node.directory = treeDir;
      return node;
    });
}

If a tree consumes other trees as input it should call readTree for each inside the read function. The readTree function returns a promise that must be fulfilled before the next call to readTree. A convienient way to enure this is provided by the promise-map-series library, also by Jo Liss.

Internally, readTree actually calls readAndReturnNodeFor recursively for all subtrees of the input tree. This of course calls read for their trees, and so on.

function readTree(subtree) {
  ...
  return RSVP.resolve()
    .then(function () {
      return readAndReturnNodeFor(subtree) // <-- recursive
    })
    .then(function(childNode) {
      node.subtrees.push(childNode);
      return childNode.directory;
    })
    .finally(...);
}

Once the build is complete you call cleanup on the builder to cascade the cleanup through the subtrees. Subtrees can sometimes just be a directory name, and in that case cleanup is not necessary.

Builder.prototype.cleanup = function () {
  function cleanupTree(tree) {
    if (typeof tree !== 'string') {
      return tree.cleanup();
    }
  }

  return mapSeries(this.allTreesRead, cleanupTree);
};

The mapSeries is from promise-map-series. It will call cleanup for each of the trees in allTreesRead sequentially, calling the next only after the promise from the last one has resolved.

The tree API has recently changed in Broccoli versions 0.14.x and 0.15.x, and you may have noticed some deprecation warnings. There was an issue discussed on Github that highlighted how read was nondescript. The job of the function is to build the output of the tree and return the directory where the output is written. Jo Liss agreed and suggested that rebuild would be a more descriptive name.

While fixing the minor naming issue, the larger issue of having to sequentially call readTree was fixed. With the new rebuild syntax, input trees are easier to manage. You simply add a property called inputTree or inputTrees to your object. Broccoli will handle calling readTree internally.

var tree = {
  rebuild: function(){
    var tmp = quickTemp.makeOrRemake(this, 'tmpDestDir');
    return RSVP.resolve().then(function(){
      // write to tmp files
      return tmp;
    });
  },
  cleanup: function(){
    quickTemp.remove(this, 'tmpDestDir');
  },
  inputTrees: [ subtree ]
};

This is the information I wish I had when I was diving into Ember CLI addons for the first time. In fact, I originally put this information together as notes while deep diving into Broccoli's source and I'm glad I invested the time. So while I think broccoli is a delicious vegetable I think it makes an even better build tool!

Hope you found this helpful. Thanks for reading!

• https://github.com/broccolijs/broccoli#plugin-api-specification
• http://simonwade.me/intro-to-broccoli/
• http://moduscreate.com/better-builds-begin-with-broccoli/