technical and social progress toward ECMAScript 6 at facebook


Ben Newman (Facebook)
NodeJS @ Pivotal Labs
19 March 2014

{ github,
  twitter,
  instagram,
  facebook
}.com/benjamn

Wikipedia

titlecapitalization.com

We have the opportunity, as technologists, to make certain kinds of problems disappear forever.

“We should have been doing it this way all along!”

GitHub is strewn with better ways of doing things that never got properly evangelized.

There has to be a way forward.

Programming languages are notoriously difficult to “fix forward.”


Why isn't every Python project using Python 3 yet?

How can ECMAScript 6
avoid the Python 3 trap?


Crazy idea: ease into the new language by simulating its most useful features in the current version of JavaScript (ECMAScript 5).

Example: => function syntax


Input (ES6):

[3, 1, 10, 28].sort((a, b) => a - b)
            


Output (ES5):

            

First, import some utilities and parse the code:


var recast = require("recast");
var types = recast.types;
var traverse = types.traverse;
var n = types.namedTypes;
var b = types.builders;

var ast = recast.parse(
  "[3, 1, 10, 28].sort((a, b) => a - b)"
);

Next, traverse and modify the syntax tree:


traverse(ast, function(node) {




















});
            

Next, traverse and modify the syntax tree:


traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {


















  }
});
            

Next, traverse and modify the syntax tree:


traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {
    var body = node.body;

    if (node.expression) {
      node.expression = false;
      body = b.blockStatement([b.returnStatement(body)]);
    }












  }
});
            

Next, traverse and modify the syntax tree:


traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {
    var body = node.body;

    if (node.expression) {
      node.expression = false;
      body = b.blockStatement([b.returnStatement(body)]);
    }

    var funExp = b.functionExpression(
      node.id, node.params, body,
      node.generator, node.expression
    );







  }
});
            

Next, traverse and modify the syntax tree:


traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {
    var body = node.body;

    if (node.expression) {
      node.expression = false;
      body = b.blockStatement([b.returnStatement(body)]);
    }

    var funExp = b.functionExpression(
      node.id, node.params, body,
      node.generator, node.expression
    );

    var bindExp = b.callExpression(
      b.memberExpression(funExp, b.identifier("bind"), false),
      [b.thisExpression()]
    );


  }
});
            

Next, traverse and modify the syntax tree:


traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {
    var body = node.body;

    if (node.expression) {
      node.expression = false;
      body = b.blockStatement([b.returnStatement(body)]);
    }

    var funExp = b.functionExpression(
      node.id, node.params, body,
      node.generator, node.expression
    );

    var bindExp = b.callExpression(
      b.memberExpression(funExp, b.identifier("bind"), false),
      [b.thisExpression()]
    );

    this.replace(bindExp);
  }
});
            

Finally, reprint the code:


console.log(recast.print(ast).code);

// Which prints:
[3, 1, 10, 28].sort(function(a, b) {
  return a - b;
}.bind(this))
            

If you already have a build step for static resources, you can be cooking with arrow functions in a matter of minutes!

recast, v.


  1. to give (a metal object) a different form by melting it down and reshaping it.
  2. to form, fashion, or arrange again.
  3. to remodel or reconstruct (a literary work, document, sentence, etc.).
  4. to supply (a theater or opera work) with a new cast.

Recast recap:


var recast = require("recast");
var ast = recast.parse(source);
transform(ast); // Anything goes.
console.log(recast.print(ast).code);
          

“Non-destructive partial source transformation”

Ariya Hidayat

Example: ...rest parameters


Input (ES6):
function max(a, ...rest) {
  rest.forEach(x => { if (x > a) a = x; });
  return a;
}

Output (ES5):
          

Example: ...rest parameters


traverse(ast, function(node) {

















});

Example: ...rest parameters


traverse(ast, function(node) {
  if (n.Function.check(node) && node.rest) {















  }
});

Example: ...rest parameters


traverse(ast, function(node) {
  if (n.Function.check(node) && node.rest) {
    var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");














  }
});

Example: ...rest parameters


traverse(ast, function(node) {
  if (n.Function.check(node) && node.rest) {
    var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");

    var sliceArgs = [
      b.identifier("arguments"),
      b.literal(node.params.length)
    ];









  }
});

Example: ...rest parameters


traverse(ast, function(node) {
  if (n.Function.check(node) && node.rest) {
    var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");

    var sliceArgs = [
      b.identifier("arguments"),
      b.literal(node.params.length)
    ];

    var sliceCall = b.callExpression(sliceCallee, sliceArgs);







  }
});

Example: ...rest parameters


traverse(ast, function(node) {
  if (n.Function.check(node) && node.rest) {
    var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");

    var sliceArgs = [
      b.identifier("arguments"),
      b.literal(node.params.length)
    ];

    var sliceCall = b.callExpression(sliceCallee, sliceArgs);

    var restVarDecl = b.variableDeclaration("var", [
      b.variableDeclarator(node.rest, sliceCall)
    ]);



  }
});

Example: ...rest parameters


traverse(ast, function(node) {
  if (n.Function.check(node) && node.rest) {
    var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");

    var sliceArgs = [
      b.identifier("arguments"),
      b.literal(node.params.length)
    ];

    var sliceCall = b.callExpression(sliceCallee, sliceArgs);

    var restVarDecl = b.variableDeclaration("var", [
      b.variableDeclarator(node.rest, sliceCall)
    ]);

    node.body.body.unshift(restVarDecl);

  }
});

Example: ...rest parameters


traverse(ast, function(node) {
  if (n.Function.check(node) && node.rest) {
    var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");

    var sliceArgs = [
      b.identifier("arguments"),
      b.literal(node.params.length)
    ];

    var sliceCall = b.callExpression(sliceCallee, sliceArgs);

    var restVarDecl = b.variableDeclaration("var", [
      b.variableDeclarator(node.rest, sliceCall)
    ]);

    node.body.body.unshift(restVarDecl);
    node.rest = null;
  }
});

Example: class syntax


  Input (ES6):

class Derived extends Base {
  constructor(value) {
    super(value + 1);
  }

  getValue() {
    return super.getValue() - 1;
  }

  static getName() {
    return "Derived";
  }
}
                    
Output (ES5):

function Derived(value) {
  Base.call(this, value + 1);
}

Derived.prototype = Object.create(Base.prototype);

Derived.prototype.constructor = Derived;

Derived.prototype.getValue = function() {
  return Base.prototype.getValue.call(this) - 1;
};

Derived.getName = function() {
  return "Derived";
};
                    

Transform code much more involved than previous examples;
see es6-class-visitors.js for all the gory details.

What our classes used to look like:


var Class = require('Class');
var copyProperties = require('copyProperties');

function Derived(value) {
  this.parent(value + 1);
}

copyProperties(Derived.prototype, {
  getValue: function() {
    return this.parent.getValue() - 1;
  }
});

copyProperties(Derived, {
  getName: function() {
    return "Derived";
  }
});

Class.extend(Derived, Base);

Original plan:


  1. Add a jslint warning that complains when you try to commit new code that assigns .prototype properties
  2. Point to a wiki article in the warning message
  3. Post about the new syntax in some internal Facebook groups
  4. Wait for the diffs to roll in
  5. Declare victory (just in time for ECMAScript 12)

Wishful thinking!


  • jslint tells you that your code might be bad after you've written it
  • Almost no one reads wiki articles unless hopelessly stuck
  • I didn't want to make reeducating thousands of Facebook engineers my full-time job
  • As long as the old style of defining classes dominated the codebase, that's the style engineers would (reasonably!) mimic

What to do?


We have lots of this (ES5):

var Class = require('Class');
var copyProperties = require('copyProperties');

function Derived(value) {
  this.parent(value + 1);
}

copyProperties(Derived.prototype, {
  getValue: function() {
    return this.parent.getValue() - 1;
  }
});

copyProperties(Derived, {
  getName: function() {
    return "Derived";
  }
});

Class.extend(Derived, Base);
We want more of this (ES6):

class Derived extends Base {
  constructor(value) {
    super(value + 1);
  }

  getValue() {
    return super.getValue() - 1;
  }

  static getName() {
    return "Derived";
  }
}

What to do?


We want more of this (ES6):

class Derived extends Base {
  constructor(value) {
    super(value + 1);
  }

  getValue() {
    return super.getValue() - 1;
  }

  static getName() {
    return "Derived";
  }
}
So that we can ship this (ES5):

function Derived(value) {
  Base.call(this, value + 1);
}

Derived.prototype = Object.create(Base.prototype);

Derived.prototype.constructor = Derived;

Derived.prototype.getValue = function() {
  return Base.prototype.getValue.call(this) - 1;
};

Derived.getName = function() {
  return "Derived";
};

We've solved this problem before.


It's just a source tranformation! What's different?


  1. ES5 to ES6 (the “wrong” direction)
  2. Only needs to be performed once!
  3. Needs to accommodate multiple existing idioms
  4. Needs to generate pretty code, as if (re)written by hand
  5. The diffs need to be human-reviewable


“Non-destructive partial source transformation”

All told:


1647 files changed
76555 insertions(+)
78260 deletions(-)

1658 classes converted
3223 classes today

Lessons

  1. If you make the output human-readable enough, reviewers may not even realize it was machine-generated.

  1. The transform script should be absolutely littered with fail-stop assertions.

  1. Set yourself up to iterate rapidly, accommodating more and more exotic cases as you encounter them.


Meaning: you should be able to change your mind as often as you like, git reset --hard, and rerun the script.

  1. Feel free to fix rare cases by hand, but stack them in separate commits.

  1. Make the transform script idempotent. No excuses!

  1. Use GNU parallel to run the transform script in many processes simultaneously.


find ~/www/html/js/lib | \
  grep "\.js$" | \
  time parallel ~/www/scripts/bin/classify --update

228.03s user 12.25s system 1229% cpu 19.548 total
          

  1. Humans have right-of-way.

  1. Identify stakeholders, and convert whole functional units of code at once.

  1. Set intermediate milestones, and prioritize them aggressively.


Killing off Class.extend in favor of extends/super only really depended on the conversion of files using Class.extend.

  1. New code mimics existing code, and the future is longer than the past.

Non-ES6 example: JSX


Input (ES6++):
          

Output (ES5):
          

Why fork the language?


  • People keep writing HTML-like templating languages, so there must be some value in that.
  • JSX affords a useful layer of indirection, allowing us to change our minds about whether to require new keywords or not, how to handle children, what to do about whitespace, React.DOM namespacing, &c.
  • Huge benefits from adding similar syntax to PHP at Facebook, and we believe the similarity is worthwhile.
  • Arguably much less aggressive than something like CoffeeScript, or even most other templating languages.
  • We're content with JSX never becoming part of the ECMAScript standard. That was never the point.

Can we express JSX in “real” ES6?


              

press the
down arrow!

Can we express JSX in “real” ES6?


            

This new kind of string is made possible by template literal syntax.

What about something trickier?


function *fibonacci(limit) {
  var a = 0;
  var b = 1;

  limit = limit || Infinity;

  while (a <= limit) {
    yield a;

    var next = a + b;
    a = b;
    b = next;
  }
}
            
var g = fibonacci(10);

console.log(g.next().value); // 0
console.log(g.next().value); // 1
console.log(g.next().value); // 1
console.log(g.next().value); // 2
console.log(g.next().value); // 3
console.log(g.next().value); // 5
console.log(g.next().value); // 8
console.log(g.next().done); // true
            
How hard could it be to translate this function into one that no longer contains function* or yield?

It's the trickiest code I've ever written.

function *fibonacci(limit) {
  var a = 0;
  var b = 1;

  limit = limit || Infinity;

  while (a <= limit) {
    yield a;

    var next = a + b;
    a = b;
    b = next;
  }














}
            
function *fibonacci(limit) {
  var a, b, next;

  a = 0;
  b = 1;

  limit = limit || Infinity;

  while (a <= limit) {
    yield a;

    next = a + b;
    a = b;
    b = next;
  }












}
            
function fibonacci(limit) {
  var a, b, next;

  return function*() {
    a = 0;
    b = 1;

    limit = limit || Infinity;

    while (a <= limit) {
      yield a;

      next = a + b;
      a = b;
      b = next;
    }
  };










}
            
function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (1) switch (context.next) {




















    }
  };
}
            
function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (1) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
















    }
  };
}
            
function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (1) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 3:
      if (!(a <= limit)) {
        context.next = 11;
        break;
      }











    }
  };
}
            
function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (1) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 3:
      if (!(a <= limit)) {
        context.next = 11;
        break;
      }








    case 11:
    case "end":
      return context.stop();
    }
  };
}
            
function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (1) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 3:
      if (!(a <= limit)) {
        context.next = 11;
        break;
      }
      context.next = 6;
      return a;






    case 11:
    case "end":
      return context.stop();
    }
  };
}
            
function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (1) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 3:
      if (!(a <= limit)) {
        context.next = 11;
        break;
      }
      context.next = 6;
      return a;
    case 6:
      next = a + b;
      a = b;
      b = next;


    case 11:
    case "end":
      return context.stop();
    }
  };
}
            
function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (1) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 3:
      if (!(a <= limit)) {
        context.next = 11;
        break;
      }
      context.next = 6;
      return a;
    case 6:
      next = a + b;
      a = b;
      b = next;
      context.next = 3;
      break;
    case 11:
    case "end":
      return context.stop();
    }
  };
}
            
function fibonacci(limit) {
  var a, b, next;

  return wrapGenerator(function(context) {
    while (1) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 3:
      if (!(a <= limit)) {
        context.next = 11;
        break;
      }
      context.next = 6;
      return a;
    case 6:
      next = a + b;
      a = b;
      b = next;
      context.next = 3;
      break;
    case 11:
    case "end":
      return context.stop();
    }
  }, this);
}
            

What else can context do?

Context.prototype.stop = function() {
  this.done = true;

  var rootEntry = this.tryEntries[0];
  var rootRecord = rootEntry.completion;
  if (rootRecord.type === "throw") {
    throw rootRecord.arg;
  }

  return this.rval;
};
            

Before returning, the runtime must check to see if an unhandled exception still needs to be thrown.

Example: for-in loops

Context.prototype.keys = function(object) {


















};
            

(runtime)

Example: for-in loops

Context.prototype.keys = function(object) {
  var keys = [];
  for (var key in object)
    keys.push(key);
  keys.reverse();














};
            

(runtime)

Example: for-in loops

Context.prototype.keys = function(object) {
  var keys = [];
  for (var key in object)
    keys.push(key);
  keys.reverse();

  return function next() {











  };
};
            

(runtime)

Example: for-in loops

Context.prototype.keys = function(object) {
  var keys = [];
  for (var key in object)
    keys.push(key);
  keys.reverse();

  return function next() {
    while (keys.length) {
      var key = keys.pop();
      if (key in object) {
        next.value = key;
        next.done = false;
        return next;
      }
    }



  };
};
            

(runtime)

Example: for-in loops

Context.prototype.keys = function(object) {
  var keys = [];
  for (var key in object)
    keys.push(key);
  keys.reverse();

  return function next() {
    while (keys.length) {
      var key = keys.pop();
      if (key in object) {
        next.value = key;
        next.done = false;
        return next;
      }
    }

    next.done = true;
    return next;
  };
};
            

(runtime)

Example: for-in loops

function *objMap(obj, callback) {
  before();
  for (var k in obj) {
    yield callback(k, obj[k]);
  }
  after();
}
            
function objMap(obj, callback) {
  var k;
  return wrapGenerator(function(context) {
    while (1) switch (context.next) {



















    }
  }, this);
}
            

Example: for-in loops

function *objMap(obj, callback) {
  before();
  for (var k in obj) {
    yield callback(k, obj[k]);
  }
  after();
}
            
function objMap(obj, callback) {
  var k;
  return wrapGenerator(function(context) {
    while (1) switch (context.next) {
    case 0:
      before();
      context.t0 = context.keys(obj);
















    }
  }, this);
}
            

Example: for-in loops

function *objMap(obj, callback) {
  before();
  for (var k in obj) {
    yield callback(k, obj[k]);
  }
  after();
}
            
function objMap(obj, callback) {
  var k;
  return wrapGenerator(function(context) {
    while (1) switch (context.next) {
    case 0:
      before();
      context.t0 = context.keys(obj);
    case 2:
      if ((context.t1 = context.t0()).done) {
        context.next = 8;
        break;
      }











    }
  }, this);
}
            

Example: for-in loops

function *objMap(obj, callback) {
  before();
  for (var k in obj) {
    yield callback(k, obj[k]);
  }
  after();
}
            
function objMap(obj, callback) {
  var k;
  return wrapGenerator(function(context) {
    while (1) switch (context.next) {
    case 0:
      before();
      context.t0 = context.keys(obj);
    case 2:
      if ((context.t1 = context.t0()).done) {
        context.next = 8;
        break;
      }








    case 9:
    case "end":
      return context.stop();
    }
  }, this);
}
            

Example: for-in loops

function *objMap(obj, callback) {
  before();
  for (var k in obj) {
    yield callback(k, obj[k]);
  }
  after();
}
            
function objMap(obj, callback) {
  var k;
  return wrapGenerator(function(context) {
    while (1) switch (context.next) {
    case 0:
      before();
      context.t0 = context.keys(obj);
    case 2:
      if ((context.t1 = context.t0()).done) {
        context.next = 8;
        break;
      }
      k = context.t1.value;







    case 9:
    case "end":
      return context.stop();
    }
  }, this);
}
            

Example: for-in loops

function *objMap(obj, callback) {
  before();
  for (var k in obj) {
    yield callback(k, obj[k]);
  }
  after();
}
            
function objMap(obj, callback) {
  var k;
  return wrapGenerator(function(context) {
    while (1) switch (context.next) {
    case 0:
      before();
      context.t0 = context.keys(obj);
    case 2:
      if ((context.t1 = context.t0()).done) {
        context.next = 8;
        break;
      }
      k = context.t1.value;
      context.next = 6;
      return callback(k, obj[k]);





    case 9:
    case "end":
      return context.stop();
    }
  }, this);
}
            

Example: for-in loops

function *objMap(obj, callback) {
  before();
  for (var k in obj) {
    yield callback(k, obj[k]);
  }
  after();
}
            
function objMap(obj, callback) {
  var k;
  return wrapGenerator(function(context) {
    while (1) switch (context.next) {
    case 0:
      before();
      context.t0 = context.keys(obj);
    case 2:
      if ((context.t1 = context.t0()).done) {
        context.next = 8;
        break;
      }
      k = context.t1.value;
      context.next = 6;
      return callback(k, obj[k]);
    case 6:
      context.next = 2;
      break;


    case 9:
    case "end":
      return context.stop();
    }
  }, this);
}
            

Example: for-in loops

function *objMap(obj, callback) {
  before();
  for (var k in obj) {
    yield callback(k, obj[k]);
  }
  after();
}
            
function objMap(obj, callback) {
  var k;
  return wrapGenerator(function(context) {
    while (1) switch (context.next) {
    case 0:
      before();
      context.t0 = context.keys(obj);
    case 2:
      if ((context.t1 = context.t0()).done) {
        context.next = 8;
        break;
      }
      k = context.t1.value;
      context.next = 6;
      return callback(k, obj[k]);
    case 6:
      context.next = 2;
      break;
    case 8:
      after();
    case 9:
    case "end":
      return context.stop();
    }
  }, this);
}
            

Check out facebook.github.io/regenerator for a convenient way to experiment with insane code and easily report any bugs that you find.


I can go into a lot more detail if there are questions, and of course https://github.com/facebook/regenerator is the final authority on how the transformation currently works.


Right now, though, I want to talk about something
much less technical.

Motivation


How can you stay motivated to finish a difficult side project that isn't directly related to your day job?

  1. Keep it fun.


Writing tests is even more important for side projects than it is for day-to-day coding.


Time your breaks for when you're pretty sure you know what you need to do next.


Guard your secrets, and don't oversell an unfinished project, but let other people in when you have something to share.

  1. Don't be discouraged by similar projects.


Even when they seem to have a head start.


You have a distinct advantage as the underdog, because you know exactly what success would look like.


Whenever you feel discouraged, realize that other people will probably be discouraged for the same reason.

  1. When should you ship?


Day job projects have to be mostly correct when you ship, but that's fortunately not the case for side projects.


Ship as soon as you're confident you can fix new problems quickly.


It makes people feel great to report real problems, especially if they're straightforward to fix.


Preemptively file known bugs and don't worry about fixing all of them before you ship.

  1. Make it easy for people to contribute, even in small ways.


"Report a bug" link next to the live editor.


Restrict the scope of your project to encourage interoperability.


As long as tests are passing, merge first and tweak later.

  1. Get non-technical friends excited about what you're working on.


It might just be that your enthusiasm is infectious,
but that counts for a lot.


Even technical friends have an emotional side that you will need to appeal to.


And let's be honest: you have the same emotional needs, and you are the primary audience for the story you're telling.

Thanks!

External links:


github.com/{
  benjamn/pivotal-meetup-talk,
  facebook/jstransform,
  benjamn/recast,
  benjamn/ast-types,
  facebook/regenerator
}

code.facebook.com/projects