regenerator

Ben Newman (Facebook)
@brooklyn_js

BENTLEY: Oh. Well, I was going to ask… eh; what is it, Major Lawrence,
that attracts you personally to the desert?
LAWRENCE: It's clean.

Sometimes software development is like a desert.

Pure possibility spreads out as far as the eye can see.

The only obstacles are in your mind.

If you build something there and it falls apart,

it was your own fault, your own poor planning.

If you feel that way about JavaScript,
then you must know something I don't.

Because there's nothing pristine about web browsers, with their ever-evolving APIs and long histories of inconsistency.

And I honestly can't say it's your fault if you expected


[3, 1, 10, 28].sort()

to evaluate to


[1, 3, 10, 28]

instead of


[1, 10, 28, 3]

(because the default sort function compares the dictionary order of the elements after turning them into strings).

JavaScript is more of a deep, dark forest than a desert.

There's always something to be grappled with, even if the lessons aren't exactly timeless.

And that impurity is part of what attracts me,
personally, to the forest.

Tonight I want to talk about the future of JavaScript,
and how we can get there more quickly.

Background

Before coming to Facebook I worked at Quora.

Near the end of my time at Quora, I began building a tool for rewriting JavaScript source code that Quora graciously let me release as open source before I left.

recast, v.

to give (a metal object) a different form by melting it down and reshaping it.
to form, fashion, or arrange again.
to remodel or reconstruct (a literary work, document, sentence, etc.).
to supply (a theater or opera work) with a new cast.

How it works:


var recast = require("recast");
var ast = recast.parse(source);
transform(ast); // Anything goes.
console.log(recast.print(ast).code);

Instead of simply pretty-printing the whole tree, recast.print tries to recyle the original source code wherever possible.

"Non-destructive partial source transformation"
– Ariya Hidayat

You get to think about Syntax in the abstract,
pristine desert of your imagination.

And what you get in return are diffs that
humans can read and review.

So far this year:

1647 files changed

76555 insertions(+)

78260 deletions(-)

What next?

Take features from the next version of JavaScript.

Rewrite them in syntax supported by all current implementations of the language.

Pretend the future is here.

Which features first?

Arrow function syntax:


[3, 1, 10, 28].sort((a, b) => a - b)

First, import some utilities and parse the code:


var recast = require("recast");
var types = recast.types;
var n = types.namedTypes;
var b = types.builders;

var ast = recast.parse(
  "[3, 1, 10, 28].sort((a, b) => a - b)"
);

Next, traverse and modify the syntax tree:


types.traverse(ast, function(node) {




















});

Next, traverse and modify the syntax tree:


types.traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {


















  }
});

Next, traverse and modify the syntax tree:


types.traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {
    var body = node.body;

    if (node.expression) {
      node.expression = false;
      body = b.blockStatement([b.returnStatement(body)]);
    }












  }
});

Next, traverse and modify the syntax tree:


types.traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {
    var body = node.body;

    if (node.expression) {
      node.expression = false;
      body = b.blockStatement([b.returnStatement(body)]);
    }

    var funExp = b.functionExpression(
      node.id, node.params, body,
      node.generator, node.expression
    );







  }
});

Next, traverse and modify the syntax tree:


types.traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {
    var body = node.body;

    if (node.expression) {
      node.expression = false;
      body = b.blockStatement([b.returnStatement(body)]);
    }

    var funExp = b.functionExpression(
      node.id, node.params, body,
      node.generator, node.expression
    );

    var bindExp = b.callExpression(
      b.memberExpression(funExp, b.identifier("bind"), false),
      [b.thisExpression()]
    );


  }
});

Next, traverse and modify the syntax tree:


types.traverse(ast, function(node) {
  if (n.ArrowFunctionExpression.check(node)) {
    var body = node.body;

    if (node.expression) {
      node.expression = false;
      body = b.blockStatement([b.returnStatement(body)]);
    }

    var funExp = b.functionExpression(
      node.id, node.params, body,
      node.generator, node.expression
    );

    var bindExp = b.callExpression(
      b.memberExpression(funExp, b.identifier("bind"), false),
      [b.thisExpression()]
    );

    this.replace(bindExp);
  }
});

Finally, reprint the code:


console.log(recast.print(ast).code);

// Which prints:
[3, 1, 10, 28].sort(function(a, b) {
  return a - b;
}.bind(this))

If you already have a build step for static resources, you can be cooking with arrow functions in a matter of minutes!

What about something a little trickier?


function *fibonacci(limit) {
  var a = 0;
  var b = 1;

  limit = limit || Infinity;

  while (a <= limit) {
    yield a;

    var next = a + b;
    a = b;
    b = next;
  }
}

var g = fibonacci(10);

console.log(g.next().value); // 0
console.log(g.next().value); // 1
console.log(g.next().value); // 1
console.log(g.next().value); // 2
console.log(g.next().value); // 3
console.log(g.next().value); // 5
console.log(g.next().value); // 8
console.log(g.next().done); // true

How hard could it be to translate this function into one that no longer contains function* or yield?

I've been jokingly referring to this side project as "my life's work" for so long that I'm terrified it might actually be finished soon.
— Ben Newman (@benjamn) May 8, 2013

It's the trickiest code I've ever written.


function *fibonacci(limit) {
  var a = 0;
  var b = 1;

  limit = limit || Infinity;

  while (a <= limit) {
    yield a;

    var next = a + b;
    a = b;
    b = next;
  }













}


function *fibonacci(limit) {
  var a, b, next;

  a = 0;
  b = 1;

  limit = limit || Infinity;

  while (a <= limit) {
    yield a;

    next = a + b;
    a = b;
    b = next;
  }











}


function fibonacci(limit) {
  var a, b, next;

  return function*() {
    a = 0;
    b = 1;

    limit = limit || Infinity;

    while (a <= limit) {
      yield a;

      next = a + b;
      a = b;
      b = next;
    }
  };









}


function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (true) switch (context.next) {



















    }
  };
}


function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (true) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;















    }
  };
}


function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (true) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 2:
      if (!(a <= limit)) {
        context.next = "end";
        break;
      }










    }
  };
}


function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (true) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 2:
      if (!(a <= limit)) {
        context.next = "end";
        break;
      }








    case "end":
      return context.stop();
    }
  };
}


function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (true) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 2:
      if (!(a <= limit)) {
        context.next = "end";
        break;
      }
      context.next = 5;
      return a;






    case "end":
      return context.stop();
    }
  };
}


function fibonacci(limit) {
  var a, b, next;

  return function(context) {
    while (true) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 2:
      if (!(a <= limit)) {
        context.next = "end";
        break;
      }
      context.next = 5;
      return a;
    case 5:
      next = a + b;
      a = b;
      b = next;
      context.next = 2;
      break;
    case "end":
      return context.stop();
    }
  };
}


function fibonacci(limit) {
  var a, b, next;

  return wrapGenerator(function(context) {
    while (true) switch (context.next) {
    case 0:
      a = 0;
      b = 1;
      limit = limit || Infinity;
    case 2:
      if (!(a <= limit)) {
        context.next = "end";
        break;
      }
      context.next = 5;
      return a;
    case 5:
      next = a + b;
      a = b;
      b = next;
      context.next = 2;
      break;
    case "end":
      return context.stop();
    }
  }, this);
}

Easy enough, you say. But what if an exception is thrown from inside a triply-nested try-finally block?


function *crazy(x) {
  try {
    try {
      try {
        mightThrow(x);
      } finally {
        yield "inner";
      }
    } catch (x) {
      throw yield x;
    } finally {
      yield "middle";
    }
  } finally {
    yield "outer";
  }
}


function crazy(x) {
  return wrapGenerator(function crazy$($ctx) {
    while (1) switch ($ctx.next) {
    case 0:
      $ctx.t0 = 31;
      $ctx.pushTry(null, 26, "t0");
      $ctx.t1 = 26;
      $ctx.pushTry(15, 21, "t1");
      $ctx.t2 = 12;
      $ctx.pushTry(null, 7, "t2");
      mightThrow(x);
    case 7:
      $ctx.popFinally(7);
      $ctx.next = 10;
      return "inner";
    case 10:
      $ctx.next = $ctx.t2;
      break;
    case 12:
      $ctx.popCatch(15);
      $ctx.next = 21;
      break;
    case 15:
      $ctx.popCatch(15);
      $ctx.t3 = $ctx.thrown;
      delete $ctx.thrown;
      $ctx.next = 20;
      return $ctx.t3;
      ...


    ...
    case 20:
      throw $ctx.sent;
    case 21:
      $ctx.popFinally(21);
      $ctx.next = 24;
      return "middle";
    case 24:
      $ctx.next = $ctx.t1;
      break;
    case 26:
      $ctx.popFinally(26);
      $ctx.next = 29;
      return "outer";
    case 29:
      $ctx.next = $ctx.t0;
      break;
    case 31:
    case "end":
      return $ctx.stop();
    }
  }, this);
}

Check out the live sandbox for a convenient way to experiment with insane code and easily report any bugs that you find.

I can go into a lot more detail if there are questions, and of course https://github.com/facebook/regenerator is the final authority on how the transformation currently works.

Right now, though, I want to talk about something
much less technical.

Motivation

How can you stay motivated to finish a difficult side project that isn't directly related to your day job?

Unless you're some kind of superhero, you can't just rely on having boundless enthusiasm for programming. It runs out.

Telling yourself, "The trick… is not minding that it hurts"
is not a sustainable strategy.

Keep it fun.

Writing tests is even more important for side projects than it is for day-to-day coding.

Time your breaks for when you're pretty sure you know what you need to do next.

Guard your secrets, and don't oversell an unfinished project, but let other people in when you have something to share.

Don't be discouraged by similar projects.

Even when they seem to have a head start.

You have a distinct advantage as the underdog, because you know exactly what success would look like.

Whenever you feel discouraged, realize that other people will probably be discouraged for the same reason.

When should you ship?

Day job projects have to be mostly correct when you ship, but that's fortunately not the case for side projects.

Rule of thumb: ship as soon as you're confident you can fix new problems quickly.

It makes people feel great to report real problems, especially if they're straightforward to fix.

Preemptively file known bugs and don't worry about fixing all of them before you ship.

Make it easy for people to contribute, even in small ways.

"Report a bug" link next to the live editor.

Restrict the scope of your project to encourage interoperability.

As long as tests are passing, merge first and tweak later.

Get non-technical friends excited about what you're working on.

It might just be that your enthusiasm is infectious,
but that counts for a lot.

Even technical friends have an emotional side that you will need to appeal to.

And let's be honest: you have the same emotional needs, and you are the primary audience for the story you're telling.

Thanks!

External links:

github.com/{
  benjamn/recast,
  facebook/regenerator,
  amiorin/regeneratorify,
  olov/defs,
  TooTallNate/gnode }

Talk to me:

{ github,
  twitter,
  instagram,
  facebook
}.com/benjamn