StealJS Roadmap Discussion

StealJS
1

My high-level vision for the next year of Steal is to keep the benefits of client-side module loaders (while adopting new technologies to improve it) while gaining the no-compromises benefit of Node-based module bundlers by improving our watch mode.

Additionally I want to lay out the road to 1.0.

Improve NPM workflow

The addition of NPM support made life much better for Steal users but there are still some holes in our implementation. Steal should be able to load anything that Browserify/Webpack can bundle so we should add support for:

  • folder/index convention of NPM modules. That is if we are unable to load folder.js, try folder/index.js. Additionally we should cache the results of these so that there are no 404s on refresh.
  • Progressively load package.jsons so that we don’t load them for modules that will never be used.
  • Minimize the amount of information kept (from package.jsons) to reduce production size.

Implement smart caching

We can speed up development with smart caching using Service Workers. We should:

  • Cache modules served from node_modules/, only fetch these if their associated package.json changes.
  • Cache the results of transpiling ES6 code, Less code, etc. so that we only perform the transpilation when the source has changed.
  • Cache the complete dependency tree and use that on page load so that all modules needed are fetched right away. (this is an option Steal supports call depCache).

Smaller production builds

To stay competitive our production builds should be reduced down to only what is needed. This means not bundling extensions that are unused in production.

By default only normalize hooks should be included in production builds. We could use tree-shaking techniques to remove extension code that doesn’t perform normalization.

Additionally if the user’s application isn’t using progressive loading, the AMD shim should be used.

Conditional loading

One problem that comes up often in DoneJS applications is the need to only load fixtures in development. steal-conditional was created to solve this problem, however it doesn’t work with the NPM plugin. There have been some changes to the conditional extension in SystemJS so we should take a look at what has been developed there and possible upgrade and make it work with NPM.

Better extension APIs

We should encourage a rich ecosystem of extensions. These extensions should be able to load their own dependencies. configDependencies is the current way to add extensions, but these cannot have NPM dependencies so they’re not as useful as they should be. This also means you can’t npm install an extension, unless you list node_modules/extension/extension as the module name in configDependencies. Gross!

Steal could add a new property: extensions which is an array of modules that will be loaded after the config and before the main.

Writing extensions should be easier as well. Wrapping is not a great API for extensions in JavaScript. We could switch to class-based extensions so they are as easy as:

export default class ForwardSlash extends System.constructor {
  normalize() {
    ...
  }
}

These extensions would need to be executed in order, so that each one replaces the global System object.

Dependency injection for testing

This is being discussed on GitHub, essentially people want to be able to inject a dependency as a substitute for testing purposes. This can be done today manually using the low level APIs provided, but could be made more user-friendly.

Steal development server

One advantage that bundlers have over client-side module loaders is that they don’t have to make compromises based on what browsers support. Steal supports continuous builds (aka watch mode) so that development mimics production.

However this only works for you app’s “main” page, not for demo pages or test pages. It would be nice if you could get the benefits of both. I’m thinking of development server middleware that parses HTML files for steal script tags and automatically includes their mains in the continuous build.

Steal 1.0

Steal is mature so we should be on 1.0 already. Since this will be the last opportunity to make breaking changes for a while there’s a couple of things we should definitely do:

  • Upgrade to Babel 6 which is not compatible with old versions.
  • Make Babel the default transpiler instead of Traceur.
  • Progressively load package.jsons. Some people might be relying on having the complete tree all of the time so this would be a breaking change, technically.
2

Progressively load package.jsons. is really important. Steal is really making too many requests and it also make configuration of packages more difficult.

Steal dev server is a good idea. actually I use my own for this, to make part of transpilation on the client. Cacheing on the client is too particuary great idea, but may introduce some caveats.

Other things are great. Just think more about balance flexibility vs abstraction, task it to make it right.

And probably you should wirte some comparison article Steal vs JSPM vs webpack vs browserfy to get attention to stealJs.

1 Like
3

I’m starting to think Better extension APIs is the most important feature on the roadmap.

One example of how this could be used, when I worked on upgrading to Babel 6 a couple of weeks ago I had to modify 3 projects; es6-module-loader, steal, and transpile. It was way more work than it should be.

With the new extensions feature a Babel extension could do transpiling in the transform hook. This would allow you to optionally transpile all code (not just import/export code, some code on NPM uses ES6 features with CommonJS). And you could make arbitrary babel plugins work. Most importantly this could be developed as a 3rd party module on NPM. Upgrading Babel would just be a matter of upgrading the extension.

There are many other use cases where this would be valuable. Right now writing extensions is too difficult. You can’t publish them to NPM, you can’t use NPM modules in your extensions. can-ssr’s steal extension would use this.

The first step to making this happen is to change our systemjs/es6ml forks to be constructor based. The System object needs to be an instance of System.constructor so that you can extend from this. The idea is that System should be defined as window.System = new SystemConstructor() . This will be hard in regards to config, since a lot of config is defined on the System loader that would need to be copied over.

4

Wrapping is not a great API for extensions in JavaScript.

I’m not sure I agree with this. In general, extending/inheritance is not a good way to go.

I’m starting to think Better extension APIs is the most important feature on the roadmap.

I’m not totally convinced that the cost of this would be worth it. I think the most important thing we are missing are better docs, better output with builds, and articles.

5

Btw, I’m assuming that what’s needed to make this happens is an overhaul of systemjs/es6ml and all of our extensions … something that would take longer than most of the other things in this roadmap.

6

Yeah, it would be a lot of work, just keep in mind that it’s something we’ll have to do at some point because the WhatWG Loader is constructor based.

Another idea that would be less work would be allowing plugins to be applied to all modules (not requiring the bang syntax or a file extension) so maybe something like:

System.config({
  plugins: {
    "*": [
      "foo_extension"
    ]
  }
});

In which case we would load this plugin that could do:

exports.normalize = function(name){
  if(name === "foo") return "bar";

  return this.parentNormalize.apply(this, arguments);
};

Something to that effect. I believe SystemJS has something like this now (or at least had discussed it at one point).

I do think this is a very important issue, we won’t have a nice ecosystem without a way to write extensions that can use npm dependencies and be published to npm themselves. It would also enable us to make Steal a smaller library (currently any time we need something we are adding it to core), which is another roadmap item.

7

Yeah, I agree it’s really important. I just hope we can do it w/o a big re-write first.

8

Well done guys, keep pushing this! So much more fun than webpack.

2 Likes
9

I was playing around with the plugins idea yesterday. It’s available in a branch, you can check out an example test here: https://github.com/stealjs/steal/blob/modules/test/modules/modules_test.js#L13-L19

In these tests it is called “extensions” but I think plugins is more appropriate. It’s very rough at this stage, I haven’t tried to get it to work when you have multiple plugins yet. When it’s a bit more fleshed out I would like to release it under a beta tag.