Creating Node packages in NPM

npm is the package manager for Node.js, the server-side JavaScript thingy. And it’s lovely. On a day-to-day basis, doing C#, I use NuGet in Visual Studio. While NuGet is a welcome addition to Visual Studio, npm seems to make it much easier to create and reuse packages. 

For instance, I’ve just pushed my parsing library, canto34, up to this location on npm, and the process was remarkably easy. The steps are –

1. create a user on the npm website.

2. Link your computer to it with npm command line;

npm adduser


3. Create a single file (package.json) which describes the package, its dependencies, and its location in git

4. Publish the package using npm again;

npm publish

And that’s it — package deployed. 

Now, that’s pretty sweet — anyone in the world can install the package using this command line;

npm install canto34

And get going with the library. But what about me? I want to continue to develop canto34 here on my dev machine, and use it in another project (in my case, a little program called Mettle that’s beginning to take shape.)

So I want to make changes to canto34 as I make changes to Mettle. I don’t want to have upload changes to canto34, and then download them in Mettle. I just want to develop both. 

npm lets you do this by CDing into canto34’s directory and typing

npm link

And then CDing into Mettle’s directory and typing

npm link canto34

Magic! npm ‘installs’ canto34 on my machine using a shortcut to my development copy, so that I now have this structure on my disk;


And I can now develop both projects together.

The thing here isn’t that the command line support is good — on its own, that’s pretty dull. The nice thing is now the npm system seems really well designed to get you sharing code, and doesn’t punish you for doing so. You can share libraries as open source, or use them in proprietary projects, or mix the two together, and it feels like npm is your buddy — a little like, I guess, an author feels about an editor; the technical guy that gets your artistry out to the world. 

All in all, a very good developer experience


Introducing Canto34.js, a lexing and parsing library in JavaScript

I mentioned it in passing in my last post, but I’ve been working on a project called Canto 34 which helps you write lexers and parsers in JavaScript. Here’s the current from the GitHub repo


Canto 34 is a library for building recursive-descent parsers.

When it comes to writing a parser, you get two main choices. Write your own, or use a parser generator like PEGJS or ANTLR.

I’ve never really had much success with parser generators, and it’s always seemed pretty easy to write a recursive-descent parser yourself, if you have some basic tools like a regex-based lexer, and some basic functions for matching tokens and reporting errors. Canto34.js gives you the functions you need to write a regex-based lexer and a recursive descent parser yourself.

A Simple Example

Here’s a simple example of a language which just defines name-value pairs;

foo 1, bar 5, baz 8.

We’ll see how to parse this language. First, you write a lexer which identifies the different kinds of token — names, integers, commas, and periods;

var lexer = new canto34.Lexer();

// add a token for whitespace
    name: "ws",       // give it a name
    regexp: /[ \t]+/, // match spaces and tabs
    ignore: true      // don't return this token in the result

// add a token type for names, defines as strings of lower-case characters
lexer.addTokenType({ name: "name", regexp: /^[a-z]+/  });

// bring in some predefined types for commas, period, and integers.
var types = canto34.StandardTokenTypes;

And here’s how you use it;

var tokens = lexer.tokenize("foo 1, bar 2.");

which returns a set of tokens;

    { content: "foo", type: "name",    line: 1, character: 1 },
    { content: 1,     type: "integer", line: 1, character: 5 },
    { content: ",",   type: "comma",   line: 1, character: 6 },
    { content: "bar", type: "name",    line: 1, character: 8 },
    { content: 2,     type: "integer", line: 1, character: 12 },
    { content: ".",   type: "period",  line: 1, character: 13 }

Now you feed these tokens into a parser. Here’s a parser for our language;

var parser = new canto34.Parser();
parser.listOfNameValuePairs = function() {
    this.result = [];
    while (!this.eof() && this.la1("comma")) {

parser.nameValuePair = function() {
    var name = this.match("name").content;
    var value = this.match("integer").content;
    this.result.push({ name:name, value: value });

And it’s used like this;

var tokens = lexer.tokenize("foo 1, bar 2, baz 3.");

parser.result now contains the value;

    {name:"foo", value:1},
    {name:"bar", value:2},
    {name:"baz", value:3},

And we’re done! That’s a basic lexer and parser, written in thirty lines of code.

What’s canto34.js good for?

Canto 34 is designed for quickly writing parsers for straightforward domain- specific languages (DSLs). Please don’t use it for writing parsers for general-purpose programming languages — it’s not designed to be that fast or powerful.

Using Twitter Bootstrap Typeahead to select a key/value pair, not a string

Twitter Bootstrap comes with a UI widget called Typeahead — an auto-complete textbox. Problem is, it works only with strings — you type a string, it searches a list of strings, when you choose one, it tells you you selected a string.

This is fine in some cases, but it’s not much cop if you want to select, say, a primary key field.

I’ve found an approach for this that lets you select in a more traditional manner — search names, return a different, unique value. It uses the ‘highlighter’ function and a little bit of JSON. Highlighter is a bit of a misnomer — it’s actually an arbitrary function for re-writing the text that appears in the dropdown.

So when you invoke, say, $(‘#mysearchbox’).typeahead(), you pass a set of options. Here’s what you need to do for each option;

source: Set the source option to a function returning an array of JSON-serialized name/value pairs;

return [
JSON.stringify({ name:”one”, value: 1 }},

    JSON.stringify({ name:”two”, value: 2 }},

    JSON.stringify({ name:”three”, value: 3 }},

    JSON.stringify({ name:”four”, value: 4 }},

    JSON.stringify({ name:”five”, value: 5 }}


highlighter: Set the highlighter to return just the name, not the value;

highlighter: function(item) {
return JSON.parse(item).name;

matcher: Set the matcher to return things that match the right name;

matcher: function (item) {


      .indexOf(this.query.toLocaleLowerCase()) != -1;

updater: Use the updater to catch the selected item;

updater: function (item) {
return JSON.parse(item).name;

This pattern lets you search for text but return a corresponding key.