Extending MSBuild with custom command line tools, Part I – Console Errors

Ever wanted to extend your build process in Visual Studio, but found VS extensions to be bag-of-snakes crazy?

If you work on a decent-sized or complex solution, you’ll probably find yourself wanting to generate some code, or validate an input file, or process some configuration data, as part of your build. For example, I recently needed to validate some JSON files against a complex set of validation rules. If the JSON files didn’t ‘make sense’, then the build process needed to fail. So let’s say we have this JSON file;

capture

Notice how the start date is after the end date, and that doesn’t make sense? I don’t want this file to accidentally go into production, so I want to validate it, and if the problem exists, fail, and fail properly, giving me an error in Visual Studio like this;

capture

So I certainly need to write a piece of code like this;

var fileContent = File.ReadAllText(args[0]);
var file = JsonConvert.DeserializeObject<DataFile>(fileContent);
if (file.StartDate > file.EndDate)
{
    // start date must be before the end date -- fail!
}

But how do insert that into your build process? If you look as Visual Studio’s ‘recommended’ way to do things, it starts to look difficult. You can write your own MSBuild task by inheriting Task, but there’s a lot of ceremony and you have to write the task in a different solution. You can also write an extension, but that’s even harder.

In this post, I describe a small trick that you can use to dramatically simplify the process of writing your own build tools. That makes it simple to write your own tools and have them integrate with the Visual Studio / MSBuild build system. You can find the source code on github.

Outline

Visual Studio detects build errors in a really simple way — it looks at the console output produced during the build process and looks for lines like this;

c:\src\BlogPost\BlogPost\Program.cs(13,17,13,18): error CS0818: Implicitly
  -typed variables must be initialized

So this occurs when I write this code in C#;

namespace BlogPost
{
    class Program
    {
        static void Main(string[] args)
        {
            var x;
        }
    }
}

When I build, Visual Studio runs MSBuild, which runs the C# compiler (csc.exe), which writes to the console; internally, there will be a line like this in the compiler;

Console.WriteLine(@"c:\src\BlogPost\BlogPost\Program.cs(13,17,13,18): err
  or CS0818: Implicitly-typed variables must be initialized");

And visual studio picks up on that line in the output, detects the build error, stops the build, and puts the error into the error window.

And here’s the ‘secret’ — absolutely anyone writing a similar message during the build process gets the same privileges. There’s no API to invoke, no DLLs to register in the GAC, no nothing clever. Just make a console app and have it write messages with Console.WriteLine. You can now handle errors as easily as C# does.

Steps

The code for this post can be found on GitHub.

Step 1: Create your console app.

First, create a new console app. If you’re improving an existing solution, you can just add the app into the solution itself.

Here’s the guts of Program.cs from the console app;

static void Main(string[] args)
{
    try
    {
        var fileContent = File.ReadAllText(args[0]);
        var file = JsonConvert.DeserializeObject(fileContent);
        if (file.StartDate > file.EndDate)
        {
            WriteBuildError("Error", args[0], 1, $"The start date is after the end date");
            return;
        }
    }
    catch
    {
        WriteBuildError("Error", new Uri(Assembly.GetExecutingAssembly().Location).AbsolutePath, 1, $"This tool expects 1 command-line argument");
    }
}

So you can see what’s happening; the main function loading the JSON file, and checking a condition – when that condition fails, it writes out a build error and quits.

That build error method looks like this;

private  static void WriteBuildError(string type, string filePath, int lineNumber, string message)
{
    var msBuildMessage = string.Format(@"{0}({1}) : {2}: {3}.", filePath, lineNumber, type, message);
    Console.WriteLine(msBuildMessage);
}

Simply writing a formatted message to the console.

So now,  you can see that it takes almost no code to write a console app which is going to be ‘compatible’ with MSBuild. Now if you want to do something — like generate content — now’s your chance. Read your input files, build source files, run tests, whatever else it is you want to do — and if there are problems with the inputs, or tests fail, you write out the errors.

So how do you integrate this console app into the build process?

Step 2: Altering the .csproj of your dependent project

So far you’ve written a console app that functions as your compiler, code generator, file vaidator, etc. You’ll also have a project that uses the output of that process — say, if you code-generate some C#, you can have a second project that then compiles it. Here’s an example solution structure;

capture

Solution Structure

We have BlogPost.csproj, which is the tool, and AppThatUsesTheTool. We want to invoke the tool to validate customConfig.json.

We open up the csproj file in a text editor and add this right at the bottom just above the tag;

  <PropertyGroup>
    <MyCustomToolPath>$([System.IO.Path]::GetFullPath("$(ProjectDir)..\BlogPost\$(OutDir)\BlogPost.exe"))</MyCustomToolPath>
    <MyInptuFile>$(ProjectDir)customConfig.json</MyInptuFile>
  </PropertyGroup>
  <Target Name="RunMyCustomTool" BeforeTargets="Build">
    <Exec Command="$(MyCustomToolPath) "$(MyInptuFile)"" IgnoreExitCode="true" />
  </Target>

Now, how does this work?

The <PropertyGroup> tag just defines a couple of properties and sets their content; those first four lines do the equivalent of C# like;

var MyCustomToolPath = System.IO.Path.GetFullPath($"{ProjectDir}..\BlogPost\{OutDir}\BlogPost.exe");
var MyInptuFile =$"{ProjectDir}customConfig.json";

The <Target> tag does the actual work. It extends the C# build with another arbitrary step. We give it an arbitrary name, and then tell it to run before the normal C# build. Inside, we say what we want to happen; we run the task, which just executes a batch command, and we feed it the path to BlogPost.exe, our command-line tool, passing it the file we want to validate. BlogPost’s Main method receives that as `args[0]`.

So now, we’re done. If you build the solution and the JSON file is invalid, BlogPost.exe writes the console error, the solution fails to build and you see a build error in Visual Studio;

capture

And you can double-click the error and it’ll jump into customConfig.json, just as it would in a C# file with a syntax error.

 

Canto34 parsing toolkit v2.0.1 released

I’ve updated my JavaScript toolkit for building recursive-descent parsers to ES2015, and updated it to v2.0.1. It’s on npm and github. The readme explains how to use it.

I’ve also just found out that Visual Studio 2015 Update 1 supports TextMate grammars — and Canto34 produces them. So you can write yourself a language, and use its lexer inside Visual Studio. This makes it great for producing DSLs like custom configuration files and testing languages. You’ll want to use the canto34-syntax.js code to produce a .tmLanguage file, and install it into Visual Studio using this tutorial.

Developing ASP.NET IIS Applications: Triggering LiveReload on a successful Visual Studio or MSBuild build

I’m all about getting rid of the friction in development. All that ‘ceremony’ that ends up built into your muscle memory to get your projects built and running.

One bit of small friction, but one you pay regularly, is pressing F5 in web applications, reloading the page after you’ve changed the source code. LiveReload is a really nice solution to that when you’re editing JavaScript and HTML, and in ASP.NET MVC when you’re editing Razor (CSHTML) files.

However, the job isn’t all done, and when you’re making server-side changes — changing controllers and such — LiveReload isn’t as easy to make fire. You really want it to trigger when it finishes a build, once all your binaries are up to date. I decided to fix that with a very small MSBuild target. The idea is that you add this into your web app (manually stitching it into your CSPROJ file) and when you build your source, it writes a small random file to disk. You use LiveReload to watch for that file, and that will trigger your browser to refresh. It’s a small — very small — improvement to your workflow if you’re already using LiveReload, but it’s something you do so often that any time you can get rid of a tiny piece of friction, you save yourself a little bit of the mental capacity you use for concentrating on real problems. In a very real way, each little bit of friction steals an IQ point or two; hoard them for the difficult tasks! As Alfred North Whitehead said:

“It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.” – ANW

You need to get LiveReload into your workflow. I’ve found the best for me is to use Grunt or Gulp, both of which are designed to work with LiveReload. On windows, it’s been the most reliable way for me to get it to run — the original app is an alpha version, there’s not been an update in a long time, and I’ve found it flakey.

So here’s what you need to do;

Make sure you’ve got node.js installed

Come on now. Get it if you haven’t already. Nodejs.

Create a package.json file in your solution directory.

If you don’t already have a package.json file, open a command window, change directory to your solution folder, and type

npm init

And hit ‘enter’ a bunch of times.  The defaults are fine, but you may need to type your project name in, in lower case, if it complains about your folder name.

Install Grunt and Grunt-Contrib-Watch

Now install Grunt, the task runner, and Grunt-Contrib-Watch, a package which watches for file changes and triggers useful work.


npm install -g grunt-cli
npm install grunt --save-dev
npm install grunt-contrib-watch --save-dev

Nice and straightforward!

Create a Gruntfile

In your Solution directory, create a file called Gruntfile.js. (Note the capitalisation.) Then add this to it;

module.exports = function(grunt) {

var reloadTokens = ['{{MyWebProjectFolder}}\LiveReloadOnSuccessfulBuild.token']

grunt.initConfig({
  watch: {
    files: reloadTokens,
    options: {
      livereload: true,
    }
  }
});

grunt.loadNpmTasks('grunt-contrib-watch');
grunt.registerTask('default', ['grunt-contrib-watch']);

};

Alter the file — you’ll see on the second line the token {{MyWebProjectFolder}} which you should replace with the relative folder path from your solution to your web application folder. Normally that’s the name of the project in Visual Studio. This is the config file that tells Grunt to watch for a file called ‘LiveReloadOnSuccessfulBuild.token’, a file that gets generated by a little bit of MSBuild trickery you’ll see in a minute. This file needs to go into source control.

Create the MSBuild target file

In your web application folder — the same folder that holds your CSPROJ file — create a file called LiveReloadOnSuccessfulBuild.targets and put in this content;

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 <ItemGroup>
 <!-- trick from https://blogs.msdn.microsoft.com/msbuild/2005/10/06/how-to-add-a-custom-build-action-to-visual-studio/ -->
 <AvailableItemName Include="UpdateOnSuccessfulBuild" />
 </ItemGroup>

<Target Name="LiveReloadOnSuccessfulBuild" AfterTargets="Build">
 <!-- If you're using LiveReload, you want your pages to reload 
 after your web app compiles. This add a build action, 
 "UpdateOnSuccessfulBuild", which you can use to mark a 
 dummy file. The build action writes a random guid to 
 the file within your web app folder; this can be watched 
 for in your Gruntfile.js, gulpfile.js, or by the 
 LiveReload app. -->
 <Message Importance="High" Text="Writing LiveReload token to LiveReloadOnSuccessfulBuild.token" />
 <WriteLinesToFile
 File="LiveReloadOnSuccessfulBuild.token"
 Lines="Live Reload Token: this changes on a successful build, to trigger LiveReloads. Exclude from source control! $([System.Guid]::NewGuid().ToString())"
 Overwrite="true"
 Encoding="Unicode"/>
 </Target>
</Project>

This is an MSBuild file — the same stuff that runs when you build your solution — and it instructs the build process to create a file called ‘LiveReloadOnSuccessfulBuild.token’ whenever the build completes. The Gruntfile I described earlier watches for this file, and when it finds it, it sends a signal to LiveReload to reload your page. This needs to go into source control.

Wire up the targets file into your web app build process

We need to wire this targets file up into visual studio’s build system. Open your web application’s CSPROJ file in a text editor, and on the penultimate line, just above </Project>, put this;


 <Import Project="$(ProjectDir)LiveReloadOnSuccessfulBuild.targets" />

Save the file, and if prompted in visual studio, allow it to reload your web app.

That’s all the editing done. Now, when you build in Visual Studio, it will alter the ‘LiveReloadOnSuccessfulBuild.token’ file and put something like this in;


Live Reload Token: this changes on a successful build, to trigger LiveReloads. Exclude from source control! ad7c25b3-dcfa-4571-98d9-ad9936c1e1d8

 

As it says, you may want to make sure this file isn’t checked into source control — if you’re using TFS, you don’t need to do anything, but if you’re using git, you may want to put it in your .gitignore file.

Install LiveReload extension in Chrome

When LiveReload works the way it does it is to watch your files, and when a change is made, it sends a signal. But what listens for the signal? You need to install an extension in Chrome.

use it!

OK, so we’re almost there. With a command window open in your solution directory, enter

grunt watch

That is, ‘run grunt with the ”watch” command’. If you look back at the grunt file, you’ll see that ‘watch’ is configured to look for that token, and send the signal to your browser.

Now that that’s started, on Chrome you’ll need to start the LiveReload extension watching for the signal. There’s ‘reload’ icon on the same line as the web address, with a little hollow circle surrounded by looping arrows. When you click it, the hollow circle is filled in, showing that the browser is connected to LiveReload. Be aware that it’s trying to connect to the LiveReload server you started with grunt watch, so you need to do it in that order — grunt, then browser.

We’re done! Now, whenever you build your web app, it will automatically refresh the browser. Although this was a bit of a slog, we’re now in the position where you don’t have to refresh your browser. If you have two monitors, you can code in one window and watch the web app develop in the other. Saves you mental energy and the RSI of constantly flicking your hand to the mouse!

Syntax highlighting added to Canto34

I’ve just extended my parser toolkit, canto34. Canto 34 is a library for building recursive-descent parsers. It can be used both on node.js, and in the browser as an AMD module.

Starting in v0.0.5, I’ve added a function to let you generate a `.tmLanguage` file. This is the syntax highlighting system of TextMate, Sublime Text, Atom, and Visual Studio code.

This is actually pretty sweet; it makes your language feel a bit more ‘first class’ when you get the richer IDE experience.

You first generate the file content, then save it wherever your text editor demands. For example, To use it with Sublime Text 3 on windows, call code like so, swapping `{myusername}` with the name of your user account;

 var content = canto34.tmLanguage.generateTmLanguageDefinition(lexer);    require(‘fs’).writeFileSync(“C:\\Users\\{myusername}\\AppData\\Roaming\\Sublime Text 3\\Packages\\User\\myparser.tmLanguage”, content);

This will write out a syntax highlighting file into your User folder, which is a personal ‘dumping ground’ for configuration bits and pieces on your machine for Sublime Text. When you restart ST, you’ll see your language in the list of language options, displayed in the bottom-right.

You’ll need to configure the tokens in your lexer a bit more. For example, if you have a variable name token, you’ll need to tell the lexer that this should be highlighted one way, and if you have a keyword like ‘for’, you’ll use another. Do this with the `roles` option on a token type;

    lexer.addTokenType({ name: “comment”, ignore: true, regexp: /^#.*/, role: [“comment”, “line”] });

lexer.addTokenType(types.constant(“for”,”for”, [“keyword”]));

lexer.addTokenType(types.constant(“[“, “openSquare”, [“punctuation”]));

lexer.addTokenType(types.constant(“]”, “closeSquare”, [“punctuation”]));

The list of roles isn’t clear to me – I’ve just figured out some key ones like `keyword`, `comment`, and `punctuation` from reverse engineering existing `tmLanguage` files.

Find out more on GitHub or NPM.

Pretty-printing recordsets from the mssql node package

I’ve been using the node mssql package in node scripts, and it’s great but it doesn’t have a pretty-printer for recordsets; console.dir() will print something out but you get some nasty-looking JSON formatted output.

This function is just a reusable printer for recordsets returned from sql.Request().query() or sql.Request().batch().methods.

Pass in the recordset and a function which prints the line, eg;

printTable(recordset, console.log);

And you get something more famililar with a SQL Server Management Studio feel;

-- SELECT id, name from table
id   name
===========
6    p1
7    p2
8    p3

Here’s the function:

// pretty-print a table returned from mssql node package 
// https://www.npmjs.com/package/mssql
var printTable = function(recordset, printLine) {

 var padEnd = function (str, paddingValue) {
   while(str.length < paddingValue) {
     str = str + " ";
   }
   return str;
 };

 var print = function(value) {
   return (value == undefined || value == null) ? "(null)" : value.toString(); 
 }

 var maxWidth = {};

 for(var c in recordset.columns) {
   maxWidth[c] = c.length;
 }

 var l = recordset.length;
 for(var r = 0; r < l; r++) {
   var row = recordset[r];
   for(var c in recordset.columns) {
     var col = recordset.columns[c];
     row[c] = print(row[c]);
     maxWidth[c] = Math.max(maxWidth[c], row[c].length);
   }
 }

 var head = "";
 for(var c in recordset.columns) {
   var head = head + padEnd(c, maxWidth[c]) + " ";
 }

 printLine(head);

 var sep = Array(head.length).join("=");
 
 printLine(sep);

 for(var r = 0; r < l; r++) {
   var row = recordset[r];
   var printedRow = "";
   for(var c in recordset.columns) {
     printedRow = printedRow + padEnd(row[c], maxWidth[c]) + " ";
   }
   printLine(printedRow);
 }
};

Calculating expected finger table entries for virtual ring distributed systems like Chord and Cassandra

Here’s a block of code I wrote to print out finger tables for Chord-like distributed hash tables. Just update the ‘nodes’ variable and the m-value to adjust to any system. Runs happily in Node.js 4.4.0.

// calculating finger tables for Chord-like distributed hash table.

// the size of the ring, as 2^m;
var m=9, ringSize = Math.pow(2,m);

// the nodes we know about
var nodes = [1, 12, 123, 234, 345, 456, 501]; 

// just make sure it's sorted
nodes = nodes.sort( (a,b) => a-b );

console.log('the finger tables for a ', ringSize,'-virtual-node system');
console.log('========');
console.log('nodes: ', nodes);

// calculate the finger table for each node
nodes.forEach(n => {

 var fingerTable = [];
 i = 1;
 for(var x = 0; x < m; x++) {
 var nodeId = (n + i) % ringSize;
 // find the node after nodeId; 
 var ftEntry = nodes.filter(x => x > nodeId).concat(nodes)[0];
 fingerTable.push(ftEntry);
 i = i << 1;
 }
 
 console.log(n + ':', fingerTable);
});

Fun With ORM Includes!

This is a cautionary tale about how an ORM (in this case, Entity Framework) can screw your performance if you don’t pay attention to the queries it generates. Particularly, the use of recusive includes.

Here’s a simplified version of the original problem;

var person = await DataContext.Set<Person>()
 .Include(p => p.Groups)
 .Include(p => p.Manager)
 .Where(p => p.Id == personId)
 .SingleOrDefaultAsync();

So here’s the intent, and the bug. The code above is supposed to say ‘load a person with the specific ID, and include his/her groups and manager’

And here’s the problem. EF sees that you want to load Person.Manager. Person.Manager is a person, so we’ll need to include the manager’s groups. But ah! Person.Manager has a Manager, which has a Manager, which has a Manager. EF goes batshit at this point, and generates over 90 joins;

person join person join person join person join person ....

And that query goes off, loads far too much data, and takes far too much time; in the order of seconds. Now this is a query that should load in small numbers of milliseconds..

The fix I put in was to divide this into two, much faster queries. Something more like this;

// don't load manager etc.
 var person = await DataContext.Set<Person>()
 .Include(p => p.Groups)
 .Where(p => p.Id == personId)
 .SingleOrDefaultAsync();

if (person.ManagerId.HasValue) {
 var manager = await DataContext.Set<Person>()
 .Include(p => p.Groups)
 .Where(p => p.Id == person.ManagerId.Value)
 .FirstOrDefaultAsync();
 }

So we hit the database twice, but with much cheaper queries. This may not be perfect (I’d like to get everything in one shot) but I couldn’t think of a better approach.

Here’s the takeaway, though. It’s almost never right to use a recursive include (like Person.Manager) unless you want to load complete branches or ancestor paths. When mixed with other includes (like Person.Groups) it leads to an explosion in the complexity of the query. So;

  • Measure the performance of any queries you write. Use the free Stackify Prefix  or some other tool to make sure your queries are small-numbers-of-milliseconds fast. If not, question whether you could make it faster.
  • Use Stackify Prefix to examine the generated SQL. Sure, EF can be a bit verbose, but any time it generates more than a page of SQL, question whether the query is as simple as it could be.
  • Don’t ask for it if you don’t need it. Don’t Include() in data just in case.
  • Two very cheap queries beats one awful one. EF’s only so clever, and it doesn’t handle lots of includes well, so don’t be afraid of two efficient queries when EF goes mental.
  • On the other hand, don’t take that as permission to write an N+1 bug. This is where you load a ‘master’ object, then iterate through children, loading each in turn. This will kill your server, and it’s the most common database performance bug written in enterprise software.