A Multilevel Cache Retrieval Design for .NET

Caching data is vital for high-performance web applications. However, cache retrieval code can get messy and hard to test without the proper abstractions. In this article, we’ll start an ugly multilevel cache and progressively refine it into something maintainable and testable.

The source code for this article is available at the Gigi Labs BitBucket repository.

Naïve Multilevel Cache Retrieval

A multilevel cache is just a collection of separate caches, listed in order of speed. We typically try to retrieve from the fastest cache first, and failing that, we try the second fastest; and so on.

For the example in this article we’ll use a simple two-level cache where:

We’re going to build a Web API method that retrieves a list of supported languages. We’ll prepare this data in Redis (e.g. using the command SADD languages en mt) but will leave the MemoryCache empty (so it will have to fall back to the Redis cache).

A simple implementation looks something like this:

    public class LanguagesController : ApiController
    {
        // GET api/languages
        public async Task<IEnumerable<string>> GetAsync()
        {
            // retrieve from MemoryCache

            var valueObj = MemoryCache.Default.Get("languages");

            if (valueObj != null)
                return valueObj as List<string>;
            else
            {
                // retrieve from Redis

                var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
                var db = conn.GetDatabase(0);
                var redisSet = await db.SetMembersAsync("languages");

                if (redisSet == null)
                    return null;
                else
                    return redisSet.Select(item => item.ToString()).ToList();
            }
        }
    }

Note: this is not the best way to create a Redis client connection, but is presented this way for the sake of simplicity.

Data Access Repositories and Multilevel Cache Abstraction

The controller method in the previous section is having to deal with cache fallback logic as well as data access logic that isn’t really its job (see Single Responsibility Principle). This results in bloated controllers, especially if we add additional cache levels (e.g. fall back to database for third-level cache).

To alleviate this, the first thing we should do is move data access logic into repositories (this is called the Repository pattern). So for MemoryCache we do this:

    public class MemoryCacheRepository : IMemoryCacheRepository
    {
        public Task<List<string>> GetLanguagesAsync()
        {
            var valueObj = MemoryCache.Default.Get("languages");
            var value = valueObj as List<string>;
            return Task.FromResult(value);
        }
    }

…and for Redis we have this instead:

    public class RedisCacheRepository : IRedisCacheRepository
    {
        public async Task<List<string>> GetLanguagesAsync()
        {
            var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
            var db = conn.GetDatabase(0);
            var redisSet = await db.SetMembersAsync("languages");

            if (redisSet == null)
                return null;
            else
                return redisSet.Select(item => item.ToString()).ToList();
        }
    }

The repositories each implement their own interfaces, to prepare for dependency injection which is one of our end goals (we’ll get to that later):

    public interface IMemoryCacheRepository
    {
        Task<List<string>> GetLanguagesAsync();
    }

    public interface IRedisCacheRepository
    {
        Task<List<string>> GetLanguagesAsync();
    }

For this simple example, the interfaces look almost identical. If your caches are going to be identical then you can take this article further and simplify things even more. However, I’m not assuming that this is true in general; you might not want to have a multilevel cache everywhere.

Let’s also add a new class to abstract the fallback logic:

    public class MultiLevelCache
    {
        public async Task<T> GetAsync<T>(params Task<T>[] tasks) where T : class
        {
            foreach(var task in tasks)
            {
                var retrievedValue = await task;

                if (retrievedValue != null)
                    return retrievedValue;
            }

            return null;
        }
    }

Basically this allows us to pass in a number of tasks, each corresponding to a cache lookup. Whenever a cache lookup returns null, we know it’s a cache miss, which is why we need the where T : class restriction. In that case we try the next cache level, until we finally run out of options and just return null to the calling code.

This class is async-only to encourage asynchronous retrieval where possible. Synchronous retrieval can use Task.FromResult() (as the MemoryCache retrieval shown earlier does) to conform with this interface.

We can now refactor our controller method into something much simpler:

        public async Task<IEnumerable<string>> GetAsync()
        {
            var memoryCacheRepository = new MemoryCacheRepository();
            var redisCacheRepository = new RedisCacheRepository();
            var cache = new MultiLevelCache();

            var languages = await cache.GetAsync(
                memoryCacheRepository.GetLanguagesAsync(),
                redisCacheRepository.GetLanguagesAsync()
            );

            return languages;
        }

The variable declarations will go away once we introduce dependency injection.

Multilevel Cache Repository

The code looks a lot neater now, but it is still not testable. We’re still technically calling cache retrieval logic from the controller. A cache depends on external resources (e.g. databases) and also potentially on time (if expiry is used), and that’s not good for unit tests.

A cache is not very different from the more tangible data sources (such as Redis or a database). With them it shares the function of retrieving data and the nature of relying on resources external to the application, which makes it incompatible with unit testing. A multilevel cache has the additional property that it is an abstraction for the underlying data sources, and is thus itself a good candidate for the repository pattern:

multilevel-cache-repository

We can now move all our cache retrieval logic into a new MultiLevelCacheRepository class:

    public class MultiLevelCacheRepository : IMultiLevelCacheRepository
    {
        public async Task<List<string>> GetLanguagesAsync()
        {
            var memoryCacheRepository = new MemoryCacheRepository();
            var redisCacheRepository = new RedisCacheRepository();
            var cache = new MultiLevelCache();

            var languages = await cache.GetAsync(
                memoryCacheRepository.GetLanguagesAsync(),
                redisCacheRepository.GetLanguagesAsync()
            );

            return languages;
        }
    }

Our controller now needs only talk to this repository, and carry out any necessary logic after retrieval (in this case we don’t have any):

        public async Task<IEnumerable<string>> GetAsync()
        {
            var repo = new MultiLevelCacheRepository();
            var languages = await repo.GetLanguagesAsync();
            return languages;
        }

Dependency Injection

Our end goal is to be able to write unit tests for our controller methods. A prerequisite for that is to introduce dependency injection.

Follow the instructions in “ASP .NET Web API Dependency Injection with Ninject” to set up Ninject, or use any other dependency injection framework you prefer.

In your dependency injection configuration class (NinjectWebCommon if you’re using Ninject), set up the classes and interfaces you need:

        private static void RegisterServices(IKernel kernel)
        {
            kernel.Bind<IMemoryCacheRepository>().To<MemoryCacheRepository>()
                .InSingletonScope();
            kernel.Bind<IRedisCacheRepository>().To<RedisCacheRepository>()
                .InSingletonScope();
            kernel.Bind<IMultiLevelCacheRepository>().To<MultiLevelCacheRepository>()
                .InSingletonScope();
            kernel.Bind<MultiLevelCache>().To<MultiLevelCache>()
                .InSingletonScope();
        }

Note: you can also set up an interface for MultiLevelCache if you want. I didn’t do that out of pure laziness.

Next, refactor MultiLevelCacheRepository to get the classes it needs via constructor injection:

    public class MultiLevelCacheRepository : IMultiLevelCacheRepository
    {
        private IMemoryCacheRepository memoryCacheRepository;
        private IRedisCacheRepository redisCacheRepository;
        private MultiLevelCache cache;

        public MultiLevelCacheRepository(
            IMemoryCacheRepository memoryCacheRepository,
            IRedisCacheRepository redisCacheRepository,
            MultiLevelCache cache)
        {
            this.memoryCacheRepository = memoryCacheRepository;
            this.redisCacheRepository = redisCacheRepository;
            this.cache = cache;
        }

        public async Task<List<string>> GetLanguagesAsync()
        {
            var languages = await cache.GetAsync(
                memoryCacheRepository.GetLanguagesAsync(),
                redisCacheRepository.GetLanguagesAsync()
            );

            return languages;
        }
    }

Do the same with the controller:

    public class LanguagesController : ApiController
    {
        private IMultiLevelCacheRepository repo;

        public LanguagesController(IMultiLevelCacheRepository repo)
        {
            this.repo = repo;
        }

        // GET api/languages
        public async Task<IEnumerable<string>> GetAsync()
        {
            var languages = await repo.GetLanguagesAsync();
            return languages;
        }
    }

…and make sure it actually works:

multilevel-cache-verify

Unit Test

Thanks to this design, we can now write unit tests. There is not much to test for this simple example, but we can write a simple (!) test to verify that the languages are indeed retrieved and returned:

        [TestMethod]
        public async Task GetLanguagesAsync_LanguagesAvailable_Returned()
        {
            // arrange

            var languagesList = new List<string>() { "mt", "en" };

            var memCacheRepo = new Mock<MemoryCacheRepository>();
            var redisRepo = new Mock<RedisCacheRepository>();
            var cache = new MultiLevelCache();
            var multiLevelCacheRepo = new MultiLevelCacheRepository(
                memCacheRepo.Object, redisRepo.Object, cache);
            var controller = new LanguagesController(multiLevelCacheRepo);

            memCacheRepo.Setup(repo => repo.GetLanguagesAsync())
                        .ReturnsAsync(null);
            redisRepo.Setup(repo => repo.GetLanguagesAsync())
                        .ReturnsAsync(languagesList);

            // act

            var languages = await controller.GetAsync();
            var actualLanguages = new List<string>(languages);

            // assert

            CollectionAssert.AreEqual(languagesList, actualLanguages);
        }

Over here we’re using Moq’s Mock objects to help us with setting up the unit test. In order for this to work, we need to make our GetLanguagesAsync() method virtual in the data repositories:

public virtual Task<List<string>> GetLanguagesAsync()

Conclusion

Caching makes unit testing tricky. However, in this article we have seen how we can treat a cache just like any other repository and hide its retrieval implementation details in order to keep our code testable. We have also seen an abstraction for a multilevel cache, which makes cache fallback straightforward. Where cache levels are identical in terms of data, this approach can probably be simplified even further.

On Unit Testing Private Methods

Unit tests are easy to write for methods that are publicly accessible. However, when it becomes necessary to test methods with modifiers other than public, things tend to get rather complicated. This article presents a brief overview of some of the techniques involved in achieving this.

Breaking Encapsulation

What does the book say about this particular problem? The Art of Unit Testing essentially says that it is okay to break your class’s encapsulation and expose its inner members if it makes the class testable. Quoting from Chapter 3:

“By now, you may be thinking to yourself that adding all these constructors, setters, and factories for the sake of testing is problematic. It breaks some serious object-oriented principles, especially the idea of encapsulation, which says, “Hide everything that the user of your class doesn’t need to see.” That’s our next topic. (Appendix A also deals with testability and design issues.)

“Some people feel that opening up the design to make it more testable is a bad thing because it hurts the object-oriented principles the design is based on. I can wholeheartedly say to those people, “Don’t be silly.” Object-oriented techniques are there to enforce some constraints on the end user of the API (the end user being the programmer who will use your object model) so that the object model is used properly and is protected from unforeseen ways of usage. Object orientation also has a lot to do with reuse of code and the single-responsibility principle (which requires that each class has only a single responsibility).

“When we write unit tests for our code, we are adding another end user (the test) to the object model. That end user is just as important as the original one, but it has different goals when using the model. The test has specific requirements from the object model that seem to defy the basic logic behind a couple of object-oriented principles, mainly encapsulation. Encapsulating those external dependencies somewhere without allowing anyone to change them, having private constructors or sealed classes, having nonvirtual methods that can’t be overridden: all these are classic signs of overprotective design. (Security-related designs are a special case that I forgive.) The problem is that the second end user of the API, the test, needs them as a feature in the code. I call the design that emerges from designing with testability in mind testable object-oriented design (TOOD), and you’ll hear more about TOOD in Appendix A.”

So the guy who wrote the book is basically telling us: “Don’t be silly. It’s okay to piss on the several-decade-old discipline of Object Oriented Programming if it lets us test our classes. In fact, I’m going to invent my own design that allows it.”

A twist on publicly exposing a class’s internals (which the book also mentions) is to make them internal and use InternalsVisibleTo. Although more restrictive, this practice is no less dangerous.

Breaking a class’s encapsulation and opening it up for public access defies the whole point of encapsulation: to restrict the access to certain data so that (a) it can be changed only via well-defined channels and is protected against corruption, and (b) it can be refactored without affecting lots of code that relies on it (“client code”). If a test can gain access to those internals, then so can regular code that shouldn’t be able to see it.

Update 30th January 2016: Roy Osherove, author of The Art of Unit Testing, pointed out on Twitter and in the comments below that the above-quoted information is based on the original 2006 edition of the book. From his Twitter comment:

“In the 2nd edition I strongly advise against testing private methods. This blog is based on a book from 2006.”

His comment at the bottom of this article further elaborates on this and provides a link to a list of things that have changed since the first edition.

Reflection

On the other end of the spectrum, there are those who don’t want to break encapsulation, but because of some religious obligation towards [insert test-related methodology here] they have to have 100% unit test coverage, including public methods, protected, internal, private, and even those written by Chuck Norris Jon Skeet which are pointless to test because they can be axiomatically taken to be correct.

Such people often go to great lengths in order to test their private methods, and their approach of choice usually involves Reflection. (A more recent variant is to use dynamic.) Here are some approaches I’ve seen:

While these approaches will allow you to test the internals of a class without breaking encapsulation per se, they are pretty messy and will make your tests hard to maintain.

More importantly, however, the very fact that the private members of a class need to be manipulated in order to write unit tests is often the sign of a flawed design, which brings us to…

Refactoring

So why do we need to test these class internals anyway? Actually, in principle, it’s fairly reasonable. Your class has a private helper method, say, to validate an email address. So why not write unit tests to ensure that that particular code is behaving correctly, before testing the class as a whole?

This is where the book quoted earlier kind of has a point. It is right in suggesting that sometimes our designs may be overprotective. This does not mean we should break encapsulation; it only means we should reconsider our design to ensure that we are following object oriented principles correctly.

So when we go back and look at our private email-validating method, is there really any reason why it should be locked up in a class? Is it so unreasonable for it to be taken out into its own EmailValidator class and used in other parts of the software? Does it really belong in the original class in the first place? Think Single Responsibility Principle. There’s a reason why serialization logic is usually not encapsulated in model classes, for example.

In general it should be sufficient to write unit tests against only the public interfaces of our classes, and careful design often enables this. This allows us to maintain encapsulation where it makes sense while at the same time making it unnecessary to resort to unorthodox techniques to test every last line of code, private or otherwise.

Naturally, there are exceptions to this, and sometimes particular use cases will mandate the need to unit test private methods. That’s okay, and that’s why it’s important to evaluate methodologies according to the need you have at the time. Software engineering is too young an industry for there to be hard-and-fast rules.

Ultima 1 Reverse Engineering: Decoding Savegame Files

This article was originally posted at Programmer’s Ranch on 29th August 2013. The version migrated here is mostly the same except for minor updates and a brand new screenshot of the latest GOG installer for Ultima 1.

It’s no secret that I’m a long-time fan of the Ultima series of games. My most significant contribution to the Ultima fan community was running Dino’s Ultima Page, a news and information hub for the community, for over a decade.

Ultima insipired such a great following that hundreds of fan-made remakes, tools and other projects appeared over the years. Many remakes were intended to be mods of games such as Dungeon Siege or Neverwinter Nights. Among other things, talented developers found ways to understand the data files of the games (including the graphics, savegames, maps, etc). This resulted in several useful tools (savegame editors, map viewers, etc), but more importantly, it allowed people to create new game engines using the original data files, with the intention of allowing the games to be played on modern operating systems. The first of these was Exult (an Ultima 7 engine), but over the years, projects appeared for most Ultimas – such as Pentagram for Ultima 8, or Nuvie for Ultima 6.

This article describes the process of reverse engineering, i.e. how to try to make sense of data files given the data files and nothing else. We will be looking at the savegame format of Ultima 1. Although Ultima 1 is not free, you can get the entire first Ultima trilogy from Good Old Games for a few bucks. Ultima 1 is a great choice to start reverse engineering because it’s relatively simple – the savegame file is only 820 bytes long and is uncompressed. This made sense for me as when I started this project I was still a budding programmer, and it also made sense because there was very little knowledge about the U1 formats around, whereas the later games had been studied in depth. Reverse engineering is a bit of an advanced topic so feel free to skip it if you feel lost, but it’s also a very interesting topic.

So first thing you need to do is install Ultima 1. If you got it off Good Old Games, it conveniently comes with DOSBox, allowing you to play it under modern operating systems:

u1revenge-install-u1-gog

After launching the game, you will find yourself in the main menu.

u1revenge-mainmenu

Press ‘a’ to go through the character creation process.

u1revenge-chargen-6

Once you have selected your attributes and saved your character, you find yourself back in the main menu. In the Ultima 1 folder, you should notice a new file called PLAYER1.U1:

u1revenge-mainmenu-savefile

That’s the savegame file we’ll be messing around with. You can use a hex editor to take a glimpse of its contents. Personally I like XVI32 because it’s pretty lightweight and even allows you to edit hex entries.

u1revenge-xvi32

This might look like gibberish, but you can already notice a few things. The first 14 bytes in the savegame file are reserved for the player’s name. Knowing the stats you chose during character creation (strength 32, agility 32, stamina 15, charisma 12, wisdom 11 and intelligence 13), you can also spot them in hex on the second line (20, 20, 0F, 0C, 0B, 0D). Windows Calculator has a Programmer mode that is quite useful for switching between decimal and hex:

u1revenge-wincalc

A good way of decoding more parts of the savegame file is interacting with the game itself. Start the game. The world view looks like this:

u1revenge-startworld

Keep a copy of your savegame file at this point. If you move around a bit and save the game, you should at the very least observe changes in your character’s X- and Y-coordinates:

u1revenge-u1moved

You can manually check which bytes in the savegame file changed. I found it more useful to write a hex diff tool that actually highlights the bytes that changed:

u1revenge-hexcompare

As you can see, it’s not so simple: there are many things that might change, including food. However, you can choose your moves carefully (e.g. 2 steps west, 3 steps north) so that you can then spot which bytes have changed that much and determine which are the world coordinates.

Another way of learning more about the savegame file is by actually tampering with it. Using XVI32, I changed the byte after ’96’ from 00 to 13:

u1revenge-tampering

After running the game, note how the Hits shot up from 150 to 5014:

u1revenge-tampering-result

That makes a bit of sense: the 96 (hex) we saw earlier corresponds to 150 in decimal – the original value of Hits. But why did it become 5014 when we tweaked the byte after it?

It’s because DOS games like this stored values as 16-bit integers in little endian format (i.e. the bigger byte is the second one). So if we have the value we tweaked, i.e. 96 13, that’s actually (13 * 100) + 96 (all hex), which results in 5014 (decimal).

Isn’t that neat? Reverse engineering requires a lot of time and patience, but it’s a bit like fitting together the pieces of a jigsaw puzzle. After a while you might end up understanding a good chunk of the data files:

u1revenge-savegame-file-format

Once you understand the data files (which also includes map and graphics files), you can then proceed to write all sorts of tools and stuff. I had called this project U1Revenge (Ultima 1 Reverse Engineering Effort) and wrote a map viewer and was working on an engine for it. Although I stopped working on it, I did release a couple of demos, the latest of which you can grab from the project page.

u1revenge-engine

Reverse engineering is certainly not a new art. The book Masters of Doom describes how fans of DOOM would hack the game’s map files to create their own level editors. Many games have similarly been studied, and a wealth of knowledge is available today. Reverse engineering is not just an achievement; it is a glimpse of history, and helps to understand how games were created even before we were born. The following links provide further reading:

Loading and Saving Images with SDL2

We have seen in past articles how we can load basic image formats (such as BMP, PNG and JPG) using SDL2 and SDL_image 2.0. In this article, we’re going to learn a bit more about the image formats we can load and save with these libraries.

Saving BMPs

Just like SDL2 gives us SDL_LoadBMP() to load BMP images, it also gives us SDL_SaveBMP() to save them. There is no need for SDL_image to use either of these functions, because they are in core SDL2.

Since we know from past articles that we can use SDL_image to load a few other formats, it is then easy to write a small program to convert from PNG (or any other of the supported format) to a BMP:

#include <SDL.h>
#include <SDL_image.h>

int main(int argc, char ** argv)
{
    SDL_Surface * image = IMG_Load("image.png");
    SDL_SaveBMP(image, "out.bmp");
    SDL_FreeSurface(image);

    return 0;
}

As you can see, it is not even necessary to initialize SDL2. We simply load the PNG file, and save it back to disk as BMP.

Saving PNGs

As someone pointed out in this forum thread, there are two undocumented functions that allow you to save a PNG:

sdl2-savepng

Thus, we can use this function to do the opposite conversion from BMP to PNG:

    SDL_Surface * image = SDL_LoadBMP("image.bmp");
    IMG_SavePNG(image, "out.png");
    SDL_FreeSurface(image);

Loading other formats

Disgracefully, there is no documentation for SDL_image 2.0 at the time of writing this article. The ‘documentation’ links on the SDL_image 2.0 homepage are actually for SDL_image 1.2.8, which is very misleading.

The old documentation for IMG_Init() shows three supported flag values you can pass in: IMG_INIT_JPG, IMG_INIT_PNG and IMG_INIT_TIF. Through intellisense I’ve discovered a fourth one for the WEBP format:

sdl2-init-webp

I’ve also found that calling IMG_Init() and IMG_Quit() is completely unnecessary, as everything seems to work without them.

Finally, despite the few init flags mentioned above, there are many more formats support by IMG_Load(). I’ve tested WEBP, PCX, GIF, PPM, TIF, TGA and BMP with success; and there are other formats in the old documentation that may be supported.

Visual Studio bug with long const strings

It turns out that constants aren’t only problematic in ASP .NET 5.

If you have a pretty long const string with a null character (\0) somewhere towards the beginning, then you’ll be pretty surprised to get the following error when you try to compile:

Unexpected error writing debug information — ‘Error HRESULT E_FAIL has been returned from a call to a COM component.’

Say you have a 3,000-character string that looks something like this:

const string s = "aaaaaaaaaaaa\0aa...aaaaa";

Although this is perfectly valid, Visual Studio fails to compile it:

vsconststringbug

I created a Stack Overflow question and a Visual Studio bug report for this.

Comments on the Stack Overflow question confirmed this to be a bug in the PDB writer (Eric Lippert). It’s Visual Studio specific, and doesn’t happen when compiling from the command line (Jon Skeet). Apparently it happens if the null character is before the 2033rd index in the string (Tommy).

One workaround suggested by Tommy is to use the string literal notation, which actually works:

const string s = @"aaaaaaaaaaaa\0aa...aaaaa";

Another workaround is to simply drop the const qualifier.

More Gulp in Practice

In my introductory Gulp article, I explained how to set up Gulp, and how to use it to concatenate and minify JavaScript files. In this article, we’re going to see a bunch of techniques that you’ll find useful when working with Gulp in practice.

We’re going to start with the following package.json:

{
  "name": "gulptest",
  "version": "1.0.0",
  "description": "Learning to use Gulp.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Daniel D'Agostino",
  "license": "ISC",
  "devDependencies": {
    "gulp": "^3.9.0",
    "gulp-concat": "^2.6.0",
    "gulp-uglify": "^1.5.1"
  }
}

…and the following Gulpfile.js:

var gulp = require('gulp'),
    uglify = require('gulp-uglify'),
    concat = require('gulp-concat');
     
gulp.task('default', function() {
  return gulp.src('js/*.js')
    .pipe(concat('all.js'))
    .pipe(uglify())
    .pipe(gulp.dest('dist/'));
});

This is where we left off in the previous Gulp article.

Restoring Packages

And as we have seen in the previous article, it takes a bit of work to set things up the first time. You have to install a bunch of packages, set up your package.json, and then write your Gulpfile. Fortunately, however, this needs to be done only once. We’ve been installing Gulp and the packages it requires using the –save-dev flag for a reason. Once they get added to the package.json, a person getting this file from source control needs only to run the following command:

npm install

We didn’t specify a package to install, so npm will look in your package.json file and install any packages you’re missing. No need to install all the required Gulp packages every time.

gulp-npm-install

It is still necessary, however, to install Gulp globally (npm install gulp -g) to be able to run the gulp command.

Watch

Some people think it’s a pain in the ass to have to run Gulp every time you change a script. Well, it turns out that there’s a Gulp plugin that will run tasks automatically when changes are detected.

All we need to do is add a watch task that will execute a particular task when changes to a specified set of files are detected:

gulp.task('watch', function() {
    gulp.watch('js/*.js', ['default']);
});

Try this by making a change to one of your JavaScript files (e.g. adding a space) and saving the file. watch detects this and runs the default task:

gulp-watch

JSHint

JavaScript is known to be a language with a lot of pitfalls, and there are many reasons why it takes extra effort to do things right. I’m not going to get into the details here; read JavaScript: The Good Parts (book) if you’re curious.

Fortunately, however, there’s a set of tools we can use to help us. These tools carry out a process called linting, which means syntactic analysis of the code. Just like with compiling just about any programming language, syntactic analysis reports any errors in your programming syntax. That is very useful in catching bugs early.

A linter called JSHint can be used by Gulp. In order to use it, we’ll need to install both jshint and gulp-jshint:

npm install jshint gulp-jshint --save-dev

With that done, add JSHint to your task:

var gulp = require('gulp'),
    jshint = require('gulp-jshint'),
    uglify = require('gulp-uglify'),
    concat = require('gulp-concat');
     
gulp.task('default', function() {
  return gulp.src('js/*.js')
    .pipe(jshint())
    .pipe(jshint.reporter('default'))
    .pipe(concat('all.js'))
    .pipe(uglify())
    .pipe(gulp.dest('dist/'));
});

gulp.task('watch', function() {
    gulp.watch('js/*.js', ['default']);
});

The order of placement of your jshint() is important. If you place it after your concat() call, you’ll see all errors coming from all.js, rather than the actual source filenames.

Now, you’d think jQuery library code would pass linting with flying colours, right? Right?

gulp-jshint-jquery

Development and Production Tasks

If we’re minifying our JavaScript, then how do we debug it? We can’t possibly read minified code.

In fact, we don’t have to. We can set up different tasks for Gulp to run in development and production. For instance:

var gulp = require('gulp'),
    jshint = require('gulp-jshint'),
    uglify = require('gulp-uglify'),
    concat = require('gulp-concat');
     
gulp.task('dev', function() {
  return gulp.src('js/*.js')
    .pipe(jshint())
    .pipe(jshint.reporter('default'))
    .pipe(concat('all.js'))
    //.pipe(uglify())
    .pipe(gulp.dest('dist/'));
});

gulp.task('prod', function() {
  return gulp.src('js/*.js')
    //.pipe(jshint())
    //.pipe(jshint.reporter('default'))
    .pipe(concat('all.js'))
    .pipe(uglify())
    .pipe(gulp.dest('dist/'));
});

gulp.task('watch', function() {
    gulp.watch('js/*.js', ['dev']);
});

Notice I’ve changed the name of the tasks, and commented out different parts of the pipeline in dev and prod. We don’t want to minify our JavaScript during development because we want to be able to debug it, and we don’t want to waste time linting it when preparing a release, because hopefully it turned out impeccable as a result of our development. 🙂

Then, just run:

gulp <taskname>

See:

gulp-named-tasks

Task Dependencies

Right, but now, we lost the ability to just run gulp without specifying the task name, because we no longer have a default task. We can fix this as follows:

gulp.task('default', ['dev']);

Instead of specifying a function for the task to execute, we’re giving it an array of task names it is dependent on. In this case, we’re saying that the default task is dependent on the dev task. So Gulp will run dev first, then default:

gulp-dependencies

This is particularly useful when you want to chain tasks. For instance, your dev task will depend on separate tasks for processing JavaScript and CSS. Speaking of which…

CSS

We’ve mostly seen examples dealing with JavaScript files, but Gulp can do much more than that.

Dealing with CSS is not very different from what we’ve seen so far, but we need a separate plugin for minification:

npm install gulp-minify-css --save-dev

Remember to add a require() for this at the top:

    minifycss = require('gulp-minify-css'),

With that available, we can now create a separate task to concatenate and minify CSS:

gulp.task('dev-css', function() {
  return gulp.src('css/*.css')
    .pipe(concat('all.css'))
    .pipe(minifycss())
    .pipe(gulp.dest('dist/'));
});

This is a good time to rename our former dev task to dev-js for clarity:

gulp.task('dev-js', function() {
  return gulp.src('js/*.js')
    .pipe(jshint())
    .pipe(jshint.reporter('default'))
    .pipe(concat('all.js'))
    //.pipe(uglify())
    .pipe(gulp.dest('dist/'));
});

Then, using the task dependency mechanism we have seen in the previous section, we can set things up so that both these tasks are run when invoking gulp dev or just gulp:

gulp.task('dev', ['dev-js', 'dev-css']);

gulp.task('default', ['dev']);

And here you go:

gulp-css-with-dependencies

Notify

Rather than explaining this one, let’s install it and see what it does:

npm install gulp-notify --save-dev

Let’s update our Gulpfile to notify us when one of the core tasks is ready:

var gulp = require('gulp'),
    jshint = require('gulp-jshint'),
    uglify = require('gulp-uglify'),
    minifycss = require('gulp-minify-css'),
    notify = require('gulp-notify'),
    concat = require('gulp-concat');
     
gulp.task('dev-js', function() {
  return gulp.src('js/*.js')
    .pipe(jshint())
    .pipe(jshint.reporter('default'))
    .pipe(concat('all.js'))
    //.pipe(uglify())
    .pipe(gulp.dest('dist/'))
    .pipe(notify({ message: 'dev-js task complete' }));
});

gulp.task('dev-css', function() {
  return gulp.src('css/*.css')
    .pipe(concat('all.css'))
    .pipe(minifycss())
    .pipe(gulp.dest('dist/'))
    .pipe(notify({ message: 'dev-css task complete' }));
});

gulp.task('prod', function() {
  return gulp.src('js/*.js')
    //.pipe(jshint())
    //.pipe(jshint.reporter('default'))
    .pipe(concat('all.js'))
    .pipe(uglify())
    .pipe(gulp.dest('dist/'))
    .pipe(notify({ message: 'prod task complete' }));
});

gulp.task('dev', ['dev-js', 'dev-css']);

gulp.task('default', ['dev']);

gulp.task('watch', function() {
    gulp.watch('js/*.js', ['dev']);
});

This, by the way, is the final Gulpfile for this article (in case you want to copy it for faster setting up).

Here’s the result of running Gulp with this:

gulp-notify

Not only does it write to the console, but you get a notification from your system tray. Nice.

Semicolon insertion

If you just bump gulp.src onto the next line like this:

gulp.task('dev-js', function() {
  return
    gulp.src('js/*.js')
    .pipe(jshint())
    .pipe(jshint.reporter('default'))
    .pipe(concat('all.js'))
    //.pipe(uglify())
    .pipe(gulp.dest('dist/'));
});

…your task will mysteriously not run. That’s because of a JavaScript feature called semicolon insertion. Basically, JavaScript sees fit to put a semicolon after your return statement, making your task ineffective. Just make sure your return statement isn’t isolated like this, and use the style in the previous sections.

Getting more out of Gulp

There are many other useful plugins for Gulp that you can use. Check out Mark Goodyear’s article for examples involving SASS and image manipulation among other things.

The final package.json

Here you go. This should save you some time setting things up. Remember: npm install.

{
  "name": "gulptest",
  "version": "1.0.0",
  "description": "Learning to use Gulp.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Daniel D'Agostino",
  "license": "ISC",
  "devDependencies": {
    "gulp": "^3.9.0",
    "gulp-concat": "^2.6.0",
    "gulp-jshint": "^2.0.0",
    "gulp-minify-css": "^1.2.3",
    "gulp-notify": "^2.2.0",
    "gulp-uglify": "^1.5.1",
    "jshint": "^2.8.0"
  }
}