Tag Archives: Resource Management

GoLang: Using defer for Scope Bound Resource Management

Most programming languages today provide some way to bind the lifetime of a resource (for instance, a file handle) to a scope within a program, and implicitly clean it up when that scope ends. We can use destructors in C++, using blocks in C#, or with statements in Python. In Go, we use defer. As far as I can tell, this pattern originates in C++ and is called either Resource Acquisition Is Initialization (RAII), or Scope-Bound Resource Management (SBRM). It has various applications that range from resource deallocation (such as file handling, smart pointers, or locks) to scoped utility functions, as I’ve shown in my 2016 article, “Scope Bound Resource Management in C#“.

To understand what we’re talking about and why it’s useful, we first need to start with life without SBRM. Consider a simple program where we read the contents of a file:

package main

import (
	"fmt"
	"os"
	"time"
)

func main() {
	file, err := os.Open("example.txt") // open the file
	if err != nil {
		fmt.Println("Failed to open file!")
		return
	}

	data := make([]byte, 4)
	_, err = file.Read(data) // read 4 bytes from the file
	if err != nil {
		fmt.Println("Failed to read the file!")
		return
	}

	fmt.Println(string(data)) // print the data from the file

	file.Close() // close the file

	time.Sleep(time.Second) // pretend to do other work afterwards
}

Here, we’re opening a file, doing something with it, and then (like good citizens) closing it. We’re also doing something afterwards. However, this is in itself quite error-prone. Here are a few things that could happen which would jeopardise that Close() call:

  • We might forget to call Close() entirely.
  • An early return (e.g. due to an error) might skip the call to Close().
  • A more complex function with multiple branching might not reach Close() due to a mistake in the logic.

We can improve the situation by using defer. Putting the Close() call in a defer statement makes it run at the end of the function, whether there was an error or not. To illustrate, we’ll try to open a file that doesn’t exist, and use a Println() instead of the actual Close() to be able to see some output:

package main

import (
	"fmt"
	"os"
)

func main() {
	_, err := os.Open("nonexistent.txt") // open the file
	defer fmt.Println("Closing")
	if err != nil {
		fmt.Println("Failed to open file!")
		return
	}
}

Because the deferred statement runs at the end of the function in any case, we see “Closing” in the output:

Failed to open file!
Closing

defer is useful to ensure resources are cleaned up, but it’s not as good as SBRM constructs from other languages. One drawback is that there’s no actual requirement to use defer when allocating a resource, whereas something like C#’s using block ensures that anything allocated with it gets disposed at the end of its scope.

Another disadvantage is that it is function-scope only. Let’s imagine we have this program where we do a series of time-consuming tasks:

package main

import (
	"fmt"
	"time"
)

func main() {
	fmt.Println("Doing hard work...")

	time.Sleep(time.Second)

	fmt.Println("Doing harder work...")

	time.Sleep(2 * time.Second)

	fmt.Println("Doing even harder work...")

	time.Sleep(3 * time.Second)

	fmt.Println("Finished!")
}

We’d like to benchmark each step. In “Scope Bound Resource Management in C#“, I was able to wrap statements in a using block with a utility ScopedTimer that I created. Like C#, Go has blocks based on curly brackets, so let’s try and (ab)use them to measure the time taken by one of the tasks:

package main

import (
	"fmt"
	"time"
)

func main() {
	fmt.Println("Doing hard work...")

	{
		startTime := time.Now()
		defer fmt.Printf("Hard work took %s", time.Since(startTime))
		time.Sleep(time.Second)
	}

	fmt.Println("Doing harder work...")

	time.Sleep(2 * time.Second)

	fmt.Println("Doing even harder work...")

	time.Sleep(3 * time.Second)

	fmt.Println("Finished!")
}

The output is:

Doing hard work...
Doing harder work...
Doing even harder work...
Finished!
Hard work took 148ns

Two things went wrong here:

  • The benchmark measured 148 nanoseconds for a task that took one second! That’s because the time.Since(startTime) got evaluated right away rather than when defer started executing its statement.
  • The benchmark for the first task only got printed at the end of the entire function. That’s because defer runs at the end of the function, not at the end of the current scope.

We can fix the first problem by wrapping the deferred statement in an anonymous function that executes itself (which in the JavaScript world would be called an Immediately Invoked Function Expression or IIFE):

// ...

	{
		startTime := time.Now()
		defer func() {
			fmt.Printf("Hard work took %s", time.Since(startTime))
		}()
		time.Sleep(time.Second)
	}

// ...

We now get a benchmark of about 6 seconds, simply because defer is still running at the end of the function:

Doing hard work...
Doing harder work...
Doing even harder work...
Finished!
Hard work took 6.002184229s

To fix this benchmark, we have to fix the second problem, which is that defer runs at the end of the function. What we want is to use defer to measure the duration of each task inside the function. We have a number of ways to do this, but since defer is function scoped, they all involve the use of functions.

The first option is to break up main() into separate functions for each task:

package main

import (
	"fmt"
	"time"
)

func runTask1() {
	startTime := time.Now()
	defer func() {
		fmt.Printf("Hard work took %s\n", time.Since(startTime))
	}()
	time.Sleep(time.Second)
}

func runTask2() {
	startTime := time.Now()
	defer func() {
		fmt.Printf("Harder work took %s\n", time.Since(startTime))
	}()
	time.Sleep(2 * time.Second)
}

func runTask3() {
	startTime := time.Now()
	defer func() {
		fmt.Printf("Even harder work took %s\n", time.Since(startTime))
	}()
	time.Sleep(3 * time.Second)
}

func main() {
	fmt.Println("Doing hard work...")

	runTask1()

	fmt.Println("Doing harder work...")

	runTask2()

	fmt.Println("Doing even harder work...")

	runTask3()

	fmt.Println("Finished!")
}

This does produce correct results:

Doing hard work...
Hard work took 1.000149001s
Doing harder work...
Harder work took 2.001123261s
Doing even harder work...
Even harder work took 3.000039148s
Finished!

However:

  • It is quite verbose.
  • It duplicates all the benchmarking logic.
  • Although many people advocate for smaller functions, I find it easier to read longer functions if the operations are sequential and there’s no duplication, rather than hopping across several functions to understand the logic.

Another way we could do this is by retaining the original structure of main(), but using IIFEs instead of curly brackets to delineate the scope of each task:

package main

import (
	"fmt"
	"time"
)

func main() {
	fmt.Println("Doing hard work...")

	func() {
		startTime := time.Now()
		defer func() {
			fmt.Printf("Hard work took %s\n", time.Since(startTime))
		}()
		time.Sleep(time.Second)
	}()

	fmt.Println("Doing harder work...")

	func() {
		startTime := time.Now()
		defer func() {
			fmt.Printf("Harder work took %s\n", time.Since(startTime))
		}()
		time.Sleep(2 * time.Second)
	}()

	fmt.Println("Doing even harder work...")

	func() {
		startTime := time.Now()
		defer func() {
			fmt.Printf("Even harder work took %s\n", time.Since(startTime))
		}()
		time.Sleep(3 * time.Second)
	}()

	fmt.Println("Finished!")
}

It works just as well:

Doing hard work...
Hard work took 1.000069185s
Doing harder work...
Harder work took 2.001031904s
Doing even harder work...
Even harder work took 3.001086566s
Finished!

This approach is interesting because we actually managed to create scopes inside a function where defer could operate. All we did was put each task and its respective benchmarking logic inside an anonymous function and execute it right away. So the sequential code works just the same whether this anonymous function is there or not; it only makes a difference for defer.

Of course, we are still duplicating code in a very uncivilised way here, so we’ll move onto the third approach, which is simply to implement the benchmarking logic in a helper function and use it to execute the task itself:

package main

import (
	"fmt"
	"time"
)

func runBenchmarked(actionName string, doAction func()) {
	fmt.Printf("Doing %s...\n", actionName)
	startTime := time.Now()
	defer func() {
		fmt.Printf("%s took %s\n", actionName, time.Since(startTime))
	}()
	doAction()
}

func main() {
	runBenchmarked("hard work", func() {
		time.Sleep(time.Second)
	})

	runBenchmarked("harder work", func() {
		time.Sleep(2 * time.Second)
	})

	runBenchmarked("even harder work", func() {
		time.Sleep(3 * time.Second)
	})

	fmt.Println("Finished!")
}

The runBenchmarked() function takes care of everything about each task: it prints a message when it’s about to start, executes the task itself, and prints the time it took using the same defer statement we’ve been using for benchmarking. To do this, it takes the name of the task (as a string) as well as the task itself (as a callback function).

Then, in main(), all we need to do is call runBenchmarked() and pass the name of the task and the task itself. This results in the code being brief, free of duplication, and nicely scoped, which I believe is the closest we can get in Go to the SBRM constructs of other languages. The output shows that this works just as well:

Doing hard work...
hard work took 1.000901126s
Doing harder work...
harder work took 2.001077689s
Doing even harder work...
even harder work took 3.001477287s
Finished!

Conclusion

defer in Go provides some degree of SBRM support for scoped cleanup or utility purposes. However, it suffers from the following drawbacks:

  • It does not enforce implicit cleanup of allocated resources as similar constructs in other languages do.
  • Any parameters that need to be evaluated at the end of the scope should be wrapped in an IIFE.
  • It is function-scoped, therefore using defer for a limited/block scope inside a function requires it to be wrapped in another function.

Scope Bound Resource Management in C#

This article explains how we can use scope to localize resource lifetime as well as other arbitrary user code effects within a method. The article starts with background from C++, because that’s where this technique comes from.

Update 5th November 2016: Some of the utility classes described here have been incorporated in my Dandago.Utilities NuGet package.

The RAII Pattern

Despite their power, pointers in C/C++ have been the cause of much outrage. Using them incorrectly typically leads to disastrous effects, from memory leaks to segmentation faults. While newer languages such as Java and C# have imposed damage control by taking control of the lifetime of objects allocated on the heap (via garbage collection), there are techniques even in C++ that make pointers a lot less messy to use.

In fact, the problem here is not even about pointers. Pointers belong to a family of resources, along with file handles, sockets, and many other things. These are typically allocated on the heap and must be released after use; otherwise bad things happen.

Scott Meyers’ excellent book Effective C++: 55 Specific Ways to Improve Your Programs and Designs has an entire section dedicated to safely working with resources. One of the first things he suggests in that section is to encapsulate a resource within an object. For example:

class MyResource
{
public:
    MyResource() // constructor
    {
        this->ptr = new int[1000]; // allocate memory on heap
    }

    ~MyResource() // destructor
    {
        delete[] this->ptr; // free memory
    }

private:
    int * ptr; // pointer encapsulated within class
};

This encapsulation is called Resource Acquisition Is Initialization (RAII), or Scope-Bound Resource Management. Apart from constructors, C++ classes can have destructors. These get called either explicitly from application code, or automatically when an object allocated on the stack goes out of scope. Let’s see how this works in practice:

int main(int argc, char ** argv)
{
    // create instance of MyResource on stack
    MyResource resource;

    return 0;
} // we went out of scope; destructor gets called

Just like declaring an int variable allocates it on the stack, the same thing happens with class types. When that object goes out of scope (i.e. reaches the next } brace), its destructor gets called. This is great because you can’t really forget to dispose of the encapsulated resource. If you return early, throw an exception, etc, the destructor will get called when control leaves the current execution block.

The IDisposable Pattern

In C#, objects allocated on the heap (via the new keyword) are typically killed by the garbage collector when it determines that they are no longer in use. While C# code typically doesn’t face the problems C++ code has with pointers, managing resources is no less important. Just like C++, C# has to deal with file handles, databases, sockets, unmanaged libraries and other stuff that must be disposed as carefully as they are initialized.

For this reason, C# provides two tools to essentially do the work of C++ destructors: the IDisposable pattern, and finalizers. Using these correctly is non-trivial and depends on the situation, but they are also pretty standard and well-documented. Check out this CodeProject article for an in-depth treatment of the topic.

For convenience, C# also provides an overloaded using keyword that works hand-in-hand with the IDisposable pattern:

            using (var fs = File.OpenWrite("file.txt"))
            using (var sw = new StreamWriter(fs))
            {
                sw.WriteLine("Hello!");
            } // automatically calls Dispose() for both

A using block, wrapping an object that implements IDisposable, will automatically call that object’s Dispose() method when it goes out of scope (and using blocks can be stacked, as above). This is essentially equivapent to:

            FileStream fs = null;
            StreamWriter sw = null;
            try
            {
                fs = File.OpenWrite("file.txt");
                sw = new StreamWriter(fs);

                sw.WriteLine("Hello!");
            }
            finally
            {
                sw.Dispose();
                fs.Dispose();
            }

Abusing the IDisposable Pattern

While IDisposable is meant to be used to safely dispose of resources, we can extend its usefulness to other scenarios that relate to a particular scope.

One example where I’ve used this idea before is to log the beginning and end of a method:

    public class ScopedLog : IDisposable
    {
        private string name;

        public ScopedLog(string name)
        {
            this.name = name;
            Console.WriteLine("Begin {0}", name);
        }

        public void Dispose()
        {
            Console.WriteLine("End {0}", this.name);
        }
    }

This pattern doesn’t really add anything, but gives you an elegant way to do scope-related stuff without that getting in the way of your application logic:

        static void Main(string[] args)
        {
            using (var log = new ScopedLog("Main"))
            {
                Console.WriteLine("Hello!");
            }

            Console.ReadLine();
        }

Here’s the output:

scope-logging

Another example is when you want to benchmark your code. Code can get really messy when you’re doing logging, benchmarking and other stuff in the same method. Instead, we can make a dedicated class:

    public class ScopedTimer : IDisposable
    {
        private string name;
        private DateTime startTime;

        public ScopedTimer(string name)
        {
            this.name = name;
            this.startTime = DateTime.Now;
        }

        public void Dispose()
        {
            var endTime = DateTime.Now;
            var elapsed = endTime - this.startTime;

            Console.WriteLine("{0} took {1}", this.name, elapsed);
        }
    }

…and then put it neatly in a using block:

        static void Main(string[] args)
        {
            using (var timer = new ScopedTimer("Main"))
            using (var log = new ScopedLog("Main"))
            {
                Console.WriteLine("Hello!");
            }

            Console.ReadLine();
        }

Here’s the output:

scope-benchmarking

A final example is changing the console colour. It’s very easy to save the old colour, set a new one, and then revert back to the old colour when going out of scope:

    public class ScopedConsoleColour : IDisposable
    {
        private ConsoleColor oldColour;

        public ScopedConsoleColour(ConsoleColor newColour)
        {
            this.oldColour = Console.ForegroundColor;

            Console.ForegroundColor = newColour;
        }

        public void Dispose()
        {
            Console.ForegroundColor = this.oldColour;
        }
    }

Note: you could use Console.ResetColor(), but then you can’t nest colour changes.

Here’s the updated Main() code for this example:

        static void Main(string[] args)
        {
            using (var timer = new ScopedTimer("Main"))
            using (var log = new ScopedLog("Main"))
            {
                Console.WriteLine("Hello!");

                using (var colour = new ScopedConsoleColour(ConsoleColor.Yellow))
                {
                    Console.WriteLine("Howdy?");
                }

                Console.WriteLine("Bye!");
            }

            Console.ReadLine();
        }

Here’s the output:

scope-colour

Other Applications in C++

The RAII pattern is awesome. Not only does it allow you to safely manage the lifetime of resources, but it enables a whole class of scope-based applications. Aside from the C# examples in the previous section, RAII enables things like smart pointers in C++ (e.g. unique_ptr) and scoped locks.