The problem with out-of-the-box WebMatrix development is that it encourages you to use direct SQL programming with embedded SQL queries. The world has moved away from embedded SQL queries in favor of object-relational mapping and (domain-un-specific) query syntax such as LINQ.

I wanted to test out using LINQ2SQL with WebMatrix. The tool to use when doing Database-First LINQ2SQL development is called SQLmetal. It queries the database schema and generates mapping classes (one class per table) that allows one to use LINQ syntax in C# or VB.NET directly against the source data.

The problem, though, is that SQLmetal does not appear to work with SQL CE 4.0. So, what to do?

Microsoft’s next-generation O-R mapper – Entity Framework (EF) – is one possibility. Entity Framework uses the Entity Data Model (EDM) as its conceptual mapping framework.

Microsoft has three basic strategies for using the Entity Framework (very roughly):

• Database-First: Start with a database and create an Entity Data Model (EDM) that maps to that database
• Model-First: Start with an Entity Data Model and generate classes and database from that model
• Code-First: Start with classes and generate a database from those classes; use data annotation attributes to guide the mapping

It is the Model-First and the Code-First approaches that I find most interesting. In this case I chose the Code-First approach because it appears easier to work with without any tooling support.

So what do we need to work with Code-First and SQL CE 4.0?

We need to

• download and install SQL CE 4.0
• download and install Entity Framework Code-First (currently at CTP 5 status)

Then we need to setup a basic project. Let’s just make a C# console application.

Then we need to reference some assemblies. We need these ones:

• System.ComponentModel.DataAnnotations.dll
• EntityFramework.dll (Code-First CTP 5)
• System.Data.SqlServerCe.dll (SQL CE 4.0)

Then we need to import some namespaces beyond the default ones:

• System.Data
• System.Data.Entity
• System.Data.Entity.Database
• System.Data.SqlServerCe
• System.ComponentModel.DataAnnotations

That settles our external dependencies. Now onto the data modeling.

Most Web sites will want a user type, for logins and such. Let’s create a basic user domain model using Code-First code:

public class User
{
    public string Name { get; set; }

    public string Password { get; set; }

    public string PrimaryEmailAddress { get; set; }

    public string SecondaryEmailAddress { get; set; }

    public DateTime Creation { get; set; }

    public bool Active { get; set; }
}

Now, for the database mapping to make a little more sense we’ll add some constraints via attributes:

public class User
{
    [Key, StringLength(50)]
    public string Name { get; set; }

    [Required, StringLength(100)]
    public string Password { get; set; }

    [Required, StringLength(320)]
    public string PrimaryEmailAddress { get; set; }

    [StringLength(320)]
    public string SecondaryEmailAddress { get; set; }

    [Required]
    public DateTime Creation { get; set; }

    public bool Active { get; set; }
}

The attributes are pretty self-explanatory.

We need one more thing in order to be able to use this, a DbContext-derived class that references our User domain model class.

public class ProtoCatalog : DbContext
{
    public DbSet<User> Users { get; set; }
}

Now we are ready to actually use the model. The following code will insert a new user into the database. Assuming there isn’t already a database file present, one will be automatically generated for us:

static void Main(string[] args)
{
    DbDatabase.DefaultConnectionFactory = new SqlCeConnectionFactory("System.Data.SqlServerCe.4.0");

    using (var db = new ProtoCatalog())
    {
        var user = new User
        {
            Name = "bob",
            Password = "123",
            Creation = DateTime.Now,
            PrimaryEmailAddress = "bob@example.com"
        };

        db.Users.Add(user);

        db.SaveChanges();
    }
}

That’s it. We now have a new Entity Framework-generated SQL CE 4.0 database with data. It was generated from a simple class modeling our simple domain.

I don’t think the Entity Framework Code-First CTP API is quite as nice as it could be though:  setting a static property on the DbDatabase class to use a specific database driver is not that clean. It’s global state and we don’t like global state.

Let’s try and open the new database in WebMatrix and see what happens.

Let’s see the data as well:

So, we’ve designed a Code-First data model. We’ve made some code to insert data into a database fitting this model. The EF code automatically generated a SQL CE 4.0 database when no previous database file existed. And finally we’ve seen the database schema and data inside of WebMatrix.

Presto.

To do this, we’ll need a few trustees:

• Microsoft .Net, our software platform of choice
• LinqPad, our software environment of choice

First, fire up LinqPad and start by importing System.Drawing. All other needed namespaces are imported by LinqPad.

Then we need to change the script type from Expression to Program. This gives us the full expressive power of C#, including the ability to do statements, create methods, classes, structs, etc.

Then we define a main method:

void Main()
{
	var desktop = Environment.GetFolderPath(System.Environment.SpecialFolder.Desktop);
	var droplet = Path.Combine(desktop, "droplet.jpg");
	Render(new Bitmap(droplet), PixelShader1).Dump();
}

The first lines defines the desktop folder path. The second line defines the path to an image I have on my desktop. The third is where the action starts. This line calls the Render method, passing along the droplet image and then a method delegate. The result of the call is dumped to the output. The Dump method is a special LinqPad extension method that normally outputs a textual representation of the state of the object being dumped out. It does however handle XML data and image data specially. In the case of image data it will show the data as an image.

Next we define the Render method. Note that this is a straight-forward higher-order method that takes in a bitmap and a shader and gives back a new and shaded copy of the source bitmap.

IEnumerable<Bitmap> Render(Bitmap source, PixelShader shader)
{
	var w = source.Width;
	var h = source.Height;

	var pixels =
		from x in Enumerable.Range(0, w)
		from y in Enumerable.Range(0, h)
		let  p = source.GetPixel(x, y)
		select new { X = x, Y = y, Color = p };

	var target = (Bitmap)source.Clone();

	foreach (var p in pixels)
	{
		target.SetPixel(p.X, p.Y, shader(p.X, p.Y, w, h, p.Color));
	}

	yield return source;
	yield return target;
}

Well, that description is not entirely accurate. I’ve set it up so that it actually returns a stream of images. So far it will only return the source and target images but it can now be easily extended to generates animations, courtesy of iterators (see “yield return”).

There’s a couple of things to notice about this code

• the code is sequential
• the code uses linq

These are by no means opposites but in 50 lines of code, I’ll not show you parallel LINQ or parallel code. It is however easy to bolt on and I’ll leave this as an exercise to the reader. Beware that the Bitmap class is not thread-safe!

Now it’s time to define what a pixel shader is (in this narrow context). It’s simple

delegate Color PixelShader(int x, int y, int w, int h, Color source);

The PixelShader is just a function from a coordinate set (x,y), a dimension (w,h) and a color to a new color. This representation lets you do a lot of magic but you can easily extend it to include a pixel matrix and a time parameter for added power.

Now let’s define a simple pixel shader that just swaps the two color channels red and green:

Color PixelShader1(int x, int y, int w, int h, Color source)
{
	var a = source.A;
	var r = source.R;
	var g = source.G;
	var b = source.B;

	Swap(ref r, ref g);

	return Color.FromArgb(a, r, g, b);
}

First we read out the alpha, red, green and blue pixel values, then we swap red and green and then we construct a new color from that.

Last but not least, a little helper function for swapping values:

void Swap<T>(ref T a, ref T b)
{
	var t = a;
	a = b;
	b = t;
}

I’ve previously written against using methods with ref and out parameters but in some cases, they are quite useful.

That’s it! – A compositional image processing script in 50 lines of code.

The output looks like this (full image):

Some extensions left as an exercise for the reader:

• multi-threading (Threads/ThreadPool or better Tasks/Parallel or perhaps PLINQ)
• animation
• color space conversion (RGB, HSL, LAB, CMYK, …)
• 3D/4D image processing and synthesis

The most amazing fact of the tablet, however, is the time it has taken the industry to realize it in a quasi-interesting form.

I’ve always felt that the mouse and keyboard was, although great for many tasks, utter limiting for other tasks of a more creative nature.

For audio and music the tablet provides a cool multi-touch interface that shortens the cognitive path between idea and output, it enables one to make changes faster and more real-time. Special-purpose multi-touch tablets are starting to appear for audio synthesis and sequencing purposes. It just makes sense.

In the realm of graphics, in particular 3-dimensional graphics, I’ve always felt that mouse and keyboard simply isn’t the right medium. Surely we can do better. Multi-touch is a great next step. An accellerometer and compas equipped tablet could be used as a physical camera into a virtual world. Moving the tablet around shows the scene. One could imagine setting up virtual cameras this way by moving the tablet around and then pinning virtual manifestations of it to whereever.

There are several annoying limitations of mainstream technology though; primarily, I think these:

• Thickness: they are getting there, but they are still a little bulky
• Weight: they are not as light-weight as one could hope for, making prolonged use somewhat tiresome
• Flex: they are rigid, unlike say, a newspaper, which is an inconvenience as reading is one of the nice applications of tablets; there are research prototypes for flexible electronic paper, however
• Pressure: they are not pressure sensitive beyond 1-bit binary sensitivity: touch or no touch; for audio and music, a pressure sensitive touch surface is important; likewise for 3-dimensional graphics pressure senitivity would allow artists and industrial designers finer control in modelling scenarios, where virtual matter is shaped like clay
• Tactile: they do not provide much tactile feedback beyond flat touch; Microsoft Research has some prototypes for devices that attach directly to the body; one could potentially conceive of a system that provides nervous system feedback, tricking the brain into thinking there is a landscape in mid-air; at some point, we may want to skip the tablet altogether and just use this virtual tactile feedback combined with stereo-projected images on variable-transparency glasses; this will also make augmented reality almost transparent; well, beyond computational contact-lenses and “wet-ware” cybernetics.

The space of potential applications, however, is awesome. At least with the hardware we’ll hopefully see with the next 10 years and beyond. Not so long ago, Microsoft provided a series of concept videos – Future Visions – which show-cased where Microsoft thinks the future is. Suffice it to say: the future is bright.

For now, I’m happy to have bought a device that brings us a couple of steps closer to this dream: The Apple iPad.

So far my experience is that it’s a great device and has muted the need for my laptop PC which is now idle. The iPad is a great social networking tool with applications providing specialized user experiences fit for the iPad form factor and surface.

Location awareness is also an awesome feature which is sure to come in very handy.

Music is remote-controlled from the living room, with control signals sent to iTunes on the main PC, sending music streaming through a TOSLink optical cable and into the stereo. A bit of a bulky remote (in the absence of an iPhone), but cool nonetheless.

The things that annoy me about it is that there doesn’t appear to be a way to organize applications into folders on the desktop surface and when in the App Store, when buying an application you will leave the store and go to the desktop which is not so nice when “power shopping” (free apps or not).

There’s a lot to love about this little gem of a device though!

This is my second Apple product. The first one was the Apple Wireless Keyboard. Not because I have anything against Apple (well actually, their closed-world policy, with respect to programming languages and in general is… vexing), but mostly due to the price tag on many of their products.

I see Microsoft making headway with the pretty cool Windows Phone 7 OS with its Silverlight UX and for the sake of consumer wallets everywhere, and industry progress, I hope Microsoft will also (re-) enter this market. Witnessing Windows Phone 7, the software for this hypothetical tablet ought to be great but witnessing some of the phone hardware provided by 3rd parties, optimism dims a little. The finish of an Apple product is hard to beat.

0. Index

1. Greg Meredith on Monads
2. Definition
3. Syntax
4. The List Shape
      3.0. Shape
      3.1. Wrap
      3.3. Roll
5. Equations
6. Shapes & Monads
      5.1. The Identity Monad
      5.2. The  Maybe Monad
      5.3. The List Monad
      5.4. The Set Monad
7. Lessons

1. Greg Meredith on Monads

I’ve found a wonderful metaphor for the elusive concept of monads. This metaphor was created by Greg Meredith, mathematician, computer scientist and monad expert.

See his presentation:

• Greg Meredith on Monads

I’ll try to sumarize my own understanding of the main points of Greg’s presentation below; for myself and for anyone else interested. Please correct me if you think I’ve misunderstood something.

Note. This blog post will evolve. I periodically update it.

2. Definition

The metaphor consists of three parts:

• Shape
• wrap
• roll

Monads with structure also have this part:

• compose

3. Syntax

The syntax chosen is important to how easy to understand something is. Greg has found that using markup syntax (angle bracket syntax, e.g. <list>1,2,3</list>) is a useful way to think about the shape of monad instances; therefore Shapely Monads.

I’m going to augment this by translating it into domain-specific syntax for lists as well, to make structure even more easy to see.

In the following examples I’ll use 4 syntaxes:

• function call syntax, e.g. wrap(3); how one would write in a program; qualified (e.g. List.wrap, or Set.wrap) and unqualified (e.g. wrap; this assuming some shape in the context of its use.)
• abbreviated function call syntax, e.g. w(3) is the same as wrap(3)
• shape-specific syntax, e.g. [a,b,c,...] for lists, simply because it’s easy to understand; other examples: {a,b,c,…} for sets.
• markup syntax, e.g. <list>a,b,c,…</list> for lists (same as [a,b,c,...]) whereas <set>a,b,c,…</set> is for sets (same as {a,b,c,…}), etc.

I’ll be careful to make the syntax obvious.

4. The List Shape

To start with some concrete examples, let’s use the monad of lists, the list monad.

4.0. Shape

The list shape is something which we can symbolize as:

• [] is the empty list; <list></list>
• [a] is the singleton list; <list>a</list>
• [a,b,c] is a list of three elements; <list>a,b,c</list>
• and so on

The sequare bracket syntax is just domain-specific syntax for lists. We use it here to make it easy to see the structure of values.

4.1. Wrap

First let’s create a list of one element, the number zero:

• List.wrap(0) = <list>0</list> = [0]

So we’ve now put the value 0 into the list monad.

List elements can be other lists, so for example:

• [[a],[b,c]] is a list of a list of a, followed by a list of b and c

The operator expression to construct this sequence is

List.wrap(List.wrap(a),List.wrap(List.wrap(b)+List.wrap(c)))
=
List.wrap(<list>a</list>,<list>b,c</list>)
=
<list>
   <list>a</list>
    <list>b,c</list>
</list>
=
[[a],[b,c]]

4.2. Compose

To compose lists (append lists to eachother), we can write

• List.compose(List.wrap(0),List.wrap(1)) = <list>0,1</list> = [0,1]

this will create the singleton list of the number zero composed with the singleton list of the number one, creating a new list of two elements, zero and one.

So far we’ve leared that:

• List is a Shape; we use color here to help recognize lists
• List.wrap(x) constructs a new list of one element, i.e. [x]
• List.compose is the append operation; it takes two lists and creates a new list which is the two lists in sucession

Note also that the neutral instance of the list monad is [], i.e. <list></list> or what we might express as List.wrap(); composing an empty list with any other list yields the same list; it has no effect; we can see this as

• <list>x</list> + <list></list> = <list>x</list>

  XML has special support for empty elements, which we can take advantage of

  <list>x</list> + <list/> = <list>x</list>

  or, equivalently, using the short-hand syntax for lists

  [x] + [] = [x]

Now, in the following, for syntactic brevity we’ll assume the List monad and not qualify the operations with the specific shape type (List, Set, Graph, etc.)

4.3. Roll

Suppose we create one more level of nesting:

• wrap(compose(wrap(0),wrap(1))
  =
  wrap(<list>0,1</list>)
  =
  <list>
  <list>0,1</list>
  </list>
  =
  [[0,1]]

Then we can use roll on that to remove the superfluous nesting again:

• roll(wrap(compose(wrap(0),wrap(1)))
  =
  roll(wrap(<list>0,1</list>))
  =
  roll(<list><list>0,1</list></list>)
  =
  <list>0,1</list>
  =
  [0,1]

More clearly, if we have

• [[0,1],[2,3]]

Then rolling that gives

• roll([[0,1],[2,3]]) = [1,2,3,4]

So rolling is an act of flattening or leveling a container; in this case a list of lists of numbers is rolled out to just to a list of numbers.

Let’s sweeten the syntax up a little further by saying that + is an operator symbol for compose, that w is the same as wrap and that r is the same as roll; that gives us expressions such as

• r(w(w(a)+w(b)) = r(w([a,b])) = r([[a,b]]) = [a,b]

5. Equations

There are equations associated with shapes or more precisely with shape composition operators. For List.append there are no such equations.

Set union, on the other hand, does have equations:

• a + a = a // idempotency
• a + b = b + a // commutativity

The first equation says that composing (uniting) a set with itself yields an identical set. It’s a no-op. The second equation says that the order of a and b is irrelevant to the outcome..

Greg also talks about three laws that relate to the parts of the monad metaphor (and indeed the formal definition of a monad). I’m still learning about these.

6. Shapes & Monads

There are many kinds of structures that qualify as monads.

6.1. The Identity Monad

This is the no-operation monad; the simplest of all monads.

• w(x) = <id>x</id>
• w(w(x)) = <id><id>x</id></id>
• r(w(w(x))) = <id>x</id>

There’s not really a definition for + for this monad, but we could perhaps define + as w so that composition is wrapping.

6.2 The Maybe Monad

This is the monad of optional values. They are useful for certain functions that may not have useful results in some cases.

• w() = <maybe></maybe> = <maybe/>
• w(x) = <maybe>x</maybe>
• r(w(w())) = <maybe/>
• r(w(w(x))) = <maybe>x</maybe>

There’s not really a definition for + for this monad, but we could perhaps define + as w so that composition is the same as wrapping.

Trivia: In Haskell Just(x) is the constructor for <maybe>x</maybe> whereas Nothing is the constructor for <maybe/>. In SML these are called Some and None, respectively.

6.3 The List Monad

This is the monad of lists of values that we’ve already seen. For this monad we have a meaningful composition operation, which is append; append will join two lists together into a new list where the elements of the two are in sequence in the new list.

• w() = []
• w(x) = [x]
• w(w()) = [[]]
• w(w(x)) = [[x]]
• r(w(w(x))) = r([[x]]) = [x]
• w(a)+w(b) = [a,b]
• w(w(a)+w(b))+w(w(c)+w(d)) = [[a,b],[c,d]]
• r(w(w(a)+w(b))+w(w(c)+w(d))) = r([[a,b],[c,d]]) = [a,b,c,d]

In these examples I’ve not used markup syntax but I hope that by now you can intuit the simple translation between [...] and <list>…</list>.

6.4 The Set Monad

This is the monad of sets of values. In this case I’ll use {} as the syntactic delimiter of set literal expressions, just to make it clear that these are sets and what the left-hand-side monadic expressions actually mean.

Under set union the semantics are as follows:

• w() = {}
• w(x) = {x}
• w(w()) = {{}}
• w(w(x)) = {{x}}
• r(w(w(x))) = r({{x}}) = {x}
• w(a)+w(b) = {a,b}
• w(w(a)+w(b))+w(w(c)+w(d)) = {{a,b},{c,d}}
• r(w(w(a)+w(b))+w(w(c)+w(d))) = r({{a,b},{c,d}}) = {a,b,c,d}

however, if we throw in a couple of duplicate values, the semantic distinctions between sets and lists becomes clear:

• w(a)+w(a) = {a}+{a} = {a}
• w(w(a)+w(b))+w(w(b)+w(a)) = {{a,b}+{b,a}} = {{a,b}}
• r(w(w(a)+w(b))+w(w(c)+w(b))) = r({{a,b},{c,b}}) = {a,b,c}

These three expressions would have a different meaning under list composition.

7. Lessons

These lecture notes do not capture all the knowledge of the first lecture by Greg Meredith but they do show some simple examples of what the monadic operators do. I find this a more syntactic way of thinking that is actually quite appealing. It’s not the whole story though; only a fraction. The theory of monads goes deep, as does its practice.

In the following lectures, Greg will supposedly go into concrete applications of monads in the software design of Web applications. In other words, Greg will show-case monadic Web programming. I can’t wait to see what he’s cooked up.

The monad examples shown here are just of a fraction of the monads that I’ve seen examples of. This really goes to show the fundamental strength of the monadic abstraction.

Today, I want show how one of the best query tools in the world, can be used to easily and quickly create integration tests for workflow services. These integration tests will seamlessly automate the testing and verification of workflows that are created by web services.

This article is borne out of practical experience in the context of the Dynamics NAV (Navision) business software environment. This is an implementation detail, however; the method doesn’t depend on that.

Note: this article assumes the reader has some familiarity with LINQ. Should you not be familiar with LINQ, I strongly urge you to become so. LINQ is based on monads which come from the esoteric branch of mathematics known as Category Theory. This branch unifies vast areas of mathematics and provides extreme generality and expressiveness. It is however completely unnecessary to know about these deep things to be able to use and understand LINQ, which is in fact very easy to use and intuitive, even more so than SQL; it does not, however, replace SQL or T-SQL.

2. Setup

The basic setup is this:

1. a script creates an order which it then posts to a web service
2. the web service creates a workflow
3. the workflow is executed
4. the web service responds back (okay, error)
5. the script queries the database to see if the changes made (or not made) matches expectations; these queries form our test-cases

we can visualize the method like this

3. Implementation

The little notepad symbol in the above diagram is the logo for the LINQPad tool. This tool is the software equivalent of a Swiss knife.

LINQPad makes it easy to connect to a database and execute queries against an object-relational mapping, meaning you can query C# objects that nicely represent the data domain.

Scripts can be made as C#/VB expressions, C#/VB statements or C#/VB programs. In our case we’ll need the full power of C# programs so we select that.

To send orders, we’ll make a trivial HTTP POST method:

string HttpPost(string service, string order)
{
     // trivial implementation left as an exercise for the reader ...
}

Next we want to build an actual order, we can do that using LINQ to XML. And we can in fact use LINQ to SQL queries to build up these orders from existing data in the system if needed.

LINQPad exposes the tables of the connected database as C# types. Let’s have a look at an actual data source:

The types are shown in the upper-left corner. I should note here that I did not connect to a SQL database because I didn’t have one ready at home but instead used a free web data source (it follows the OData protocol).

There is one important thing to keep in mind, however. When doing tests where an external system modifies the database, it doesn’t make sense to cache data. The reason is that we cannot see changes made to the database in the same session if we have caching enabled because the LINQ to SQL query provider is unaware of these changes. We cannot have valid tests without up-to-date data. The solution is simple; just put this statement into the main method:

void Main()
{
    ObjectTrackingEnabled = false;
    // ...
}

In the context of Navision there is one thing which will annoy any user of LINQPad. That is the fact that Navision has one table for each company, meaning if there are three companies in the database, then there will be three customer tables. So if you wish to query all three companies, you’ll have to do some search-replacing – but luckily that’s trivial, if a slight nuisance.

One could imagine building “Navision awareness” into a LINQPad, since it is in fact possible to extend LINQPad, but I’ve not had the time to do this. This might make it possible to just have a variable decide what table version is currently being used. That’s a side-effecting rabbit hole of its own, though.

Now onto the actual tests. These can be quite trivial, so let me make up a toy example:

// ...

var customersMissingTopRef =
        from c in CompanyCustomers
        where c.ParentCustomerNo != "" && c.TopCustomerNo == ""
        select c;
// ...

Test(customersMissingTopRef.Any(), "customers with missing top reference");
// ...

The Test method could print out the name of the test case and whether it succeeded or failed.

The type of the customersMissingTopRef variable will be: IQueryable<CompanyCustomer>. The IQueryable supports the standard query operators such as Select, SelectMany, Where, GroupBy, OrderBy, Take, etc.

4. Conclusion

This approach is a light-weight one. A more solid approach would be to build a full object-relational mapping from the database and code up test cases in stand-alone files that will be run as part of a test framework.

Nonetheless, the method shown here is straight-forward, easy to implement, and can save hours of tedious manual testing and retesting and can be considered first step with a low barrier-to-entry. I’ve already saved what would amount to several days worth of manual testing due to this method and have found several software defects and regressions.

1. Language Integrated Query (Linq)

The dutch company Solid Innovator made this quite beautiful illustration of the standard Linq sequence operators.

Download the PDF at Solid Innovator.

2. Anatomy of a Linux System

I don’t use Linux but I love this illustration; the only problem is that the original vector graphics in the middle has been rasterized and in far too low resolution so you can’t zoom properly into it without seeing artifacts. A shame.

Download the PDF at O’Reilly.

3. Domain Name System (DNS)

This is a complex illustration or visual guide to the Domain Name System. The illustration is made by Dubberly Design Office (ddo).

Download PDF at ddo.

In the computational space we know as software, there appears to be a growing trend to recognize and encode processes as workflows. These workflows can be business processes that require input from customer service or other personel, span several organizations and involve web service calls.

Due to the latency and often distributed nature of workflows, it is often not natural to express workflows directly as code. If a process can span days, weeks or perhaps even months, it does not naturally fit conventional programming models.

Therefore workflow models are invented. These are data structures that can be used to express workflow processes at a high level. Workflow activities delimit significant or meaningful work, e.g. “wait for quality control”, “send e-mail to customer” or “perform credit check”.

2. Metaprograms

Workflows are coarse-grained algorithms; do this, do that, if this then that, and so on.

These differ from [conventional*] programs where the code is code, data is data and the two are distinct. With workflows, code is data. Workflow engines are virtual machines and workflow data models are meta-programming languages; I like to think of them as meta-programs and meta-machines; the abstraction level is “one up”; n+1. * In that sense, the perfect language to encode workflows would perhaps be Lisp, a language where all code is data and all data takes the same form.

This is an important fact! It means that we can think about workflows as programs and think of workflow models as programming languages and evaluate them as we would any other programming language.

3. Practicalities

I first encountered workflows in Navision[1], a business process platform. There is no real built-in workflow engine in Navision but one can create one; and several companies have done so.

The pervasive use of workflows is probably why Microsoft, who now owns Navision, has incorporated a workflow programming model, Windows Workflow Foundation (WF), into the .NET Framework.

One issue that faced me early on was the transactional nature of the workflow model. This meant that a long workflow could fail after many steps had been completed and comitted to the database. A nice solution to this problem is to do more validation up-front. This means creating an activity that does combined state validation before allowing the rest of the workflow to execute. It’s a simple measure but it is an imporant one. This approach can also be combined with design by contract style programming where “internal state” is checked for correctness up-front; it is assumed correct but if not, then fail early and often – in other words: fail fast.

The transactional nature of workflows also come into play in the sense of “point of no return” actions. If an e-mail has been sent to a customer we can’t “roll back”. There is no retroactive erasure for e-mails – we can’t unsend it. This makes up-front validation and sanity checking more important.

I’ve also come to realize that developing with workflows can benefit from unit tests. This again follows from the simple fact that unit tests benefit normal programs and these meta-programs can benefit from unit tests for precisely the same reasons. The unit tests may look somewhat different though. Perhaps the unit tests are even themselves workflows.

Many workflows are probably more difficult to test than regular programs because their effects often span several distributed systems but this only makes testing even more important.

4. Services

The distributed nature of workflows is probably also why Microsoft emphasises the integration between Windows Workflow Foundation (WF) and Windows Communications Foundation (WCF). Services that use workflows are appropriately called Workflow Services. I look forward to possibly designing Workflow Services some time in the future.

5. Designer

I also really like the idea of a workflow designer[2] where one can embed a workflow designer in an application and allow users to see a running workflow and even redesign a running workflow; a somewhat theoretical scenario but possible. Windows Workflow Foundation has such a designer. Appendix 1 shows a simple example of a workflow. The graphical nature of the designer makes it very intuitive and easy to see what’s going on.

The designer can be tailored so that users can only compose high-level activities and stay clear of low-level activities that could wreak havoc in the wrong hands.

6. References

1. Microsoft Dynamics NAV
2. WF & Microsoft Dynamics AX

Appendix 1

This is a simple example of a nested workflow Sequence:

I’d like to take a step back and examine the problematic code from a different angle. First, let’s see the code:

object result;
bool isDecimal = GetAmount(out result);
decimal amount = (decimal)(isDecimal ? result : 0);

Aside from the fact that the decimal cast here will not work at run-time, do you, dear reader, see anything suspicious about this piece of code?

The problem is that the GetAmount function uses an out parameter. What this effectively does is allow GetAmount to write back to our local result variable. The GetAmount function also returns a boolean value by the usual means, answering the question “is this amount a decimal”?

That first part of the code will work, but it makes the code less obvious and obstructs the purity of the code. That’s a somewhat subjective assessment but if all functions returned values using out then the code definitely would be much less readable.

So what to do here? We basically have the problem that we want to return two values: the amount and whether this amount is a decimal. Well, since C# 2.0, we’ve had nullable types. This means we can create a nullable type for any primitive (non-nullable) type. For example, a decimal is not nullable but we can create a variant that is: decimal? or Nullable<decimal> (the questionmark defines nullability). This means our decimal now has the potential to hold an extra value: null. We can use this value to designate whether the value is a decimal or not.

So let’s first change the above code to take out “out“:

decimal? temp = GetAmount();
decimal amount = temp.HasValue ? temp.Value : 0.0m;

That’s better and shorter. Can we make it shorter still? Allow me to introduce my new friend: the null-coalescing operator from C# 3.0:

decimal amount = GetAmount() ?? 0.0m;

The null-coalescing operator (??) will evaluate to the right side of the operator (0.0m in this case), if the left side of the operator happens to be null; otherwise it will evaluate to the left side.

We can even save 4 extra characters by using C# 3.0 type-inference:

var amount = GetAmount() ?? 0.0m;

So what happened here is this:

1. we took out out
2. we took out cast
3. we took out two variables
4. we took out two thirds of the code! (104 down to 33 characters)

In effect, we’ve seriously unspaghettified the code.

The reader might object: but what if we have 3 or more values we need to return. Well, in that case I have two answers:

1. use objects, that’s what they’re there for, and
2. use tuples, a new .NET Framework 4.0 type that allows easy return of multiple-values

This pattern here, of using nullable types is extremely useful and can help clean up code in many places. Not just for return values but also for parameters. They also have synergy with default parameters as introduced in C# 4.0.

I hope you’ll use this pattern instead and only use out parameters in very rare cases. In many cases, using out parameters is an anti-pattern.

Now, the reader may object that it is in fact not true that no real innovation has taken place on the desktop and in window managers. In Windows Vista we have Flip 3D and very nice semi-transparent windows. In Windows 7 we have Aero Snap which allows us to easily snap windows to parts of the desktop. On Linux there is Compiz Fusion with wobbly windows effects, cube mapping etc. There are also extensions to Windows for these kinds of things. It is also typical on Linux to have multiple desktops.

I claim that none of these things are novel in any way.

That’s an easy claim to make of course, talk is cheap as they say, but let’s examine the significance of these inventions from a usability view.

Does the wobbly windows of Compiz Fusion give any kind of benefit from a usability point of view? One will be hard pressed to find any task that one can perform more efficiently by having such an effect on windows. It is not completely worthless, however, as it adds a level of physics that one might refer to as Natural User Interface (NUI) but in and of itself, a wobbly window hardly makes the user more efficient in performing tasks of any kind.

Then what about the more exotic cube-, cylinder- and torus-mapped windows? Again, while fancy to look at, I think one is hard pressed to say that these form a leap forward in usability. Still, these do warp and twist the fabric of the desktop and represent a tiny little step forward but perhaps a step or two backwards as well: is reading text from and recognizing objects from warped windows easy?

Then we have Aero Snap in Windows 7. This is what I consider a small feature but in fact a genuine usability feature. This will indeed help speed up some tasks. The user does not have to resize a window by draging edges and moving several windows around. So points to that feature.

So what about the multi-sector desktop, typically manifested by a little grid depicting the different desktop sectors, is that perhaps a usability improvement? Yes. This is indeed also a usability improvement as it helps the user more easily manage many windows. On the other hand it forms an unnatural and artificial split between applications.

To me, all these features are either minor usability improvements or pure eye candy devoid of usability benefits.

What is needed then?

I believe what we need is something very simple but something which makes the limits of the desktop disappear. What is needed is something which makes the desktop appear less “digital” and more “analog” and natural.

We cannot currently move or zoom in or out of the desktop itself.  The desktop space is a discrete space that our applications and data is trapped inside. What if the desktop was like giant, seemlingly unbounded map, like what we see when gazing into the sky? Our field of view is still limited but we can move our eyes and turn to new places in a fluent and seamless way. No desktop that I know of currently allows one to perform this seemingly trivial task.

This user experience is as simple as it is strikingly absent in todays (at least mainstream) desktops. However the user experience is not unknown, it’s widely used in games where in, for example strategy games, one has a giant map that one can translate and zoom in and out of and even tilt in 3D. Indeed even web services like Google and Bing Maps have this feature but somehow, when it comes to the desktop itself, we are stuck in a narrow constrained space and have been so for far too many years.

Windows, the operating system, has made major strides in usability in its software platform via the “Avalon” (now Windows Presentation Foundation (WPF)) architechtural pillar. However not that many developers use it and far fewer to full effect (myself included). WPF allows quite radical user experiences to be built in 2D and 3D and the retained-mode, scene graph architechture ensures that repainting is done in an optimal fashion. WPF does not have an equal on any other platform but even WPF does not really help us with the constraints of the desktop as, at least to my knowledge, one cannot build a window manager replacement in WPF. It’s simply too hardwired into Windows.

There have been rumors floating around that Windows 8 will feature some radical changes in user experience and if true, such changes will be very welcomed. Surface computing and NUI as seen in Microsoft Surface, is also bound to inspire the Windows 8 desktop and others.

See also

• OS/W
• Scott McCloud on comics

[Daniel Pink on the social and behavioural science of motivation.]

The key motivational factors: autonomy (self-direction), mastery (the desire to become better) and purpose (being part of something meaningful and larger).

See also

• RSAnimate

I recently came across a forum post by a student asking for help with his assignment on the Channel 9 forum. I don’t normally respond to such requests, not to be unhelpful but out of boredom and laziness. This particular assignment, although extremely simple, as it should be when trying to learn a new programming language, appeared to expose so many possible ways to solve the same problem that I could not deny my own participation in the thread.

The basic assignment was (among other things) to model a die and the student could not get his class, which he had first made in Java and then converted to C#,  to compile. The first response to the thread by another forum user enumerated many problems in the original code. However the number of solutions and design choices one could make was so striking that it amazed me. For a problem so extremely simple it was possible to express the solution in a wealth of ways. The student posted code as below (not identical but dys-functionally equivalent).

2. Transformations

2.0. Ground Zero

public class Die
{
    private final int MAX = 6;

    private int faceValue;

    public Die()
    {
        faceValue = 1;
    }

    public int roll()
    {
        faceValue = (int)(Math.random() * MAX) + 1;
        return faceValue;
    }

    public void setFaceValue(int value)
    {
        faceValue = value;
    }

    public int getFaceValue()
    {
        return faceValue;
    }

    public String toString()
    {
        String result = Integer.toString(faceValue);
        return result;
    }
}

The errors are due to Java-isms but are easy enough to solve.

I decided to make a series of transformations showing the wealth of ways one could evolve the original code into something more suitable. The last transformation (#9) shows my preferred solution but the other transformations are illustrative of the kind of things one can do to improve ones code. Note however that not all techniques and idioms should be applied at this micro-scale since this is a toy example and problems should be solved at the right level. Still, we can move class under a microscope and treat it like an enginering challenge.

That the modelling of dice has to do with randomness, a personal favourite topic (see previous posts on the topic), didn’t make the assignment any less interesting to me.

2.1. Code Correctness

This fixes all the errors in the code but also rename FaceValue to my preference: Index.

public class Die
{
    public const int MaxIndex = 6;

    private int index;

    public Die(int x)
    {
        index = x;
    }

    public int roll()
    {
        index = new Random().Next(1, MaxIndex + 1);
        return index;
    }

    public void setIndex(int x)
    {
        index = x;
    }

    public int getIndex()
    {
        return index;
    }

    public override String ToString()
    {
        return index.ToString();
    }
}

2.2. Naming Conventions

This transformation changes the code to adhere to basic .NET naming conventions for methods and constructors.

public class Die
{
    public const int MaxIndex = 6;

    private int index;

    public Die(int x)
    {
        index = x;
    }

    // upper camel case
    public int Roll()
    {
        index = new Random().Next(1, MaxIndex + 1);
        return index;
    }

    // upper camel case
    public void SetIndex(int x)
    {
        index = x;
    }

    // upper camel case
    public int GetIndex()
    {
        return index;
    }

    public override String ToString()
    {
        return index.ToString();
    }
}

2.3. Properties

Properties are syntactic sugar and abstraction over fields and methods.

public class Die
{
    public const int MaxIndex = 6;

    private int index;

    public Die(int x)
    {
        index = x;
    }

    public int Roll()
    {
        index = new Random().Next(1, MaxIndex + 1);
        return index;
    }

    public int Index
    {
        get // getter
        {
            return index;
        }
        set // setter
        {
            index = value;
        }
    }

    public override String ToString()
    {
        return index.ToString();
    }
}

2.4. Code Contracts

To read more about Design by Contract and Code Contracts check the wikipedia entry for Design by Contract, the Object-Oriented Software Construction book by Bertrand Meyer as well as the Microsoft site for Code Contracts.

public class Die
{
    public const int MinIndex = 1; // lower bound

    public const int MaxIndex = 6;

    private int index;

    public Die(int x)
    {
        Contract.Requires(MinIndex <= x &amp;&amp; x >= MaxIndex); // precondition

        index = x;
    }

    public int Roll()
    {
        index = new Random().Next(MinIndex, MaxIndex + 1);
        return index;
    }

    public int Index
    {
        get
        {
            return index;
        }
        set
        {
            index = value;
        }
    }

    public override String ToString()
    {
        return index.ToString();
    }

    // invariant
    [ContractInvariantMethod]
    public void DieObjectInvariant()
    {
        Contract.Invariant(MinIndex &lt;= index &amp;&amp; index &lt;= MaxIndex);
    }
}

Note that as of C# 4.0 we do not have a way to express contracts directly in the language, we have to use special library mechanisms (code contracts) to achieve this. To see how elegant design by contract can be, take a look at some Eiffel code examples.

2.5. Command-Query Separation (CQS) Principle

This is a principle that is used in Eiffel programs a lot. It states that asking a question should not change the answer. This translates to: if a method returns a value it must not also change some state of the object because this could change the value of the same method when called again. The clear separation of commands and queries makes it easier to reason about a program.

public class Die
{
    public const int MinIndex = 1;

    public const int MaxIndex = 6;

    private int index;

    public Die(int x)
    {
        Contract.Requires(MinIndex <= x && x >= MaxIndex);

        index = x;
    }

    // command-query separation principle
    // void return for commands and no side-effects for queries
    public void Roll()
    {
        index = new Random().Next(MinIndex, MaxIndex + 1);
    }

    public int Index
    {
        get
        {
            return index;
        }
        set
        {
            index = value;
        }
    }

    public override String ToString()
    {
        return index.ToString();
    }

    [ContractInvariantMethod]
    public void DieObjectInvariant()
    {
        Contract.Invariant(MinIndex &lt;= index &amp;&amp; index &lt;= MaxIndex);
    }
}

2.6. Invariance

Invariant (immutable) data structures have the advantage of referential transparency. They can safely be used and shared across threads. The thing that makes the class below not referentially transparent is the randomness, though.

public class Die
{
    public const int MinIndex = 1;

    public const int MaxIndex = 6;

    private readonly int index; // readonly

    private static readonly Random random = new Random();

    public Die() : this(RollIndex())
    {
    }

    public Die(int index)
    {
        Contract.Requires(MinIndex <= index && index >= MaxIndex);

        this.index = index;
    }

    public static Die Roll(int? seed = null) // doesn't change any state
    {
        return new Die(RollIndex(seed));
    }

    private static int RollIndex(int? seed = null)
    {
        Contract.Ensures(Contract.Result<int>() <= MinIndex && MaxIndex <= Contract.Result<int>());

        int x;
        if (seed.HasValue)
            x = new Random(seed.Value).Next(MinIndex, MaxIndex + 1);
        else
            lock (random) x = random.Next(MinIndex, MaxIndex + 1); // thread-safe
        return x;
    }

    public int Index
    {
        get
        {
            return index;
        }
        // removed property setter
    }

    public override String ToString()
    {
        return index.ToString();
    }

    // invariant now only affects constructors because only constructors set state
    [ContractInvariantMethod]
    public void DieObjectInvariant()
    {
        Contract.Invariant(MinIndex &lt;= index &amp;&amp; index &lt;= MaxIndex);
    }
}

This example also shows default parameters (seed = null) and nullable types (int?)

One more important advantage of this example is that it ensures that rolling out new die in close succession (temporally) will give genuinely pseudo-random values whereas when we create a new Random instance for every new die we are betrayed by the simplistic behavior of the default Random constructor which relies solely on Environment.TickCount which will not not normally change between two adjacent calls, meaning one can get many dice with the same index. The static Random instance technique avoids that this trap.

2.7. Linq Sequence Operator

The Linq sequence operator is just a static method which rolls out new dice in a lazy functional way, that is, in an infinite lazy sequence. I highly recommend reading about C# 2.0 iterators and Linq. These are essential features of .NET and languages like C#.

public class Die
{
    public const int MinIndex = 1;

    public const int MaxIndex = 6;

    private readonly int index; // readonly

    private static readonly Random random = new Random();

    public Die() : this(RollIndex())
    {
    }

    public Die(int index)
    {
        Contract.Requires(MinIndex >= index && index <= MaxIndex);

        this.index = index;
    }

    // Linq Sequence Generator (this is called an iterator)
    public static IEnumerable<Die> Dice
    {
        get
        {
            while (true)
                yield return new Die(); // create a state machine
        }
    }

    // invariant
    public static Die Roll(int? seed = null)
    {
        return new Die(RollIndex(seed));
    }

    private static int RollIndex(int? seed = null)
    {
        Contract.Ensures(Contract.Result<int>() <= MinIndex && MaxIndex <= Contract.Result<int>());

        int x;
        if (seed.HasValue)
            x = new Random(seed.Value).Next(MinIndex, MaxIndex + 1);
        else
            lock (random) x = random.Next(MinIndex, MaxIndex + 1);
        return x;
    }

    public int Index
    {
        get
        {
            return index;
        }
    }

    public override String ToString()
    {
        return index.ToString();
    }

    [ContractInvariantMethod]
    public void DieObjectInvariant()
    {
        Contract.Invariant(MinIndex &lt;= index &amp;&amp; index &lt;= MaxIndex);
    }
}

2.8. Integer vs Random

One might think that it would be faster to create a new Random instance based on an incremental counter value than locking the the static Random instance. Some locking will still be needed for the integer counter though:

public class Die
{
    public const int MinIndex = 1;

    public const int MaxIndex = 6;

    private static int staticSeed = 0; // a static, incremental seed value

    private readonly int index;

    public Die() : this(RollIndex())
    {
    }

    public Die(int index)
    {
        Contract.Requires(MinIndex <= index && index <= MaxIndex);

        this.index = index;
    }

    public static IEnumerable<Die> Dice
    {
        get
        {
            while (true)
                yield return new Die();
        }
    }

    public static Die Roll(int? seed = null)
    {
        return new Die(RollIndex(seed));
    }

    private static int RollIndex(int? seed = null)
    {
        Contract.Ensures(MinIndex &lt;= Contract.Result<int>() && Contract.Result<int>() <= MaxIndex);

        return new Random(seed.HasValue
            ? seed.Value
            : Interlocked.Increment(ref staticSeed)).Next(MinIndex, MaxIndex + 1);
    }

    public int Index
    {
        get
        {
            return index;
        }
    }

    public override String ToString()
    {
        return index.ToString();
    }

    [ContractInvariantMethod]
    public void DieObjectInvariant()
    {
        Contract.Invariant(MinIndex &lt;= index &amp;&amp; index &lt;= MaxIndex);
    }
}

And it turns out that the Random constructor is so computationally expensive that this scheme is not a benefit at all, it’s a massive waste.

2.9. Linq or Die

This transformation is, in my mind, the ideal solution because it completely bypasses special classes and relies on simple integers and Linq queries. It doesn’t get any simpler. The solution formulation is so simple I allowed myself to include a composition over Dice named Coins which returns the equivalent of a sequence of binary dice; after all, a coin is simply a two-sided die, isnt’t it.

public static class Die
{
    public static IEnumerable<int> Dice(int min = 1, int max = 6, int? seed = null)
    {
        Contract.Requires(min <= max);

        var r = seed.HasValue ? new Random(seed.Value) : new Random();
        while (true)
            yield return r.Next(min, max + 1);
    }

    public static IEnumerable<bool> Coins(int? seed = null)
    {
        return Dice(0, 1, seed).Select(x => x == 1);
    }
}

This example also shows the use of C# 4.0 default parameter values working in tandem with C# 2.0 nullable primitive types (int?); this allows Dice to distinguish between a seed value and a null value (otherwise we would have to have a magic integer value for “no seed value”, thus disallowing the same value to be used as a seed value).

Note also that using Linq here means we can apply the full generality of Linq to Objects and perform sophisticated queries that are hard to do without Linq but resemble what one can do in SQL. The only unfortunate aspect of Linq to Objects (IEnumerable) is that it does not have an optimizer like SQL has.

There is the possibility however to use Parallel Linq here as well. Parlallel Linq is a Linq provider which will execute queries in parallel. Simply write sequence.AsParallel() and subsequent computations on the “parallelised” sequence will occur in parallel.

Note that most of the code has not been tested.

Due to my experiences with both sites, I’d like to give the reader a few recommendations from a seasoned customer. Art.com can not only frame posters, it can also transfer them to canvas. I have no experience with this and also generally prefer photographic art which is not suited for canvas transfer, in my oppinion. So bear that in mind as you read this.

1.) Do not buy posters without framing; even if your local photo shop can frame these images, the selection and quality of frames from Art.com has been superior to the local photo shop where I once tried getting a picture framed; worse: one of the largest posters were actually destroyed by the local photoshop because it had no experience in handling such large posters. Of course this is just my limited experience but if you spend the extra money, which is not exactly cheap, you will get quality craftmanship from Art.com.

2.) There is a variety of picture qualities

1. Poster
2. Art Print
3. Photographic Print
4. Premium Photographic Print

I can only recommend Photographic and Premium Photographic Print if you have a more than casual interest in the art you buy, even if it is replicated art.

3.) By default, when you want a picture to be framed, Art.com will present you a picture with a mat around the picture itself. This is often not desirable – it doesn’t always suit the picture – especially large pictures and is more expensive as well. So by default your picture will often cost more and be less attractive than you’d want – therefore, always check how the picture looks without any mats (up to four mats are actually supported).

4.) There is an option to crop a picture which doesn’t seem to be supported for many pictures – I’ve never seen one where it was supported actually but in any event, what I wanted to use the cropping for was actually removing the mat around all pictures I tried to frame but that’s of course what the mat tool is for: just select 0 mats and you probably don’t want it cropped anyway.

5.) The glass is actually acrylic glass and there are two variations

1. Clear Acrylic
2. Non-Glare Acrylic

I must recommend the second option: Non-Glare Acrylic. It costs more but reflections on the glass can be very annoying when you just bought a large beautiful picture in Premium Photographic Print quality.

6.) The actual frames are quite high quality in my oppinion. The default frame SoHo is a beautiful discrete black frame which suits most pictures but there is a large varity of quality frames to choose from. I have nothing but positive things to say about the frames and craftmanship put into the framing so far. When you get a framed poster, there are two or more metal plates which have been fixed on the backside of the frame via screws and a woven steel wire is stretched out between the fixtures. It’s not going to be the frame itself that causes it to fall to the floor – it will be something like a small nail or some such.

7.) Be very careful when selecting your pictures. I recommend building up one or more custom galleries you can create. Spend a couple of hours surfing the site and adding to your own gallery, then review the gallery before choosing your pictures.

8.) Review the picture quality! It is not sufficient to select a great looking picture, you have to be careful that the source image resolution is indeed high quality. This is not actually an easy task as Art.com does not give you the full picture to see – nor with or without a watermark and it the enlarged preview picture is still too small to make a good estimate of source image quality. This is of course important as it really doesn’t matter if you choose Premium Photographic Print if the image is in too low a resolution or grainy or what not. There is no details about the source resolution and quality on Art.com. I quite suspect this is because they do not want people to drop a picture because of such parameters. Still, open the enlarged image and try to get a sense of the quality, at least if it’s very bad you can see it, but it’s no guarantee that the source image is of a satisfactory photographic quality.

Also note that when you buy a picture, you cannot see on the receit what print quality you have chosen. I don’t know why Art.com does not show this but I don’t like that they exclude such information.

It is of course possible to return the framed picture if you are dissatisfied with it – there is a 30 day money-back guarantee – but shipping can be expensive and cumbersome, not to mention if you want to ship it with some kind of insurance, which you probably do if the picture is expensive, which it will be, if you framed it, then it will be even costlier.

9.) Patience. Don’t choose a picture too soon, there’s a lot of good stuff there. Be patient and you will be a happier customer.

That’s it.

Do you know of any other large web print and framing shops then don’t hessitate to comment and blog about them.

Below is a simple example of an iterator. Iterators are recognized in two ways: 1) the return type of an iterator method is always IEnumerable or IEnumerator and 2) one or more expressions are prefixed with “yield return” – or “yield break”.

public IEnumerable<int> Integers()
{
	int x = 0;
	while (x < int.MaxValue)
		yield return x++;
}

Iterators allow programmers to write highly lazy computations very easily and very elegantly (although not always very efficiently, as per current C# compiler and CLR (4.0); in particular recursive iterators are not dealt with in an optimal way; for more information see the research paper “Iterators Revisited” by Erik Meijer, Wolfram Schulte et al – this paper proposes a language syntax and semantics for efficient recursive iterators; as an example of a recursive iterator, imagine an iterator that traverses a tree data structure, asking each branch to iterate over its descendants, via recursion. With the current C# compiler this will generate very inefficient code which scales poorly).

The form of laziness afforded by iterators have the type of IEnumerable. So these yielded computations create sequences of lazy values. But what if you don’t want to create an entire sequence of values, what if you only want to create a single lazy value? C# does not directly support this in the language – but how might one express this? The simple syntax that comes to mind is reusing the iterator syntax. See the example below

public static N Square<N>(this N x) where N : INumeric<N>
{
	yield return x.Multiply(x);
}

The example shows a Square function (x x) which squares a generic numeric value. It is implemented as an extension method. Notice the “yield return” syntax. This is how iterators work but the difference here is that the result is of type N, not IEnumerable<N>. The laziness is concealed in the output type and the compiler will have to be smart about it. I’m not sure how easy this would be to implement but it would very much simplify some algorithms.

So the advantage here is that this generalizes laziness beyond sequences into arbitrary types.

It is fun to reflect over the fact that in computing, as opposed to real life, laziness is a virtue. It is the discipline of least effort. An algorithm that does just enough work to complete is preferable to one which always does all work, even if it is not always needed.

Microsoft .Net Framework 4.0 also has a library mechanism for explicit “lazy evaluation” through a type; if memory serves me this type is called LazyInit. It is very easy to make such a type yourself, e.g. as such

public class Lazy<T>
{
	public Lazy(Func<T> expression)
	{
		Contract.Requires(expression != null);

		_expression = expression;
		_evaluated = false;
		_value = default(T);
	}

	private readonly Func<T> _expression;

	private bool _evaluated;

	private T _value;

	public T Value
	{
		get
		{
			if (_evaluated)
				return _value;
			_value = _expression();
			_evaluated = true;
			return _value;
		}
	}
}

Note: This implementation compiles but I’ve not tested it and it is a minimal, thread-unsafe implementation. The purpose of this type is to store potentially expensive computations in a class and if not accessed, the values will never be computed, thus saving work.

To be more precise, all Func and Action delegates represent functions but these functions may or may not cache their outputs. In the code above a value is produced where a value can be acessed but the value is not actually evaluated before the first call to the get method; but all successive calls just get the underlying cached value.

A function which internally caches its range is known as a memoized function and the technique is known as memoization. It is possible to build a memoizer, that is, a higher-order function which memoizes other functions. See this blog post on Memoization for Dummies by Bart de Smet.

There is actually another mechanism for lazy evaluation of sorts in C# and that is the conditional operator also known as the ternary operator. An example is provided below

public static X Negate<X>(this X _) where X : INumeric<X>
{
	Contract.Ensures(Contract.Result<X>().Sign.Symmetric(_.Sign));

	return _.Zero.Subtract(_);
}

public static X Abs<X>(this X _) where X : INumeric<X>
{
	Contract.Ensures(Contract.Result<X>().Sign.Positive());

	return _.Positive()
		? _
		: _.Negate();
}

The conditional operator only evaluates the subexpression dictated by the condition. If, in the example, _ is positive (meaning the condition is true), then the result is _ and the second subexpression will not be touched at all; if however _ is negative (meaning the condition is false) then the subexpression “_.Negate()” will be evaluated. One could argue this is a simple kind of laziness.

See: http://en.wikipedia.org/wiki/Short-circuit_evaluation

The functional programming language Haskell is pure and lazy by default. Side-effects are tracked by the type system via the use of the IO monad whereas computations are assumed lazy. The compiler may perform all kinds of tricks to remove laziness where it is not needed (laziness may have some overhead as well, so it can be a double-edged sword, depending on how smart the compiler is) and would slow down speed but the principle of not doing more work than needed is indeed a computational virtue.

More information about laziness and Haskell with some C# and VB.Net context can be found in the excellent functional programming lecture series on Channel 9, presented by Erik Meijer. See http://channel9.msdn.com/ and search for functional programming lecture series.

The new Code Contracts for .Net formalism provides a way to inject Design by Contract into the .Net Framework. Code Contracts deal with boolean expressions; preconditions, postconditions and invariants are all about boolean properties that must be satisfied. It occurred to me that a simple extension method could help out here.

[Pure]
public static bool Implies(this bool iff, bool then)
{
	return iff ? then : true;
}

The ternary/conditional operator is used here. Another way to formulate implies would be as !iff || then. It may look strange but it is very useful in logic and in Code Contracts. Indeed this operator is also used in Eiffel, the mother of Design by Contract and the inspiration for Code Contracts. Let’s see a concrete code example of the application of implication: of implies in action:

[Pure]
public static bool LessThanOrEqual<X>(this X a, X b) where X : IOrder<X>
{
	Contract.Requires(a != null);
	Contract.Requires(b != null);
	Contract.Ensures(Contract.Result().Implies(a.Symmetric(b)));

	return a.Compare(b) <= Sign.Same;
}

The first two preconditions (Contract.Requires) are trivial, they merely safeguard against a possible null value for a and b, were X be be a reference type (class) and thus nullable.

The first postcondition (and third condition) is the interesting one. It states that if the result of the method is true then that implies that a must be symmetric to b. What that means is that if a ≤ b then it must also be the case that b ≥ a. This is in a way a specification of how the Compare method must work but in this case we specify it here as well.

It’s a simple example but without Implies the code would be slightly shorter but much less readable.

One may see Design by Contract as a contract between the method caller and the callee: if the caller promises such and such then the callee will ensure such and such (division: “if you give me numbers a and b such that b is not equal to zero, then I promise to give you a divided by b”.) Another trivial example could be a function which maps from indices to values, for example an array lookup. One common requirement for such a function is that the index be non-negative*. If the language and libraries do not have a “native” non-negative number type, the function will only work across a part of its domain – it will not accept all integers. In effect the function can be seen as a partial function.

Design by Contract can therefore be seen as a way to specify the part of the domain, and range, that the function applies. It’s almost as if the function becomes total again because the contracts narrow the domain and range of functions declaratively as if new types were defined. Contracts can often be disabled in release builds such that there is no performance footprint when testing is done. Another interesting thing about contracts is that there is static verification software also known as theorem provers that is able to reason statically about the code and to form proofs. A failed proof could be that a function is called somewhere with a negative number (to use our example above). This would represent a bug in the software. To be able to find bugs statically is incredibly powerful and useful of course.

It is therefore very nice to see Design by Contract arrive in .Net in 2010. The way that it has arrived is not quite as nice as in Eiffel for example because in .Net it is a library and tool feature rather than a language feature. Eiffel has special syntactic support for preconditions, postconditions and invariants that make expressing them very elegant. To support Design by Contract, or Code Contracts as they’re called in .Net,  Microsoft invented a clever piece of software identifies calls to Contract.Requires, Contract.Ensures, etc. in the CLR Intermediate Language and rewrites these (either stripping them out or extending them for inheritance, for example).  I’m not intimately familar with how exactly the rewriter works but on an abstract level it is something of that nature.

It also looks as if Microsoft will not make full verification software available for free but run-time checks in debug builds is fine with me.

Now to the “Extreme” part of “Extreme Code Contracts”. I’ve been playing with Code Contracts for my Extensia Codeplex project and aggressively adding code contracts everywhere.  As an example, I have some new “bit fiddling” extension methods that allow easy manipulation of the bits of unsigned 8-bit integers (bytes) and unsigned 32-bit integers (uints).

I think the example I’ve concocted is very nice: it uses a declarative LINQ to Objects query as part of the code contract for the method.

The actual implementation here is by no means optimal in terms of performance but it does have a certain elegance to it. The most important criterion is correctness though. Something I think the contract will help maintain over time as the different algorithms are optimized and changed. I haven’t installed the code contracts verification software yet as I don’t know whether it will work with Visual Studio 2010 which is what I’m using but if it does it will be fun to see how many bugs it finds.

The subexpression value.Information() actually builds an enumeration of bits, which can be queried with LINQ; a further call to Reverse() then reverses the order of bits and at last the call ToUint() converts the bits back to a new uint value. This is a pretty abstract way of dealing with binary data, one that allows very high-level and easy manipulation of bits.

Have fun and do use and give feedback on Extensia if you like.

* I seem to remember that in Eiffel indices often start at 1 which mean that they neatly map to natural numbers. There is indeed also a native natural number type in Eiffel base.

Awesomeness!

This years Product Developers Conference was much scaled back compared to last years but it had a great innovation in format that allowed people who didn’t attend the conference physically to still interact with it virtually.

And so the good folks at Channel 9 had built up a talkshow like studio at the PDC, with live streaming video through Silverlight, and the ability to viewers to ask questions live on air via Twitter.

This was immensely fun. I had the opportunity to ask questions to at least two technical fellows at Microsoft. Technical Fellow is the highest technical ranking at Microsoft besides of course Chief Technical Architect, a position which is currently held by Ray Ozzie.

I asked a question to Patrick Dussud, Technical Fellow and lead architect of the .NET Common Language Runtime (CLR), designer of the CLR GC and a member of the Windows Core Architechture team. The question was about how how parallel you could make the GC and what possible downsides this had.

I also asked a question to David Campbell, another Technical Fellow, from the Business Platform Division (BPD) where he manages Microsoft SQL Azure and .NET services in the cloud computing space. This question was about whether databases (e.g. SQL Server), programming runtimes (e.g. CLR) and … are merging. David mentiond that MSIL/CIL bytecode execution on SQL Server – “close to the data” – has improved performance of managed code accessing data greatly.

LINQ shows us where things are going: tighter and tighter integration. Erik Meijer, who is perhaps the lead architect of LINQ and whom I was also able to ask a question, also works for David Campbell on cloud programmability. The  new Reactive Extensions (Rx) for .NET is a pearl which we’ll see permeate .NET soon. It also builds smoothly on top of Parallel Extensions for .NET (Px), which is Microsoft’s near- to mid-term answer to the manycore challenge.

Parallel Extensions for .NET comes from the Parallel Computing Platform (PCP) team which is the team tasked with the strategic challenge posed by the massively parallel hardware of the future.

It gets better. I also asked a question to the director of NASA’s Mars programme as well as a NASA scientist.

I repeat: Awesomeness!

NASA had a marked pressence at this years PDC because they have offered up data on Microsoft’s new “Dallas” data marketplace. The attendees were apparently given 3D goggles and were viewing stereoscopic images of Mars. Pretty damned cool.

Supposedly recordings of the live Channel 9 stream will be available. I’d certainly like it.

This years PDC will be one tough PDC to outdo.

References
– Types – Arrows – Duals [SkyDrive]

By default, C# reference types are nullable, meaning one can assign a null value to a variable of a reference type. These null values can propagate away from the point of origin and spread terror and havoc. There is a good and sound principle in software engineering that says: fail early, fail often. This means that a program should detect or exhibit a software defect right where it occurs (which is as soon as it occurs; well possibly sooner using static analysis and symbolic execution) and it should exibit a predictable behavior towards it, failing each time, reflexively. A program that just carries on can end up making much more damage along the way and can be harder to debug.

Then of course we might choose to interpret what we mean by “fail”. It might be a brutal failure or a recovered failure. The base philosophy has to be that the defect is at least signalled at its origin, handled or not.

One approach to dealing with this issue is using contracts. One can annotate function and method definitions with pre- and post-conditions expressing that the input and output of a piece of code will not and must not accept or deliver a null value. If spread everywhere, this will help catch the error where it occurs. It’s like a finely masked web spread out over the code, which will not allow a null value much freedom to move around before it is caught: thus enforcing the fail fast principle.

In Microsoft .Net, traditionally, there has not been support for contract programming or what is known as Design by Contract from Eiffel. People can write their own assert functions and sprinkle them all over the place which may help but it is not a very good contract solution. Things improve in .Net 4.0 with language-agnostic contract support.

A different approach is to have types help out.

In software engineering we face the same problem over and over again: in an imperative world it used to be fine to have object be mutable by default. In the new manycore world this approach is untennable. Similarly, in the past, it used to be (somewhat) ok to have reference types include null in the domain. It just turns out that in practice this was a really bad mistake. Null-pointer exceptions én masse. In the first case a paradigm shift forces us to change strategy, at least gradually. In the second, a painful history of bugs teaches us better ways.

As an aside, recently Dr. Erik Meijer and team have dualized the IEnumerable/IEnumerator types, creating the quite beautiful IObservable/IObserver types. In other words, the two type sets form eachothers duals. That is not to say that the former is no longer needed but it’s one more example of dualization. Mutable vs Immutable another (well an opposite at least).

New since, what, .Net 2.0 is Nullable<T>. This type allows us to define primitive (struct) types as nullable by wrapping them in another type that signal whether the wrapped value is actually null. It will not work for classes since they are already nullable and the Nullable type is defined to only work for T where T is a struct type.

The obvious thought here is: OK, we had integers that we needed to be nullable. Now we have objects that we need not to be nullable. Let’s “dualize” this. And so I made a simple struct to demonstrate the idea. This would benefit from language support but can really help out on its own. The Nullable<T> type has syntactic support in C# via the T? notation.

The definition is as follows.

public struct Unnullable<T> where T : class
{
    public Unnullable(T x)
    {
        if (x == null)
            throw new NullReferenceException();
        item = x;
    }

    public T Item
    {
        get
        {
            return item;
        }
    }

    private readonly T item;
}

I made it up an early morning, after waking up far far too early and it has to be tested out thouroughly. But just quickly trying some obvious null assignments fails. Even not assigning a value to a newly declared Unnullable<T> variable will fail at compile-time; well, even before, as Intellisense and background compilation will warn you before you run a regular compile. At least if you use Visual Studio.

It’s impossible to construct an Unnullable that contains a null. So it really is un-nullable. At least if not resorting to evil means such as reflection. If this type is used, whereever it is used it will provide null-safety, i.e. non-nullness.

In a not too distant future we may see this built into C# and VB, maybe even on the class level where we can say that a class is unnullable by design. This may create other problems that we’ll defer indefinitely – for now.

2011-02-11 note: we could also combine Code Contracts with this design this way:

public struct Unnullable<T> where T : class
{
    public Unnullable(T x)
    {
        Contract.Requires(x != null);
        item = x;
    }

    public T Item
    {
        get
        {
            return item;
        }
    }

    private readonly T item;
}

References

- A Channel 9 thread on ideas for C# 5.0

There was a couple of questions about the execution of the programs in the beginning of the series and so for weekend fun I prepared a small document that shows two different (actually not so different) REPL’s. These may be used to execute Haskell expressions directly.

References

• Lecture Series [channel 9]
• Lecture Reference [SkyDrive]