The D Programming Language, and my video game engine

permalink         categories: programming         originally posted: 2005-04-13 19:20:31

I've been writing a video game for Windows for more than two years now. At its heart is a game engine of my own fiendish design. I say "fiendish" because it's, well, awful. I plan on finishing my current game, then doing another game with this source tree. After that, if I am any sort of even mild success at this, I shall throw away this source tree and begin anew—which is something I'm quite looking forward to.

In my current engine, I wrote nearly everything myself. I feel in retrospect that was one of my worst mistakes. There is a lot of good source code out there, under liberal licenses, written by people who know a lot more about their problem domain and have spent a lot of time thinking about it than I ever will. It is foolish to reimplement all my own wheels, when I'm not always a very good wheelsmith.

Python

One definite change for the new engine: I will use Python wherever feasible. It's a wonderful, modern, joyful language to program in. Its only real drawback: as an interpreted language, it's not good for performance-critical code.

There are several implementations of the Python language, as I shall soon discuss. I mention this now merely so I can say that the traditional implementation of Python is often called CPython, because it's the Python written in C. (As opposed to...? Read on!)

C + CPython = SeaSnake

My next engine is code-named SeaSnake, because it's to be a mixture of C(++) and Python. My goal with SeaSnake is to use existing libraries wherever possible and write as little of my own code as I can. SeaSnake, therefore, will be mostly comprised of third-party libraries. I'll glue them together with C++, write enough additional C++ to handle all the performance-critical code, then make all of the above callable from an embedded CPython interpreter. The rest of the engine I'll write in Python. The games will also be written in Python. I'll be cruisin'.

But thinking about working in C++ these days just makes me sad. I look around at my fellow programmers, working in Python, and Java, and C#, and I feel like the kid who is stuck indoors on a sunny day. Yes, hopefully I can leave the C++ part behind soon enough and just live in my Python ivory tower. But getting all those libraries glued together will be a pain; C/C++ libraries aren't the most interoperable lot you've ever seen. C/C++ lacks much of the common functionality offered by other environments, so library authors feel behooved to reinvent their own wheels... their own memory management, their own error handling schemes, and so on. It'll be no small feat to integrate this Tower of Babel of libraries.

C# + IronPython = SharpIron

I had considered what I called the SharpIron engine, built on C# and IronPython. IronPython is a CLR reimplementation of Python by Jim Hugunin, of Jython fame.

(In case you haven't heard of Jython, well, it's miraculous. It's a Python compiler for the JVM. You write Python in one end, and JVM bytecodes come streaming out th' other. It supports all the native Java types, and Java and Jython classes are totally interoperable. A real class act. Sadly, it's fallen a bit behind the times; the current Jython release is from December of 2001, and only implements Python 2.0. But there are signs that the project is re-emerging from its slumber.)

There are many compelling benefits to moving to C# and IronPython:

• C# is a much nicer language than C++.

• I get all the benefits of the CLR... garbage collection, introspection, native code generation for the user's computer.

• I get to use "real" Python.

• IronPython generally runs Python code faster than CPython! (Though it'd still a good idea to move performance-critical code to C#.)

• Not only is inter-module interoperability fantastically good, I get instant, automatic, fantastically-good inter-language interoperability. No glue code necessary. I can write a class in C#, and instantiate an object, and I can automatically see and use that object from Python.

• I can work with my Python code in the full MSVS debugger! I can examine variables, set breakpoints, set watchpoints... all the good stuff that's implictly support for every CLR language.

• I expect learning C# would be a good career move.

• Other people are already making .Net bindings for many of the libraries I want to use, which cuts down on the amount of code I'd have to write.

There was only one downside, but it's a doozy: the .Net runtime. I believe .Net 1.0 shipped with Windows XP, but most people aren't running that yet. You can download the .Net runtime for any version of Windows you like, but it's 25MB—and shareware gamers are notoriously fickle about large downloads. Still, I figured if I only required 1.0 that would be enough people that I could limp along. And it'd be worth it for the rest of it.

However! Following the initial announcement of IronPython, and likely because of it, Mr. Hugunin went to work at Microsoft in the CLR runtime group, sometime in mid-2004. Specifically he's working on enhancing future versions of the CLR so they work better with "dynamic" languages like Python. And, well, he's been "drinking the Kool-Aid". IronPython, which is still in early beta, now only supports the .Net 2.0 runtime (and above). And, guess what, kids: .Net 2.0 itself is only in early beta. I give Mr. Hugunin plenty of slack; I'm sure the .Net 2.0 CLR makes his life implementing IronPython plenty easier. But it does mean I can't ship a game on this platform for several years.

With this sad realization, the SharpIron engine has been relegated to the back burner, and it's back to SeaSnake.

Boo

Before giving up on C# and the CLR, I briefly considered Boo, a Python-like language designed for .Net. But Boo is also very beta, and diverges dramatically from real Python, in the quest tailor itself to running in the CLR. For instance, Boo is statically-typed! Quite a departure from the duck-typed Python. (If Python can be said to have any typing whatsoever.)

In many ways Boo is a more native language than IronPython. IP shields you from some of the assumptions the CLR makes so it can be more like real Python, wheras Boo changes the language so it fits in better with the CLR's assumptions. But that meant diverging from Python syntax. And, worse yet, sometimes they made incompatible changes to the language seemingly for pure aesthetic reasons. "As long as we're reimplementing Python," I suppose they thought to themselves, "we might as well fix the things about it we didn't really like." Agree or disagree with these decisions, I decided don't want to learn a purposely-different dialect of Python; it'd trip me up too much switching between it and real Python.

So in the final analysis it just wasn't compelling enough to overcome my problems with .Net's installed base problem or its own beta-ness, and it just wasn't Python enough.

D

The other day, during my usual web stumbling, I saw a recommendation for the D programming language. I guess I'd seen it before, but never really considered it. But suddenly it was shiny and new.

D is the creation of one Walter Bright. Mr. Bright wrote Zortech C++, the very first native C++ compiler (before that all we had was cfront, which translated C++ to C). Back in the early-mid '90s, when there was a diverse and competitive market for Windows compilers, Symantec bought Zortech and rebranded the compiler as Symantec C++. During his subsequent tenure at Symantec, Mr. Bright wrote the compiler for Symantec's Visual Cafe, which he says is still "the world's fastest". As time passed, the Windows compiler market went away as everyone switched to Visual C++. One by one the other players dropped out, and today there are only two viable commercial compilers for Windows: Microsoft and Intel. In the late '90s Mr. Bright got the rights to his compiler back from Symantec, and now gives it away for free as the Digital Mars C / C++ compiler.

Now he gives away the Digital Mars D compiler too. D is Mr. Bright's own creation, started apparently in 1999 and first released in 2001. It is meant as a successor to—and hopefully improvement on—C++. It's available for Win32 and Linux, both x86 only so far.

It doesn't suck as much as C++ (how could it?). Indeed, its feature list looks much like C# and Java:

• It has built-in garbage collection.

• It has dynamic arrays as core types, and it supports array slices too.

• It is interoperable with C, and even interoperable with COM on Windows. (You can call COM interfaces, and even write COM servers in pure D.)

• It has templates, and they couldn't possibly suck as hard as C++'s. It has "mixin" classes and interfaces. (Belt and braces!) It has delegates and closures. All the modern OOP candy.

• It has exceptions, and "Design By Contract" ideas built in to the language (like assert() and pre/post conditions).

• It's a 100% compiled language—no JIT, no bytecodes, no VM to speak of.

• Interoperability between third-party modules is very high. The language provides so many facilities that folks don't have to reinvent their own incompatible wheels.

• It's currently at the top of the Great Compiler Shootout leaderboard. However! That's an artifact of how scores are calculated. D is the only language where all tests are available, and that means it can score more points than any other language. (Note how, generally, the fewer tests a language has implemented, the lower its score is.)

• The compiler is free (as in beer), and apparently I can ship programs with it for free too.

• People have already made D bindings for Python and DirectX.

• The compiler outputs native debugging information, so you can use your existing debuggers; Visual Studio and WinDBG under Windows, gdb under Linux.

• It natively supports Unicode.
• It doesn't support multiple inheritance, which is just fine by me thank you very much.

General downsides to D:

• It is the product of one guy, Walter Bright. Progress is slow.

• While the language is not fresh-off-the-boat new, it is a tiny corner of the world. So it hasn't gotten millions of lines of code run through it like Java, C++, C#, or Python. There is only a small, but growing, list of projects for D at the D Source portal. Note the "status" fields on those projects. Not a one of 'em is out of beta!

• I doubt I know anyone who has ever used it, is currently using it, or has any plans to use it.

• Debugging isn't perfect. You can't browse dynamic arrays in VC++; you can only examine their contents via the Memory window, and even there you have to tease out the address of the array by hand.

Shortcomings of the D language itself:

• D falls pretty far from the modern "everything's an object" language design. I gather Mr. Bright is quite concerned about performance, so perhaps this was an explicit design choice on his end. Never-the-less, I find I quite enjoy making method calls on strings and arrays in Python.

• Strings are not objects unto themselves in D. It's like C; a string is an array of chars. Of course, in D those arrays are dynamic, so it's the next-best thing to "real" strings, and a definite improvement on C. (There's a third-party string class library; it looks lovely and all, but of course that will suffer the same problems as every other reinvented wheel out there.)

• In D, the == operator is the "equality" operator; when used on two objects, it recursively compares all its members to ensure that the objects are exactly equivalent. This means that the common C/C++/Java/C#/every-other-C-like-language idiom if (object == NULL) is guaranteed to segfault in D. To determine whether or not two pointers point to the same object, you must use the is operator, as in if (object is NULL); you can also use ===, which means exactly the same thing. (So I guess is == ===, but == != ===. Hope that's clear!)

  While having an equality operator seems like a good idea, swapping it for the == operator sure seems like an awful one. My fifteen years of C and C++ programming experience means I would constantly trip over this. I certainly feel this is in opposition to Mr. Bright's stated rule of thumb: "if D changes the rules, if one tries it the C++ way, one gets an obvious failure rather than a subtle, erratic one." I would prefer to see == and === switch, so that if (object == NULL) worked. Perhaps it's not too late to make this kind of change to the language; after all, it has yet to reach 1.0, and the installed base isn't that large. But this is Mr. Bright's language, not mine, and I sincerely expect he is uninterested in my opinions on the subject.

  [Added 2006-02-26] A minor update: as of version 0.126, the keyword === is deprecated, and should be replaced with is. Same goes for !==, replaced with !is. == still means this recursive "equality" operator though.

• [Edit 2006-02-26] Great Caesar's Ghost, he's fixed this one! As of version 0.148, D now has a proper one-byte boolean type. There is a minor fly in the ointment—D 0.148 auto-casts bool to int, which I don't agree with. But this is a wonderful change, and one which a year ago I wouldn't have thought would happen.

  Here was my original bullet point, preserved for posterity.

  D doesn't have a native boolean type. (Yet, strangely, D has built-in keywords for true and false.) Now, you can limp along without a built-in boolean type, as C did for twenty years. But Mr. Bright feels, pretty strongly, that D doesn't want or need a real boolean type. D has a built-in alias for bool that maps it to the bit type; a bit is an unsigned one-bit integer, either 0 or 1 (aka true and false). One side-effect of this implementation is that you cannot specify a bool as an output parameter to a function.

  This has been a major source of contention in the D world, but Mr. Bright stands firm; he thinks bools being 1-bit integers is a good idea, and it's his language, and that is that.

  Since it's my blog, I get to have the last word. I disagree with nearly every aspect of this approach. I would prefer to see booleans be real native types, stored as bytes, and limit their values at compile-time to true and false which would become 1 and 0 at runtime. If this is a performance concern, add a peephole optimization for ((boolean expression) ? 1 : 0). But if it's a syntactic sugar "shorthand"... it's a bad one. It's bad news to auto-promote between bools and ints in a strongly-typed language. (Yes, C did it—but if C didn't make mistakes or need improvement there wouldn't even be a D.) Booleans and ints aren't conceptually related, and allowing people to mix them is a potential source of confusion and errors. Here, too, is a change I think that could still be made, if Mr. Bright was interested in making it, which he's not.

  [Added 2005-04-16] Oh dear, this is even worse than I thought. If you run the DMD compiler and ask for "warnings", you will get a warning every time you assign the results of a boolean expression to a bool variable, like so:

  bool foo = (this && that);

  This is because boolean expressions actually return int but a bool is internally a bit, and you can lose valuable precision when you assign an int to a bit. Again, this is a really dumb move; the return type of a boolean expression should be the native boolean type. But, again, I expect Mr. Bright has no interest in changing it.

• [Added 2005-04-14] The following is a valid D program:

  int returnNothing()
  	{
  	}
  
  int main()
  	{
  	returnNothing();
  	}
  

  This compiles! If you run it you get an "assertion" failure at line 3. This is apparently by design; when I asked about this on the newsgroups, all the replies mentioned that this was a well-trod point of contention.

  In other languages, having control paths that don't return a value would be an error. In D, it is a warning, and warnings are turned off by default. But, aha!, if you turn on compile-time warnings with -w you do get a warning:

  warning - testmain.d(1): function testmain.returnNothing no return at end of function
  warning - testmain.d(5): function testmain.main no return at end of function

  Guess I'll have to stick with -w from now on, huh!

• D doesn't have real introspection (what Java calls "reflection"), just some simple "class information". Improved introspection was on Mr. Bright's to-do list as far back as 2001. But he is just one guy.

• [Added 2005-04-16] Normally the DMD compiler does not emit "warnings". This is by design, see No Warnings near the bottom of the D overview page. You can enable warnings with the -w flag. But the DMD compiler thinks warnings are just as bad as errors, so your build will stop cold if DMD emits even a single warning.

• The existing garbage collector built-in to D is The Boehm-Weiser Conservative Collector. While it has a lot of wonderful attributes, it's not an incremental collector. When it collects, it collects everything at once. This can impact real-time simulations like games if not carefully managed (see below).

• [Retracted on 2005-04-14] D doesn't support multidimensional arrays, though you can have "jagged" arrays (an array of pointers to arrays). Update: D does support multidimensional arrays. Not sure where I read that, but it was apparently old information. D appears to have supported multidimensional arrays as recently as version 0.54, released February 14^th, 2003. (Valentine's Day!)

So what about this "D is the product of one person" stuff? Would D die if Mr. Bright got hit by a bus? Well, he has released the source code to the D front-end and the D standard library, including the (Boehm-based) garbage collector. Mr. Bright's sources are dual-licensed, under GPL and the Perl "Artistic" license, which is basically compatible with commercial software. The D front-end is currently being converted to work as a GCC front-end. But that's still in the early stages.

The existing back-end is entirely proprietary; if Mr. Bright got run over by our hypothetical bus, development on it it might very well die with him. A couple years ago, another guy named Burton Radons started writing his own D back-end. But he apparently stopped soon after, as his distributions are dated 2002... perhaps work on that stopped when Mr. Bright ported D to Linux himself.

D + Python = my next engine?

I'm sure you can see what I'm pondering: should I write my next game engine in D? Instead of writing the glue and performance-sensitive code in C++, I could write it in D. D doesn't suck nearly as much as C++, it could talk to all the libraries I want to deal with, and people have already done some of the work for me. The sooner I could get the glue code written, the sooner I could leave that language behind and start writing in Python.

But, I just don't know. Switching to D just seems so... regrettable. Certainly, spending time on D would be counter-productive as a career move. And when there's a bug, you don't have many options. There's no company to get on the phone. Either you appeal to Mr. Bright's sense of propriety or, if it's in something for which you have source code, you can try to fix it yourself. This is true of many open-source projects, of course, but few projects are so fundamental, so interesting, and so small.

Anyway, I already feel like I'm way off the beaten path. Switching to D would be the technological equivalent of moving to a desert island or a remote cave. Though I wouldn't be the first person to switch to D for writing games. A fella named Kenta Cho has switched to writing his games in D. All his games for Windows since rRootage have been written in D using OpenGL, SDL, and Ogg Vorbis. They aren't polished games up to modern standards, but they're more than worthy as a proof-of-concept. It is possible to write videogames for Windows under D.

(Though I did notice, during one particularly long game of Torus Troopers, the game came screeching to a halt at one point for maybe 300 milliseconds. I can only assume that D's non-incremental garbage collector kicked in. If I switched to D, I'd have to carefully manage the gc myself.)

I guess I'm looking for reasons why D would be a bad idea. Switching to D is very appealing, for all the reasons I just described. I suspect part of that attraction is because I haven't spent any time with it, and I don't know its warts. But from a distance it looks survivable, even enjoyable.

One additional point to consider: D is close enough to C++ that it is concievable to undertake downgrading a D program to C++. If I invested in D, and then worst-case I had to give up on it, I could recover most of my investment.

Logo

One last note. Another of my artistic outlets is designing logos for things. I got inspired and made what I feel is an appealing logo for D:

It's hard coming up with new things to do with the letter D, and I don't claim that this is something entirely new under the sun. But it looks nice, doesn't it?