backward compat is good, bad

i read ian murdock's blog entry on the importance of backwards compatibility i found myself at once agreeing and wincing.

i agree that backwards compatibility is down right important, particularly for our users not to mention third party developers. the example ian used from joel's blog about sim city causing microsoft to patch their allocator so it behaved differently just for that one app (!!!) is so amazingly bad, though, that it's a bit unfortunate. a number of the comments on his blog got sidetracked by this poor choice of an example.

however, backwards compatibility, like most things in life, is neither all good or all bad. (dualism: blech!) with all the upside of it, there's also the drawbacks that come with it. namely increased engineering costs due to increased complexity of both development and testing, greater exposure to security risks and a real stunting of innovation since you can't just change things willy nilly.

we all want quickly developing, solid and innovative software ... but we also want backwards compatibility. how do we get both? that's a very good question.

this is an interesting conversation right now for kde as we're about to screw the backwards compat pooch with the release of 4.0 since there are so many api changes. for third party developers, the api changes will require changes to their source code. we've tried to limit the pain with documentation and tools but let's face it: kde4 isn't backwards compatible with kde3 and that's not the nicest thing if you are a third party developer.

fortunately, however, you should be able to run kde3 and kde4 apps side by side just as running kde1 apps alongside kde2 apps was possible. this addresses much of the issue for our users, at the cost of additional memory usage and disk space consumed.

so why do we do not just stay the conservative route? because we can provide a significantly better experience for both application developers and users.

at the same time we strive to provide binary compat throughout major revisions of our public API. we've messed up once or twice in the past on this, but generally we've done pretty well. so it's not like we don't care, we just see the both/and rather than only an either/or =)

that said, i think it would be really interesting for ian to point out some of the hot points of incompatibility that he sees and has to deal with. concrete examples are so much easier to work with than abstract philosophical meanderings.

for instance, g++'s incompatibilities year after year were insanely difficult to live with. that seems to be better now and, i'm hoping, all but solved so we won't have to deal with this nightmare in the future as we have in the past.

distros that swap bits and pieces about which results in unpredictable levels of quality and loss of knowledge investment for users are also maddening.

then again, in each of these scenarios there were often good reasons for taking the path that was taken. simply switching to a conservative methodology of backwards compatibility would address one set of issues and deliver us another.

perhaps we can find ways to evolve a system while keeping compatibility over time. you know, something in the middle. a middle way. =)