by Leroy on June 29th, 2023.
Current software, including desktop software, phone apps, and websites, have become bloated and full of errors. I think the cause for the now common slew of software problems is entirely in the hands of the corporate structure of software development. Public corporations are incentivized to publish more and more features to appease shareholders and investors. To accomplish this, corporations have structured themselves as fungible groups with individual autonomy rather than as a cohesive unit to produce software. The resulting software is connected piecemeal as a hodgepodge of features from disparate groups and then shipped as-is without consideration of the software as whole.
I have grown numb to the number of software errors I face on a daily basis. Desktop applications update automatically and no longer work when less than 24 hours prior they worked fine. I am forced to dig inside config files, or worse, delete them. I have to find caches and delete those too. I cannot revert to an older version or choose to stay at a particular version. With an ad-blocker, I open websites and am faced with constant pop-ups about cookies and sign-ups and warnings about my browser being out-of-date, but my browser updates automatically against my will. I expect that if I am on a corporate website doing something non-trivial, where non-trivial isn't explicitly the most glaringly obvious workflow, that something won't work. Even my operating system requires me to restart some system-level task almost daily. It's not easy to see how we've gotten here, but I want to lay out how I've come to understand the current state of software development.
Software development laid bare, with all of its vanities removed, is simply about writing text. There must be some logic and structure to enforce this text, of course, but it is mostly a very loose enforcement. The strictest enforcement of structure is the compiler itself which is, in most cases today, determined by the language of the software. Most other forms of enforcement are merely social, like specifications or agreements that code shouldn't be used if it doesn't pass basic automated testing.
All of this together means that most corporate software is really just an endeavor to write a collaborative text document where potentially hundreds, or thousands, of different authors add and edit text. It's a very hard problem to collaborate effectively with this many number of collaborators. But we attempt to do it anyway because there is a gross assumption made by many that software is somehow fungible-by-default: that text can simply be appended again and again to build up features of some software application over time.
Because of the constructive nature of software development, it's easy to see how the error of fungibility has manifested in corporate structures. The slippery slope toward the assumption of fungibility begins when one realizes that software can be broken down into concrete parts. It is taken as fact that an enlightened software architect can sit down and architect some software, from beginning to end, as a sum of some number of functions. All the corporation needs then is developers to write the text of those functions. Overlooking the glaring mistake which assumes that software can be broken into constituent parts before being written -- when in reality design must take place as it is being written -- none of this is really wrong. Software is literally a sum of functions.
The mistake is made when thinking about adding, changing, or otherwise creating new features. New features are the bread-and-butter, so to speak, of corporate software development even if no one wants the new features. This is a problem that starts at the executive level and seeps all the way down to individual contributor: everyone from the CEO to the project manager to the developer must validate their own worth to the corporation and new features are an easy sell.
Here is the mistake magnified: If software can be architected as 1000 different functions, then it is conceivable that a new feature could be created by appending another 100 functions. This append-only mode of software development also assumes that the latter 100 functions can make use of the existing 1000 functions without modifying them. This breaks down in practice in very obvious ways, but the inertia of separate groups within the corporation keeps the development happening despite the obvious problems.
I think an analogy would better illustrate my point: let's use a play written by Shakespeare as an example because a play is also merely text. The first error of software development is realizing that more developers does not necessarily yield better or quicker or more robust software. Imagine that one of Shakespeare's plays were not written by a single author, or even a collaboration of a few authors, but hundreds of different authors. This obviously breaks down at any kind of scale because the level of knowledge required to work in a codebase is dismissed in this kind of structure. Developers inevitably end up overwriting the work of others, repeating the work of others, and solving the same problem in multiple ways which leads to spaghetti code, or a play which makes no cohesive sense. This kind of analogy is nearly trite nowadays because this kind of software development methodology has been lambasted time and time again, but I just wanted to state a premise -- my analogy isn't finished.
Because the idea that a play, and software, doesn't get better with more and more authors is so pervasive, corporate structures have changed to protect themselves against this kind of chaotic development. Corporate structures have been designed to create teams or groups of developers who work independently on some feature of the software. Yes, more developers aren't going to solve a problem better, but, because it is taken as fact that software is inherently fungible, there's a consensus that the same amount of developers distributed into different groups can work on independent features without the same disastrous results. There's then a naivete that those features can then be summed together to create a final end-product.
Imagine now that a single play of Shakespeare were written not by hundreds of different authors, but ten different groups, each with 100 writters. Each group gets their own scene to write in the play. Because these groups are independent and often have competing interests, each scene of this single nightmarish play may have a different tone and perhaps none of the characters or drama line up between scenes. Characters that are killed in one scene may be resurrected in the next, magically, because the group in charge of that scene needed that character to be alive for whatever reason. Motifs laid out in one scene are missing entirely from the rest of the play so the motif loses all meaning and becomes a random and ultimately useless decoration. The overall story of the play has been lost because each group is focused only their own scene. At the end of some deadline, all of the scenes are joined together into one play. No one reads the entire play from beginning to end because each team only focuses on their scene.
Who ends up reading the play? The audience, of course, who has the unfortunate task of deciphering why the login button redirects their browser five times before landing on a page that complains that their browser is out-of-date despite the page amounting to nothing more than this:
<form action="/login" method="POST">
<input type="text" name="username">
<input type="password" name="password">
<input type="submit" value="Login">
</form>Unfortunately, this seems to be the exact scenario of modern software development today. The audience, the end-user, gets the grating experience of a discordant sum of scenes which barely connect with one another. The software, be it a website, phone, or desktop application, is never operated on as a whole by the corporation. The user actually is relied on as a crucial member of the development team -- they are the ones who see the crashes, the random errors, and the buttons which do nothing or go nowhere. Corporations rely on error reports from users because the corporation has lost the ability to craft an entire play. The corporation is structured as disparate teams which can only produce scenes, or features, and as a result can only see scenes and not a play.