10 April 2017

The Problem with Today’s Software Thought Leaders

Albert Einstein once said:  "What is right is not always popular and what is popular is not always right."   The current zeitgeist in software development has this exact same feel.  Our lives as software developers have been shaped by the ideas of many people.  Some of those ideas are right and some are popular.  In time the wrong ideas will fade and the right ideas will dominate.  The problem is: How do we make that determination in the present?  What are the good ideas and what are the bad ideas? 

As software developers our working days usually consist of two types of professional activities, one is obviously creating software (software development) or related technical tasks configuration management, troubleshooting, deployments, etc. and the other is being managed or other duties that relate to providing information (status) to managers (software development management).  The software industry is very young and constantly changing.  At any given moment there are dominant de facto standards that are followed for both software development and its management.  Although it varies across industry sectors, many development practices and tooling have become standards, things like IDE’s, continuous integration, the use of tests and testing frameworks, etc.  Some of these have been influenced by Agile and XP.   On the management side the standard seems to be largely dominated by Agile like scrum and associated practices from XP.

The people who dominate our industry with their ideas could be described as our thought leaders.   Wikipedia has the following definition for thought leader

A thought leader can refer to an individual or firm that is recognized as an authority in a specialized field and whose expertise is sought and often rewarded.

The creation of Agile was done in a ceremonious fashion, depicted in an odd ritualistic looking and now famous photograph, by signatories to a manifesto.   Let’s call these people the Agile Thought Leaders or just the Agilistas (since brevity is our thing).  The Agilistas have come to have their ideas dominate the industry and in many cases dictate how we now spend our days. 

It seems that Agile’s rise in popularity is directly proportional to my increasing skepticism about it.  I find myself internalizing that skepticism into frustration and even feelings of anger towards it and the Agilistas.  These feelings become exacerbated by the various incarnations of Agile created by legions of certified Agile consultants that roll into my life to tell me how to spend my days.

Now those feelings are not constructive and one really needs to step back and ask: Why is what I see in the industry wrong and how can I form and a constructive criticism of it? 

I was lucky to come across Greg Wilsons’s talk: "What We Actually Know About Software Development, and Why We Believe It's True".  This is a very enlightening talk which I highly recommend.  In it he looks objectively at how we have arrived where we are and how we can possibly better move forward.   I got the term Agilistas from his talk, although I don’t know if he coined it.

The talk touches on issues that I feel plague both my career and our industry. The problem manifests itself in how we make our decisions about what we do on projects both from the day to day and to the larger industry practices. Many of these decisions seem to be influenced all too often by people who claim some authority, sometimes it’s positional authority, sometimes it’s due to some form of gravitas and all too often right or wrong it boils down to someone’s opinion.  To better illustrate this about our thought leaders let's look at Greg Wilson’s critique of one of Martin Fowler’s publications about DSL’s (Domain Specific Languages).  The relevant quotes with time stamps are as follows:

11:26 [Martin Fowler] is a very deep and careful thinker about software and software design and last summer in IEEE Software he had an article on Domain Specific Languages, this notion that build a tiny language in which the solution to you original problem in easy to express. And in that he said using DSLs leads to improved programmer productivity and better communication with domain experts.

11:51 I want to show you what happened here. One of the smartest guys in our industry made two substantive claims of fact in an academic journal and there is not a single citation in the paper.  Not a single foot note or reference. There is no data to backup this claim. I am not saying that Martin is wrong in his claims. I am saying is what we have here is a Scottish verdict. Courts in Scotland are allowed to return one of three verdicts, guilty, innocent, or not proven.  Arguments in favor of DSLs, arguments in favor of functional programming languages making parallelism easier or Agile development leading to tenfold return on investment or whatever are unproven.  It doesn’t mean they are or wrong, it doesn’t mean they are right, it means that we don’t know and we should be humble enough to admit that.

12:38: Carrying on in that same article "Debate still continues about how valuable DSLs are in practice. I believe that debate is hampered because not enough people know how to develop DSLs effectively."   Crap I think the debate is hampered by low standards of proof.

12:55:  Listening computer scientists argue, listening to software engineers in industry argue, it seems that the accepted standard of proof is I’ve two beers and there’s this anecdote about a tribe in new guinea from one of Scott Berkin’s books that seems to be vaguely applicable therefore I’m right.  Well no, sorry that’s not proof and you should have higher standards than that.

I think these are excellent points. I disagree with Greg Wilson here and feel that Martin Fowler is not one of the smartest guys in our industry, although he maybe one of the most influential. I also question whether he can be considered a careful thinker.  I think his approach like many of the other Agilistas is self serving and shows very little humility or willingness to be self critical. This is something that Bertrand Meyer commented on when reaching out to the Agilistas in conjunction with his book on Agile! The Good, the Hype and the Ugly that almost all of the Agilistas treated him with "radio silence" after receiving his draft.  The problem is that the Agile business is just too lucrative for consulting and on the speaking circuit.  A great example of this business model is "Uncle Bob" Martin’s infomercial-esque site promoting his wisdom for sale. I refuse to link to it, so google it, if you want to see that spectacle.  All of these guys make a lot of money selling Agile and if they started being critical of it that would be a risk to their revenue streams.  Also how many of these guys still do significant software development work?  Ironically I think Uncle Bob still codes, but I am talking about opening up an IDE or text editor after checking out a big code base from version control and fixing bugs or doing actual coding or refactoring.  Now maybe that’s not fair to ask that, or is it?  After all they are driving how we spend our days being managed on software projects.

Another point of contention I have can be found on Martin Fowlers website under the FAQ section:

Why is your job title "Chief Scientist"?

It's a common title in the software world, often given to someone who has a public facing technical role. I enjoy the irony of it - after all I'm chief of nobody and don't do any science.

He openly admits he does no science (the black bolding is mine). Therein lays the problem with all of the Agilistas. There seems to be no scientific method involved in many if not all of their claims.  These claims as we have seen are sometimes made in "academic" journals without any citations. Of course I am skeptical about much that is written about software in journals published by ACM or IEEE and of the organizations themselves.  I think Greg Wilsons observation validates my skepticism and further draws attention to the lack of scientific method involved in software.  We need better science and people who are intellectually honest about the fact that they are just expressing unproven and in some cases unfounded opinions.

It seems to me a more conscientious approach would be for the Agilistas to promote and fund studies. Use the scientific method to back up their claims and help move the industry forward.  Whether they like it or not that is what will most likely eventually happen and they will either need to get on board or be left behind.

The irony here is that while we need more of a scientific method in defining software methodologies, much of our infrastructure comes from mathematics, another area that seems to be lacking with the Agilistas.  For example: Noam Chomsky’s formalization of grammars were adapted by Jim Backus and further refined algebraically by Donald Knuth to give us modern compiler theory and compilers.  There’s a long list of people who have laid down the mathematical foundations, people like Alan Turing, John Von Nueman, Alonzo Church, Stephen Cole Kleene, Haskell Curry and the list goes on.  Unfortunately I think too many people in our industry are ignorant of our field’s true mathematical underpinnings. I have also encountered a large degree of anti-intellectualism from programmers when it comes to applying modern math to programming.  Again I think this is an area where one should only ignore it at their own peril.

If you want to see the work that should and probably will drive our future I would recommend looking at what the researchers are doing, people like:

These are just a few. There are many people doing research that will eventually find its way into our industry. It has been happening this way all along.  The problem with these people’s work is it requires real intellectual effort as opposed to the Agile approach which, while in some cases maybe common sense, all to often seems to come across as a bunch of platitudinous management speak.

I think we need to hold the Agilistas to higher standards of proof!

Disclaimer: I still code personal projects but have not been on a dev team doing real dev work in about 3-4 years after doing it for over 25 years.  Actually from what I have seen of Agile management techniques and war room/open office layouts, I am in no hurry to do any of that any time soon.

References and Further Reading