The Problem with Today’s Software Thought Leaders

Albert Einstein once said:  "What is right is not always popular and what is popular is not always right."   The current zeitgeist in software development has this exact same feel.  Our lives as software developers have been shaped by the ideas of many people.  Some of those ideas are right and some are popular.  In time the wrong ideas will fade and the right ideas will dominate.  The problem is: How do we make that determination in the present?  What are the good ideas and what are the bad ideas? 

As software developers our working days usually consist of two types of professional activities, one is obviously creating software (software development) or related technical tasks configuration management, troubleshooting, deployments, etc. and the other is being managed or other duties that relate to providing information (status) to managers (software development management).  The software industry is very young and constantly changing.  At any given moment there are dominant de facto standards that are followed for both software development and its management.  Although it varies across industry sectors, many development practices and tooling have become standards, things like IDE’s, continuous integration, the use of tests and testing frameworks, etc.  Some of these have been influenced by Agile and XP.   On the management side the standard seems to be largely dominated by Agile like scrum and associated practices from XP.

The people who dominate our industry with their ideas could be described as our thought leaders.   Wikipedia has the following definition for thought leader: 

A thought leader can refer to an individual or firm that is recognized as an authority in a specialized field and whose expertise is sought and often rewarded.

The creation of Agile was done in a ceremonious fashion, depicted in an odd ritualistic looking and now famous photograph, by signatories to a manifesto.   Let’s call these people the Agile Thought Leaders or just the Agilistas (since brevity is our thing).  The Agilistas have come to have their ideas dominate the industry and in many cases dictate how we now spend our days. 

It seems that Agile’s rise in popularity is directly proportional to my increasing skepticism about it.  I find myself internalizing that skepticism into frustration and even feelings of anger towards it and the Agilistas.  These feelings become exacerbated by the various incarnations of Agile created by legions of certified Agile consultants that roll into my life to tell me how to spend my days.

Now those feelings are not constructive and one really needs to step back and ask: Why is what I see in the industry wrong and how can I form and a constructive criticism of it? 

I was lucky to come across Greg Wilsons’s talk: "What We Actually Know About Software Development, and Why We Believe It's True".  This is a very enlightening talk which I highly recommend.  In it he looks objectively at how we have arrived where we are and how we can possibly better move forward.   I got the term Agilistas from his talk, although I don’t know if he coined it.

The talk touches on issues that I feel plague both my career and our industry. The problem manifests itself in how we make our decisions about what we do on projects both from the day to day and to the larger industry practices. Many of these decisions seem to be influenced all too often by people who claim some authority, sometimes it’s positional authority, sometimes it’s due to some form of gravitas and all too often right or wrong it boils down to someone’s opinion.  To better illustrate this about our thought leaders let's look at Greg Wilson’s critique of one of Martin Fowler’s publications about DSL’s (Domain Specific Languages).  The relevant quotes with time stamps are as follows:

11:26 [Martin Fowler] is a very deep and careful thinker about software and software design and last summer in IEEE Software he had an article on Domain Specific Languages, this notion that build a tiny language in which the solution to you original problem in easy to express. And in that he said using DSLs leads to improved programmer productivity and better communication with domain experts.

11:51 I want to show you what happened here. One of the smartest guys in our industry made two substantive claims of fact in an academic journal and there is not a single citation in the paper.  Not a single foot note or reference. There is no data to backup this claim. I am not saying that Martin is wrong in his claims. I am saying is what we have here is a Scottish verdict. Courts in Scotland are allowed to return one of three verdicts, guilty, innocent, or not proven.  Arguments in favor of DSLs, arguments in favor of functional programming languages making parallelism easier or Agile development leading to tenfold return on investment or whatever are unproven.  It doesn’t mean they are or wrong, it doesn’t mean they are right, it means that we don’t know and we should be humble enough to admit that.

12:38: Carrying on in that same article "Debate still continues about how valuable DSLs are in practice. I believe that debate is hampered because not enough people know how to develop DSLs effectively."   Crap I think the debate is hampered by low standards of proof.

12:55:  Listening computer scientists argue, listening to software engineers in industry argue, it seems that the accepted standard of proof is I’ve two beers and there’s this anecdote about a tribe in new guinea from one of Scott Berkin’s books that seems to be vaguely applicable therefore I’m right.  Well no, sorry that’s not proof and you should have higher standards than that.

I think these are excellent points. I disagree with Greg Wilson here and feel that Martin Fowler is not one of the smartest guys in our industry, although he maybe one of the most influential. I also question whether he can be considered a careful thinker.  I think his approach like many of the other Agilistas is self serving and shows very little humility or willingness to be self critical. This is something that Bertrand Meyer commented on when reaching out to the Agilistas in conjunction with his book on Agile! The Good, the Hype and the Ugly that almost all of the Agilistas treated him with "radio silence" after receiving his draft.  The problem is that the Agile business is just too lucrative for consulting and on the speaking circuit.  A great example of this business model is "Uncle Bob" Martin’s infomercial-esque site promoting his wisdom for sale. I refuse to link to it, so google it, if you want to see that spectacle.  All of these guys make a lot of money selling Agile and if they started being critical of it that would be a risk to their revenue streams.  Also how many of these guys still do significant software development work?  Ironically I think Uncle Bob still codes, but I am talking about opening up an IDE or text editor after checking out a big code base from version control and fixing bugs or doing actual coding or refactoring.  Now maybe that’s not fair to ask that, or is it?  After all they are driving how we spend our days being managed on software projects.

Another point of contention I have can be found on Martin Fowlers website under the FAQ section:

Why is your job title "Chief Scientist"?

It's a common title in the software world, often given to someone who has a public facing technical role. I enjoy the irony of it - after all I'm chief of nobody and don't do any science.

He openly admits he does no science (the black bolding is mine). Therein lays the problem with all of the Agilistas. There seems to be no scientific method involved in many if not all of their claims.  These claims as we have seen are sometimes made in "academic" journals without any citations. Of course I am skeptical about much that is written about software in journals published by ACM or IEEE and of the organizations themselves.  I think Greg Wilsons observation validates my skepticism and further draws attention to the lack of scientific method involved in software.  We need better science and people who are intellectually honest about the fact that they are just expressing unproven and in some cases unfounded opinions.

It seems to me a more conscientious approach would be for the Agilistas to promote and fund studies. Use the scientific method to back up their claims and help move the industry forward.  Whether they like it or not that is what will most likely eventually happen and they will either need to get on board or be left behind.

The irony here is that while we need more of a scientific method in defining software methodologies, much of our infrastructure comes from mathematics, another area that seems to be lacking with the Agilistas.  For example: Noam Chomsky’s formalization of grammars were adapted by Jim Backus and further refined algebraically by Donald Knuth to give us modern compiler theory and compilers.  There’s a long list of people who have laid down the mathematical foundations, people like Alan Turing, John Von Nueman, Alonzo Church, Stephen Cole Kleene, Haskell Curry and the list goes on.  Unfortunately I think too many people in our industry are ignorant of our field’s true mathematical underpinnings. I have also encountered a large degree of anti-intellectualism from programmers when it comes to applying modern math to programming.  Again I think this is an area where one should only ignore it at their own peril.

If you want to see the work that should and probably will drive our future I would recommend looking at what the researchers are doing, people like:

• Roland Backhouse
• Richard Bird
• Jeremy Gibbons
• Ralf Hinze
• Donald E. Knuth
• Lambert Meertens
• Erik Meijer
• Martin Odersky

These are just a few. There are many people doing research that will eventually find its way into our industry. It has been happening this way all along.  The problem with these people’s work is it requires real intellectual effort as opposed to the Agile approach which, while in some cases maybe common sense, all to often seems to come across as a bunch of platitudinous management speak.

I think we need to hold the Agilistas to higher standards of proof!

Disclaimer: I still code personal projects but have not been on a dev team doing real dev work in about 3-4 years after doing it for over 25 years.  Actually from what I have seen of Agile management techniques and war room/open office layouts, I am in no hurry to do any of that any time soon.

References and Further Reading

• 5 minute book review: Facts and Fallacies of Software Engineering
• A Pedagogical Framework for Domain-Specific Languages(paywalled, of course)
• Agile Bullshit: Agile Thought-Leaders Know It All
• Agile Manifesto
• "An Experiment on Unit Increase in Problem Complexity" Woodfield 1979
• Burden of Proof
• Challenging the fundamental assumptions of DDD
• CUSEC 2010 Keynote: Greg Wilson – "Bits of Evidence"
• Greg Wilson’s What We Actually Know About Software Development
• The Leprechauns of Software Engineering
• The Leprechauns of Software Engineering (reddit)
• Lutz Prechelt's Homepage
• No Silver Bullet — Essence and Accidents of Software Engineering (1986)

Software Development is a Risky Business

Software projects are inherently risky.  Some statistics show that roughly seventy percent to fifty percent of software projects get classified as “challenged”.   Where challenged can mean cost overruns and/or delivery delays, or it can mean complete failure and the cancellation of the project.  Some statistics indicate as the size of the project increases so does the probability of problems and failure.  Many articles and blog posts have been written about the long history of software project failure.  At the time of writing this post the most recent and very public software project debacle was the healthcare.gov project.  It seems as IT becomes more pervasive and essential the problems with projects may be getting worse not better.

In spite of these numbers and all too often in my experience I have found that individuals and organizations seem to have a very optimistic outlook about their own projects especially at the beginning of the project.  Perhaps it’s the classic: “that won’t happen to me” attitude.  Even as projects start to run into problems I have seen people especially managers not take issues seriously or ignore them all together.  In some of the analysis of healthcare.gov, this was cited as one of the problems.

Risk management and risk assessment/analysis are developed areas in project management and engineering and are often discussed in software projects.  Many risks will end up falling into two categories.  One category is actual risks to the end product, such as unforeseen security vulnerabilities in software systems or other engineering issues that lead to flaws or critical failures of the system or components. The other is risks in the management and execution of constructing product or system. 

Risk management is one of those areas where the engineering aspects and the engineering management aspects can become tightly coupled.  An example can be derived from the XKCD comic “Tasks”: https://xkcd.com/1425/.  Now the point of this comic is to demonstrate the difference between an easy problem and hard one, and presumably the conversation takes place between a manager and developer.  In a risk setting, perhaps solving the hard problem is the goal of the project.  So a risk might be that the research team fails to solve that problem at the end of the five years, or that it takes ten. So in a sense the task is high risk but there are team related risks, maybe the budget is too low and the team is subpar due to low salaries so they can never even solve it.  Another possible scenario is that the conversation never happened and the project management and team think it is solvable in six months.  This second scenario is the risk I have seen though out most of my career, the project management and team are overly optimistic or really don’t have a good understanding of the complexity of the task at hand.  In some cases these types of issues seem to be due in part to pluralistic ignorance.  Often in cases where developers don’t want to contradict management.  There’s a great dramatization of this type of behavior in the show Silicon Valley when the Hooli developers are working on the Nucleus project, where the information is conveyed up the management chain but nobody wants to tell Gavin the truth.

Sadly it seems like the risks that are identified are almost peripheral to the real risks on projects.  It is usually things like the hardware delivery will be delayed which will delay setting up our dev/test/prod environments and delay the start of development.  While these are legitimate risks, there is always the risk of all of the unknowns that will cause delays and cost over runs, even things like developer turnover never seem to be taken into account, at least in my experience.  Also there seems to no cognizance of the long term cost risks like the project was poorly managed or rushed or was done on the cheap which caused the delivered codebase to be an unmaintainable big ball of mud that will incur a much higher long term costs.

In order for software engineering to grow up and become a real engineering discipline it will need formal risk assessment and risk management methodologies both on the project management side and on the work product side.  This is also an area where ideas and methodologies can potentially be taken from existing work in other engineering disciplines.  These methodologies will also most likely draw from probability and game theory.

References and Further Reading

• Officials Were Warned About Health Site Woes
• Red Flags: How Politics and Poor Management Led to the Meltdown of HealthCare.gov
• Senate details HealthCare.gov crash
• HealthCare.gov Diagnosis: The Government Broke Every Rule Of Project Management
• HealthCare.gov's fatal mistake
• The Staggering Impact of IT Systems Gone Wrong
• Why Do Projects Fail?
• Gartner Survey Shows Why Projects Fail
• Study: 68 percent of IT projects fail
• The Non-Existent Software Crisis: Debunking the Chaos Report
• Why Software Projects Fail and How to Make Them Succeed
• The Standish Group Report CHAOS Report
• The scariest software project horror stories of 2012
• 2013 IT Project Success Rates Survey Results
• 14 Surprising Project Management Statistics
• FAA’s Modernization Program: Clear Skies, or Still In Heavy Turbulence?
• Anatomy of an IT disaster: How the FBI blew it
• Why Software Fails
• The Long, Dismal History of Software Project Failure
• CIO analysis: Why 37 percent of projects fail: New research identifies five important reasons that projects fail
• IT Project Failure Rates: Facts and Reasons

Why You Can’t Have a Real Software Engineering Discipline

As we all know the fields of computer science and software engineering are in their infancies.  Many a blog post has been written lamenting the fact that software engineering is not a real engineering discipline, and while I have not written that exact post, I have written about the subject and have deconstructed what others have said about it.  A number of these points do apply to why software projects routinely fail, yet another topic that has received considerable attention.  Now admittedly all engineering disciplines regardless of their maturity and formalization have project failures and creating a real software engineering discipline will not eliminate this problem but one would hope that it would abate it.

I find it odd that at a time when so many people are involved in IT there seems to be little discernible progress in creating a real software engineering disciple.  There are probably millions of people working on creating software.  Also there are many researchers trying to solve this problem from many different angles.  Practitioners have been attempting to solve it with ideas like Agile and Software Craftsmanship, etc.  Additionally there is a long list of failures surrounding the formalization software engineering.  

My biggest complaint is the fact that there really are no good formal standards in general software engineering principles and methodologies lack a formal foundation and even the more vague principles can be easily thwarted and misused, and often, as I have previously complained, it often ends up being based on pure opinion which is usually won through positional authority, perseverance or by those who are just more politically savvy.

The intent of this post is not to attempt to solve or even offer solutions to the problem of creating a real software engineering discipline but to look at what I think are some barriers to creating this discipline and in some of these cases offer thoughts on overcoming those barriers.

Low Entry Requirements

If we are to think software engineering as an engineering discipline, I challenge you to find another engineering discipline that routinely has practitioners with no formal training in the field.  I very much doubt that someone who did not hold a degree in engineering and who built a shed in his back yard is now working as a structural engineer on a construction project, the mere idea seems ludicrous.  Yet I have worked with non technical degree holders who became software developers, one who started by building web pages and now calls himself an "Internet Application Architect", and is probably one of the biggest cargo cult coders I’ve ever had the misfortune to work with.  This egalitarian aspect isn’t all bad as it does let good people into the field as well.   Another non technical degree holder I once worked with became an accomplished developer and went on to get an advanced degree in CS and is now a CS professor.   Although it’s probably the case that for every good non technical degree holder who joins the field it is likely that dozens of mediocre and bad practitioners will also join.   It seems that in other engineering disciplines there are much more stringent educational requirements to ensure proper training.   To be clear here I am not equating having a degree with being competent because I have met many incompetent people with degrees, but still you need something.   I confess I am at a loss for a solution for this one.

The Lack of Differentiation between Science and Engineering

Chemistry is a scientific discipline, chemical engineering is an engineering discipline, and you can make this comparison between other engineering disciplines and the sciences that they employ e.g. electrical, mechanical, and structural map to various areas of physics.  Engineering and scientific disciplines are different types of disciplines often taught in separate schools.  Although each engineering discipline has some scientific overlap and it would probably be possible to move from the appropriate scientific field to the corresponding engineering field in general you probably would not do so without returning to school.   In recent years software engineering curricula have been added to the rosters of many colleges and while this is potentially a step forward, discounting for the moment that software engineering is still not really real engineering, a software engineering degree and a computer science degree in many cases gets you the same job!   

So in other fields there would be a differentiation between a degree that would put you on a track to do scientific research and one that would put you on a track to do engineering work.  As I understand it, if you are fresh out of school with a BS in CS that qualifies you to be a tester at Microsoft¹, developers hired right out of school need at least a Masters degree in CS.   This is an extreme case of how this really breaks down in our industry.  In most cases engineering practitioners are the software engineering, MIS, CS or even non technical degrees and the few scientific careers generally go to the advanced degree holders.  I admit this probably fairly normal a BS in chemistry or biology is also more likely to get you a job in IT than a job as a research scientist.  Still compared to more mature engineering fields there seems to be a lack of real differentiation between the degrees that yield a career as a software engineer versus one as a computer scientist.

Bad Management

The entry requirements for managers in software make the low entry requirements for software engineering practitioners look downright rigorous.  I have previously criticized the fact that many organizations find the cheapest people, especially for software management and give them an obligatory certification.  Now I know that engineering management regardless how formal or established the engineering discipline is an area that has problems and failures, but again I very much doubt that you would find a twenty something business or liberal arts major with a freshly minted PMP suffix managing construction or civil works projects.   Yet in my experience this is pretty normal in IT especially in the government sector.  Of course bad software management is executed by older managers as well.

A post by Larry White titled "Engineering Management Is Dying" delves into some these issues.  It is definitely the case that methodologies like Agile have changed how software projects work and the traditional corporate management approach to software really doesn’t work.  One point he makes is that it is not uncommon to see a two hundred person project at Google headed by an engineer.  To me this implies that someone is in some way managing that project.  I think all of this is indicative of the need for software engineering mangers to also be practitioners in engineering or at least very well versed in how software development works.  The situation he talks about at Google is not the case everywhere, in that case the company is its own client, but there are many cases where software is being built for clients and this does create a need for management that deals with the client, perhaps this should be a role that is separated from management and called a client liaison, which is what often happens.  Also I am not sure how things work at Google but I have also seen internal development in organizations where the IT department provides services to an internal client, I have also seen this go horribly awry with contentious unproductive relationships between departments.  In short I feel that just as we need a real software engineering discipline we also need a real incarnation of software engineering management.

The Disconnect between Academia and Industry

Most practitioners do not keep up with, or read academic research, actually many practitioners don’t read at all, but that’s another issue, and most academicians don’t work in the field so they lack the hands on knowledge.  Unfortunately this rift can take on a somewhat disdainful tone as is the case in my exploration of one practitioner’s attempt to define "Real Engineering".   Fortunately some people, I’d like to count myself among them, take a more constructive approach to building bridges across this rift.  Daniel Lemire has an interesting critique the quality of software produced in academia. 

I think the solution here is for both sides to become more engaged in problems faced on each side. I am optimistic about this one as I feel that these walls are breaking down as the field is growing up and many of the emerging technologies are forcing developers to be more cognizant of and engaged in research topics.  I also have encountered much more research that has ties to the practical concerns of day to day software development, some I have mentioned with more to come.

The Lack of an Effective CS Math Curriculum 

If you read my blog you know it to be a mix of math and software practices. I would describe myself as a software practitioner and a math enthusiast.  My math journey has taken me on some interesting math excursions into areas of math that seem to get little or no mention in CS curricula and I feel that this is a major problem in really applying math to the field of software engineering.  Another problem is in the way that it is taught, my experience was that it was not only taught badly but in a way that made it seem irrelevant to many of the programming courses.  Specifically I would shift the CS math curriculum to include less differential and integral calculus and include more logic, combinatorics, abstract algebra, graph theory, order theory, category theory and probably some topology among others.

Over the last few years I have been progressively learning more math which has been in part motivated by necessity for understanding research papers.  This approach has changed the way I see things, I now see the patterns of math in software. They are there and I believe that they can be exploited to create a real software engineering discipline.

For the math learning problem I do have some ideas many of which are expressed in my blog.  Some of my posts like refactoring if statements with Demorgan’s Laws or the String Monoid, the Object Graph, etc. are about making these mathematical ideas more relevant and accessible to programmers. Other posts like my series on naming and my post on generic programming and entropy and complexity are about my own ideas and ideas in the research literature that I feel may help to create a real software engineering discipline.  These are my attempts at solutions to these problems, so expect more of both of these and more thoughts on the CS math curriculum.

The Software Architect Debacle

This is one that really bothers me.  I feel the software architect role in general has a very negative effect on creating quality software and it dissuades the industry from developing a real engineering discipline.  The Software Architect role tends to be broadly defined, you have enterprise architects, software architects, etc., some architects tend to be involved with hardware and networking, some work on configuration management, some do software design, some do all of these things and more.  I feel this is a problem as each of these is a separate engineering area with different types of problems. By not breaking these down it often leads to a lack of focus on specific problems, also I have seen cases where the architect focuses on their preference and ignores other issues.  The architect role is often divorced from the code.  I have met many architects that were responsible for software but had never even looked at the code.  To me this is a huge failing that architects that are responsible for building software often lack the interest, time, or even aptitude to know the quality or underlying of structure of the software that they are delivering.

To really do justice to these ideas I might need a separate post, my solution would be to break down the architect role into specific engineering roles, which could include:

• Software Process Engineering - this would be involved with software team and resources planning, some aspects of configuration management such as defining policies, requirements analysis and general project management aspects of software construction including the definition and refinement of the SDLC.  This role is tightly coupled with software engineering management.
• Software Structural Engineering - this would include requirements comprehension, code infrastructure planning, prototyping work, hands on code structural work including custom frameworks and reusable components, third party library products, and general code quality including reviews and static analysis.
• Software Quality Engineering - This would include the QA roles, software testing and testing tools, requirements validation and refinement, usability and reliability and general software quality issues.
• Software Infrastructure Engineering - This would include hardware, networking, database, app server and general service infrastructure, non functional requirements fulfillment and possibly the implementation of configuration management, continuous integration, etc.

This is a rough "sketch" of these possible roles and each of these areas has some overlap implying that all of the people in these roles would work closely together, also some combination of, or all of these roles might be performed by a single individual depending on the organization’s size and structure.  What this approach does is clearly defines roles and responsibilities as opposed to some nebulous software architect role.

The Continuous Turnover of the Work Force

A young CEO once commented that he thinks young programmers are superior.  In general companies prefer younger workers as they often don’t have family commitments so they can work more hours and you can pay them lower salaries, DC area government contractors rely on this to keep their profit margins up.

As an older developer I often find this frustrating.  I confess that I have not moved up the ladder and still mostly work as a developer or what might be called a hands on architect, yes I know but that’s the current term, actually my preferred title would be: Software Structural Engineer.  Working as a contractor I have on several occasions found myself on projects that were dominated by younger developers and unfortunately on more than one occasion I watched as teams would make many mistakes due to a lack of experience, in some cases I was able to help in others I was ignored by the younger developers.  This is not to say that I do not still make mistakes, I’ve just been around long enough to have made a lot of them already.

If you buy into the software craftsmanship thing, I admit to being skeptical of this idea, you might be inclined to draw a parallel to craftsman of the past when older established masters took on apprentices and passed on their wisdom to the next generation.  Given the access to information these days such process is somewhat antiquated and perhaps impractical.  Nevertheless I feel, and the industry supports me here, that older developers who keep their skills up to date have value.  I know several younger developers who have sought me out to learn from me so I think I can safely say I still have some value.

Many older developers have let their skills get rusty and have not kept up on current technologies I have seen this many times.  Also many companies seem to lack the vision to allow for real hands on technical career growth.  I can’t help but feel that this inclination to recycle developers in each new generation is hurting or at least slowing our ability to develop a real software engineering discipline, this potential loss of continuity seems like it leads to some lost wisdom.

The Social Networking Drain and Hollywoodification of Silicon Valley

As I write this Hollywood gears up to deliver a reality TV show that takes place in silicon valley and Mark Zuckerberg now gets the paparazzi treatment.  Another article that I previously mentioned was about how CS enrollment was up because the movie The Social Network had enticed many aspiring Zuckerberg wannabes.  At present there does seem to be a gold rush mentality and it is noticeable and even discussed on sites like Hacker News. 

As someone who has never worked in the valley I am very much an outsider and may not have a good perspective on these things, but it seems that there are a lot of startup companies focusing on crap. Now I get it.  We live in a shallow consumer society with a rapacious appetite for crap. But are not these supposed to be some of the smartest people and I am not the only one to wonder why so many smart people who should have better taste and higher standards are settling for working on serving up ads and other vapid crap instead of doing more meaningful work like working on real problems that would advance human society or even just work on advancing software engineering. 

¹I believe this was the case in the late 90’s and may have changed, this information was conveyed to me by a former Microsoft employee.

Real Software Engineering: The Good, The Bad, and The Ugly

Deconstructing Glenn VanderBurg’s "Real Software Engineering"

Many years ago when I started my career, fresh out of school with a Bachelors in Computer Science, I chose to answer that ubiquitous question "What do you do?" with the title of "Software Engineer". Now you must realize that this was back at a time when any technical title including Programmer was as rare and mysterious as any engineering field to the average person who probably had never used a computer, not like today when everyone seems to have one in their hand at all times. When I was younger I always felt weird calling myself a programmer and I had a number of associations with other engineering disciplines, including starting my college studies in Chemical Engineering. Actually I was a chemistry nerd before I was a computer nerd, which you might have guessed as much if you read my previous post on Software Engineering. I also felt weird and perhaps even a bit phony using the title Software Engineer, I still do, so now I mostly go by Software Developer.

My original foray into the contemplation of Software Engineering lead me to do some more research which yielded an enlightening and informative presentation by Glenn Vanderburg called "Real Software Engineering", PDF Slides here, which attempts to explore ideas of what a real discipline of Software Engineering might be.  I strongly recommend watching it. It’s about 52 minutes long. I feel that his presentation has some very good points but it also has some flaws and biases.

The good

He starts with a critique about the state of software engineering today, which includes the following points:

Doesn’t reliably control costs.

Doesn’t reliably produce quality software.

Sometimes doesn’t produce software at all.

Software Engineering doesn’t work.

The term engineering is reserved for practices that work, a good capsule definition of engineering independent of any particular discipline.

The set of practices and techniques that have been determined to work reliably through experience.

Software engineering is a caricature of engineering.

In software we have these practices that don’t work and that is called engineering

I find myself agreeing with these, and sadly Software Engineering does seem to be a ridiculous farce that often seems to resemble a real engineering practice only minimally.

He then goes on to talk about the NATO Software Engineering Conference in 1968 which he cites as the origin of the waterfall process as a misinterpretation of a document called:
MANAGING THE DEVELOPMENT OF LARGE SOFTWARE SYSTEMS by Dr. Winston W. Royce who it seems inadvertently started the waterfall process through what might be called "Poor Information Design".

He continues with a quote from Parness about the document oriented path into engineering captured by the following quote:

In Enginering ... people design through documents.

-Parness

This is followed by a list that might be considered failed attempts at such an approach:

• CASE Tools
• OOSE
• VDM
• UML
• MDA
• OMT
• Booch

He refers to this as:

Modelling and what I call ‘Math Envy’

While I agree with this last statement in practice, I feel that he goes too far with this opinion, more on that later.

His presentation provides a good historical survey of Software Engineering with respect to the 1968 NATO Software Engineering Conference and is worth watching for that alone. He continues with some great observations about engineering in general:

Engineering is a creative activity it’s about designing and building and creating new things and there are always blind alleys and missteps and mistakes and discoveries and reactions to those reactions and adjustments.

I especially like the trial and error aspect in relation to creativity. I think this is a fundamentally important and potentially overlooked aspect of software development. 

The following succinctly sums up what might be the kernel of general engineering:

Different Engineering Disciplines are Different

• Different materials, physical effects, forces
• Different degrees of complexity in requirements, designs, processes, and artifacts
• Varied reliance on formal modeling and analysis vs. experimentation, prototyping, and testing
• Varied use of defined and empirical processes

This leads to a comparison between the empirical vs. formal predefined models:

The defined process control model requires that every piece of work be completely understood. A defined process can be started and allowed to run until completion, with the same results every time.

The empirical process control model provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs.

He then takes the following ideas from Structural Engineering:

Structural engineering is the science and art of designing and making, with economy and elegance, […] structures so that they can safely resist the forces to which they may be subjected.

—Structural Engineer’s Association

Science and art, it employs the findings of science and yet is done with creativity "designing and making" engineers don’t sketch something on paper and validate it with math and throw it over the wall and never look at it again they participate in the building of what they design structural engineers have hard hats in their office because they visit the site and talk to the construction people and work with them during the process. "With economy and elegance" cost is always an object.

From these ideas he derives a brief description of Software Engineering:

Software engineering is the science and art of designing and making, with economy and elegance, Systems so that they can readily adapt to situations which they may be subjected.

With the following caveat:

Software engineering will be different from other kinds of engineering.

We know that software engineering will be different from other kinds of engineering we know that because other kinds of engineering are different from each other. Software systems are usually more complex in terms of number of moving parts and things like that than artifacts in other engineering disciplines we’re not working with physical materials we have different laws we work with and it is rare for a software project to run up against the limitations of physical materials, very rare.

He reiterates this from the Structural Engineering piece:

Cost is always an object.

This is more eloquently stated by Arthur Mellen Wellington:

Engineering is not the art of constructing. It is rather the art of not constructing: or, it is the art of doing well with one dollar what any bungler can do with two.

—Arthur Mellen Wellington

The following are elements that in part drive the agile philosophy:

The source code is the model and the document.

Source code is the detailed design

The design and construction of software is tightly coupled.

He draws the following inferences from the above:

Encourage continued innovations in processes

And do like other engineering disciplines do. Take the processes that work and call those engineering.

Agile software development, far from being anti-engineering, represents the responsible application of engineering ideals to the field of software.

I know that was a lot of quotes with sparse commentary, but it seemed like the best way to capture the really good points that he makes in his talk and again I am trying to create a written artifact that I can reference and build upon in the future.

The Bad

One thing that I really don’t like about his presentation is the comparison of the two bridge builders: Robert Maillart who designed the Valtschielbach Bridge in Switzerland among others and Leon Moisseiff.

The distinction being that Maillart used testing and all but one of his bridges are still standing while Moisseiff used modeling most notably Deflection Theory and was responsible for designing the ill fated Tacoma Narrows Bridge aka Galloping Gertie possibly one of the most famous engineering disasters in history. Moisseiff was also involved in the design of Manhattan Bridge which is still standing and his Deflection Theory was used in the design of the Golden Gate Bridge and was later determined to be of sound design and is also still standing. The Frommers travel guide considers the Golden Gate Bridge "possibly the most beautiful, certainly the most photographed bridge in the world", what bridge Maillart design?

I will not dispute that the collapse of the Tacoma Narrows Bridge was probably one of the most famous if not the most famous engineering disaster of all time and Moisseiff certainly suffered as a result, his career was shattered and he died shortly thereafter. Wikipedia states: "Though he [Moisseiff] had designed many other famous spans, the Narrows collapse overshadowed them all. It became a symbol of failed engineering and the dangers of arrogance in design." I am in no position nor do I care to debate the soundness of deflection theory or the reasons for the collapse, actually it seems that reasons for the collapse are pretty complicated.

From my limited understanding of Civil Engineering it seems that problems with wind are a continuous problem for Civil and Structural Engineers, I know that the John Hancock Tower in Boston was plagued by problems where wind induced twisting which caused large plate glass windows to careen hundreds of feet to the street making it a deadly hazard.  These were replaced with plywood earning it nicknames like the "Plywood Palace" and causing it to be condemned as a fire hazard, eventually these problems were resolved and it won awards for enhancing the Boston skyline. More recently, only twelve years ago in 2000 the Millennium Bridge in London exhibited similar resonance problems to those of Galloping Gertie, it was shut down and reinforced.  So while all of this makes for interesting history of engineering fodder and good showmanship, I am not sure how directly relevant it is to software engineering. I Again I agree with his assertion that testing, perhaps not necessarily TDD in my opinion, is most likely a very important component in creating a real software engineering discipline but I feel this particular comparison argument is flawed as it is completely irrelevant to software engineering and in no way can be used to conclude that TDD should be used in software engineering. Also he contradicts his own argument with the statement, from above:

Software engineering will be different from other kinds of engineering.

He also contradicts this in his slides:

Software is very unlike bridges and buildings.

He ends his presentation abruptly with:

Today software engineering is called Agile.

I hate to say it but the way he states it, it sounds a bit like "Mission Accomplished" to me and I may be misinterpreting him here but to me it sounds like his whole presentation is designed to conclude that Agile and TDD are the solution to the problem and I agree that they are probably the best solutions to the problem today, but I think they are severely lacking as an effective and precise framework for a real engineering discipline, I think we still need more cowbell.

The Ugly

I feel that his presentation has anti-math and anti-academic undertones and I have to say: "No Sir I don’t like it." The two main themes that I take issue with are:

• He mostly dismisses the work of non practitioner academics.
• He dismisses Math and views it as only a cost saving tool.

At various points He states the following:

Advances come from practitioners not from academia.

"Advances come from practitioners." And then are brought into academia.

Important advances often from practitioners not necessarily from academia and they are brought into the academic world and are refined and so forth. But they come from practitioners.

They were practitioners … and there were a few academics, but the academics had good things to say as well.

Perhaps I am misinterpreting his tone here, but the in the last statement he seems surprised that the academics had something of value to contribute. Actually I feel that he has this somewhat backwards, for example I think the work that companies like Google are doing and the general trend in analytics fundamentally contradict this sentiment. I think the way that both the fields of Computer Science and Software Engineering will inevitably move forward in both efficiently and efficacy will be through more collaboration between practitioners and academicians with both groups refining each other’s ideas. I anticipate and hope this is a trend that we will see more of. In fact some really interesting things are happening in that regard one interesting example is the work that Mike McCandless has done with Lucene using Finite State Transducers which was based on academic work, not to mention the Entropy-Framework relationship that we have already looked at.

He goes on to say:

Mathematical Models were introduced to engineering as a cost saving tool it would be easy if you hear software engineering people talk about math and modeling, it would easy to get the idea that it’s the only way to do things right and that it is a tool for robustness and you don’t know it really works that it is really works unless have math to prove it, I’ve heard all of those statements. But, that’s not why it was introduced to engineering at all it was introduced to save cost because in other engineering fields they are working with real materials and things that require people and labor to go out and build and building prototypes and testing them especially at scale is extremely costly and if you can build a model, an approximation of reality you can save money.

Ask your friendly neighborhood aerospace engineer how much math he would do if building a model of his design and testing it were was effectively instantaneous and free. The answer would probably not be none, but a lot less than I do now.

I am not sure if I agree with the idea, that seems to be implied, which is that mathematical models are strictly a cost saving measure.

Towards the end he states the following are approaches needed to advance software engineering:

Not Math

Not Models

Not Documents

Not copying other disciplines

Learn From practioners

Bias towards empirical processes

Ironically he mentions the following:

Programming languages are formal languages some of them even have mathematically specified semantics ... and we understand the math behind them at least as well as your average structural engineer understands the math behind the procedures that he was taught to do to validate his designs.

Programs themselves are models of the solution to the problem, our very tools are mathematical in nature.

In the slides he has the following quote, which I think he skips in his presentation due to time constraints, but I believe it is intended to reinforce his anti math-argument:

What is needed is not classical mathematics, but mathematics. Systems should be built at levels and modules, which form a mathematical structure.

—Friedrich Bauer

I was unable to find it elsewhere, so I don’t know the context of the Friedrich Bauer quote, but I find the quote interesting. I feel the key here is: "not classical mathematics, but mathematics". My interpretation of this is that he is saying that math is important, but not the math many people think about, hence "not classical mathematics".

In Conclusion

I hope I didn’t come across as too harsh on Glenn Vanderburg but I am just being honest and as he said about Bruce Eckel, "He said it so I get to pick on him". 

The real strength of his presentation is that he really does step back from software and give some perspective on general engineering, and while we can all debate the utility of comparing different engineering fields it does hold that they all have a several common threads, including the application of artistry and creativity to design and construction, the objective to design and build things, working within certain constraints including cost, and the organization of people and resources to accomplish the tasks of design and construction.

I admit that my experience with real Agile is limited and while I like what I have seen. I still doubt that it truly fulfills very little beyond an empirical approach and it seems that its focus is primarily on the process of creating software, which is not necessarily a bad thing.  The structural aspect of software is not ignored by Agile, but I do feel that it is the structural aspects of software which is where we currently lack a real engineering framework.  This is really what needs to become more established.  Ideas like Coupling, Cohesion, Refactoring, Patterns, Anti-patterns, etc. are the state of the art in terms of structure but they seem too "squishy" which leads me to ask the question, is it not possible to have more comprehensive math backed concepts like the types of things you find in other engineering disciplines?

The answers to these questions in the past have been dominated by ideas like formal methods and metrics, ideas which have mostly failed to live up to a real promise, however, that does not mean that we should completely discount these and similar ideas in the future.  These and other more math oriented approaches may represent a failed lineage that will go extinct or they may represent the nascent ideas from which we are only now developing the theoretical and mathematical ideas to truly conquer these problems. For example the Nato Conference of 1968 is often cited as a milestone in the development of software engineering. It was only about forty years proir that Category Theory was discovered. I wonder how many attendees or general practitioners knew about it, unlike today where it seems that almost every software blogger writes about Monads and many also mention Category Theory.  I feel that the new era of software engineering will be forged from math but it won’t be "Classical Math" like Calculus and Algebra over the Field of Real Numbers, it will be from areas like Graph Theory, Lattice Theory, Category Theory, Information Theory and other related disciplines and not only will it be built on those disciplines but also the interrelation of those disciplines. I may be wrong about this but I have seen research that seems to point in these directions, ideas I will explore in future posts.  It seems crazy to assume that the failures of the past imply failures of the future. If math can make software cheaper and more reliable to produce it will become an inevitability.

Driven Development

I have recently made the transition to working at a TDD (Test Driven Development) shop, I have no previous experience with it and I am still not sure if I am really feeling it, this may be based on my perception that in order to do it right it seems SO tedious that it is going to seriously harsh my creative coding buzz. Also based on my, albeit limited, knowledge of Combinatorics it seems like it should be called "Smoke Test Driven Development" as the number of possible combinations needed to fully test any software is truly astronomical. On the other hand I have seen it be very useful and the idea of a verification process for software is invaluable, additionally many respected Software Process and Design luminaries such as Martin Fowler and Bob Martin promote it.

I admit part of my problem is that there are probably a lot of things about TDD that I still don’t really get, actually one of my shortcoming as a developer is that I am really not that interested in software process, it’s the structure of software that gets me jazzed. Also I think I have not been lucky enough to see good process applied by good people and my lack of passion for this discipline has resulted in my taking the path of least resistance.

I have been doing this Software Development thing for quite a number of years, more than I want to say, and I have to confess I still don’t really have an explicit methodology, I have acted as a lead developer, although I hate to officially have that title, on many small and mid size projects over the years and in many cases feel that I have made the difference between success and failure of projects, I of course mean that the projects were considered successful, some of which were quite complicated, and in several case have been recognized for helping the success of the project, yet I feel that I am still essentially methodologyless.

Of my last three projects, two claimed themselves as Agile and one declared itself RUP CMMI level 3. The two Agile projects were not even close, they were mostly chaotic waterfall processes, the RUP CMMI project was true to its name, but the process implementation was ridiculously self absorbed in the process itself to the point of being essentially useless, as it was guided by the assumption that checking off all of the process step checkboxes meant the project would run fine on autopilot with people who had little or no software management experience and no thought need be given to the structure or architecture of the software. Admittedly being a seasoned developer this has left me a bit jaded and thinking: "Process, I don’t need no stinking process."

As for TDD I am still on the fence, when I first switched to it I did some light research on it and in the process I encountered BDD (Behavior Driven Development) this started to seem like a YADD (Yet Another Driven Development), so then I googled "Driven Development" to see what else was out there, this of course is still mostly TDD and BDD, but there are many new YADDs, the best one was a humorous yet enlightening blog entry by Scott Berkun called "Asshole Driven Development" which lays out the following (and many more in the comments):

• Asshole Driven Development (ADD)
• Cognitive Dissonance Development (CDD)
• Cover Your Ass Engineering (CYAE)
• Development By Denial (DBD)
• Get Me Promoted Methodology (GMPM)

Ironically I think one of my Agile projects was a combination of Scott Berkun’s ADD and CDD, and what I would call RDD, Resume Driven Development, that’s where developers pick technologies solely to add them to their resumes, I was against this practice and reviled for it, thus I was on one side of the CDD conflict.

As I have wandered through the desert of pseudo-process for the last few years, I have been increasingly researching, observing and thinking about why am I able to succeed where others fail and what is the best way to do this, also there are many things I have observed where the management of the project was incredibly inefficient and dysfunctional, sometimes I would openly comment on and other times I would keep to myself.

I figured that I would jump on the YADD proliferation bandwagon and propose two of my own YADD’s, the first is my FDD, Framework Driven Development, the basic structural principles of which are outlined in my blog entry "What does your framework look like?"

In my recent experience I have noticed that software projects can become too focused on process and that process can become a magic bullet. It seems that in some cases process trumps structural concerns. The process component is easier for mangers to grasp and is more congruent with their formal methods, naturally. However, I think that the two need to be balanced, and I feel that something like my FDD concept is a step in that direction, as how to integrate it into process, that is something that still needs to be thought through.

The second YADD is what I would call DSDD, Developer Skill Driven Development. My previous ADD, CDD, and RDD "Agile" project was run by very young non-technical "managers", who had or were in the process of acquiring their PMI (Project Management Institute) PMP (Project Management Professional) suffixes, this one tends feel like a GMPM to me. Needless to say these "mangers" had only a rudimentary, at best, understanding of software development, I should also mention that this was government work so the tolerance for inefficiency was high, and there were a number of very good and good developers so the project mostly ran, as one developer pointed out, by itself. The most frustrating thing about the project was how the work was assigned, the better developers, myself included would often watch, sometimes in anger, as poor and junior developers were given the more challenging work, with which they always stumbled. Another developer would often say to me in jest, "We are all just 25 horsepower engines," I guess sometimes you have to laugh otherwise you’ll be crying.

I have since seen and also reflected back to previous projects in regards to how often managers seem to not understand or take into account the strengths of their team members, and it seems to be a reoccurring antipattern which I would call SDD, Slot Driven Development. (How many YADD’s are we up to now?) I think part of the problem is the management philosophy of divide and concur, where you split up the work into tasks, apply your "resources" and repeat.

DSDD is the idea that not all project tasks have the same level of difficulty and not all developers have the same level or set of skills and the two should be matched as effectively as possible. Another ramification of this approach is to probably a need rethink how project work is partitioned, think framework builder i.e. the 20%.

Of course I am just having some fun ranting here and am not really seriously proposing the creation of more YADDs. Some of my bad experiences may be due to the type of contracts I have taken recently, more on that later, but it does seem that in general software projects are poorly managed, please feel free to share your thoughts, experiences and YADDs in the comments.