As we all know the fields of computer science and software engineering are in their infancies. Many a blog post has been written lamenting the fact that software engineering is not a real engineering discipline, and while I have not written that exact post, I have written about the subject and have deconstructed what others have said about it. A number of these points do apply to why software projects routinely fail, yet another topic that has received considerable attention. Now admittedly all engineering disciplines regardless of their maturity and formalization have project failures and creating a real software engineering discipline will not eliminate this problem but one would hope that it would abate it.
I find it odd that at a time when so many people are involved in IT there seems to be little discernible progress in creating a real software engineering disciple. There are probably millions of people working on creating software. Also there are many researchers trying to solve this problem from many different angles. Practitioners have been attempting to solve it with ideas like Agile and Software Craftsmanship, etc. Additionally there is a long list of failures surrounding the formalization software engineering.
My biggest complaint is the fact that there really are no good formal standards in general software engineering principles and methodologies lack a formal foundation and even the more vague principles can be easily thwarted and misused, and often, as I have previously complained, it often ends up being based on pure opinion which is usually won through positional authority, perseverance or by those who are just more politically savvy.
The intent of this post is not to attempt to solve or even offer solutions to the problem of creating a real software engineering discipline but to look at what I think are some barriers to creating this discipline and in some of these cases offer thoughts on overcoming those barriers.
Low Entry Requirements
If we are to think software engineering as an engineering discipline, I challenge you to find another engineering discipline that routinely has practitioners with no formal training in the field. I very much doubt that someone who did not hold a degree in engineering and who built a shed in his back yard is now working as a structural engineer on a construction project, the mere idea seems ludicrous. Yet I have worked with non technical degree holders who became software developers, one who started by building web pages and now calls himself an "Internet Application Architect", and is probably one of the biggest cargo cult coders I’ve ever had the misfortune to work with. This egalitarian aspect isn’t all bad as it does let good people into the field as well. Another non technical degree holder I once worked with became an accomplished developer and went on to get an advanced degree in CS and is now a CS professor. Although it’s probably the case that for every good non technical degree holder who joins the field it is likely that dozens of mediocre and bad practitioners will also join. It seems that in other engineering disciplines there are much more stringent educational requirements to ensure proper training. To be clear here I am not equating having a degree with being competent because I have met many incompetent people with degrees, but still you need something. I confess I am at a loss for a solution for this one.
The Lack of Differentiation between Science and Engineering
Chemistry is a scientific discipline, chemical engineering is an engineering discipline, and you can make this comparison between other engineering disciplines and the sciences that they employ e.g. electrical, mechanical, and structural map to various areas of physics. Engineering and scientific disciplines are different types of disciplines often taught in separate schools. Although each engineering discipline has some scientific overlap and it would probably be possible to move from the appropriate scientific field to the corresponding engineering field in general you probably would not do so without returning to school. In recent years software engineering curricula have been added to the rosters of many colleges and while this is potentially a step forward, discounting for the moment that software engineering is still not really real engineering, a software engineering degree and a computer science degree in many cases gets you the same job!
So in other fields there would be a differentiation between a degree that would put you on a track to do scientific research and one that would put you on a track to do engineering work. As I understand it, if you are fresh out of school with a BS in CS that qualifies you to be a tester at Microsoft1, developers hired right out of school need at least a Masters degree in CS. This is an extreme case of how this really breaks down in our industry. In most cases engineering practitioners are the software engineering, MIS, CS or even non technical degrees and the few scientific careers generally go to the advanced degree holders. I admit this probably fairly normal a BS in chemistry or biology is also more likely to get you a job in IT than a job as a research scientist. Still compared to more mature engineering fields there seems to be a lack of real differentiation between the degrees that yield a career as a software engineer versus one as a computer scientist.
Bad Management
The entry requirements for managers in software make the low entry requirements for software engineering practitioners look downright rigorous. I have previously criticized the fact that many organizations find the cheapest people, especially for software management and give them an obligatory certification. Now I know that engineering management regardless how formal or established the engineering discipline is an area that has problems and failures, but again I very much doubt that you would find a twenty something business or liberal arts major with a freshly minted PMP suffix managing construction or civil works projects. Yet in my experience this is pretty normal in IT especially in the government sector. Of course bad software management is executed by older managers as well.
A post by Larry White titled "Engineering Management Is Dying" delves into some these issues. It is definitely the case that methodologies like Agile have changed how software projects work and the traditional corporate management approach to software really doesn’t work. One point he makes is that it is not uncommon to see a two hundred person project at Google headed by an engineer. To me this implies that someone is in some way managing that project. I think all of this is indicative of the need for software engineering mangers to also be practitioners in engineering or at least very well versed in how software development works. The situation he talks about at Google is not the case everywhere, in that case the company is its own client, but there are many cases where software is being built for clients and this does create a need for management that deals with the client, perhaps this should be a role that is separated from management and called a client liaison, which is what often happens. Also I am not sure how things work at Google but I have also seen internal development in organizations where the IT department provides services to an internal client, I have also seen this go horribly awry with contentious unproductive relationships between departments. In short I feel that just as we need a real software engineering discipline we also need a real incarnation of software engineering management.
The Disconnect between Academia and Industry
Most practitioners do not keep up with, or read academic research, actually many practitioners don’t read at all, but that’s another issue, and most academicians don’t work in the field so they lack the hands on knowledge. Unfortunately this rift can take on a somewhat disdainful tone as is the case in my exploration of one practitioner’s attempt to define "Real Engineering". Fortunately some people, I’d like to count myself among them, take a more constructive approach to building bridges across this rift. Daniel Lemire has an interesting critique the quality of software produced in academia.
I think the solution here is for both sides to become more engaged in problems faced on each side. I am optimistic about this one as I feel that these walls are breaking down as the field is growing up and many of the emerging technologies are forcing developers to be more cognizant of and engaged in research topics. I also have encountered much more research that has ties to the practical concerns of day to day software development, some I have mentioned with more to come.
The Lack of an Effective CS Math Curriculum
If you read my blog you know it to be a mix of math and software practices. I would describe myself as a software practitioner and a math enthusiast. My math journey has taken me on some interesting math excursions into areas of math that seem to get little or no mention in CS curricula and I feel that this is a major problem in really applying math to the field of software engineering. Another problem is in the way that it is taught, my experience was that it was not only taught badly but in a way that made it seem irrelevant to many of the programming courses. Specifically I would shift the CS math curriculum to include less differential and integral calculus and include more logic, combinatorics, abstract algebra, graph theory, order theory, category theory and probably some topology among others.
Over the last few years I have been progressively learning more math which has been in part motivated by necessity for understanding research papers. This approach has changed the way I see things, I now see the patterns of math in software. They are there and I believe that they can be exploited to create a real software engineering discipline.
For the math learning problem I do have some ideas many of which are expressed in my blog. Some of my posts like refactoring if statements with Demorgan’s Laws or the String Monoid, the Object Graph, etc. are about making these mathematical ideas more relevant and accessible to programmers. Other posts like my series on naming and my post on generic programming and entropy and complexity are about my own ideas and ideas in the research literature that I feel may help to create a real software engineering discipline. These are my attempts at solutions to these problems, so expect more of both of these and more thoughts on the CS math curriculum.
The Software Architect Debacle
This is one that really bothers me. I feel the software architect role in general has a very negative effect on creating quality software and it dissuades the industry from developing a real engineering discipline. The Software Architect role tends to be broadly defined, you have enterprise architects, software architects, etc., some architects tend to be involved with hardware and networking, some work on configuration management, some do software design, some do all of these things and more. I feel this is a problem as each of these is a separate engineering area with different types of problems. By not breaking these down it often leads to a lack of focus on specific problems, also I have seen cases where the architect focuses on their preference and ignores other issues. The architect role is often divorced from the code. I have met many architects that were responsible for software but had never even looked at the code. To me this is a huge failing that architects that are responsible for building software often lack the interest, time, or even aptitude to know the quality or underlying of structure of the software that they are delivering.
To really do justice to these ideas I might need a separate post, my solution would be to break down the architect role into specific engineering roles, which could include:
- Software Process Engineering - this would be involved with software team and resources planning, some aspects of configuration management such as defining policies, requirements analysis and general project management aspects of software construction including the definition and refinement of the SDLC. This role is tightly coupled with software engineering management.
- Software Structural Engineering - this would include requirements comprehension, code infrastructure planning, prototyping work, hands on code structural work including custom frameworks and reusable components, third party library products, and general code quality including reviews and static analysis.
- Software Quality Engineering - This would include the QA roles, software testing and testing tools, requirements validation and refinement, usability and reliability and general software quality issues.
- Software Infrastructure Engineering - This would include hardware, networking, database, app server and general service infrastructure, non functional requirements fulfillment and possibly the implementation of configuration management, continuous integration, etc.
This is a rough "sketch" of these possible roles and each of these areas has some overlap implying that all of the people in these roles would work closely together, also some combination of, or all of these roles might be performed by a single individual depending on the organization’s size and structure. What this approach does is clearly defines roles and responsibilities as opposed to some nebulous software architect role.
The Continuous Turnover of the Work Force
A young CEO once commented that he thinks young programmers are superior. In general companies prefer younger workers as they often don’t have family commitments so they can work more hours and you can pay them lower salaries, DC area government contractors rely on this to keep their profit margins up.
As an older developer I often find this frustrating. I confess that I have not moved up the ladder and still mostly work as a developer or what might be called a hands on architect, yes I know but that’s the current term, actually my preferred title would be: Software Structural Engineer. Working as a contractor I have on several occasions found myself on projects that were dominated by younger developers and unfortunately on more than one occasion I watched as teams would make many mistakes due to a lack of experience, in some cases I was able to help in others I was ignored by the younger developers. This is not to say that I do not still make mistakes, I’ve just been around long enough to have made a lot of them already.
If you buy into the software craftsmanship thing, I admit to being skeptical of this idea, you might be inclined to draw a parallel to craftsman of the past when older established masters took on apprentices and passed on their wisdom to the next generation. Given the access to information these days such process is somewhat antiquated and perhaps impractical. Nevertheless I feel, and the industry supports me here, that older developers who keep their skills up to date have value. I know several younger developers who have sought me out to learn from me so I think I can safely say I still have some value.
Many older developers have let their skills get rusty and have not kept up on current technologies I have seen this many times. Also many companies seem to lack the vision to allow for real hands on technical career growth. I can’t help but feel that this inclination to recycle developers in each new generation is hurting or at least slowing our ability to develop a real software engineering discipline, this potential loss of continuity seems like it leads to some lost wisdom.
The Social Networking Drain and Hollywoodification of Silicon Valley
As I write this Hollywood gears up to deliver a reality TV show that takes place in silicon valley and Mark Zuckerberg now gets the paparazzi treatment. Another article that I previously mentioned was about how CS enrollment was up because the movie The Social Network had enticed many aspiring Zuckerberg wannabes. At present there does seem to be a gold rush mentality and it is noticeable and even discussed on sites like Hacker News.
As someone who has never worked in the valley I am very much an outsider and may not have a good perspective on these things, but it seems that there are a lot of startup companies focusing on crap. Now I get it. We live in a shallow consumer society with a rapacious appetite for crap. But are not these supposed to be some of the smartest people and I am not the only one to wonder why so many smart people who should have better taste and higher standards are settling for working on serving up ads and other vapid crap instead of doing more meaningful work like working on real problems that would advance human society or even just work on advancing software engineering.
1I believe this was the case in the late 90’s and may have changed, this information was conveyed to me by a former Microsoft employee.
