26 September 2016

Software Development is a Risky Business





Software projects are inherently risky.  Some statistics show that roughly seventy percent to fifty percent of software projects get classified as “challenged”.   Where challenged can mean cost overruns and/or delivery delays, or it can mean complete failure and the cancellation of the project.  Some statistics indicate as the size of the project increases so does the probability of problems and failure.  Many articles and blog posts have been written about the long history of software project failure.  At the time of writing this post the most recent and very public software project debacle was the healthcare.gov project.  It seems as IT becomes more pervasive and essential the problems with projects may be getting worse not better.


In spite of these numbers and all too often in my experience I have found that individuals and organizations seem to have a very optimistic outlook about their own projects especially at the beginning of the project.  Perhaps it’s the classic: “that won’t happen to me” attitude.  Even as projects start to run into problems I have seen people especially managers not take issues seriously or ignore them all together.  In some of the analysis of healthcare.gov, this was cited as one of the problems.


Risk management and risk assessment/analysis are developed areas in project management and engineering and are often discussed in software projects.  Many risks will end up falling into two categories.  One category is actual risks to the end product, such as unforeseen security vulnerabilities in software systems or other engineering issues that lead to flaws or critical failures of the system or components. The other is risks in the management and execution of constructing product or system. 


Risk management is one of those areas where the engineering aspects and the engineering management aspects can become tightly coupled.  An example can be derived from the XKCD comic “Tasks”: https://xkcd.com/1425/.  Now the point of this comic is to demonstrate the difference between an easy problem and hard one, and presumably the conversation takes place between a manager and developer.  In a risk setting, perhaps solving the hard problem is the goal of the project.  So a risk might be that the research team fails to solve that problem at the end of the five years, or that it takes ten. So in a sense the task is high risk but there are team related risks, maybe the budget is too low and the team is subpar due to low salaries so they can never even solve it.  Another possible scenario is that the conversation never happened and the project management and team think it is solvable in six months.  This second scenario is the risk I have seen though out most of my career, the project management and team are overly optimistic or really don’t have a good understanding of the complexity of the task at hand.  In some cases these types of issues seem to be due in part to pluralistic ignorance.  Often in cases where developers don’t want to contradict management.  There’s a great dramatization of this type of behavior in the show Silicon Valley when the Hooli developers are working on the Nucleus project, where the information is conveyed up the management chain but nobody wants to tell Gavin the truth.


Sadly it seems like the risks that are identified are almost peripheral to the real risks on projects.  It is usually things like the hardware delivery will be delayed which will delay setting up our dev/test/prod environments and delay the start of development.  While these are legitimate risks, there is always the risk of all of the unknowns that will cause delays and cost over runs, even things like developer turnover never seem to be taken into account, at least in my experience.  Also there seems to no cognizance of the long term cost risks like the project was poorly managed or rushed or was done on the cheap which caused the delivered codebase to be an unmaintainable big ball of mud that will incur a much higher long term costs.


In order for software engineering to grow up and become a real engineering discipline it will need formal risk assessment and risk management methodologies both on the project management side and on the work product side.  This is also an area where ideas and methodologies can potentially be taken from existing work in other engineering disciplines.  These methodologies will also most likely draw from probability and game theory.



References and Further Reading





31 May 2016

My Ant Symbiosis Stories



This post is a little off topic from my normal posts.  Although, since I consider my blog to be about programming, software engineering, math and computer science, perhaps taking a brief foray into the natural sciences is warranted as it is science and I am a science nerd, and of course it is my blog.  The book that I am referencing for this post is: The Ants by E.O. Wilson and Bert Hölldobler. It has many great examples of data visualization.  There is also a CS relevant area of study called ant hill algorithms.  Additionally, I believe that the study of social biology is basically just game theory which again is a topic that comes up in CS.


The natural world is fascinating and it teams with many interesting life and death stories that play out every minute of every day in the theatre of nature.  Some of the most fascinating plots include various types of symbiosis that occur between species. Symbiosis, also called mutualism, is a cooperative relationship between two species that sometimes includes an exchange of “survival favors”.  A fairly famous example of this type of species interdependence is the story of why Brazil nuts produce very poorly in plantations.  It turns out the Brazil nut tree Bertholletia excelsa is dependent on a pollinator insect, an orchid bee Euglossa, which is attracted by an orchid, Coryanthes Vasquezii  which does not grow on the tree but is present in the undisturbed forest habitat.  Finally the tough outer shell needs large rodents like the Agouti to gnaw it open, they much like squirrels consume some of the seeds and also bury some for later which allows some of the forgotten seeds to germinate.  There are many examples of symbiotic relationships in rainforest and other habitats.


Ants are notorious for having symbiotic relationships with plants and other insects and many such cases are well documented. One case is the Neotropical Cecropia tree which provides hollow stems for Azteca ants to inhabit and even food in the form of gylcogen-rich Müllerian bodies.  There was another story about a relatively recent discovery of a plant that grows on the harsh cliffs of Pyrenees that depends on three species of ants for pollination and seed dispersal.  These types of stories are not uncommon in the natural world, however, one often reads of them playing out in the above and often exotic and/or tropical locations, so I found myself rather surprised to discover this type of drama transpiring literally inches from my living room window in suburban Arlington Virginia. 


The story I am going to tell happened during the summer of 2011 and I had originally labored under the assumption that it was a somewhat unique undocumented story.  However, my subsequent research has yielded a number of pictures which depict many of the events that I will describe and show here often with better photographic quality than what I what I captured.  Not that my photos are bad but they lack the fidelity of others I have seen.  I make up for this in quantity which captures various events over an extended period of time.


One of my hobbies is identifying eastern forest native plants and experimenting with growing them in my yard.  This activity, probably to the chagrin of my mcmansioned neighbors, has turned my yard into a small native plant and wildlife habit.   A few years ago I collected some seeds of a plant known as Yellow Leaf Cup, Bearpaw, or Bearsfoot, which I believe goes by the Latin name Smallanthus Uvedalia formerly known as Polymnia Uvedalia.  It is in the Aster Family (Asteraceae).  I planted some in pots and scattered some seeds in my yard, the potted ones germinated but did not make it however some took root right in front of my living room window.  To me this plant really stands out, for a temperate zone plant it has very large leaves, 12 to 18 inches long, and as a result it almost completely obscures my living room window shrouding it in giant green veined leaves as you can see in the picture at the top of this post.  The location of these leaves put me in a unexpected and unique position to watch an incredible and probably epoch old symbiotic drama of nature play out between Ants(Hymenopterans) and Hemipterans


In June 2011 as the stand of Large Flowered Leaf-Cup reached about eight or ten feet in height and completely blocked my living-room window providing a nice screen of green privacy, an odd looking insect, pictured below, showed up.  As you can see it’s shaped like a thorn which has garnered its ilk the general common name of thorn bug it is also referred to as a tree hopper this class of insects that falls under the order Hemiptera, and I believe it is the species Entylia carinata. It does not have a common name.  I think it needs a common name and in that picture I see a rhinoceros looking insect, thus I am coining it the rhinoceros thorn bug or just the shorter version of rhino thorn bug.  If you look closely at the picture below you can see its eyes and legs which are hard to see in many of the photos, my guess is this is by design as thorns don’t have eyes and legs.



Shortly after its arrival the ants showed up to tend to it:



I contacted a biologist who specializes in ants and he identified them as a species of carpenter ants in the genus Camponotus. He told me the possible species for this area are: Pennsylvanicus, Nearcticus, and Subbarbatus.


Over the following months I watched the interesting drama play out with these two species. Owning a copy The Ants gave some perspective to the traits and behaviors of these species.  Over time I captured over one thousand pictures of what played out between these two species.  This collection is really a dataset of photos and it revealed some interesting observations, which while not novel, still make for an interesting story.


Rhino Thorn Bugs: Masters of Camouflage


First let’s start with how Rhino Thorn Bug disguises itself.  A number of the photos of it were taken in different lighting conditions.  In the above photos the light is more direct and the yellow and green “veins” of both the plant and insect mesh and it looks like a thorn on the stem.  In the following lighting the Rhino Thorn Bug appears to have purplish veins that blend with the background of plant veins.



 Another outcome of the Rhino Thorn Bug living on smallanthus uvedalia is the damage to the leaves which also makes for a nice abstract photo:



I’m making a leap here, but it seems that this damage extends the camouflage range for the rhino thorn bug such that it can hang out on the leaf and look like damaged parts of the leaf that more closely match its brownish color and pseudo-veins in back lit conditions:



I also noticed that it seemed to stay in one place and was able to feed by making multiple bites into the thick main leaf vein, I suspect that the major vein might be a bundle of veins and it was tapping into individual sub-veins each time.  This is visible in a subsequent photo.


Protection and Feeding


Ant Symbiosis with other insects takes a number of forms. A common symbiosis is known as Trophobiosis where ants feed on what is known as honeydew.  This substance is a mixture of sugars that the insect excretes from its posterior.  It is often the case that the provider of this mixture breaks down more complex polysaccharides into simpler sugars that the ants can digest these include glucose, fructose, etc.   The not so great image below shows this an ant feeding from the posterior of the rhino thorn bug:



This relationship is not unfamiliar to humans. It seems to mirror our use of livestock especially dairy cows, although we get the fluid we consume from a different organ.  However, the behavior of protecting, caring for and in some cases herding these insects is exactly what we do with our livestock.  The ants have been doing this millions of years prior to our existence as the ant Hemiptera symbiosis behavior is thought to have originated in the Eocene period.  There is evidence of this, shown below, from about 15 million years ago found in Dominican Amber which shows a queen from the extinct species Acropyga glaesaria on her mating flight with a “seed” mealybug in her mandibles for use when she establishes her new colony.



Another ant behavior that I believe I caught is ants transferring food to each other, which is known as Trophallaxis.  In the image below you can see the two ants at the top with their mandibles interlocked, in a previous picture one of them was feeding and now seems to be transferring food to the other.



Castes


Ants have different castes of workers. The image below shows two different castes I observed.  The one at the top is a Major worker.  Ants of this caste are in some cases referred to as soldiers as they in some species take on roles like acting as “doormen” at colony entrances to ensure that only ants of that colony are allowed.  It easy to see that the larger body size with the large head and large mandibles makes them more formidable, although I don’t know if the solder distinction truly applies in this case.  The other two are minor workers which have smaller bodies and heads. 



Rhino Thorn Bugs, the Next Generation


As the story unfolded it got even more interesting. I recall, although I don’t have photos of it that the ants seems to escort the thorn bugs to meet each other for mating.  The evidence of mating was soon obvious as can be seen in the photo below with the ants in attendance of the brood of recently hatched nymphs.  The ants tended to the nymphs which included chasing them around as they wandered off. Some of the nymphs seemed to stray and many of them died.  Also visible in this photo are the multiple bites left by what was presumably the female rhino thorn bug made over her extended time staying in one location.



Pretty soon the nymphs are providing the same trophobiosis as their parents as shown below:



As the nymphs grow up you can see that they start to take on their adult shape:




The Next Year, New Ants and Aphids


I was curious to see if the thorn bugs would return the following year. They did not.  However, I was surprised to a whole new ant drama play out with new actors.   I reached out to the ant biologist who helped me the previous year and he identified these as Tapinoma sessile.  I was not able to identify the aphids.  I was told they would be plant specific.


Their activity was concentrated around the flowers and spanned the time that they were in bloom.  The “infestation” of the flower heads can be seen in the following photo:



Another difference was the ratio of ants to honeydew producers, as you can see there are many aphids to each ant which in contrast to the rhino thorn bugs which often had multiple ants in attendance of each one.  The biologist that I contacted said that the ants looked fat and happy, and their abdomens or gasters in ant biology speak, seem to be bursting and almost glistening, presumably with honeydew.   I once saw David Attenborough eat a honeypot ant and it got me thinking I should have tried it with these guys. Well that was until I learned that their common names include odorous house ant and stink ant.



The photo below shows how the flower’s sepal seems to serve as a protected staging area for the aphids to develop to full maturity:



All of this happened in 2012. I have not seen the aphids since. I do see the rhino thorn bugs at least one or two each year over the last few years.  I have never seen the level of activity that I saw back in 2011, I am always hopeful that each new year will bring some new drama. Actually last year it was an opiliones explosion, but that’s a whole other story. 


The story about the Brazil nut symbiosis is interesting and was a little bit of a setup to show my cute seed disperser below.  Although the plant is obscured by another (Wingstem, Verbesina alternifolia) It is collecting Large Flower Leafcup seeds, the flower heads are visible slightly right and below it.



References and Further Reading







24 May 2016

Software Development is a Risky Business




Software projects are inherently risky.  Some statistics show that roughly 70% to 50% of software projects get classified as "challenged".   Where challenged can mean cost overruns and/or delivery delays, or it can mean complete failure and the cancellation of the project.  Some statistics indicate as the size of the project increases so does the probability of problems and failure.  Many articles and blog posts have been written about the long history of software project failure.  At the time of writing this post the most recent and very public software project debacle was the healthcare.gov project.  It seems as IT becomes more pervasive and essential the problems with projects may be getting worse not better.


In spite of these numbers and all too often in my experience I have found that individuals and organizations seem to have a very optimistic outlook about their own projects especially at the beginning of the project.  Perhaps it’s the classic: "that won’t happen to me" attitude.  Even as projects start to run into problems I have seen people especially managers not take issues seriously or ignore them all together.  In some of the analysis of healthcare.gov, this was cited as one of the problems.


Risk management and risk assessment/analysis are developed areas in project management and engineering and are often discussed in software projects.  Many risks will end up falling into two categories.  One category is actual risks to the end product, such as unforeseen security vulnerabilities in software systems or other engineering issues that lead to flaws or critical failures of the system or components. The other is risks in the management and execution of constructing product or system. 


Risk management is one of those areas where the engineering aspects and the engineering management aspects can become tightly coupled.  An example can be derived from the XKCD comic "Tasks".  Now the point of this comic is to demonstrate the difference between an easy problem and hard one, and presumably the conversation takes place between a manager and developer.  In a risk setting, perhaps solving the hard problem is the goal of the project.  So a risk might be that the research team fails to solve that problem at the end of the five years, or that it takes ten. So in a sense the task is high risk but there are team related risks, maybe the budget is too low and the team is subpar due to low salaries so they can never even solve it.  Another possible scenario is that the conversation never happened and the project management and team think it is solvable in six months.  This second scenario is the risk I have seen though out most of my career, the project management and team are overly optimistic or really don’t have a good understanding of the complexity of the task at hand. In some cases these types of issues seem to be due to pluralistic ignorance, usually where the developers didn’t want to contradict management.


Sadly it seems like the risks that are identified are almost peripheral to the real risks on projects.  It is usually things like the hardware delivery will be delayed which will delay setting up our dev/test/prod environments and delay the start of development.  While these are legitimate risks, there is always the risk of all of the unknowns that will cause delays and cost over runs, even things like developer turnover never seem to be taken into account, at least in my experience.  Also there seems to no cognizance of the long term cost risks like the project was poorly managed or rushed or was done on the cheap which caused the delivered codebase to be an unmaintainable big ball of mud that will incur a much higher long term cost.


In order for software engineering to grow up and become a real engineering discipline it will need formal risk assessment and risk management methodologies both on the project management side and on the work product side.  This is also an area where ideas and methodologies can likely be taken from existing work in other engineering disciplines.  These methodologies will also draw from probability and game theory.


References and Further Reading



30 April 2015

What is Software Engineering?


The terms software engineering, software engineer, and sometimes just engineer as in "full stack engineer" appear a lot in the software development field.  Some people including myself find the use of the term "Software Engineering" to be somewhat dubious. I also feel that there are too many people calling themselves engineers who really shouldn’t be.  As a result of my skepticism I have been trying to devise a way to determine exactly what is Software Engineering?


The happy path is to go to the web and search for it or just find a book. Both will lead to many answers.  But that is really just getting someone else’s opinion.  A little later we will look at other people’s opinions as there are very established authors who readily answer this question.


To get some perspective, let’s step back and ask a more basic question. What is engineering?  For this I will rely on the ultimate oracle of our time. Some top responses from the Google are:


The Google dictionary definition:


en·gi·neer·ing

noun

the branch of science and technology concerned with the design, building, and use of engines, machines, and structures.

• the work done by, or the occupation of, an engineer.

• the action of working artfully to bring something about.

"if not for Keegan's shrewd engineering, the election would have been lost"


The Georgia Tech School of Engineering provides the following:


Engineering is the practical application of science and math to solve problems, and it is everywhere in the world around you. From the start to the end of each day, engineering technologies improve the ways that we communicate, work, travel, stay healthy, and entertain ourselves.

What is Engineering? | Types of Engineering:


Engineering is the application of science and math to solve problems.  Engineers figure out how things work and find practical uses for scientific discoveries.

Wikipedia: defines it as


Engineering (from Latin ingenium, meaning "cleverness" and ingeniare, meaning "to contrive, devise") is the application of scientific, economic, social, and practical knowledge in order to invent, design, build, maintain, research, and improve structures, machines, devices, systems, materials and processes.

So to summarize and synthesize the above for a working definition, we get:


Engineering is a branch of science and technology where mathematical, scientific, economic, and social knowledge are applied artfully, creatively, and efficiently to invent, design, build, maintain, research, and improve structures, machines, devices, systems, materials and processes.

Another aspect of engineering that should be considered is its diversity. Just searching around I came up with the following list of types of engineering, which should by no means be considered complete, they include:  Aeronautical, Agricultural, Audio, Biomedical, Biomechanical, Building, Ceramics, Chemical, Civil, Computer, Electrical, Electronics, Environmental, Food, Geological, Industrial, Marine, Mechanical, Military, Mineral, Military, Nuclear, Optical, Petroleum, Photonics, Systems, and Textile.


Often discussions of software engineering draw parallels to construction and civil engineering and it is easy to see why, most people often see the results of these disciplines and even live in them.  Not to trivialize these disciplines, but bricks and mortar, in addition to wood, glass, and rebar are things that support our lives and are physical thus somewhat easy to conceptualize.  But all engineering disciplines now touch and perhaps run our lives in ways that we rarely think about.  I previously posited that chemical engineering, prior to the software era, was probably the most pervasive engineering in our lives not to mention that the computer hardware we use wouldn’t exist without it.  It serves as an interesting area to look at in regards to software engineering comparisons too, as it is a relatively recent discipline and has some parallels like the control of processes and "flows" in this case of materials.  Perhaps the closest "relative" to software engineering might be industrial engineering.  Like software it deals with human factors.  It also deals with stochastic processes, something that is becoming increasingly more important in software systems.


Now that we have some general engineering perspective, let’s revisit the question "What is software engineering" by looking at what the established organizations and authors have to say.


Wikipedia gives the following definition:


Software engineering is the study and an application of engineering to the design, development, and maintenance of software.

Typical formal definitions of software engineering are:


  • the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software
  • an engineering discipline that is concerned with all aspects of software production
  • the establishment and use of sound engineering principles in order to economically obtain software that is reliable and works efficiently on real machines.

K.K. Aggarwal provides us with a bit of history:


At the first conference on software engineering in 1968, Fritz Bauer defined software engineering as "The establishment and use of sound engineering principles in order to obtain economically developed software that is reliable and works efficiently on real machines".

Stephen Schach defined the same as "A discipline whose aim is the production of quality software, software that is delivered on time, within budget, and that satisfies its requirements".


The ACM weighs in with:


Software engineering (SE) is concerned with developing and maintaining software systems that behave reliably and efficiently, are affordable to develop and maintain, and satisfy all the requirements that customers have defined for them. It is important because of the impact of large, expensive software systems and the role of software in safety-critical applications. It integrates significant mathematics, computer science and practices whose origins are in engineering.

There are a number of books on software engineering which have an obligation to address the question.


The answer in Ian Sommerville’s 8th Edition of Software Engineering  is:

Software engineering is an engineering discipline that is concerned with all aspects of software production from the early stages of system specification to maintaining the system after it has gone into use. In this definition, there are two key phrases:

1. Engineering discipline Engineers make things work. They apply theories, methods and tools where these are appropriate,. but they use them selectively and always try to discover solutions to problems even when there are no applicable theories and methods. Engineers also recognise that they must work to organisational and financial constraints, so they look for solutions within these constraints.

2. All aspects of software production Software engineering is not just concerned with the technical processes of software development but also with activities such as software project management and with the development of tools, methods and theories to support software production.


What Every Engineer Should Know about Software Engineering by Philip A. Laplante offers the following:


Software engineering is "a systematic approach to the analysis, design, assessment, implementation, test, maintenance, and reengineering of software, that is, the application of engineering to software. In the software engineering approach, several models for the software life cycle are defined, and many methodologies for the definition and assessment of the different phases of a life-cycle model" [Laplante 2001]

In Software Engineering A Practitioner's Approach 8th Edition Roger S. Pressman and Bruce R. Maxim provide a terse explanation:

Software Engineering encompasses a process, a collection of methods [practice] and an array of tools that allow professionals to build high-quality computer software.


Roger Y. Lee describes it in Software Engineering:A Hands-On Approach as:


Software engineering is the process of designing, developing and delivering software systems that meet a client’s requirements in an efficient and cost effective manner. To achieve this end, software engineers focus not only on the writing of a program, but on every aspect of the process, from clear initial communication with the client, through effective project management, to economical post-delivery maintenance practices. The IEEE Standard glossary of software engineering terminology defines software engineering as the application of a systematic, disciplined, quantifiable approach to the development, operation and maintenance of software (Ghezzi et al. 1991). As made clear by the above definition, software engineering came about in order to reverse early development practices by establishing a standard development method that could unify and normalize development operations, regardless of the people involved or the goals sought.

The laborious effort of wading through all of this erudition gives me the following distillation:


Software Engineering is:


  • An engineering discipline
  • Application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software"
  • Software engineering (SE) is concerned with developing and maintaining software
  • Software engineering is the process of designing, developing and delivering software systems
  • A standard development method that can unify and normalize development operations, regardless of the people involved or the goals sought.

Software Engineering Includes:


  • All aspects of software production from the early stages of system specification to maintaining
  • The system after it has gone into use
  • Non technical activities such as software project management and with the development of tools, methods and theories to support software production.
  • The integration of significant mathematics, computer science and practices whose origins are in engineering.
  • The application of theories, methods and tools where these are appropriate
  • Many methodologies for the definition and assessment of the different phases of a life-cycle model
  • Several models for the software life cycle
  • A process, a collection of methods [practice] and an array of tools that allow professionals to build high-quality computer software.
  • The application of sound engineering principles to the analysis, design, implementation (development), assessment, test, maintenance and reengineering of software that is reliable and works efficiently on real machines

Software Engineering Produces:


  • Quality software
  • Software that is delivered on time
  • Cost effective production of software (within budget)
  • Software that satisfies its requirements.
  • Software systems that behave reliably and efficiently
  • Software that is affordable to develop and maintain
  • Solutions that conform to organizational and financial constraints

I tried to break the multitude of ideas into three categories, but they are admittedly a bit fuzzy and you could probably move some of these around.  The point of the categorization exercise is to attempt to separate a definition from the broader idea of what the discipline encompasses and the attributes of the structures and systems that it produces, again things can get fuzzy here.  In thinking about these and my own experience with software development I came up with a list of things that I feel should be defined to make software engineering a real discipline and they are:


  1. What is the definition of the engineering discipline we are describing?
  2. What are the produced artifacts, e.g. structures and processes that the discipline creates?
  3. What non-managerial processes and methodologies are used to create the produced artifacts?
  4. What types of documentation (descriptive artifacts) are used to describe the produced artifacts and the processes to create those produced artifacts?
  5. How does the underlying math and science support the engineering discipline and how do these foundational disciplines allow one to reason about the produced artifacts that the engineering discipline is trying to construct?  How do these disciplines define and drive the descriptive artifacts?
  6. How does one organize and manage the effort including various roles people will play to actually carry out the work to bring the system or structure to fruition?

So coming back to the original purpose of this post, my answer to the question of what is software engineering, number one above, is:


Software engineering is the branch of science and technology where mathematical, computer science, linguistic, social knowledge and system relevant domain specific knowledge are applied artfully, creatively, and efficiently to invent, design, build, maintain, research, and improve software systems.

That’s the easy part. The hard part is what are the answers to the other five questions? I know that there a number of industry standard answers for these that include things like UML, Agile, Scrum, etc., but are those really valid and are they really working? I don’t think so, so maybe it’s time to look for other answers.