23 March 2015

What is Software Correctness?

Software correctness which is really software quality is not one thing.  It is comprised of a number of different and sometimes conflicting attributes.  As always Gerald J. Sussman provides interesting insights and in this case it is in his "We really don’t know how to compute" talk.   He makes the point that "correctness" may not be the most important attribute of software, the example he give is if Google’s page craps out on you, you will probably try again and it will probably work.  Actually this example might be more about reliability but it got me thinking as I have always thought that "software correctness" meant the combination of both accuracy and reliability and that was this the "correctness" that I always thought was the most important feature of software.  But this correctness is really two software quality attributes and the Google search software exemplifies these two aspects of "software correctness", one is availability and the other is search accuracy or the fulfillment of the users expectations.  One could argue that availability is not a software attribute but it is affected by the quality of the software and it is an attribute of a software system even if some of that might be driven by hardware.   So these two attributes for Google search do not have to be perfect.  The search quality needs to be at least better than the competition and the availability has to be high enough so that people won’t be discouraged from using it and for these two attributes they are both very high.   These parameters of quality vary in other domains, a machine administering a dose of radiation to a cancer patient would need to have stringent requirements as to the correctness of the dose, and in fact there have been deaths due to exactly this type of problem.  Another is software guiding a spacecraft or a Mars rover which needs to be extremely reliable and correct and when it hasn’t been, well there are a few space bricks floating around the solar system due to software errors.  In the real world, especially now, software is everywhere and it now lives in a variety of contexts.  So which quality attributes are important will vary based on its domain and its context.

So what makes software good?  Google first conquered search with Map-Reduce and other processes running on custom commodity hardware that allowed for flexible fault tolerant distributed massive data crunching.  Amazon has commoditized its server infrastructure to sell spare capacity that can be easily used by anyone.  Apple created easy to use attractive interfaces for the iPhone and iPad that consumers loved with similar usability now being offered on Android.  NASA has successfully landed on the frigid world of Titan, touched the edge of the heliosphere, and has run rovers on Mars.  While there are obviously hardware components to these systems, it was the various types of software quality that played a pivotal role in these and the many other great software successes.

It is easy to look back at these successes.  With the exception of NASA, I doubt there was a deliberate effort on which quality aspects to focus on.  Most were probably chosen intuitively based on business needs.  Software quality is essential to any organization that creates software.  It is an established subdomain or perhaps a related separate sibling field of software development.  It has consortiums, professional organizations and many books written on the subject.  So I delved into some of these and you find mention of Agile and TDD.  You also start to see things like CMMI, ISO 9000, Six Sigma.  Some of the resources seem to be written in some alien language, perhaps called ISO900ese, that seems related to software development but leaves me feeling very disconnected from it.   I think that the problem might be that these attempts to formalize software quality as a discipline are suffering from the same problem that plagues the attempt to formalize software engineering as a discipline.  It conflates the development process with the final product and the software quality formalization might be further exacerbated by the influence of the domain of manufacturing and its jargon.  I think these established methodologies have value, for example makes sense, but it seems like in general with these standards there is a lot to weed through and the good parts seem to be hard to pick out.  So I thought it might be fruitful to look at the problem from another angle that might provide a better way to reason about the quality of a system not only after and during its development but also before development begins.

A list of "quality parameters" for software is a requirement for software engineering to be a real engineering discipline.  These parameters could then be viewed qualitatively and quantitatively and perhaps even be measured.  In engineering disciplines various parameters are tradeoffs, you can make something sturdier but you will probably raise the cost and or the weight, for example plastic bumpers on cars, metal would be more durable but would be heavier and more costly.  If one were to think of some parameters in software as tradeoffs you can see that improvements on some parameters may cause others to go down.  In my post "The Software Reuse Complexity Curve" I posit that reuse and complexity are interrelated and that the level of reuse can affect understandability and maintainability.  Another example is performance, increasing performance of certain modules or functions might lead to an increase in complexity which without proper documentation could lead to harder understandability which might cause a higher long term maintenance cost.

So to come back to the question and to rephrase it:  What are the attributes that constitute software quality?   I am not the only person to ask this question as I have seen other blog posts that address this. There are quite a few people who have asked this question and have compiled good lists.  One post by Peteris Krumins lists out some code quality attributes that Charles E. Leiserson, in his first lecture of MIT's Introduction to Algorithms, mentions  as being more important than performance.   Additionally Bryan Soliman’s post "Software Engineering Quality Aspects" provides a nice list with a number of references.  I thought I’d create my own list derived from those sources and my own experience.  I am categorizing mine in three groups, Requirements based (including nonfunctional requirements), structural, and items that fall into both categories.


  • Performance
  • Correctness (Requirements Conformance)
  • User Expectations
  • Functionality
  • Security
  • Reliability
  • Scalability
  • Availability
  • Financial Cost
  • Auditability


  • Modularity
  • Component Replaceability
  • Recoverability
  • Good Descriptive Consistent Naming
  • Maintainability/ Changeability
  • Programmer's time
  • Extensibility/Modifiability
  • Reusability
  • Testability
  • Industry standard best practices (idiomatic code)
  • Effective library use, leveraging open source (minimal reinvention)
  • Proper Abstractions and Clear code flow
  • Good and easy to follow architecture/ Conceptual Integrity
  • Has no security vulnerabilities
  • Short low complexity routines


  • Robustness/Recoverability
  • Usability
  • Adaptability
  • Efficiency
  • Interoperability/Integrate-ability
  • Standards
  • Durability
  • Complexity
  • Installability

This is by no means intended as a definitive list.  The idea here is to put forth the idea of enumerating and classifying these attributes as a way to reason about, plan and remediate the quality of software projects, basically create a methodical way to address software quality throughout the software development lifecycle.  In the case of planning software quality attributes can be potentially rated with respect to priority.  This could also allow conflicting attributes to be more effectively evaluated. 

The attribute of availability is one that is fairly easy to quantify and often gets addressed usually in some type of Service Level Aggreement (SLA).  Although availability cannot be predicted, the underlying system that "provides" it can be tested. Netflix has developed chaos monkey to in part to verify (test) this quality attribute of their production system.  Tests and QA testers are a way to verify the requirements correctness and the correctness of system level components and services contracts of software. Static analysis tools like PMD, Checkstyle, and Sonarqube can help with code complexity, code conventions and even naming. 

Complexity is an elusive idea in this context it also crosses many of the above ideas.  Code can be overly complex, as can UIs, Usability (think command line), Installability, etc.  Another cross concept is understandability which can affect code, UIs, etc. which is in part related to complexity but can manifest itself in the use of bad or unconventional nomenclature as well.  There may be other cross concerns and it seems that the spectrum of quality attributes might better reasoned about by separating out these concepts and developing general ways to reason about them.

I should note that I include financial cost as one of the attributes.  While it might not be a real quality attribute, higher quality may necessitate higher cost, but not necessarily.  If one were to develop two systems that had the same level of quality across all of the chosen attributes, but one system had twice the cost, the cheaper one would be a better value, hence more desirable. So cost is not a quality attribute per se, but it is always an important factor in software development. 

In summary my vision is a comprehensive methodology where the quality attributes of a software system are defined and prioritized and hopefully measured.  They are planned for and tracked throughout the life cycle and perhaps adjusted as new needs arise or are discovered.  The existing tools would be used in a deliberate manner and perhaps as thinking about the quality attributes becomes more formalized new tools will be developed to cover areas that are not currently covered.  It will no longer be about code coverage but quality coverage.

References and Further Reading

Anthony Ferrara, Beyond Clean Code,  http://blog.ircmaxell.com/2013/11/beyond-clean-code.html

David Chappell,  The Three Aspects of Software Quality: Functional, Structural, and Process, http://www.davidchappell.com/writing/white_papers/The_Three_Aspects_of_Software_Quality_v1.0-Chappell.pdf

Rikard Edgren, Henrik Emilsson and Martin Jansson , Software Quality Characteristics,


Bryan Soliman, Software Engineering QualityAspects, http://bryansoliman.wordpress.com/2011/04/14/40/

Ben Moseley and Peter Marks , Out of the Tar Pit, http://shaffner.us/cs/papers/tarpit.pdf

Brian Foote and Joseph Yoder, Big Ball of Mud, http://laputan.org/mud/

Wikipedia , Software quality, http://en.wikipedia.org/wiki/Software_quality

Franco Martinig, Software Quality Attributes, http://blog.martinig.ch/quotes/software-quality-attributes/

MSDN, Chapter 16: Quality Attributes, http://msdn.microsoft.com/en-us/library/ee658094.aspx

Advoss, Software Quality Attributes, http://www.advoss.com/software-quality-attributes.html

Eric Jacobson, CRUSSPIC STMPL Reborn, http://www.testthisblog.com/2011/08/crusspic-stmpl-reborn.html