Fibonacci: The Math of Agile

Agile planning poker¹ uses the Fibonacci sequence or in some cases a bastardized version of it.  I do not know the exact reason that the series was chosen, I do know the intent is that there is a standard set of numbers and using these deters the player from the temptation of using false precision, like a value of 10.5 for example.  Also it captures growth so each next number is a "unit" of growth larger than the last.  I suspect that the sequence of powers of two (1,2,4,8,16,32,64...) would also work, and might be more apropos to the computer industry, but these two sequences are similar in that they both increase by a constant factor, each subsequent power of two is two times larger and in the Fibonacci sequence each term is roughly 1.618... larger than the previous term, also each has a closed form exponential formula for each term 2ⁿ or for Fibonacci sequence it's Binet’s Formula, see below.   I don’t know if the Fibonacci sequence is the right way to go, but it seems to work and it is fairly prevalent in nature and art so it’s probably not a bad choice.  I confess that while this in one of my math of programming series, I am using it primarily as an excuse to write a post about some interesting results relating to the Fibonacci sequence and the Golden Mean, so if you are only interested in Agile you may want to treat this as a short post and stop here, but I encourage you to read on, the math is mostly high school level and it really is interesting.

The Fibonacci sequence is generally attributed to a man, Leonardo Pisano Bigollo who went by Fibonacci and did not actually discover the sequence, it was known about five hundred years earlier in the tenth century by Pingala and it was probably known much earlier.  Ironically Fibonacci’s greatest mathematical contribution was the introduction of Hindu-Arabic Numeral System to Europe, just imagine trying to do multiplication and fractions with Roman numerals.

The Fibonacci sequence is what is known as a recurrence relation or recursive sequence each term is defined as the sum of the two previous terms where the first two terms, which are the generator or seed values F₀ = 0 and F₁ = 1, the formula is:

It can alternatively be written as:

This form will be more useful to us in a moment.

The sequence is:

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393,...

The Fibonacci sequence has a very interesting relation with a very interesting number known as the Golden Mean which is sometimes referred to as phi and has the following value:

An irrational number that is approximately: 1.6180339887498948482045868343656...

The ratio of a Fibonacci number divided by the previous number in the sequence eventually converges to phi, the first few ratios are:

3/2	=	1.5
5/3	=	1.666666...
8/5	=	1.6
13/8	=	1.625
21/13	=	1.615384...
34/21	=	1.619047...
55/34	=	1.617647...
89/55	=	1.6181818...

As you can see it is already heading there.  It is interesting that a ratio which is in the set of rational numbers converges to an algebraic irrational number. The following equation describes this relationship, the Limit of consecutive quotients:

Additionally the following is true:

If we take the following formula:

And divide it by F_n we get:

We then take the limit as n goes to infinity, replacing the terms using the two limit equations from above:

And if we multiply by phi:

Solving for zero gives us:

Using the quadratic equation we get the following two roots (conjugates):

This is phi and its conjugate, which we will call tau:

Now that we have defined phi and tau we can describe Binet’s formula, mentioned above, which is an exponential form closed formula:

Binet’s formula is an easy way to compute Fibonacci Sequence values without having to calculate all of the values prior to the value you wish to calculate. 

Additionally there is an interesting matrix based calculation:

The above matrix raised to the nth power gives the above matrix of Fibonacci numbers, the first few are:

The determinant of this matrix is (-1)ⁿ which is described by Cassini’s Identity:

The Fibonacci sequence and the Golden mean have many interesting identities and relationships with other mathematical disciplines and natural phenomena, for example there is the famous relation rabbit breeding, it relates to bee ancestry, the packing of seeds in flowers, and other plant growth proportions. The Fibonacci sequence occurs in the shallow diagonals of Pascal’s Triangle, it relates to Pythagorean Triples. It relates to the similar Lucas Sequence , the Fibonacci Spiral form above which is a Logarithmic Spiral and the list goes on, a lot of this can be found on the previously linked Wikipedia Page and this BBC radio program.

¹ http://renaissancesoftware.net/papers/14-papers/44-planing-poker.html

Real Software Engineering: The Good, The Bad, and The Ugly

Deconstructing Glenn VanderBurg’s "Real Software Engineering"

Many years ago when I started my career, fresh out of school with a Bachelors in Computer Science, I chose to answer that ubiquitous question "What do you do?" with the title of "Software Engineer". Now you must realize that this was back at a time when any technical title including Programmer was as rare and mysterious as any engineering field to the average person who probably had never used a computer, not like today when everyone seems to have one in their hand at all times. When I was younger I always felt weird calling myself a programmer and I had a number of associations with other engineering disciplines, including starting my college studies in Chemical Engineering. Actually I was a chemistry nerd before I was a computer nerd, which you might have guessed as much if you read my previous post on Software Engineering. I also felt weird and perhaps even a bit phony using the title Software Engineer, I still do, so now I mostly go by Software Developer.

My original foray into the contemplation of Software Engineering lead me to do some more research which yielded an enlightening and informative presentation by Glenn Vanderburg called "Real Software Engineering", PDF Slides here, which attempts to explore ideas of what a real discipline of Software Engineering might be.  I strongly recommend watching it. It’s about 52 minutes long. I feel that his presentation has some very good points but it also has some flaws and biases.

The good

He starts with a critique about the state of software engineering today, which includes the following points:

Doesn’t reliably control costs.

Doesn’t reliably produce quality software.

Sometimes doesn’t produce software at all.

Software Engineering doesn’t work.

The term engineering is reserved for practices that work, a good capsule definition of engineering independent of any particular discipline.

The set of practices and techniques that have been determined to work reliably through experience.

Software engineering is a caricature of engineering.

In software we have these practices that don’t work and that is called engineering

I find myself agreeing with these, and sadly Software Engineering does seem to be a ridiculous farce that often seems to resemble a real engineering practice only minimally.

He then goes on to talk about the NATO Software Engineering Conference in 1968 which he cites as the origin of the waterfall process as a misinterpretation of a document called:
MANAGING THE DEVELOPMENT OF LARGE SOFTWARE SYSTEMS by Dr. Winston W. Royce who it seems inadvertently started the waterfall process through what might be called "Poor Information Design".

He continues with a quote from Parness about the document oriented path into engineering captured by the following quote:

In Enginering ... people design through documents.

-Parness

This is followed by a list that might be considered failed attempts at such an approach:

• CASE Tools
• OOSE
• VDM
• UML
• MDA
• OMT
• Booch

He refers to this as:

Modelling and what I call ‘Math Envy’

While I agree with this last statement in practice, I feel that he goes too far with this opinion, more on that later.

His presentation provides a good historical survey of Software Engineering with respect to the 1968 NATO Software Engineering Conference and is worth watching for that alone. He continues with some great observations about engineering in general:

Engineering is a creative activity it’s about designing and building and creating new things and there are always blind alleys and missteps and mistakes and discoveries and reactions to those reactions and adjustments.

I especially like the trial and error aspect in relation to creativity. I think this is a fundamentally important and potentially overlooked aspect of software development. 

The following succinctly sums up what might be the kernel of general engineering:

Different Engineering Disciplines are Different

• Different materials, physical effects, forces
• Different degrees of complexity in requirements, designs, processes, and artifacts
• Varied reliance on formal modeling and analysis vs. experimentation, prototyping, and testing
• Varied use of defined and empirical processes

This leads to a comparison between the empirical vs. formal predefined models:

The defined process control model requires that every piece of work be completely understood. A defined process can be started and allowed to run until completion, with the same results every time.

The empirical process control model provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs.

He then takes the following ideas from Structural Engineering:

Structural engineering is the science and art of designing and making, with economy and elegance, […] structures so that they can safely resist the forces to which they may be subjected.

—Structural Engineer’s Association

Science and art, it employs the findings of science and yet is done with creativity "designing and making" engineers don’t sketch something on paper and validate it with math and throw it over the wall and never look at it again they participate in the building of what they design structural engineers have hard hats in their office because they visit the site and talk to the construction people and work with them during the process. "With economy and elegance" cost is always an object.

From these ideas he derives a brief description of Software Engineering:

Software engineering is the science and art of designing and making, with economy and elegance, Systems so that they can readily adapt to situations which they may be subjected.

With the following caveat:

Software engineering will be different from other kinds of engineering.

We know that software engineering will be different from other kinds of engineering we know that because other kinds of engineering are different from each other. Software systems are usually more complex in terms of number of moving parts and things like that than artifacts in other engineering disciplines we’re not working with physical materials we have different laws we work with and it is rare for a software project to run up against the limitations of physical materials, very rare.

He reiterates this from the Structural Engineering piece:

Cost is always an object.

This is more eloquently stated by Arthur Mellen Wellington:

Engineering is not the art of constructing. It is rather the art of not constructing: or, it is the art of doing well with one dollar what any bungler can do with two.

—Arthur Mellen Wellington

The following are elements that in part drive the agile philosophy:

The source code is the model and the document.

Source code is the detailed design

The design and construction of software is tightly coupled.

He draws the following inferences from the above:

Encourage continued innovations in processes

And do like other engineering disciplines do. Take the processes that work and call those engineering.

Agile software development, far from being anti-engineering, represents the responsible application of engineering ideals to the field of software.

I know that was a lot of quotes with sparse commentary, but it seemed like the best way to capture the really good points that he makes in his talk and again I am trying to create a written artifact that I can reference and build upon in the future.

The Bad

One thing that I really don’t like about his presentation is the comparison of the two bridge builders: Robert Maillart who designed the Valtschielbach Bridge in Switzerland among others and Leon Moisseiff.

The distinction being that Maillart used testing and all but one of his bridges are still standing while Moisseiff used modeling most notably Deflection Theory and was responsible for designing the ill fated Tacoma Narrows Bridge aka Galloping Gertie possibly one of the most famous engineering disasters in history. Moisseiff was also involved in the design of Manhattan Bridge which is still standing and his Deflection Theory was used in the design of the Golden Gate Bridge and was later determined to be of sound design and is also still standing. The Frommers travel guide considers the Golden Gate Bridge "possibly the most beautiful, certainly the most photographed bridge in the world", what bridge Maillart design?

I will not dispute that the collapse of the Tacoma Narrows Bridge was probably one of the most famous if not the most famous engineering disaster of all time and Moisseiff certainly suffered as a result, his career was shattered and he died shortly thereafter. Wikipedia states: "Though he [Moisseiff] had designed many other famous spans, the Narrows collapse overshadowed them all. It became a symbol of failed engineering and the dangers of arrogance in design." I am in no position nor do I care to debate the soundness of deflection theory or the reasons for the collapse, actually it seems that reasons for the collapse are pretty complicated.

From my limited understanding of Civil Engineering it seems that problems with wind are a continuous problem for Civil and Structural Engineers, I know that the John Hancock Tower in Boston was plagued by problems where wind induced twisting which caused large plate glass windows to careen hundreds of feet to the street making it a deadly hazard.  These were replaced with plywood earning it nicknames like the "Plywood Palace" and causing it to be condemned as a fire hazard, eventually these problems were resolved and it won awards for enhancing the Boston skyline. More recently, only twelve years ago in 2000 the Millennium Bridge in London exhibited similar resonance problems to those of Galloping Gertie, it was shut down and reinforced.  So while all of this makes for interesting history of engineering fodder and good showmanship, I am not sure how directly relevant it is to software engineering. I Again I agree with his assertion that testing, perhaps not necessarily TDD in my opinion, is most likely a very important component in creating a real software engineering discipline but I feel this particular comparison argument is flawed as it is completely irrelevant to software engineering and in no way can be used to conclude that TDD should be used in software engineering. Also he contradicts his own argument with the statement, from above:

Software engineering will be different from other kinds of engineering.

He also contradicts this in his slides:

Software is very unlike bridges and buildings.

He ends his presentation abruptly with:

Today software engineering is called Agile.

I hate to say it but the way he states it, it sounds a bit like "Mission Accomplished" to me and I may be misinterpreting him here but to me it sounds like his whole presentation is designed to conclude that Agile and TDD are the solution to the problem and I agree that they are probably the best solutions to the problem today, but I think they are severely lacking as an effective and precise framework for a real engineering discipline, I think we still need more cowbell.

The Ugly

I feel that his presentation has anti-math and anti-academic undertones and I have to say: "No Sir I don’t like it." The two main themes that I take issue with are:

• He mostly dismisses the work of non practitioner academics.
• He dismisses Math and views it as only a cost saving tool.

At various points He states the following:

Advances come from practitioners not from academia.

"Advances come from practitioners." And then are brought into academia.

Important advances often from practitioners not necessarily from academia and they are brought into the academic world and are refined and so forth. But they come from practitioners.

They were practitioners … and there were a few academics, but the academics had good things to say as well.

Perhaps I am misinterpreting his tone here, but the in the last statement he seems surprised that the academics had something of value to contribute. Actually I feel that he has this somewhat backwards, for example I think the work that companies like Google are doing and the general trend in analytics fundamentally contradict this sentiment. I think the way that both the fields of Computer Science and Software Engineering will inevitably move forward in both efficiently and efficacy will be through more collaboration between practitioners and academicians with both groups refining each other’s ideas. I anticipate and hope this is a trend that we will see more of. In fact some really interesting things are happening in that regard one interesting example is the work that Mike McCandless has done with Lucene using Finite State Transducers which was based on academic work, not to mention the Entropy-Framework relationship that we have already looked at.

He goes on to say:

Mathematical Models were introduced to engineering as a cost saving tool it would be easy if you hear software engineering people talk about math and modeling, it would easy to get the idea that it’s the only way to do things right and that it is a tool for robustness and you don’t know it really works that it is really works unless have math to prove it, I’ve heard all of those statements. But, that’s not why it was introduced to engineering at all it was introduced to save cost because in other engineering fields they are working with real materials and things that require people and labor to go out and build and building prototypes and testing them especially at scale is extremely costly and if you can build a model, an approximation of reality you can save money.

Ask your friendly neighborhood aerospace engineer how much math he would do if building a model of his design and testing it were was effectively instantaneous and free. The answer would probably not be none, but a lot less than I do now.

I am not sure if I agree with the idea, that seems to be implied, which is that mathematical models are strictly a cost saving measure.

Towards the end he states the following are approaches needed to advance software engineering:

Not Math

Not Models

Not Documents

Not copying other disciplines

Learn From practioners

Bias towards empirical processes

Ironically he mentions the following:

Programming languages are formal languages some of them even have mathematically specified semantics ... and we understand the math behind them at least as well as your average structural engineer understands the math behind the procedures that he was taught to do to validate his designs.

Programs themselves are models of the solution to the problem, our very tools are mathematical in nature.

In the slides he has the following quote, which I think he skips in his presentation due to time constraints, but I believe it is intended to reinforce his anti math-argument:

What is needed is not classical mathematics, but mathematics. Systems should be built at levels and modules, which form a mathematical structure.

—Friedrich Bauer

I was unable to find it elsewhere, so I don’t know the context of the Friedrich Bauer quote, but I find the quote interesting. I feel the key here is: "not classical mathematics, but mathematics". My interpretation of this is that he is saying that math is important, but not the math many people think about, hence "not classical mathematics".

In Conclusion

I hope I didn’t come across as too harsh on Glenn Vanderburg but I am just being honest and as he said about Bruce Eckel, "He said it so I get to pick on him". 

The real strength of his presentation is that he really does step back from software and give some perspective on general engineering, and while we can all debate the utility of comparing different engineering fields it does hold that they all have a several common threads, including the application of artistry and creativity to design and construction, the objective to design and build things, working within certain constraints including cost, and the organization of people and resources to accomplish the tasks of design and construction.

I admit that my experience with real Agile is limited and while I like what I have seen. I still doubt that it truly fulfills very little beyond an empirical approach and it seems that its focus is primarily on the process of creating software, which is not necessarily a bad thing.  The structural aspect of software is not ignored by Agile, but I do feel that it is the structural aspects of software which is where we currently lack a real engineering framework.  This is really what needs to become more established.  Ideas like Coupling, Cohesion, Refactoring, Patterns, Anti-patterns, etc. are the state of the art in terms of structure but they seem too "squishy" which leads me to ask the question, is it not possible to have more comprehensive math backed concepts like the types of things you find in other engineering disciplines?

The answers to these questions in the past have been dominated by ideas like formal methods and metrics, ideas which have mostly failed to live up to a real promise, however, that does not mean that we should completely discount these and similar ideas in the future.  These and other more math oriented approaches may represent a failed lineage that will go extinct or they may represent the nascent ideas from which we are only now developing the theoretical and mathematical ideas to truly conquer these problems. For example the Nato Conference of 1968 is often cited as a milestone in the development of software engineering. It was only about forty years proir that Category Theory was discovered. I wonder how many attendees or general practitioners knew about it, unlike today where it seems that almost every software blogger writes about Monads and many also mention Category Theory.  I feel that the new era of software engineering will be forged from math but it won’t be "Classical Math" like Calculus and Algebra over the Field of Real Numbers, it will be from areas like Graph Theory, Lattice Theory, Category Theory, Information Theory and other related disciplines and not only will it be built on those disciplines but also the interrelation of those disciplines. I may be wrong about this but I have seen research that seems to point in these directions, ideas I will explore in future posts.  It seems crazy to assume that the failures of the past imply failures of the future. If math can make software cheaper and more reliable to produce it will become an inevitability.

You Can Control What You Can’t Measure

A couple of years ago, Tom Demarco published a brief article titled: "Software Engineering:An Idea Whose Time Has Come and Gone?". Now I know this is old news, and in it he claims that the role of measurement and hence control has not been a factor in some notable successful projects, he cites Google Earth and Wikipedia, now I cannot comment on those or many famous successful projects as I have no firsthand knowledge, however, I suspect that control has been important it just was not employed as he has anticipated and as he points out he is no longer involved in hands on development. So perhaps the premise of the famous quote "You can’t control what you can’t measure" is what is flawed.  I believe that the majority of successful software projects and successful companies built around successful software have exhibited at least some if not a large degree of control over the both the software construction process and the software structure.  I admit my assertion is tenuous at best as I have very little data to support this but I do have what I consider some interesting anecdotal support.

My own experience also tells me this is true, I am not well versed in the world of software metrics but what we have see m to be largely unwieldy and even potentially dubious (KSLOC) not to mention a lack of tooling and methodologies to apply them.  Still I feel that I can look at code base and make "aesthetic" judgments about good and bad, cohesion and coupling are fairly abstract, understanding them can shed light on how to refactor code to improve it and to give a general indication of quality.  In looking at code, both good and bad patterns emerge and can be utilized to assess and improve a code base.  One thing I can often do is reduce code size by refactoring general code into a framework and refactor it to more effectively use API’s like the Java API and API’s like Apache’s StringUtil and DateUtil classes or things like effectively employing the Spring Framework’s data binding.  Also code can often be better abstracted by composition, inheritance and general modularization and the use of patterns like the SOLID principles. These are all things that even the best developers occasionally miss, I often apply these refactoring to my own code as well, and if you are a good practitioner, what I am saying here is probably what you do every day.

While my personal control has mostly been limited to small and midsize projects and I can tell you that I have received some complements about my work to create cleaner more structured systems, my professional experience pales in comparison to that of others and I find their stories more interesting as you probably do also.  I’ll start with Steve Yegge’s "Done, and Gets Things Smart" post where he talks, not about his role, but his observations about what went on at Google:

... but I've realized that one of the Google seed engineers (exactly one) is almost singlehandedly responsible for the amazing quality of Google's engineering culture. And I mean both in the sense of having established it, and also in the sense of keeping the wheel spinning. ... Done, and Gets Things Smart folks aren't necessarily your friends. They're just people you're lucky enough to have worked with.

At first it's entirely non-obvious who's responsible for Google's culture of engineering discipline: the design docs, audited code reviews, early design reviews, readability reviews, resisting introduction of new languages, unit testing and code coverage, profiling and performance testing, etc. You know. The whole gamut of processes and tools that quality engineering organizations use to ensure that code is open, readable, documented, and generally non-shoddy work.

But if you keep an eye on the emails that go out to Google's engineering staff, over time a pattern emerges: there's one superheroic dude who's keeping us all in line.

The black bolding is his and the blue bolding is my highlighting. Clearly this demonstrates a pretty high degree of control.  The next comes from Martin Fowler’s "Who Needs an Architect?":

Architectus Oryzus [is the] kind of architect must be very aware of what's going on in the project, looking out for important issues and tackling them before they become a serious problem.

...

In many ways, the most important activity of Architectus Oryzus is to mentor the development team, to raise their level so that they can take on more complex issues. Improving the development team's ability gives an architect much greater leverage than being the sole decision maker and thus running the risk of being an architectural bottleneck. This leads to the satisfying rule of thumb that an architect's value is inversely proportional to the number of decisions he or she makes.

A similar sentiment is found in Lean Software Development: An Agile Toolkit By Mary Poppendieck and Tom Poppendieck:

Master Developers

In an extensive study of large system design, _[22] Bill Curtis and his coauthors found that for most large systems, a single or small team of exceptional designers emerge to assume primary responsibility for the design. Exceptional designers exercise leadership through their superior knowledge rather than bestowed authority. Their deep understanding of both the customers and the technical issues gain them the respect of the development team. Exceptional designers are people who are extremely familiar with the application domain and are skilled at communicating their technical vision to the development team. They are usually consumed with the success of their systems.

^[22]Curtis, Kransner, and Iscoe, "A field Study of the Software Design Process for Large Systems," 1272.

Another person who seems to support this is Joshua Block in his talk about API design, I have already blogged about that and the relevant quote is here. Also John Carmack of Id Software talks about controlling the software development process, among other things, and he makes many points that I think jibe with my control argument.

I my opinion these all show that good control does occur but it’s probably done more intuitively and it is more art than science and probably occurs without traditional metrics. That is why software projects can be insanely successful without measurement.  Actually it is my experience that control and success is not a binary relation in that control does not unconditionally imply success but rather it is a continuum such that control increases the probability of success.  Also I am not saying that software metrics is an idea whose time has come and gone, actually I believe quite the opposite, but that is another topic for another time. Another takeaway from these quotes is that having the right people and empowering them in that very critical role is a very important point in software management, again another entire topic.