26 May 2011

Seven Attributes of Job Evaluation

I recently came across Erik Pettersen's "The Four Legs of Job Satisfaction", I have a similar list of my own which I have been using for the last few years to assess possible jobs when I am trying to decide about taking a new position and thought this was a good opportunity to share them.  Here they are in non-metaphorical form and in no particular order:

  • Compensation
  • Commute
  • Culture and management quality
  • The work you are doing
  • The people you work with
  • The weekly hourly load/life Impact
  • The physical work environment

Compensation is pretty strait forward and will include salary and benefits or strait fee or hourly rate for contractors.

Commute is generally thought of in terms of time, but can include cost and hassle, do you have to take two buses and transfer twice on the train?  Also parking probably falls under this one as well, but could fall under compensation if it’s included.

The work you are doing is in my opinion one of the more crucial attributes. Are you doing work that will move you and your career forward? Is it interesting and challenging? This one can vary over time depending on the phase of the project or which project you are working on at a given time.

Culture and management quality - I am lumping these two together, as they often go hand in hand but not always. Is your manager competent and/or a decent person? This can make a big difference in your day to day stress and enjoyment of the work you are doing. Also culture can affect quality of life as well.

The people you work with are pretty important and will be more of a distribution than an absolute value. If you are lucky the people that you mostly see day to day are people whose company you enjoy.

The weekly hourly load/life Impact - this one can overlap with commute but the bottom line is how much potential overtime will you get hit with and how often? Also the life impact can be emotional in that a frustrating situation can be draining and affect your quality of life and eat into your personal time if you are thinking about work when you aren't working.

The physical work environment is important.  Are you in a cube or office?  Do you have a quiet work environment?  Do you have any natural light? I also lump work neighborhood into this one. 

Now of course you can break these things out however you want, for example where would telecommuting fit in that list?  It could easily fit under commute or work environment.  The nice thing about this type of approach is that you can use it analytically to compare two jobs.  When I am planning a job change I often list these items out and give each one a delta value to compare them, for example a previous transition looked like this:

Compensation - Slight decrease in hourly rate
Commute + Still able to bike commute, five miles shorter. Same road danger factor.
Culture and management quality 0 About the same
The work you are doing +++ Went from doing legacy support work with Struts 1 to green fields RestfulSpring 3 & Hibernate
The people you work with 0 About the same
The weekly hourly load/life Impact 0 About the same
The physical work environment + Cube slightly worse but much better work neighborhood with good restaurants

Unfortunately it’s often hard to tell about some of these before you start but you have to do the best you can and you can ask about some of these like work environment, if you think about it.  Since the pay cut didn’t put me in a negative financial situation, the above transition was a no brainer.

I have wanted to share this in my blog from the time when I was just thinking about writing it and it’s a nice light break from some of my heavier topics.

17 May 2011

The Combinatorial and Other Math of the Java Collections API

The Java Collections API both involves some explicit Math and presents some nice artifacts which parallel other Math concepts.

Combinatorics is probably one of the more fundamental Maths for Computer Science and is also prevalent throughout other Maths. In one example of Combinatorial counting problems, there is a collection of items, say a deck of cards, from which we want to determine the number of selections, say hands. The formula to compute the possible Combinations is written as:

This can also be stated as “n choose k.” My mnemonic for remembering this, and it is worth remembering, is to look at the left side above note that the n is on top and the k is on the bottom, which visually maps the n! to the numerator and k! to the denominator, and then rotate the left part by 90 degrees counter clockwise so that it forms (n - k)! and put it in the denominator. Also every number calculated by this equation is a Binomial Coefficient found in Pascal’s Triangle, which in my opinion is one of the most amazing mathematical structures.

Our example for possible five card hands from a deck of cards computes as follows:

Combinations give you the number of elements with no repetitions and where order does not matter. The possible cases and Mathematical structures for the criteria of order and repetition are as follows:

  Order Does not Matter Order Matters
No Repetitions Combination/Set Permutation
Repetitions Multiset/Bag Enumeration/List/String

The Combinatorial equations are as follows:

  Order Does not Matter Order Matters
No Repetitions


The interesting thing about lists is that if you make lists of varying lengths of the following characters [0,1,2,3,4,5,6,7,8,9] , you have the natural numbers in base ten, so a list of length 3 gives you 103 or 1000 possibilities, which are: 000-999. Also lists have the highest cardinality of the four:

|List| > |Multiset| > |Permutation| > |Combinitation|

Where |object| means the cardinality or size of the collection, which is the number of elements in the set, list, etc.

The corresponding Java classes/interfaces are:

  Order Does not Matter Order Matters
No Repetitions java.util.Set java.util.SortedSet
Repetitions org.apache.commons.collections.Bag java.util.List/java.lang.String

Sadly the Java Collections API is incomplete in this regard. Also this example is somewhat abstract but it does demonstrate how to view these objects from a Combinatorial perspective, which in turn can help with choosing the appropriate data structure, also it gives insight into possible ways that they may be populated and perhaps even tested.

This is an introductory example to Combinatorics, for more perspective there is an expansion on this known as the Twelvefold way.

The next example is more concrete, in fact it comes directly from the java docs for Comparator and Comparable:

For the mathematically inclined, the relation that defines the imposed ordering that a given comparator c imposes on a given set of objects S is:
{(x, y) such that c.compare(x, y) <= 0}.
The quotient for this total order is:
{(x, y) such that c.compare(x, y) == 0}.
It follows immediately from the contract for compare that the quotient is an equivalence relation on S, and that the imposed ordering is a total order on S. When we say that the ordering imposed by c on S is consistent with equals, we mean that the quotient for the ordering is the equivalence relation defined by the objects' equals(Object) method(s):
{(x, y) such that x.equals(y)}. 

I have to admit when I first read that a few years back, I was like, WTF does that mean! Obviously whoever wrote that was thinking deeply and mathematically about the underlying principals here which is clearly needed if you are designing fundamental low level framework aspects of base classes and collections.

Comparators and Comparable are about ordering, so what is being defined above is a Total Order which means that all elements can be compared to every other element, compare this to a Partial Order where elements may or may not be comparable to each other, both of these fall under the more general Order Theory. The Quotient aka Equivalence Class concept is a bit more complicated and beyond what I am trying to talk about here.

I believe the above Comparator Javadoc quote was written by Joshua Bloch, the following is from his book Effective Java:

The equals method implements an equivalence relation. It is:
  • Reflexive: For any non-null reference value x, x.equals(x) must return true.
  • Symmetric: For any non-null reference values x and y, x.equals(y) must return true if and only if y.equals(x) returns true.
  • Transitive: For any non-null reference values x, y, z, if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) must return true.
  • Consistent: For any non-null reference values x and y, multiple invocations of x.equals(y) consistently return true or consistently return false, provided no information used in equals comparisons on the objects is modified.
  • For any non-null reference value x, x.equals(null) must return false.

The first three are the Mathematical definition of an Equivalence Relation (which relates back to the Equivalence Class) can be found in “Abstract Mathematics” or “Transition to Advanced Mathematics” type books which describe Functions and Relations. CS Majors usually are exposed to these concepts as part of the intro into discrete math.

What I find interesting about these concepts is how the comparator/comparable and equals relation and their interrelationship for the individual object map back to control the collections behavior of ordering and repetitions.

04 May 2011

Q:Are we not engineers?

A:We are developers.

Worst Post Ever

When I originally wrote this post I was trying to poke fun at a couple of posts that I thought were pretty lame, I have since come to the conclusion that my resulting post came out pretty bad, terrible in fact.

I also feel that within the original post was a kernel of a good idea that was bogged down by the rest of it, so I have rewritten this post using just the good parts.

Another unfortunate consequence of this post is that a Russian guy translated it and used it on his blog and linked back to it.  Due to the nature of the Page Rank Algorithm it made this post one the top results for Googling my name, as a result people interested in me were being directed to my worst post. This effectively became my Google problem.

The rewritten version is here.

If you are interested in the original post you can find the Russian version here, it’s pretty awesomely bad if you translate it, parts of it are quite funny actually.

02 May 2011

Monoid for the Masses

If you don’t know what a Monoid is, it is an Algebraic Structure, and before we go any further, let me just say: Don’t Panic! If you have a negative or skeptical reaction because the idea of Algebra makes you think of something that you learned in middle school and don’t really use, then you’ll be relieved to hear that you mostly don’t even need any prior knowledge of Algebra to understand the concept that I want to talk about here. In fact if you are programmer you probably already know it, you just don’t know that you know it.

So what is this mysterious Monoid that you already know? It is String concatenation. Take the following Java code for example:

String string1 = "aaaa";

String string2 = "bbbb";

String string3 = string1 + string2;

See I told you that you already knew it.

If your CS Math education was similar to mine, which I hope for your sake it wasn’t, you may have learned about Group Theory and it seemed completely abstract and unrelated to the programming related classes. Well it turns out it is relevant. Also whether you learned this or not I am hoping that you can follow what I am presenting here without any prior knowledge.

A Monoid can be thought of as a simpler version of a Group, and we will define it more formally, later. Our Monoid, strings under the operation of concatenation uses the plus sign [+] operator and a String will be bounded by quote marks ["], pretty standard stuff, and it isn’t even hard.

To talk about this properly we will restrict the character set to strings that only contain "a" or a "b", so that any string will only contain these two characters, in fact think of all strings that are all the possible combinations of only these two characters, so the Set is all strings that can have an "a" or "b" in any position, some examples are:

"a","b","ab","abb", "aaa", "bbb", "baba",...

You get the idea. This forms a Set of all possible "ab" string combinations. This Set is Closed under + (concatenation), which means if you concatenate any two strings in the set you get a string that is still in the set. So every new string is still made up of only a’s and b’s, for example:

"aa" + "abab" = "aaabab"

Concatenation is Associative, for example:

("aa" + "bb") + "aba" = "aa" + ("bb" + "aba") ="aabbaba"

Concatenation has an identity element, an Identity means that when you combine it with any element of the set you get the same element back, it’s like zero in addition, the Identity element is the empty string "", think of it like the string version of zero, after all it has zero length, for example:

"ab" + "" = "ab"

One thing of note is that the Set of all "ab" strings is actually all strings with either "a" or "b" plus (Union with) the empty string meaning that it also includes the empty string "", this is an important distinction. I did not mention it earlier to simplify the initial explanation.

To recap, Strings under concatenation form a Monoid and a Monoid is defined as:

  • A Set and a Binary Operation where the Operation is Closed on the Set.
  • The Operation is Associative.
  • The Set contains an identity element under the Operation.

So why isn’t this a Group? A Group requires an Inverse, in a sense a Group is a Monoid with an Inverse and there is none here, for example can you think of an X for:

"ab" + X = ""

Additionally you should note that concatenation is not Commutative, for example:

("aa" + "bb" ≠ "bb" + "aa") ⇔ ("aabb" ≠ "bbaa")

However, the concatenation of the identity is Commutative, it also referred to as a double sided identity:

"" + "ab" = "ab" + "" = "ab"

Why only [a,b]? Well to keep it simple, this is referred to as the alphabet and you can define your alphabet as anything you want such as [0,1], [a-zA-Z], all ASCII characters, etc.

The really cool thing about this Monoid is that it’s the basis for Formal Language Theory, and actually this is really a first lesson in Formal Language Theory and it is also good introduction to a basic concept from Abstract Algebra. Not to mention this could probably be taught in middle or even grade school.

01 May 2011

Driven Development

I have recently made the transition to working at a TDD (Test Driven Development) shop, I have no previous experience with it and I am still not sure if I am really feeling it, this may be based on my perception that in order to do it right it seems SO tedious that it is going to seriously harsh my creative coding buzz. Also based on my, albeit limited, knowledge of Combinatorics it seems like it should be called "Smoke Test Driven Development" as the number of possible combinations needed to fully test any software is truly astronomical. On the other hand I have seen it be very useful and the idea of a verification process for software is invaluable, additionally many respected Software Process and Design luminaries such as Martin Fowler and Bob Martin promote it.

I admit part of my problem is that there are probably a lot of things about TDD that I still don’t really get, actually one of my shortcoming as a developer is that I am really not that interested in software process, it’s the structure of software that gets me jazzed. Also I think I have not been lucky enough to see good process applied by good people and my lack of passion for this discipline has resulted in my taking the path of least resistance.

I have been doing this Software Development thing for quite a number of years, more than I want to say, and I have to confess I still don’t really have an explicit methodology, I have acted as a lead developer, although I hate to officially have that title, on many small and mid size projects over the years and in many cases feel that I have made the difference between success and failure of projects, I of course mean that the projects were considered successful, some of which were quite complicated, and in several case have been recognized for helping the success of the project, yet I feel that I am still essentially methodologyless.

Of my last three projects, two claimed themselves as Agile and one declared itself RUP CMMI level 3. The two Agile projects were not even close, they were mostly chaotic waterfall processes, the RUP CMMI project was true to its name, but the process implementation was ridiculously self absorbed in the process itself to the point of being essentially useless, as it was guided by the assumption that checking off all of the process step checkboxes meant the project would run fine on autopilot with people who had little or no software management experience and no thought need be given to the structure or architecture of the software. Admittedly being a seasoned developer this has left me a bit jaded and thinking: "Process, I don’t need no stinking process."

As for TDD I am still on the fence, when I first switched to it I did some light research on it and in the process I encountered BDD (Behavior Driven Development) this started to seem like a YADD (Yet Another Driven Development), so then I googled "Driven Development" to see what else was out there, this of course is still mostly TDD and BDD, but there are many new YADDs, the best one was a humorous yet enlightening blog entry by Scott Berkun called "Asshole Driven Development" which lays out the following (and many more in the comments):

  • Asshole Driven Development (ADD)
  • Cognitive Dissonance Development (CDD)
  • Cover Your Ass Engineering (CYAE)
  • Development By Denial (DBD)
  • Get Me Promoted Methodology (GMPM)

Ironically I think one of my Agile projects was a combination of Scott Berkun’s ADD and CDD, and what I would call RDD, Resume Driven Development, that’s where developers pick technologies solely to add them to their resumes, I was against this practice and reviled for it, thus I was on one side of the CDD conflict.

As I have wandered through the desert of pseudo-process for the last few years, I have been increasingly researching, observing and thinking about why am I able to succeed where others fail and what is the best way to do this, also there are many things I have observed where the management of the project was incredibly inefficient and dysfunctional, sometimes I would openly comment on and other times I would keep to myself.

I figured that I would jump on the YADD proliferation bandwagon and propose two of my own YADD’s, the first is my FDD, Framework Driven Development, the basic structural principles of which are outlined in my blog entry "What does your framework look like?"

In my recent experience I have noticed that software projects can become too focused on process and that process can become a magic bullet. It seems that in some cases process trumps structural concerns. The process component is easier for mangers to grasp and is more congruent with their formal methods, naturally. However, I think that the two need to be balanced, and I feel that something like my FDD concept is a step in that direction, as how to integrate it into process, that is something that still needs to be thought through.

The second YADD is what I would call DSDD, Developer Skill Driven Development. My previous ADD, CDD, and RDD "Agile" project was run by very young non-technical "managers", who had or were in the process of acquiring their PMI (Project Management Institute) PMP (Project Management Professional) suffixes, this one tends feel like a GMPM to me. Needless to say these "mangers" had only a rudimentary, at best, understanding of software development, I should also mention that this was government work so the tolerance for inefficiency was high, and there were a number of very good and good developers so the project mostly ran, as one developer pointed out, by itself. The most frustrating thing about the project was how the work was assigned, the better developers, myself included would often watch, sometimes in anger, as poor and junior developers were given the more challenging work, with which they always stumbled. Another developer would often say to me in jest, "We are all just 25 horsepower engines," I guess sometimes you have to laugh otherwise you’ll be crying.

I have since seen and also reflected back to previous projects in regards to how often managers seem to not understand or take into account the strengths of their team members, and it seems to be a reoccurring antipattern which I would call SDD, Slot Driven Development. (How many YADD’s are we up to now?) I think part of the problem is the management philosophy of divide and concur, where you split up the work into tasks, apply your "resources" and repeat.

DSDD is the idea that not all project tasks have the same level of difficulty and not all developers have the same level or set of skills and the two should be matched as effectively as possible. Another ramification of this approach is to probably a need rethink how project work is partitioned, think framework builder i.e. the 20%.

Of course I am just having some fun ranting here and am not really seriously proposing the creation of more YADDs. Some of my bad experiences may be due to the type of contracts I have taken recently, more on that later, but it does seem that in general software projects are poorly managed, please feel free to share your thoughts, experiences and YADDs in the comments.