Becoming generalist - dealing with constraints in software development

Recently I read the fascinating book - the goal, it is really an eye opener for me. I learned the theory of constraints from the book. Everything of TOC looks like common sense, but the conclusion is really striking.

TOC tells us the importance of bottle neck (constraint). We need to identify it, and protect it. In the example of manufacturing assembly line, to make sure the task in bottle neck is the highest priority, other jobs even have to be idle instead of working, otherwise it will make the whole productivity even worse!

I believe TOC can apply to software development as well: because the software developing also needs a series dependent activities. And similar to manufacturing, software development also has constraints(bottlenecks), it might be in: test,coding,deploy,maintenance. On the other side, compared to manufacturing, software development is people centered vs. machine centered; most of the constraints is about people; software process is lot easier to change compare to assembly line.
In software industry, we could have a big advantage over manufacturing: to deal with bottleneck, we don't need put other people to sit in idle, we can put more people working on the bottle neck! But this advantage depends on a requirement - if our developers are generalists.

If every developer in a team is a generalist, which would be a great help to maintain a high productivity, then we can deal with almost any constraint, since when there is a bottleneck, we can put more people on it. So the developing flow will be well controlled in a high productivity speed.

Becoming a generalist raise the bar for developers: we are not just a geek who focus on a single area, we are not only J2EE developers, but we also need to fix the bug of a J2ME application; we might need to mange data base as DBA; or improve the build system by writing Ant or shell script.

Becoming a generalist is really challenging, and much harder than focusing on one single area. But based on my experience, software developing job is not a rocket science, every technology can be learned if given an enough time. Technology is not a problem for becoming a generalist, the problem is attitude and mindset. We need to be open minded, be flexible, and ready to wear different hat at different context.

A generalist has following benefits:

• Doing different jobs, give developer a chance to learn different new technology and tools, learn by doing will make you learn faster.
• Have chance to know the big picture of your work.
• Better for team work and collaboration
• Improve your competence and marketability, since you can put more stuff on your resume.
• More changes to become linchpin and less likely being laid off, who wants to fire a developer who can do multiple jobs?

I will suggest our managers should encourage developers to become generalist. For example, he can deliberately assign different jobs for one developer; rotate developer to another team; foster pair programming and mentoring, etc.

Can software be estimated?

As a software developer, I am always asked by project managers:How long do you think you need to finish project X or fix  issue Y? Most of the time I feel annoyed and guilty, because usually I don't really know the answer but I have to give them  a number - which is far from accurate.

I believe how to get a good estimation is one of the major issue for all the developer and managers. An accurate estimation help manager control and plan the project, which make sense. But it is based on an assumption: the software can be estimated. Is this assumption really true? Let's investigate it.
First, if we assume software is estimatable, => then we can see software is predictable, => then we can plan ahead, => waterfall model. We all know that waterfall and CMM is not workable, try to harden the software project is myth.
Next, let's check the software development model. From Lean/Agile software development we know that software is a discovery process, an explorative process, a theory building process, an process of reducing ignorance level. In summary, it is a learning process, the problem solving process itself is also a learning process.  That is why agile uses iterative, feedback process. Which means at the beginning you know very little, you only understand the problem when you solve the problem at the end. So how could you estimate or predict at the beginning? How could you give an accurate estimation at the beginning? You can predict only if you know all the variables, but in software development, there are so many unknown variables: third party library issues, platform compatible issues, IDE, target constraints, etc and other issues you might not aware(you don't know you don't know). So based on analysis the above , we can say estimation in software is impossible.

Let's look at another question: can we estimate bug fix? Bug fix is also a learning process, especially when you are assigned a bug which code base is not written by you. You need to take time to understand the bug, reproduce the issue, understand the code base, familiar with the context, all this stuff are learning process and takes time. How can we estimate this time, because this is really case by case. And we all know that bug fix is actually we spend 90% of time on finding the bug, another 10% time on fixing the bug. Once we found the bug, we know how to fix it. So we can only see we can estimate the bug only after we found the bug, but at this moment it is pointless to give an estimation since we already solve the 90% of the problem.

I would like to share my own experience:In an iterative development process,  there is no need to estimate, just do it. Before I start fixing a bug, I knew nothing; the time I know the estimation answer is when I solve the issue, so the process of finding the answer of estimation is the same process of solving the problem. So at this case why we need the estimation?

 So we know that software nature is impossible to estimate and needless to estimate. But why project managers need it? Maybe they don't have technical background and still trying to use the concept of waterfall methodology; or maybe the psychological reason: human usually not comfortable with uncertainty and needs for closure .

Forcing estimation cause some worse consequences:

• Force developer to plan up front, which will fallback to waterfall;
• Developer faces the pressure to lie, and give the wrong number, which cause,
• Wrong measurement is worse than no measurement. Like weather forecast.

What can we do when the project manager ask you estimation next time:

•  Live with the uncertainty, that is the nature of the software development;
• To be honest, tell them you don't know yet, you only estimate what you can deliver after the current iteration;
• Timebox the estimation, spend 30 minutes to investigate the problem, you should know if you can finish it by the end of the day or not. Avoid yak-shaving.

Learning the theory of Constraint - "The goal" Review

Yesterday I finished the book:The Goal: A Process of Ongoing Improvement. The book is so fascinating, when I read it, I can not put it down, just like reading one of Agatha Christine's books. I never imagine that a business book can be written like this.
What I am most interested is the principle described in book: the theory of constrains. Here I copied some key points from the TOC WIKI:


Key assumption
The underlying premise of Theory of Constraints is that organizations can be measured and controlled by variations on three measures: throughput, operational expense, and inventory.
Throughput is the rate at which the system generates money through sales.
Inventory is all the money that the system has invested in purchasing things which it intends to sell.
Operational expense is all the money the system spends in order to turn inventory into throughput.
"The Goal" itself is to "make money". All other benefits are derived, in one way or another, from that single primary goal.

The five focusing steps
Theory of Constraints is based on the premise that the rate of goal achievement is limited by at least one constraining process. Only by increasing flow through the constraint can overall throughput be increased.Assuming the goal of the organization has been articulated (e.g., "Make money now and in the future") the steps are:

1. Identify the constraint (the resource or policy that prevents the organization from obtaining more of the goal)
2. Decide how to exploit the constraint (get the most capacity out of the constrained process)
3. Subordinate all other processes to above decision (align the whole system or organization to support the decision made above)
4. Elevate the constraint (make other major changes needed to break the constraint)
5. If, as a result of these steps, the constraint has moved, return to Step 1. Don't let inertia become the constraint.

Applying to software development
This book is about manufacturing, but I found it can be easily applied to the software development. And I found it is very similar with Kanban method: Kanban help us visualize the bottleneck, improve the productivity by limiting the Work-In-Progress. Which is exactly the same as the theory of constraints. I feel that the TOC can be regarded as the scientific theory behind the Kanban system. Before when I learn Kanaban, I just accept the principles and conclusion from the Kanban, but I never thought of why it is like this or that,  Since this book uses lots of examples and step by step from question to conclusion. Now I understand more about Kanban. So I can say Kanban is an implementation of TOC.
If you knows Lean/Agile, you will find there are lots of concepts shared between lean and TOC:

• lead time
• Inventory is bad
• WIP (Work-In-Progress)
• small batch increase the productivity
• local optimization is not working only optimize the whole works.
• The dependent event queue can be mapped to the value stream map, or Kanban Queue.

Even the subtitle of the book - "A Process of Ongoing Improvement", which is perfectly match one of the lean philosophy - Kaizen, which means continuous improvement.

I just found David J. Anderson's book: Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results. I haven't read that, but based on the title, it must be about applying the TOC into software development. In his recent blog post: REFLECTIONS ON THE PASSING OF ELI GOLDRATT,  he describe how TOC inspired him for his book and Kanban system.
Actually when I read this article from twitter last week, and this is the reason to let me read the book of "The goal".

And I found this youtube video is interesting: Standing on the Shoulder of Giants by Eliyahu M. Goldratt, Godratt talks about the TOC, Kanban and Toyota Production System.

I realized I need to read this book one more time. the TOC is so powerful, but I still have lots of questions, like:
- How can we apply it into our daily software development?
- How to find the bottleneck?
- What are the measurements we need?

Hope I can find the answers from the goal, kanban and Lean.