Why I decided to take Scrum training?

Yesterday I registered an Certified Scrum Master training.  I choose the seminar on Jan 12, 2012  from Berteig Consulting.  Actually I am not so interested in SCRUM, rather I am super fan of Lean/Kanban, XP and software craftsmanship.
I have a bias on SCRUM because I feel its process is oversimplified, and focus on manager side only, ignores developer side; and it lack the whole picture of Lean and flexibility of Kanban. Another reason is so far I did not find a good SCRUM example in my past experience, so I really doubt the its performance.
The third reason is recently I found many articles which pointed out the issues of SCRUM.

As a many years of working experience as a software developer, I really doubt a certificate is useful.  I don't think that you will become an SCRUM master just for 2 days training, which means master Agile philosophy is too easy.

But recently I changed my mind, and decided to get a SCRUM Master certificate. You may wonder why I changed my mind so quickly.

Why learn Scrum?
1. I realized that the current IT situation is that  Scrum is still the de-facto process, even though Lean/Kanban is really hot inside the Agile community, but outside of the community, Scrum is still the most popular one. I will say 90% of the IT managers does not know Kanban. They are  more familiar with SCRUM.  If you have a Scrum training or get a certificate, it will be helpful for your career path. My long term goal is Agile coach, so I hope I can use CSM as a business card to help me find better jobs.

2. I hope the CSM certificate can help me make changes in my company. 6 month earlier, I tried to suggest to my managers thinking of using Lean/Kanban, they are familiar with Scrum, but not Kanban. I found I get stuck, because I don't know how to persuade them, since neither I have Kanban experience, nor I have Scrum certificate. Now there are 3 Scrum Master in our company. I hope if I have a Scrum certificate, then I could use their familiar "Scrum" language to communicate, then it would be easier for me help the company to transform from Scrum to Lean/Kanban.

What do I want to know about Scrum?

• Clear the misunderstandings about Scrum;
• Focus on the principles and philosophy behind Scrum;
• Figure out the fundamental assumption which support Scrum;
• Find out the good and bad parts of Scrum;

Cynefin framework and software code

Recently I read some article about Cynefin framework, it is really inspiring. As the diagram shows, cynefin framework categorize the problems in to 4 domains:

• simple
• complicated
• complex
• chaotic

For different domain, the approach and strategy are quite different.
Today I would like to analyze the software code using cynefin framework.

Which domain does software belong to?
In General I will say software belongs to complex domain. One reason is software product environment changes so fast, the other reason is its entropy nature: always grow from simple to complex. Software product is so hard to predict and plan. That is the reason why we choose agile methodology: we follow TDD, keep refactoring, iterative developing and continuous integration. These methods are to deal with rapid the software's rapid changing context.
Let's compare the complex domain with agile practices:

complex, in which the relationship between cause and effect can only be perceived in retrospect, but not in advance, the approach is to Probe - Sense - Respond and we can sense emergent practice.

Here The probe - sense - Respond is same as TDD and iterative development, since we don't understand the domain, so we have to rely on immediate feedback.
And the emergent practice and retrospection, which is the same as refactoring, refactoring itself is emergent practice.

We can also conclude that the reason why waterfall does not working is because they assume the problem domain in software is always in simple domain.  

How about other domains?
If we don't care code base and let it grows, then code base will becomes more and more complex until it becomes chaotic. As a developer we all have the experience of working with messy legacy code base: the messy code is hared to understand, hard to maintain; you fix one issue, then will generate another issue unexpectedly; the software is unstable, a small change will cause a big issue; the issue is always unpredictable;nobody understand the system; we spent all the time putting fires. Which is totally unstable and uncontrollable. These symptoms fit nicely with the chaotic domain model.

But if we keep writing clean code, the code are easy to understand, easy to maintain, easy to change. In agile world writing clean code also means simple design. So this  will fit nicely in simple or complicated domain. Since they are both ordered. so they are manageable and controllable. That is software's ultimate goal -  flexibility, maintainability and reusability. So we will say clean code will help the software reduce the complexity.

Why clean code will help reduce the complexity?
The following tools and method help us writing clean code:

• SOLID principles
• high cohesion/loose couple
• Design patterns
• DRY principle
• LoD principle
• Refactoring

Lets' dig them deeper:
- Break the unnecessary dependencies

   Uncle Bob once said writing software is all about dependency management. SOLID principle, high cohesion/loose couple means reducing dependencies. We all know that the more interactions between objects, the more complex the software will be, so reducing the dependencies means making the software simpler.
- Set Rules and Constraints
  Design patterns basically is describing the rules of the relationship between objects in a specific context. To make software reusable, stable and manageable, we do not allow objects interact each other randomly, they have to follow certain rules. This is quite similar in human society, we have to maintain the orders via laws, rules or regulations. 

- Encapsulate changes

   One of the advantage of design patterns is how it deals with change: encapsulate changes. Which means when we identify a concept which it changes, then we encapsulate it by putting it into a class.  we don't want them randomly change. we put them in a box, isolate them. Once we put those details evil in the box, then we can understand or control the system by using abstraction. Then the system becomes stable, manageable and controllable

Conclusion

• In General software belongs to complex domain. The agile practices are the right practices to solve the complex domain problems;
• Writing clean code will help us reduce the complexity, will move from complex to simple/complicated domain;
• Writing messy code will increase the complexity, will move from complex into chaotic domain, this is what we are trying to avoid.

Strategy
So our strategy will be constantly refactoring our code base, keep code clean, avoid messy code.

Resources

1. Cognitive Edge
2. Systems Thinking and the Cynefin Framework
3. Video by Dave Snowden - Practice without sound theory will not scale

How javascript call java in Phonegap android

Phonegap is a great mobile solution, since I am working in Grapple mobile now, our platform use quite similar solution with Phonegap, so I really want to learn Phonegap to see if we can steal some ideas from it.
What troubled me a long time is in its android platform: I wonder how does its javascript code calls java native code? Before I tried to read its android source code, but did not find the answer. This week I spent some time on its source code, finally I find the answer. I would like to share this result with you, this might be valuable especially if you want to dig more to phonegap, since I seldom find the answer from google or books about phonegaps, none of them talks about this.

The phonegap source code I checked out is from github, the current version is 1.2:
1. phonegap android source code: https://github.com/callback/callback-android
2. phonegap platform sample code: https://github.com/callback/phonegap

Digging the source code
From phonegap-1.2.0.js

For example looking at accelerometer:  in javascript layer, the accelerometer is implemented by calling Phonegap.exec() method:
The PhoneGap.exec() method will be like this: So the exec method will invoke a javascript prompt()- which is popping up an dialog, huh, it sounds weird.

Let's look at the java code:
Inside com.phonegap.DroidGap.java, find the inner class GapClient, which is a subclass of WebChromeClient, it override the following 3 methods: onJsAlert(), onJsConfirm() and onJsPrompt()
Here we only look at the onJsPrompt(), look at the android API for onJsPrompt():

Tell the client to display a prompt dialog to the user. If the client returns true, WebView will assume that the client will handle the prompt dialog and call the appropriate JsPromptResult method. If the client returns false, a default value of false will be returned to to javascript. The default behavior is to return false.

So it is quite clear that: if prompt() is called in javascript layer, then onJsPrompt() method will get invoked in java layer. Phonegap just uses this way to handle the javascript to java communication.

Let's look at the onJsPrompt() method in details:  So here the phoneGap PluginManager will delegate the the javascript call to different native plugins.

From the article of http://stackoverflow.com/questions/7251557/phonegap-and-prompt
and phonegap release note: http://phonegap.com/2011/04/28/phonegap-0-9-5-released/, it mentioned that

Worked around JavaScript bridge exception for Android 2.3. Use “prompt” instead of calling objects directly.

So it seems that this is a workaround for an issue in Android 2.3.

Summary
To recap, this is the flow of javascript call java in phonegap android is like this:
1. in javascript layer, all the javascript method will finally delegate to Phonegap.exec() method
2. PhoneGap.exec() will call prompt()
3. In java layer, DriodGap.GapClient override onJsPrompt(), which will capture the prompt() method call in java, which then delegate to PluginManager to invoke the native java method call.

3 types of work - yesterday, today and tomorrow

I found it is useful to describe software developer's daily job into 3 different types: yesterday, today and tomorrow.
Yesterday: bug fix, maintenance job. since this kind of job is based on the product which is supposed to be finished, so I describe it as yesterday.
Today: developing phase, the work on the product is under development,
Tomorrow: some research and R&D work.
These 3 different type has different nature, and they need different mindset and strategy to deal with them.
Let me dig them into deeper.

Yesterday's work
Yesterday's work are bug fixing and maintenance job. This kind of job is putting off fire, they are urgent, and undeniable. We need to fix them ASAP.

Yesterday's work has no value or negative value, because customer already paid us up front, this work is support and service. This kind of job is not adding value, but removing the negative value.
The more time and effort we spend on the yesterday's work, the less value remained for us. The cost of fixing a bug in production phase is much expensive than fixing it in developing phase( today's work).

Yesterday's work is quite stressful. A minor issue delayed in yesterday will cause higher priority work. Yesterday's job is quite critical for time. we just need the job get done, there is no time for refactoring, no time to focus on better design. There is no room for developer to improve the system (code base, design, code quality etc) at this stage.

If there are too many yesterday's work, then this is not a good signal: It means we have a large amount of technical debts from today's work, it means bad design and careless decision.
So yesterday's job is highly dependent on today's work.
Ideally we should keep the number of yesterday's work as low as possible.

Today's work
Today's work is about current realistic issues. Today's work is the tasks which is belong to developing phase. We develop a new feature which is our customer wanted, so this job will bring values. They are urgent and important, but some features can be negotiable. Customer will focus on feature than time, so the time is not so critical as Yesterday's work.
Developer has lots of room to improve the system, he can have time to improve the design, refactoring.

The strategy of today's work will be - Build quality in. Focus on quality over time. If we focus on quality at this phase, then time must be shorten on maintain phase.
Quality means more than functionality, also means maintainability and readability.  We should focus on simple design, write clean code, keep refactoring. and do not easily leak the bug to maintenance phase.

Tomorrow's work
Tomorrow's work is focused on research and innovation. It will bring us potential value in future, so I call it tomorrow's work, means work for tomorrow.

It for a long term goal, we might not realize its value at present. it's important but not so urgent.

Tomorrow's job is like investment. there is a risk, we don't know how much value it will bring us.

Usually the priority of tomorrow's work is less than yesterday and today, if there are too many work spending on yesterday and today, then there is no time for us on investment. But if we did a good job on today's work, then we need less time on yesterday, then we have more time spending on tomorrow.

Mapping to First thing first
I found these 3 types of work map perfectly with Stephen Covey's first thing first :
Yesterday - Urgent but not important
today - Urgent and important
tomorrow - Not urgent but important

Conclusion
Yesterday is your debt, today is your earning, tomorrow is your investment.
We need to focus on today's work, keep improving the quality of today's work.
If we want to be productive, we need to balance these 3 types of work wisely.

Be careful "this" in Javascript Object

Consider the following javascript code:
What if I remove the this in method increment()?, like this:

I must admit at first I wrote the code like the second, But it was wrong.
As a Java programmer, I thought keyword this in javascript will like java this always refer to the object owner, so this can be omitted. But I was wrong, the code above is wrong, because javascript requires you to use this keyword explicitly, otherwise it will treat variable value as undefined. It seems javascript will treat variable value as local variable, except you specify this explicitly.

From this mistake, I understand that the experience in java is not working in javascript, if you want to access the class level variable in javascript method, you have to explicitly using this. Be careful for "this"!

The function definition in Javascript

I highly recommended Douglas Crockford's book "Javascript:The good parts". In the page 133 of the book, the author mentioned 2 ways of defining a function:

• use function statement,
• use function expression

Their results are exactly the same. For example,
Function statement:

Function expression:

The above two have the exactly same result. they will both define a function foo. But I found there are still one difference: scope.
The function statement is globally scoped, it can be put anywhere in the javascript, and you can call the function before or after it is defined.
For example:

But in function expression is different, the function call statement must be after the function is defined, otherwise the function call will be failed.
For example:

When you think a little deeper, the error of function expression make sense, because the function expression means first declare a variable, then assign it with a function object. Why it failed before the function defined is because the variable is not declared and assigned yet.

So if you use the function expression to define a javascript function, you need to aware this issue.

Learning javascript: understanding prototype property

Prototype in javascript is so important, because javascript implements the object inheritance via prototype delegation. Here I tried to summarize what I found for the javascript property. The examples I used are tested on Firefox Javscript shell .
Please note that in Javascript function is also an object, so here function and function object are interchangeable.

1. Every function object is automatically created with its own prototype property, the prototype property itself is an javascript object.You can add property or functions to the prototype property via function object.
Example:
2. When a object is created by using new keyword, it will get the prototype object from the constructor function;  created object can not access prototype property directly like function object, but it can access the property of the prototype object:
example:

3. When the created object instance read a property of the prototype, which is directly linked to the prototype object of its function constructor; while you want to modify the property, you end up by creating a new property in the object itself, which will not impact the same property name in the original function prototype. I believe javascript in this way implements the OO inheritance and overriding.
Example:

4. The function object prototype property, which itself is an object, its default value contains only one hidden property: constructor, whose value is the same as the function itself.
Example:

Code is constraint

Last week I watched Michael Feathers presentation: Code blindness.This is a great presentation, and it inspired me to think deeper.

Before I think code is important from the programmer's perspective: craftsmanship, and professionalism. But it seems quite far to the managers. They don't know understand it and don't know how to connect this with their daily management.

Today I would try to think from manager's perspective: borrowing the idea from theory of constraints, I will say code is just another constraint, like other constraints manager usually considering: time, budget, people. More specifically code quality will impact developers productivity and on time delivery.

We know that the clean code base will easier to understand, easy to make change, and easy to maintain,but bad code base will hard to understand, hard to change, and hard to maintain, which greatly slow down developer's productivity.To avoid this happens, we have to deliberately keep code lean.

There are two main metaphor to support the clean code:

• Metaphor of technical debt: code base is like technical debt, we have to pay off first by taking effort to refactoring, otherwise it will become harder and harder to change and maintain in future, so refactoring is the interest we pay for the debt.
• Metaphor of software entropy: code base is like entropy, by nature it will gradually go to messy, we have to constantly refactoring and keep our codebase clean.

So we can get the conclusion: code can be asset if it is clean; and can be debt as well if it is messy.

So far we know that code is very important to developer's productivity, it should be consider as a constraint by the manager. But I found many managers are totally unaware of this issue:
They just assume developers are all professional and their code will always good enough;
They only care the schedule, only care something get done, but does not care if the code is well maintained or not.
And since managers unaware or ignore this issue, they might make decisions to make code base even messier, for example,
- they will pressure developer to push them make quick-and-dirty fix, which cause the long term issue;
- They do not care for the code, so they measure the performance only by task been finished, not by how good the code is written, so we can imagine developer will not actively care for the code.

Code is important, but if you unaware or ignore it, then you will not manage it.
If you don't manage it, then you will finally lost the control and then we will get real problems.
When you have the problems, but you don't understand the root cause is code quality, then you probably trying to solve the issue in a wrong way.
I heard so many times from managers said something like "we need to work harder", "we should work more careful", which are totally useless slogans.

Suggestions

1. Manager should aware the code quality's importance, bring the connection between good code quality to developer's productivity and developing schedule.
2. Manager should cover code quality into their daily management, right now there are so many different tools to do code analysis and measurement.
3. Manager should create a culture to encourage software craftsmanship, clean code is a foundation of good software product,  clean code fit nicely the "build quality in" lean principle.

Making developer multitasking is a bad idea

Following is copied from a job description for a mobile software developer from my previous employer QuickPlay Media:

"Well organized with the ability to effectively manage multiple projects simultaneously. Extensive experience in a fast-paced deadline-oriented environment".

When I first look at it, I was really surprised by the word "multiple" and "simultaneously", it is pretty obvious that they requires developers to be able to do multitasking jobs. Today I would like to dig this topic a little bit.

What does "manage multiple projects simultaneously" exactly mean?
- Working project A for 5 minutes, then working on prjoect B another 5 minutes, and project C for another 5 minutes?
- Or when you are working on project A, then interrupt by project B, you have to suspend the project A, then switch to project B;
- Does it mean the developer has more project working simultaneously, the better for his performance review?
- Does multitasking really improve the productivity? get faster delivery?

This idea is totally wrong
In the last 2 years I read a lot books about agile and software craftsmanship, so I am quite confident to say this is totally wrong, it will only make things worst!

Fist it is against human nature. Human is not machine, he is non-linear, a person can only focus on one thing at a time. Interrupt and context switching is very expensive. The book Pragmatic Thinking & learning and  peopleware cover this topic, and explain why managing focus is so important to maintain a higher productivity. and that is the reason why the Pomodoro technique will force you only focus on one task in each 25 minute session. Managing people's focus is the key of improving his productivity.

Second it is against the agile principles and and other proven theories. The goal of agile is to deliver the product as fast as possible, but the interesting thing is the agile trying to reach this goal by limiting the Working-In-Progress items (WIP).  Scrum is trying to limit WIP in one iteration, WIP is called inventory in Lean, which is a waste and should be limited. Limiting WIP is the key concept of Kanban technology. and in theory of constraints also proved that putting to many task in the queue will only slow down the productivity.The reason is pretty obvious, think about this scenario, if you have 3 projects, each will take one week. If you work sequentially, you will deliver one project after 1 week, 2 projects after 2 weeks, and you will get 3 projects after 3 weeks. But if you work simultaneously, you will get any project done until after 3 weeks, but considering the context switch cost, it will take you longer time to finish all 3 projects. So based on Scrum, or Lean/Kanban, or Theory of Contraints,  working simultaneously will not deliver fast, it will only delay the delivery time.

Third is it is a shift burden to from mangers to developers. Managing backlog is manager's responsibility, but requiring developer multitasking means putting some more backlogs into developers. It might looks better for managers, but actually it is bad, since those extra backlogs are either under working or putting into developer's own backlog. This is actually hide the visibility, which is even harder for manager to track the progress. I can feel that the multitasking is a solution which managers are trying to solve too many backlogs, but it is a bad idea, since it only move the backlog from manager layer to developer layer, which making things even worse; the manager should figure out the root cause and find a true solution to increase the burden down rate or increasing the productivity. I like to call this issue is leaky management issue(copied the idea from leaky abstraction from OO design), because it is supposed to be solved in the management layer, but it leaks to the developer layer.

Based on the above analysis, I can feel multitasking will generate following result:
developer will have low productivity, projects will get more delayed, developer have to work harder and over time; since more projects get delayed, so manger put more projects to developer to work simultaneously, then becomes a positive feedback loop, which will make things even worse...

Summary
I feel very sad for my old colleagues, because they are not treated well;  and I feel sorry for my ex-managers, because they are unaware of the basic principle of agile. To make a developer working efficiently, the only thing is trying to make him to work on one thing only. I strongly suggest QuickPlay to remove multitasking in its job requirements, but removing that from managers minds are much harder.

learning clojure: how to compile clojure source code

Currently I am learning clojure now, I focused on the book "The joy of Clojure", but actually I did not enjoy the joy of clojure so far. For me the book is still difficult. As the author mentioned, this book is not for beginner. Most of my time I have to refer on the Clojure API document. Today I want to share my experience on learning chapter 10.2.1, how to compile a clojure source code.

The source code is in page 212, listing 10.3, called DynaFrame.clj:

The author mentioned how to ompile the source code:

 You can compile this namespace by saving it in a directory joy/gui, located on the classpath, in a file named DynaFrame.clj and executing the function (compile 'joy.gui.DynaFrame) in a fresh REPL.

As a clojure beginner, I found it is described not detailed enough, it took me almost a night to figure out how to build it successfully.
After fixing different weird problems, finally I got it working. Here is what I found:

1. Be careful clojure version: at first I use clojure 1.0 to launch the REPL to compile it, but I got Exception to complain not found java.lang. Later I realize the problem is in ^{:static true}, you have to use clojure 1.2 to compile it; if you still want to use clojure 1.0, then use #^{:static true} instead.

2. Use cljr tool, I found it is really handy to add classpath: just run cljr add-classpath <your folder>; run cljr list-classpath will display your classpath.

3. Add source code folder in the classpath
   put DynaFrame.clj under joy/gui, and
   run cljr add-classpath ./

4. Add *compile-path*
   create an directory called "classes", and
   run cljr add-classpath classes/
   Based on the clojure API document:*compile-path* defaults to "classes", the directory must be in the classpath for 'compile' to work.

5. run cljr repl to launch the REPL,  type
    (compile 'joy.gui.DynaFrame)
   and
    (joy.gui.DynaFrame. "1st")

In summary, to compile the clojure code successfully, you need:
 - Add source folder in the classpath;
 - Create and add "classes" folder in the classpath

Developing Blackberry application using NetBeans Plugin - part 2, fixing debug issue

On my previous post, I reviewed the new blackberry Netbeans Plugin. Latre I found Shai also published his post about how to use the new plugin in developing the LWUIT blackberry application.
But I found the new plugin could not debug the blackberry application.

Reason
The reason is if you want to debug your blackberry application, you have to copy all the .debug files into the blackberry simulator folder, include xxx.debug, xxx-1.debug, xxx-2.debug, ... etc. From the generated build-impl.xml script file, we know that blackberry plugin only copied the xxx.debug file and missing all other debug files.

Solution
Add the following targets in the build.xml, which will override the same target in the build-impl.xml:
1.  Update target blackberry-pre-debug, it will copy all the debug files into the simulator folder.

2.   Update target clean-blackberry, it will delete all the files of the application in the simulator folder.

3.  If you want to build BlackBerry CLDC application, you need to add the target create-cod:


The only difference with build MIDlet application is in <arg value="-cldc"/>, by default, the Blackberry plugin already choose MIDlet by using <arg value="-midlet"/>.

Developing Blackberry application using NetBeans Plugin - part 1, first impression

Updates: I got some feedbacks that the download link in Netbeans is not working anymore, I uploaded the plugin binary into my google doc, please try my link if you have problem to download.

Recently when I checked out the latest version of the source code of LWUIT, I realized that LWUIT used a new solution of developing blackberry application - using Netbeans Blackberry plugin. So today I tried to give an review about this new tool.

Plugin Installation
1. go to this link (or my link)to download the blackberry netbeans plugin, save it in your local drive.
2. In your Netbeans IDE, choose Tools - Plugins, click Downloaded tab, press "Add Plugins..." button, choose the downloaded blackberry plugin .nbm file,
3. Restart your Netbeans IDE, the thing is it seems the blackberry plugin has an issue, you can not see it under the Installed tab of the Plugins UI Form. I was confused it at the beginning as well.

Add BlackBerry SDK platforms
The plugin download link (or here)gives some screen shots about how to add blackberry SDK platform into your NetBeans IDE environment, here I just give an rough description, if you are familiar with the steps of adding other J2ME SDK into the NetBeans IDE, you will know it is the exactly the same steps.
Compare this step with the older solution, you will feel how easy it is.

After Adding the BlackBerry SDKs in your NetBeans IDE, you can use the same way as J2ME Mobility package for blackberry development. Really Nice!!! Good job for the author!

Test
To test how the Plugin works, I choose 2 tests, one is for MIDlet and the other is for CLDC application (RIMlet).

Test1 - build an BlackBerry MIDlet application
I imported an JavaME SDK sample application - SunSamples-Demos into my NetBeans IDE, then add another configuration, choose an BlackBerry platform, I just choose JDE 4.5, then everything else is copied from the default configuration.
1. Test Build
Click "Clean and Build", the IDE build successfully and generate .cod file and bunch of .debug files.
So Build test passes.
2. Test Run
When I click "Run", the IDE launches BlackBerry simulator, then I found the several applications (since SunSamples-Demos contains multiple MIDlets), I choose some apps, they work perfectly.
Application Running also passed.
3. Test Debug
I set several break points in the IDE, then I click "Debug", the IDE launches the BlackBerry JDWP application and the blackberry simulator, when choose an application, the application did not stop at a specific break point.
The debug test failed.

Test2 - build an BlackBerry CLDC application
This time I choose a LWUIT blackberry port demo application - LWUITDemoRIM, just check out the LWUIT source code from its SVN, open the LWUITDemoRIM project. If you installed the plugin and added the blackBerry SDK properly, it is very easy to configure the project in the NetBeans.
Here I would like to mention, for native CLDC application, you need to do a little extra work in your build script. Check this link for details: http://forums.netbeans.org/topic24475.html

1. Test Build
Click "Clean and Build", the IDE build successfully and generate .jad file, .cod file and bunch of .debug files. If you open the jad file, it will tell you the application is CLDC application.
So Build test passes.
2. Test Run
When I click "Run", the IDE launches BlackBerry simulator, you will find the familiar LWUIT demo icon, the application works perfectly as I saw before.
So running test passed.
3. Test Debug
When I click "Debug", I saw the same behaviour as the MIDlet application test, the application never stops at the break point.
Debug test failed as well.

Conclusion
Pros:
- The Blackberry Plugin for NetBeans is pretty easy to install and use. The developer will feel it looks pretty same as other J2ME SDK.
- The Build process is really simple and almost same as regular J2ME application, it will convert jar file into blackberry .cod file.
- The IDE can launch Blackberry native simulator and can run the application successfully.
- The plugin built-in support MIDlet build, need a little work for RIMlet.

Cons:
- Debug is not working, after I read the build script, I know why: based on my experience, you need to copy all the .debug files into the simulator folder, but in build-impl.xml, the target "blackberry-pre-debug" only copied 1 .debug file but missing others.  I will post the detailed solution in future.

Next Action
In future, I will continue post my investigation about this plugin, includes:
- How to build the MIDlet and RIMlet
- Dig deeper on how it implement the blackberry build
- How to fix the debug issue

Becoming generalist - dealing with constraints in software development

Recently I read the fascinating book - the goal, it is really an eye opener for me. I learned the theory of constraints from the book. Everything of TOC looks like common sense, but the conclusion is really striking.

TOC tells us the importance of bottle neck (constraint). We need to identify it, and protect it. In the example of manufacturing assembly line, to make sure the task in bottle neck is the highest priority, other jobs even have to be idle instead of working, otherwise it will make the whole productivity even worse!

I believe TOC can apply to software development as well: because the software developing also needs a series dependent activities. And similar to manufacturing, software development also has constraints(bottlenecks), it might be in: test,coding,deploy,maintenance. On the other side, compared to manufacturing, software development is people centered vs. machine centered; most of the constraints is about people; software process is lot easier to change compare to assembly line.
In software industry, we could have a big advantage over manufacturing: to deal with bottleneck, we don't need put other people to sit in idle, we can put more people working on the bottle neck! But this advantage depends on a requirement - if our developers are generalists.

If every developer in a team is a generalist, which would be a great help to maintain a high productivity, then we can deal with almost any constraint, since when there is a bottleneck, we can put more people on it. So the developing flow will be well controlled in a high productivity speed.

Becoming a generalist raise the bar for developers: we are not just a geek who focus on a single area, we are not only J2EE developers, but we also need to fix the bug of a J2ME application; we might need to mange data base as DBA; or improve the build system by writing Ant or shell script.

Becoming a generalist is really challenging, and much harder than focusing on one single area. But based on my experience, software developing job is not a rocket science, every technology can be learned if given an enough time. Technology is not a problem for becoming a generalist, the problem is attitude and mindset. We need to be open minded, be flexible, and ready to wear different hat at different context.

A generalist has following benefits:

• Doing different jobs, give developer a chance to learn different new technology and tools, learn by doing will make you learn faster.
• Have chance to know the big picture of your work.
• Better for team work and collaboration
• Improve your competence and marketability, since you can put more stuff on your resume.
• More changes to become linchpin and less likely being laid off, who wants to fire a developer who can do multiple jobs?

I will suggest our managers should encourage developers to become generalist. For example, he can deliberately assign different jobs for one developer; rotate developer to another team; foster pair programming and mentoring, etc.

Can software be estimated?

As a software developer, I am always asked by project managers:How long do you think you need to finish project X or fix  issue Y? Most of the time I feel annoyed and guilty, because usually I don't really know the answer but I have to give them  a number - which is far from accurate.

I believe how to get a good estimation is one of the major issue for all the developer and managers. An accurate estimation help manager control and plan the project, which make sense. But it is based on an assumption: the software can be estimated. Is this assumption really true? Let's investigate it.
First, if we assume software is estimatable, => then we can see software is predictable, => then we can plan ahead, => waterfall model. We all know that waterfall and CMM is not workable, try to harden the software project is myth.
Next, let's check the software development model. From Lean/Agile software development we know that software is a discovery process, an explorative process, a theory building process, an process of reducing ignorance level. In summary, it is a learning process, the problem solving process itself is also a learning process.  That is why agile uses iterative, feedback process. Which means at the beginning you know very little, you only understand the problem when you solve the problem at the end. So how could you estimate or predict at the beginning? How could you give an accurate estimation at the beginning? You can predict only if you know all the variables, but in software development, there are so many unknown variables: third party library issues, platform compatible issues, IDE, target constraints, etc and other issues you might not aware(you don't know you don't know). So based on analysis the above , we can say estimation in software is impossible.

Let's look at another question: can we estimate bug fix? Bug fix is also a learning process, especially when you are assigned a bug which code base is not written by you. You need to take time to understand the bug, reproduce the issue, understand the code base, familiar with the context, all this stuff are learning process and takes time. How can we estimate this time, because this is really case by case. And we all know that bug fix is actually we spend 90% of time on finding the bug, another 10% time on fixing the bug. Once we found the bug, we know how to fix it. So we can only see we can estimate the bug only after we found the bug, but at this moment it is pointless to give an estimation since we already solve the 90% of the problem.

I would like to share my own experience:In an iterative development process,  there is no need to estimate, just do it. Before I start fixing a bug, I knew nothing; the time I know the estimation answer is when I solve the issue, so the process of finding the answer of estimation is the same process of solving the problem. So at this case why we need the estimation?

 So we know that software nature is impossible to estimate and needless to estimate. But why project managers need it? Maybe they don't have technical background and still trying to use the concept of waterfall methodology; or maybe the psychological reason: human usually not comfortable with uncertainty and needs for closure .

Forcing estimation cause some worse consequences:

• Force developer to plan up front, which will fallback to waterfall;
• Developer faces the pressure to lie, and give the wrong number, which cause,
• Wrong measurement is worse than no measurement. Like weather forecast.

What can we do when the project manager ask you estimation next time:

•  Live with the uncertainty, that is the nature of the software development;
• To be honest, tell them you don't know yet, you only estimate what you can deliver after the current iteration;
• Timebox the estimation, spend 30 minutes to investigate the problem, you should know if you can finish it by the end of the day or not. Avoid yak-shaving.

Learning the theory of Constraint - "The goal" Review

Yesterday I finished the book:The Goal: A Process of Ongoing Improvement. The book is so fascinating, when I read it, I can not put it down, just like reading one of Agatha Christine's books. I never imagine that a business book can be written like this.
What I am most interested is the principle described in book: the theory of constrains. Here I copied some key points from the TOC WIKI:


Key assumption
The underlying premise of Theory of Constraints is that organizations can be measured and controlled by variations on three measures: throughput, operational expense, and inventory.
Throughput is the rate at which the system generates money through sales.
Inventory is all the money that the system has invested in purchasing things which it intends to sell.
Operational expense is all the money the system spends in order to turn inventory into throughput.
"The Goal" itself is to "make money". All other benefits are derived, in one way or another, from that single primary goal.

The five focusing steps
Theory of Constraints is based on the premise that the rate of goal achievement is limited by at least one constraining process. Only by increasing flow through the constraint can overall throughput be increased.Assuming the goal of the organization has been articulated (e.g., "Make money now and in the future") the steps are:

1. Identify the constraint (the resource or policy that prevents the organization from obtaining more of the goal)
2. Decide how to exploit the constraint (get the most capacity out of the constrained process)
3. Subordinate all other processes to above decision (align the whole system or organization to support the decision made above)
4. Elevate the constraint (make other major changes needed to break the constraint)
5. If, as a result of these steps, the constraint has moved, return to Step 1. Don't let inertia become the constraint.

Applying to software development
This book is about manufacturing, but I found it can be easily applied to the software development. And I found it is very similar with Kanban method: Kanban help us visualize the bottleneck, improve the productivity by limiting the Work-In-Progress. Which is exactly the same as the theory of constraints. I feel that the TOC can be regarded as the scientific theory behind the Kanban system. Before when I learn Kanaban, I just accept the principles and conclusion from the Kanban, but I never thought of why it is like this or that,  Since this book uses lots of examples and step by step from question to conclusion. Now I understand more about Kanban. So I can say Kanban is an implementation of TOC.
If you knows Lean/Agile, you will find there are lots of concepts shared between lean and TOC:

• lead time
• Inventory is bad
• WIP (Work-In-Progress)
• small batch increase the productivity
• local optimization is not working only optimize the whole works.
• The dependent event queue can be mapped to the value stream map, or Kanban Queue.

Even the subtitle of the book - "A Process of Ongoing Improvement", which is perfectly match one of the lean philosophy - Kaizen, which means continuous improvement.

I just found David J. Anderson's book: Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results. I haven't read that, but based on the title, it must be about applying the TOC into software development. In his recent blog post: REFLECTIONS ON THE PASSING OF ELI GOLDRATT,  he describe how TOC inspired him for his book and Kanban system.
Actually when I read this article from twitter last week, and this is the reason to let me read the book of "The goal".

And I found this youtube video is interesting: Standing on the Shoulder of Giants by Eliyahu M. Goldratt, Godratt talks about the TOC, Kanban and Toyota Production System.

I realized I need to read this book one more time. the TOC is so powerful, but I still have lots of questions, like:
- How can we apply it into our daily software development?
- How to find the bottleneck?
- What are the measurements we need?

Hope I can find the answers from the goal, kanban and Lean.

NoClassDefFoundError in Blackberry

Background
Since lastweek I spent couple of days trying to fix an issue: The application keep throwing NoClassDefFoundError Exception in a callback method. Through JDE debugger, I can figure out exactly where the exception is threw and which object cause the exception, but when I check the code back and forth, I could not find the problem, since the class file is there. Finally I realized that I also changed the build script: before I use antenna wtkbuild task to generate the library, the new build I use regular javac task, since I used the blackberry native API. Will it cause the problem? It does not make any sense to me. I tried to switch back the build script using twkbuild task and adding the rim api in the class path, this time everything works! No more NoClassDefFoundError any more!

Finding the root cause
I compared the wtktask and the javac task with blackberry build script:

build target(with issue):

<target name="compile_rim" depends="init">
 
  <javac destdir="${outputclasses}" encoding="${src.encoding}" source="1.3" srcdir="${src.dir}:${rim.src.dir}">
    <classpath location="${rimlib.loc}"/>
   </javac>
</target> 

Created by Pretty R at inside-R.org


wtk build ( no issue)



Form the above, I can not tell the difference, and can not figure out why there is an issue. So I have to look at the antenna source code. When I looked at the WtkBuild.java, I noticed that antenna sets the javac target to 1.1 by default! This is interesting.

Check the ant document for javac task, for target attribute, it says:

Generate class files for specific VM version (e.g., 1.1 or 1.2). Note that the default value depends on the JVM that is running Ant. In particular, if you use JDK 1.4+ the generated classes will not be usable for a 1.1 Java VM unless you explicitly set this attribute to the value 1.1 (which is the default value for JDK 1.1 to 1.3). We highly recommend to always specify this attribute.

So based on the above statement, I suspect that the Ant require us to specify the target version, if we don't specify ant will use the default version.and the default target version cause the blackberry device throws the exception.

What is the target version for J2ME and Blackberry
In Antenna, the target is set to 1.1, is this the correct value for J2ME and Blackberry? To prove this, I googled and find some articles:
Managing Wireless Builds with Ant, this article from oracle mentioned that J2ME build requires target value to be 1.1:

<javac bootclasspath="${midp_lib}" destdir="build/classes" srcdir="src" target="1.1">

and I found this article in BlackBerry forum:javac -target parameter used by the rapc.exe tool
It shows that you need to use target 1.1 to build blackberry compatible application:

Rapc.exe automatically ensures that these .class files are targeted for BlackBerry® Java® Virtual Machine (BlackBerry JVM) 1.1 or later. This is done to maintain compatibility on the BlackBerry smartphones when the .class files are actually packaged in a .cod file.

If you are using a custom script that calls javac.exe and rapc.exe separately to package your application, you will have to make sure that you are calling javac.exe with -target option so that all generated .class files are compatible. This is a requirement to make sure that the generated .class files are consistent with Connected Limited Device configuration (CLDC) 1.1 BlackBerry JVM implementation.

Conclusion
Based on what I found, I know that Both J2ME and Blackberry Build need target option to 1.1. otherwise the blackberry application might have a weird issue like NoClassDefFoundError.

How I improved my company's build system - build automation

Last 2 weeks I haven't check twitter, did not go to infoQ, all my spare time is focused on one thing: improve the build system, the goal - one button click triggered the build and deploy automation.
This goal is finally achieved last week. Now the build system is scheduled deployment once a day, the automation jobs include following:

1. check out source code from CVS;
2. build client and server artifacts;
3. generate the change log
4. update the wiki and download page
5. upload the artifact into server and downloading repository
6. execute the server deploy script
7. send new release email 

Before only step 1) and 2) are automatic using the ANT script. All other steps are manual, I found this kind of job is boring,painful and quit easy to make mistakes.

I have the idea for build automation for a long time, especially I was inspired by the book "Continuous Delivery" and "Pragmatic Project automation".  My plan to implement the build automation is pretty simple: simulate the human deployment behaviour, implement an ANT target for each manual step. It sound easy, but when I actually did it, it was harder than imagined. But I finally I made it.
Right now everyday we will automatically release a new build, this is a milestone and a great improvement. I feel so happy and so proud of it.

The tools I used
- ANT,  I used scp, filterset, filterchain, sshexec, cvschangelog, I found Ant has lots of gem.
- Groovy, I used AntBuilder to integrate with the existing ANT task, parsing XML, generating text file, quite neat.
- Jenkins,  everyone recommend Hudson/Jenkins CI server, when I used it, it is so easy to use, and has lots of useful plugins. I use Jenkins to schedule and trigger the build.
- ViewVC, a very good tool to browse CVS source code from web, I configure the ViewVC in Jenkins, Jenkins will provide the actually source code link in the changes page.

What I learned

• Pay more attention to your feeling. If you feel boring and painful, don't just complain, try to find a way to improve it. This is a good chance to improve your self, my previous blog describe this.
• A good developer must be able to delegate the dirty jobs to the computers, it would be shame for a good developer to do the repetitive manual jobs over and over.
• I implemented the automatic build script based on the manual process, so I learned: if you can define a repeatable step by step manual process, then you can automate it by program.
• I finished this job usually in down time, I finished one small piece each time, so the improvement for the legacy system has to be emergent.

Next Actions
This is only the first step, there are way more things needs to be done, my next steps will be:
- Add Unit Test and Regression test automation
- Add the code quality analysis report
- Use Ivy for dependency management
- Use Puppet to manage the configuration

Ant mail task issue in Mac

Last week I get stuck again when using the Ant mail task, based on the document from Ant site, the mail task requires the following 2 jar files:
 - mail.jar from JavaMail API 
 - activation.jar from  JAF API

I already downloaded and copied them into /usr/share/ant/lib folder, but still I got the error.
After google a while, I realized that Mac missed another jar file in its ant package, this time it is ant-javamail.jar! So when I just copied the ant-javamail.jar from the downloaded ant package into /usr/share/ant/lib folder, then every thing worked perfectly.

In summary, to make the mail task work in Mac, you need 3 jar files:
 - mail.jar
 - activation.jar
 - ant-javamail.jar

Ant target to send email example (using gmail smtp server):

<target name="sendmail" description="send email">
        <mail mailhost="smtp.gmail.com"
                  mailport="25"
                  user="${your.gmail.account}"
                  password="${your.gmail.password}"
                  ssl="true"
                from="${your.email.address}"
                tolist="${mail.recipient}"
              subject="Results of nightly build">
          <message>This is a test email</message>
       </mail>
</target>

Ant scp and sshexec task issues in Mac

Last week when I tried to run my ant target using scp and sshexec task in my macbook pro, I got the following issues:

1) scp task issue
Problem: failed to create task or type scp
Cause: the class org.apache.tools.ant.taskdefs.optional.ssh.Scp was not found.
        This looks like one of Ant's optional components.
Action: Check that the appropriate optional JAR exists in
        -/usr/share/ant/lib
        -/Users/steve/.ant/lib
        -a directory added on the command line with the -lib argument

Do not panic, this is a common problem.
The commonest cause is a missing JAR.

2) sshexec task issue
Problem: failed to create task or type sshexec
Cause: the class org.apache.tools.ant.taskdefs.optional.ssh.SSHExec was not found.
        This looks like one of Ant's optional components.
Action: Check that the appropriate optional JAR exists in
        -/usr/share/ant/lib
        -/Users/steve/.ant/lib
        -a directory added on the command line with the -lib argument

Do not panic, this is a common problem.
The commonest cause is a missing JAR.

I already download the jsch-0.1.44.jar from Jcraft,  and copied the jar file into folder /usr/share/ant/lib, and I also tried to put the jar file into different folder as the error suggested, but no help.
Finally I found this article:, it mentioned that in Mac, the ant-jsch.jar is missing! 
and when I look at the /usr/share/ant/lib folder, yes there is no ant-jsch.jar file!  So I downloaded the ant 1.8.2 from ant website, unpack it, and copy it into the /usr/share/ant/lib folder, it just works!

In summary, to make the ant scp and sshexec task work in Mac, you need to do 2 things:

1. Install ant-jsch.jar file: download the ant package from ant website, then copy the ant-jsch.jar into /usr/shar/ant/lib folder
2. Go to Jcraft website, download the latest version of jsch jar file, copy it into /usr/share/ant/lib

Handling incoming call interruption in BlackBerry audio player

Last week I spent two days to fix an issue of blackberry audio player: the audio stream need to pause the player during incoming call interruption, and resume when call is finished.
At the beginning, I feel it is easy to fix, because I fixed this kind of issue many times before for blackberry platform.
So I just program based on my old experiences, but it did not working.

My Mistake

At first I use BlackBerry API PhoneListener to register the call interruption event. But it did not work, I found some issues:

- If you use PhoneListener, the application will prompt use to agree to use Phone API, which is annoying.

- When you delete the application, you have to restart the phone.

- The application can get the incoming call event via callIncoming() method, but some time the application get exception;

- The application get the call end event via callDisconnected() method, but when you call MMAPI player play() method, it always throws MediaException.

  It seems the application the the event earlier than the voice call app release the resource.

- I found the Phone already stop the media player before it send the incoming call event the the application, so you have to use a workaround to deal with this issue.

I tried the different workarounds, but none of them working, it seems the PhoneListener solution is not reliable, and even not working. I don't remember why it works before.

maybe because of current phone is 5.0 while before are 4.3 or 4.6? Actually I don't know.

New solution

After I googled and I found a really simple solution: handle the DEVICE_AVAILABLE and DEVICE_UNAVAILABLE event in the playerListener. Based on this article:

The DEVICE_UNAVAILABLE event occurs when there is an incoming call. The DEVICE_AVAILABLE event occurs when the call is ended.

So all you need to do is pause the player in DEVICE_UNAVAILABLE, and resume when DEVICE_AVAILABLE event comes.

Code snippet like this:

playerUpdate(Player player, String event, Object eventData) 
{
    ...

    if(event == PlayerListener.DEVICE_UNAVAILABLE)
    {
        if(player.getState() == Player.STARTED)
        {
              player.stop();
              bPausedByIncomingCall = true;
        } 
    }
    else if(event == PlayerListener.DEVICE_AVAILABLE)
    { 
        if(bPausedByIncomingCall)
        {
            player.start();
            bPausedByIncomingCall = false;
        }

    }

   ...
}

The solution is very easy and it really works. I tested it when the DEVICE_UNAVAILABLE event comes, the player is still playing, which means the event is sent before the incoming call interruption.

Retrospection

I checked the MMAPI API and Blackberry API document, I found this event is not new (Blackberry support this since 4.0!), but I did not notice this until now. Why I did not notice these two events before? I did not find them in the books, nor I did not heard from them from previous colleagues. This is my another blind spot and it exposed my ignorance of MMAPI. This issue tells me: sometime your old experience is unreliable; you still need to empty your mind, like Andy Hunt said: always have beginner's mind!

Functional programming in OO languages

Yesterday I watch the presentation: Functional Programming: A Pragmatic Introduction in Infoq.com. Jim Duey explains functional programming through Java code samples. He covers 3 parts:

•  high order functions
•  laziness
•  composing

I am more interested in the first part - high order functions. Jim shows 3 important functions:
      reduce, filter and map.
Comparing with the java code, Jim describes how these 3 methods can help us reduce the duplicated boilerplate java code: like loop, iteration. He even emphasizes that these 3 methods - reduce, filter and map can reduce 90% of the loop code!  
Copied from his slide, the benefits of the high order functions are:

•   Allows common code structures to reused 
•   Abstracts away details 
•   Isolates the essential parts of the code 
•   Allows easier unit testing
•   Lets you think at a higher level 
•   Allows easy parallelism

If you don't want to watch the video, then I found this article is really great to explain the map, filter and reduce method, also gives Java implementation.
As a java developer, the implementation code is not new to me, mostly the template method pattern is used. Actually I already saw the similar benefit in Spring framework: it is so amazing the JdbcTemplate can removed lots of the JDBC boilerplate code! But I did not aware that it shares the same idea with the function programming.
Since I learned that high order functions are so important, so I decide to investigate more and see how to implement them in other OO languages beside Java.

Javascript
Joel has a nice article about function programming using javascript: Can Your Programming Language Do This?

Groovy
I checked the Groovy API and recalled what I learned before, so I found the equivalent methods for reduce, filter and map in Collection API are:
   reduce  => inject()
   filter  => findAll()
   map     => collect()

Groovy functional programming link

Ruby
Groovy is very similar with Ruby, especially when I learned Groovy, I suspect if Groovy copies some concept from Ruby, since the collection API is so similar.
      reduce  => reduce(), inject()
      map     => map(), collect()
      filter    => select(), find_all()

This article is really great to explain them more in Ruby

Other resources
Why functional programming matters

Conclusion
- To use functional programming, it would be better to use the language that support functional programming in first class. I have to say Java implementation is too ugly, Groovy and Ruby are much better.
- Changing mindset to functional programming is hard but really important. Functional programming is another way of programming to elevate your abstraction layer.

Next Action
To pursue the way of Functional Programming, you need to pick up a new Functional Programming language ASAP, since I learned LISP through SICP, so Clojure would be my next choice.

Rediscover the power of Ant

My last post talks about using groovy template to generate a text file, but I did not realize it can be done in Ant until reading the book- "Ant in action".
Recently when I refactored the ant build script of our comany's build system, I realize I need to learn ant systematically to improve my skills in Ant, so I choose this famous book.
When I learned the "filterset" datatype, I realize I can use that to generate the text file from a template, for example the last example can be written in Ant:

<copy file="test2.template" toFile="test2.txt" overwrite="true">
   <filterset begintoken="%" endtoken="%">
     <filter token="firstname" value="Steve"/>
     <filter token="lastname" value="Zhang"/>
     <filter token="city" value="Toronto"/>
   </filterset>
 </copy>

It is even simpler than Groovy in this case, right?
This example exposes my ignorance level of Ant, I realize that there a still lots of Gems in Ant. As Douglas Crockford said most language contains good parts and bad parts, we need to use only good parts and avoid bad parts, I think what he said also can apply to Ant. So don't easily discard a tool or a language, right now in my case, I love Groovy, I love Gant, they look so cool, but I am not 100% precent for sure Groovy/Gant can fully replace our build script. I have to rely on Ant now.
I start understand the reason why our build script is a kind of ugly: probably we did not fully use the good parts of ant and too much bad parts instead.
So my next step is start reading Ant in Action!

Invoke Groovy script from Ant or Gant

Recently I tried to use groovy script to improve our build system. I found Groovy TemplateEngine is so powerful and very easy to use, quite similar to erb in Ruby. I want to use it to generate a text file. The following example shows how to generate a text file based on a template.

file test.template

Dear "$firstname $lastname",
So nice to meet you in <% out << (city == "New York" ? "\"The Big Apple\"" : city) %>.

file test.groovy

import groovy.text.SimpleTemplateEngine
import groovy.text.GStringTemplateEngine

def f = new File(this.args[0])
def binding = ["firstname":args[1], "lastname":args[2], "city":args[3]]
engine = new GStringTemplateEngine()
template = engine.createTemplate(f).make(binding)
println template.toString()

If in a terminal if I type:  groovy test.groovy Steve Zhang Toronto, and I will get output

Dear "Steve Zhang",
So nice to meet you in Toronto.

Invoke Groovy from Ant
I followed this article from Groovy site
Following is the ant script:
build.xml

<project>
 <taskdef name="groovy"
          classname="org.codehaus.groovy.ant.Groovy"
          classpath="lib/groovy-all-1.7.10.jar"/>
 
 <target name="helloGroovy">
    <groovy>
     println "hello world"
    </groovy>
 </target>
   
    <target name="testTemplate">
   <groovy src="test.groovy">
    <arg line="test.template"/>
    <arg line="Steve" />
    <arg line="Zhang"/>
    <arg line="Toronto"/>
    </groovy>
 </target> 
</project>

It pretty straight forward, first declare <taskdef>, you need to put groovy-all-version.jar in the class path, but it took me a while to figure out where the groovy-all-xxx.jar is, finally I got it: when you download the binary zip file, unzip it, find embeddable folder, inside this folder you will find the groovy-all-1.7.x.jar!
Then you can put any kind of groovy code inside <groovy>.

Invoke Groovy Script from Gant
Gant is a great tool to replace Ant, I can definitely convert the above build.xml to build.gant, but I realize it is a kind of weird, since Gant is already based on Groovy, it already depends on Groovy jars, why we still need to declare the dependency in groovy-all-xxx.jar? There must be a better solution. Through my exploration, I found two ways to call Groovy script from Gant:

A) Use GroovyShell

target( runGroovy:"run a groovy script") {
 def shell = new GroovyShell()
    shell.run(new File("test.groovy"),
               ["test.template","Steve","Zhang","Toronto"] as String[])
    
}

The above gant script also tells you how to invoke a groovy script from another Groovy script file. GroovyShell can do this for us.

B) Embed the groovy source code directly in the Gant script file
During the exploration, I have some feeling that Gant script might can be thought of Groovy code, what if we add the groovy source code directly into the Gant script, so I did this:

import groovy.text.GStringTemplateEngine

target( hello: "hello") {
  println "hello"
}

target( runGroovy:"run a groovy script") {
 def shell = new GroovyShell()
    shell.run(new File("test.groovy"),
               ["test.template","Steve","Zhang","Toronto"] as String[])
    
}

target(runTemplate :"run a groovy template") {
 def f = new File('test.template')
 def binding = ["firstname":"Steve", "lastname":"Zhang", "city":"Toronto"]
 engine = new GStringTemplateEngine()
 template = engine.createTemplate(f).make(binding)
    println template.toString()
}

The target runTemplate will have the same result output, when you look at the import statement at the top, and the groovy code inside the target, I realize that Gant script is actually Groovy code!, the Gant target can be regarded as Groovy method. This conclusion is so important, because it means you can easily integrate the Groovy(or Java) code in the build script, I am sure it will definitely improve developer's productivity.

Foscam FI8918W wireless IP camera review

I am looking for a video camera as baby monitor for a while. Currently all the baby video cameras in the market are too expensive, over $200 - $300, even that the features are quite limited. So I start looking for the replacement: the IP camera, and finally in Amazon I found FOSCAM FI8918W, it has almost all the features I need: pan/tilt remote control, wireless-IP, info-red, built-in micro phone and speaker. The most feature attract me is the wireless-IP, you can watch your camera anywhere over the internet through your browser or iPhone!

Yesterday I finally got my Foscam wireless IP camera from amazon.ca. I can't wait to unpack it start setting up. I just followed the manual step by step, it did not take me any longer to watch the video over the IE browser; but the hard part is making it work from the internet:
 - At first I choose UPnP, but it did not work, this took me a while;
 - Then I choose forward port as the manual suggested, configure the camera address and port number in my router, tried several times, finally get it working;
 - Last I went to dyndns.org to create an free domain, then configure the parameters in the camera. finally I got it working.

There is another issue: as the manual mentioned, the camera only support IE to get the audio working, but there is no audio on my windows 7, after I followed this article, finally I got it working. Since the camera relies on ActiveX to make audio working, so currently audio does not support on other browsers: Firefox, Chrome, Safari.

Overall the camera does exactly what it mentioned. I like this camera:
Pros:
- info-red feature is awesome, I can see the picture clearly at night when I turned on the info-red;
- remote control is very handy, you can control the camera in 8 direction via browser or iPhone application.
- I just found the motion detection feature, when it detects the motion, the computer will beep and automatically record the video.

Cons:
- No audio output on non IE browser or iPhone application, I heard it due to the video output is Microsoft ASF format.
- No Zoom support
- Only support Wireless b/g, not support wireless-N network
- Picture quality is OK, you can see the pause, quite similar to the web cam

Comparing its feature and the price $100 in amazon.ca, less than the half of the regular video monitor, I think it is a good choice for a baby monitor.

The iPhone application
I downloaded the foscam surveillance pro application in my iPhone, iPod touch and iPad, it costs me $5, the app is really good, except no audio output. And at first I found the left/right control is reversed, after searching the support forum, the solution is choose FI8918W-reversed, then every thing is perfect.

Other resources
watch video with audio (works in vlc player):
 http://[your ip]/videostream.asf?user=[your_user]d&pwd=[your_pass]

watch video without audio(works in the browser):
 http://[your ip]/videostream.cgi?user=[your_user]&pwd=[your_pass]

watch video with camera in vertical stroll mode:
 http://[your ip]/decoder_control.cgi?user=[your_user]&pwd=[your_pass]&command=25

Foscam surveillance application support forum:
  Faq
  forum

foscam in gadget victims

Deming's philosophy - the root of Lean

In the Agile community, almost every one talking about Toyota's Lean technology. When we talking about lean, we can not ignore Deming, since Japanese learned a lot from Deming and goes further and have today's Lean.
Yesterday I spent some time watch Deming's videos on youtube:
W. Edwards Deming - Part 1,  Part 2,  Part 3
Dr. Deming - The 5 Deadly Diseases 1984

From these videos, you will easily find some of the lean principles do get from Deming, like: continuous improvement, eliminate wast, build quality in, respect people, optimize whole, focus on long-term goal, etc.I realize that the core of his philosophy is: respecting people, people first.

Here I copied his 14 points and 5 deadly diseases about management.
Deming's 14 points:

1. Create constancy of purpose.
2. Adopt the new philosophy.
3. Cease dependence on mass inspection.Eliminate the need for massive inspection by building quality into the product in the first place.
4. Minimize total cost when dealing with suppliers.
5. Improve continuously products, services, and processes.
6. Institute training on the job.
7. Adopt and institute leadership.
8. Drive out fear.
9. Break down barriers between departments.
10. Eliminate management by slogan and exhortation.
11. Eliminate numerical quotas and goals.
12. Eliminate barriers to joy in work.
13. Institute a program of education and self-improvement for everyone.
14. Put everyone to work on the transformation of the organization.

Deming's Five Deadly Diseases

1) Lack of Constancy of Purpose
- No planning for the future
- Lack of short term definition and goals
2) Emphasis on Short Term Profits
- Worship of the quarterly dividends
- Sacrificing long term growth of the organization
3) Annual Rating of Performance
- Arbitrary and unjust system
- Demoralizing to employees
- Nourishes short term performance
- Annihilates team work, encourages fear
4) Mobility of Management
- No roots in the company
- No Knowledge of the company
- No understanding of problems of company
5) Use of Visible Figures Only
- No use of figures that are unknown or unknowable, i.e., customer or employee satisfaction
- Encouraged by business schools

From the above, you will find these problems are quite common today, we need a long time to go to adopted his philosophy, this is why I am pessimistic about lean, because I feel the largest problem for lean is coming from managers, not from software developers.

Why learning philosophy is so important?
Some one might ask: we can just follow the practices or process, why we need to learn the philosophy?
I think if technology is good and worth doing, it must have some good principles or philosophy behind the concrete technologies. Once you understand the behind philosophy, you can master the technology, you will not be confused by the different variation or context. And also the detailed technologies or practices are context based, they are easily changed or deprecated with the time and context change; but principle/philosophy is different, they are abstract and can transcend the detailed context, the philosophy can be so powerful that they can apply to different context without change. That is why the art of war and the book of five ring is still useful today, because their principles are general and timeless. This is the meaning of Tao, that is why I am so obsessed by the principle and philosophy. The philosophy is hard to mastered, I remembered the author of The Toyota Way once mentioned, many companies adopted the Toyota production System, but Toyota find few companies really understand the behind philosophy. That is my suggestion, when you learn lean, you still need to understand the behind philosophy. If you just do the lean practices without knowing the philosophy, I am sure you will not do Lean correctly; you can do lean correctly only if you fully understand the lean philosophy.
Here I would like Quote What Deming said:(from his wiki)

"Once the individual understands the system of profound knowledge, he will apply its principles in every kind of relationship with other people. He will have a basis for judgment of his own decisions and for transformation of the organizations that he belongs to. The individual, once transformed, will:

• Set an example;
• Be a good listener, but will not compromise;
• Continually teach other people; and

Help people to pull away from their current practices and beliefs and move into the new philosophy without a feeling of guilt about the past."

Blackberry trick: retrieving the MNC correctly

Recently I need to retrieve the MNC value from the blackberry device, but though the real network testing, I realize that the value might not be correct for some cases.

The 3 different solutions of retrieving the MNC

1. via GPRSInfo

int mnc = GPRSInfo.getHomeMNC(); 

2. via RadioInfo

int currentIndex = RadioInfo.getCurrentNetworkIndex();  
      int mnc = RadioInfo.getMNC(currentIndex); 

3. via GPRSCellInfo

GPRSInfo.GPRSCellInfo cellInfo = GPRSInfo.getCellInfo();  
       int mnc = cellInfo.getMNC(); 

Note:in blackberry the returned MNC value is decimal, which is different with the MNC standard, which is hexadecimal.

I found that solution 1-2 are always consistent, they are always same if you are not in the roaming network. But in solution 3, in some cases you will get the wrong value. For example I tested the following UK carriers, the retrieved mnc values are all wrong:

Carrier Name	MNC(Hex)	retrieved(dec)	retrieved(hex)
3 UK	20	527	20F
Orange UK	33	831	33F
Vodafone UK	15	351	15F
T-Mobile	30	783	30F
O2	10	271	10F

From the above table, we can tell that all the mnc values are appended letter F at the end, if we removed the F, then the value is correct. This is the first time I have ever met. Actually I found this article mentioned this issue a little bit:

...therefore the corresponding hexadecimal value returned by getMCC()or getMNC()may contain the letter F as padding. For example, for T-Mobile®, the MCCs can be identified as either 260 or 26F.

If you want to use cellInfo ( ex. MCC, MNC, LAC, cellID) to get the current location via openCellID for example, then you need to be careful for this issue.

Conclusion
So the solution to get the correct MNC value from blackberry API should be: first check the last 4 bits value, if it is F (or more than 9), then right shift 4 bits, that is it!

Learning SICP: resources and tool

I started learning SICP since the beginning this year. The reason why I learn SICP is because I was inspired by Michael Feathers and Uncle Bob. In 2009, I watched the Michael Feathers presentation "Self-education and the craftsman" for SCNA 2009, he mentioned the SICP book, I did look at it at first, but LISP's lots of parenthesis scared me. Then last year when I read Uncle Bob's blog post "Why Clojure?", he recommend this book, and I also listened his podcast and watched his video about SICP, finally  make to decide to start learning it.

Here I would like to share What I found the useful resources and tools to learn SICP.

SICP text book
Official Web site: http://mitpress.mit.edu/sicp/
You can also get PDF file from here.

SICP Video
MIT open course ware

IDE
I tried several LISP IDE, but finally I found DrScheme is the best one. DrScheme is included in the software called Racket, it is formally called PLT scheme. You can download Racket software from this site. It works on my Mac Pro, I haven't tried it on the Windows environment.

To run the program and exercises on the book, you need to download the package for SICP,
SICP support in PLT scheme. Just follows the installation instruction step by step. The latest version is SICP (PLaneT 1.13)

To use the SICP language in DrScheme, first choose Language|Choose Lanuage..., in Teaching language, choose SICP(PLneT 1.13),
and another most important thing: click "Show Details" Button, in "Output Syntax" group, under "Output Style", choose "write" radio button, then select OK.
I found if you use default selection, which "print" is selected, the output for list is quite annoying.


SICP exercise solutions
I found following two sites are pretty good:
http://community.schemewiki.org/?SICP-Solutions
http://wqzhang.wordpress.com/sicp-solutions/

Uncle bob's blog post about SICP
why clojure?
http://thecleancoder.blogspot.com/2010/08/why-clojure.html

STUB1: SICP & Assignment
http://cleancoder.posterous.com/stub-1-sicp-assignment

Robert Matin interview in The Pragmatic BookShelf
http://pragprog.com/podcasts/show/32

michael feathers 
SCNA 2009 presentation:Self-education and the craftsman
http://scna.softwarecraftsmanship.org/media/michaelfeathers

About DrScheme

 Learning to Program with DrScheme

I am interested in the chaos graphs in this article.