Developing Blackberry application using NetBeans Plugin - part 1, first impression

Updates: I got some feedbacks that the download link in Netbeans is not working anymore, I uploaded the plugin binary into my google doc, please try my link if you have problem to download.

Recently when I checked out the latest version of the source code of LWUIT, I realized that LWUIT used a new solution of developing blackberry application - using Netbeans Blackberry plugin. So today I tried to give an review about this new tool.

Plugin Installation
1. go to this link (or my link)to download the blackberry netbeans plugin, save it in your local drive.
2. In your Netbeans IDE, choose Tools - Plugins, click Downloaded tab, press "Add Plugins..." button, choose the downloaded blackberry plugin .nbm file,
3. Restart your Netbeans IDE, the thing is it seems the blackberry plugin has an issue, you can not see it under the Installed tab of the Plugins UI Form. I was confused it at the beginning as well.

Add BlackBerry SDK platforms
The plugin download link (or here)gives some screen shots about how to add blackberry SDK platform into your NetBeans IDE environment, here I just give an rough description, if you are familiar with the steps of adding other J2ME SDK into the NetBeans IDE, you will know it is the exactly the same steps.
Compare this step with the older solution, you will feel how easy it is.

After Adding the BlackBerry SDKs in your NetBeans IDE, you can use the same way as J2ME Mobility package for blackberry development. Really Nice!!! Good job for the author!

Test
To test how the Plugin works, I choose 2 tests, one is for MIDlet and the other is for CLDC application (RIMlet).

Test1 - build an BlackBerry MIDlet application
I imported an JavaME SDK sample application - SunSamples-Demos into my NetBeans IDE, then add another configuration, choose an BlackBerry platform, I just choose JDE 4.5, then everything else is copied from the default configuration.
1. Test Build
Click "Clean and Build", the IDE build successfully and generate .cod file and bunch of .debug files.
So Build test passes.
2. Test Run
When I click "Run", the IDE launches BlackBerry simulator, then I found the several applications (since SunSamples-Demos contains multiple MIDlets), I choose some apps, they work perfectly.
Application Running also passed.
3. Test Debug
I set several break points in the IDE, then I click "Debug", the IDE launches the BlackBerry JDWP application and the blackberry simulator, when choose an application, the application did not stop at a specific break point.
The debug test failed.

Test2 - build an BlackBerry CLDC application
This time I choose a LWUIT blackberry port demo application - LWUITDemoRIM, just check out the LWUIT source code from its SVN, open the LWUITDemoRIM project. If you installed the plugin and added the blackBerry SDK properly, it is very easy to configure the project in the NetBeans.
Here I would like to mention, for native CLDC application, you need to do a little extra work in your build script. Check this link for details: http://forums.netbeans.org/topic24475.html

1. Test Build
Click "Clean and Build", the IDE build successfully and generate .jad file, .cod file and bunch of .debug files. If you open the jad file, it will tell you the application is CLDC application.
So Build test passes.
2. Test Run
When I click "Run", the IDE launches BlackBerry simulator, you will find the familiar LWUIT demo icon, the application works perfectly as I saw before.
So running test passed.
3. Test Debug
When I click "Debug", I saw the same behaviour as the MIDlet application test, the application never stops at the break point.
Debug test failed as well.

Conclusion
Pros:
- The Blackberry Plugin for NetBeans is pretty easy to install and use. The developer will feel it looks pretty same as other J2ME SDK.
- The Build process is really simple and almost same as regular J2ME application, it will convert jar file into blackberry .cod file.
- The IDE can launch Blackberry native simulator and can run the application successfully.
- The plugin built-in support MIDlet build, need a little work for RIMlet.

Cons:
- Debug is not working, after I read the build script, I know why: based on my experience, you need to copy all the .debug files into the simulator folder, but in build-impl.xml, the target "blackberry-pre-debug" only copied 1 .debug file but missing others.  I will post the detailed solution in future.

Next Action
In future, I will continue post my investigation about this plugin, includes:
- How to build the MIDlet and RIMlet
- Dig deeper on how it implement the blackberry build
- How to fix the debug issue

Becoming generalist - dealing with constraints in software development

Recently I read the fascinating book - the goal, it is really an eye opener for me. I learned the theory of constraints from the book. Everything of TOC looks like common sense, but the conclusion is really striking.

TOC tells us the importance of bottle neck (constraint). We need to identify it, and protect it. In the example of manufacturing assembly line, to make sure the task in bottle neck is the highest priority, other jobs even have to be idle instead of working, otherwise it will make the whole productivity even worse!

I believe TOC can apply to software development as well: because the software developing also needs a series dependent activities. And similar to manufacturing, software development also has constraints(bottlenecks), it might be in: test,coding,deploy,maintenance. On the other side, compared to manufacturing, software development is people centered vs. machine centered; most of the constraints is about people; software process is lot easier to change compare to assembly line.
In software industry, we could have a big advantage over manufacturing: to deal with bottleneck, we don't need put other people to sit in idle, we can put more people working on the bottle neck! But this advantage depends on a requirement - if our developers are generalists.

If every developer in a team is a generalist, which would be a great help to maintain a high productivity, then we can deal with almost any constraint, since when there is a bottleneck, we can put more people on it. So the developing flow will be well controlled in a high productivity speed.

Becoming a generalist raise the bar for developers: we are not just a geek who focus on a single area, we are not only J2EE developers, but we also need to fix the bug of a J2ME application; we might need to mange data base as DBA; or improve the build system by writing Ant or shell script.

Becoming a generalist is really challenging, and much harder than focusing on one single area. But based on my experience, software developing job is not a rocket science, every technology can be learned if given an enough time. Technology is not a problem for becoming a generalist, the problem is attitude and mindset. We need to be open minded, be flexible, and ready to wear different hat at different context.

A generalist has following benefits:

• Doing different jobs, give developer a chance to learn different new technology and tools, learn by doing will make you learn faster.
• Have chance to know the big picture of your work.
• Better for team work and collaboration
• Improve your competence and marketability, since you can put more stuff on your resume.
• More changes to become linchpin and less likely being laid off, who wants to fire a developer who can do multiple jobs?

I will suggest our managers should encourage developers to become generalist. For example, he can deliberately assign different jobs for one developer; rotate developer to another team; foster pair programming and mentoring, etc.

Can software be estimated?

As a software developer, I am always asked by project managers:How long do you think you need to finish project X or fix  issue Y? Most of the time I feel annoyed and guilty, because usually I don't really know the answer but I have to give them  a number - which is far from accurate.

I believe how to get a good estimation is one of the major issue for all the developer and managers. An accurate estimation help manager control and plan the project, which make sense. But it is based on an assumption: the software can be estimated. Is this assumption really true? Let's investigate it.
First, if we assume software is estimatable, => then we can see software is predictable, => then we can plan ahead, => waterfall model. We all know that waterfall and CMM is not workable, try to harden the software project is myth.
Next, let's check the software development model. From Lean/Agile software development we know that software is a discovery process, an explorative process, a theory building process, an process of reducing ignorance level. In summary, it is a learning process, the problem solving process itself is also a learning process.  That is why agile uses iterative, feedback process. Which means at the beginning you know very little, you only understand the problem when you solve the problem at the end. So how could you estimate or predict at the beginning? How could you give an accurate estimation at the beginning? You can predict only if you know all the variables, but in software development, there are so many unknown variables: third party library issues, platform compatible issues, IDE, target constraints, etc and other issues you might not aware(you don't know you don't know). So based on analysis the above , we can say estimation in software is impossible.

Let's look at another question: can we estimate bug fix? Bug fix is also a learning process, especially when you are assigned a bug which code base is not written by you. You need to take time to understand the bug, reproduce the issue, understand the code base, familiar with the context, all this stuff are learning process and takes time. How can we estimate this time, because this is really case by case. And we all know that bug fix is actually we spend 90% of time on finding the bug, another 10% time on fixing the bug. Once we found the bug, we know how to fix it. So we can only see we can estimate the bug only after we found the bug, but at this moment it is pointless to give an estimation since we already solve the 90% of the problem.

I would like to share my own experience:In an iterative development process,  there is no need to estimate, just do it. Before I start fixing a bug, I knew nothing; the time I know the estimation answer is when I solve the issue, so the process of finding the answer of estimation is the same process of solving the problem. So at this case why we need the estimation?

 So we know that software nature is impossible to estimate and needless to estimate. But why project managers need it? Maybe they don't have technical background and still trying to use the concept of waterfall methodology; or maybe the psychological reason: human usually not comfortable with uncertainty and needs for closure .

Forcing estimation cause some worse consequences:

• Force developer to plan up front, which will fallback to waterfall;
• Developer faces the pressure to lie, and give the wrong number, which cause,
• Wrong measurement is worse than no measurement. Like weather forecast.

What can we do when the project manager ask you estimation next time:

•  Live with the uncertainty, that is the nature of the software development;
• To be honest, tell them you don't know yet, you only estimate what you can deliver after the current iteration;
• Timebox the estimation, spend 30 minutes to investigate the problem, you should know if you can finish it by the end of the day or not. Avoid yak-shaving.

Learning the theory of Constraint - "The goal" Review

Yesterday I finished the book:The Goal: A Process of Ongoing Improvement. The book is so fascinating, when I read it, I can not put it down, just like reading one of Agatha Christine's books. I never imagine that a business book can be written like this.
What I am most interested is the principle described in book: the theory of constrains. Here I copied some key points from the TOC WIKI:


Key assumption
The underlying premise of Theory of Constraints is that organizations can be measured and controlled by variations on three measures: throughput, operational expense, and inventory.
Throughput is the rate at which the system generates money through sales.
Inventory is all the money that the system has invested in purchasing things which it intends to sell.
Operational expense is all the money the system spends in order to turn inventory into throughput.
"The Goal" itself is to "make money". All other benefits are derived, in one way or another, from that single primary goal.

The five focusing steps
Theory of Constraints is based on the premise that the rate of goal achievement is limited by at least one constraining process. Only by increasing flow through the constraint can overall throughput be increased.Assuming the goal of the organization has been articulated (e.g., "Make money now and in the future") the steps are:

1. Identify the constraint (the resource or policy that prevents the organization from obtaining more of the goal)
2. Decide how to exploit the constraint (get the most capacity out of the constrained process)
3. Subordinate all other processes to above decision (align the whole system or organization to support the decision made above)
4. Elevate the constraint (make other major changes needed to break the constraint)
5. If, as a result of these steps, the constraint has moved, return to Step 1. Don't let inertia become the constraint.

Applying to software development
This book is about manufacturing, but I found it can be easily applied to the software development. And I found it is very similar with Kanban method: Kanban help us visualize the bottleneck, improve the productivity by limiting the Work-In-Progress. Which is exactly the same as the theory of constraints. I feel that the TOC can be regarded as the scientific theory behind the Kanban system. Before when I learn Kanaban, I just accept the principles and conclusion from the Kanban, but I never thought of why it is like this or that,  Since this book uses lots of examples and step by step from question to conclusion. Now I understand more about Kanban. So I can say Kanban is an implementation of TOC.
If you knows Lean/Agile, you will find there are lots of concepts shared between lean and TOC:

• lead time
• Inventory is bad
• WIP (Work-In-Progress)
• small batch increase the productivity
• local optimization is not working only optimize the whole works.
• The dependent event queue can be mapped to the value stream map, or Kanban Queue.

Even the subtitle of the book - "A Process of Ongoing Improvement", which is perfectly match one of the lean philosophy - Kaizen, which means continuous improvement.

I just found David J. Anderson's book: Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results. I haven't read that, but based on the title, it must be about applying the TOC into software development. In his recent blog post: REFLECTIONS ON THE PASSING OF ELI GOLDRATT,  he describe how TOC inspired him for his book and Kanban system.
Actually when I read this article from twitter last week, and this is the reason to let me read the book of "The goal".

And I found this youtube video is interesting: Standing on the Shoulder of Giants by Eliyahu M. Goldratt, Godratt talks about the TOC, Kanban and Toyota Production System.

I realized I need to read this book one more time. the TOC is so powerful, but I still have lots of questions, like:
- How can we apply it into our daily software development?
- How to find the bottleneck?
- What are the measurements we need?

Hope I can find the answers from the goal, kanban and Lean.

NoClassDefFoundError in Blackberry

Background
Since lastweek I spent couple of days trying to fix an issue: The application keep throwing NoClassDefFoundError Exception in a callback method. Through JDE debugger, I can figure out exactly where the exception is threw and which object cause the exception, but when I check the code back and forth, I could not find the problem, since the class file is there. Finally I realized that I also changed the build script: before I use antenna wtkbuild task to generate the library, the new build I use regular javac task, since I used the blackberry native API. Will it cause the problem? It does not make any sense to me. I tried to switch back the build script using twkbuild task and adding the rim api in the class path, this time everything works! No more NoClassDefFoundError any more!

Finding the root cause
I compared the wtktask and the javac task with blackberry build script:

build target(with issue):

<target name="compile_rim" depends="init">
 
  <javac destdir="${outputclasses}" encoding="${src.encoding}" source="1.3" srcdir="${src.dir}:${rim.src.dir}">
    <classpath location="${rimlib.loc}"/>
   </javac>
</target> 

Created by Pretty R at inside-R.org


wtk build ( no issue)



Form the above, I can not tell the difference, and can not figure out why there is an issue. So I have to look at the antenna source code. When I looked at the WtkBuild.java, I noticed that antenna sets the javac target to 1.1 by default! This is interesting.

Check the ant document for javac task, for target attribute, it says:

Generate class files for specific VM version (e.g., 1.1 or 1.2). Note that the default value depends on the JVM that is running Ant. In particular, if you use JDK 1.4+ the generated classes will not be usable for a 1.1 Java VM unless you explicitly set this attribute to the value 1.1 (which is the default value for JDK 1.1 to 1.3). We highly recommend to always specify this attribute.

So based on the above statement, I suspect that the Ant require us to specify the target version, if we don't specify ant will use the default version.and the default target version cause the blackberry device throws the exception.

What is the target version for J2ME and Blackberry
In Antenna, the target is set to 1.1, is this the correct value for J2ME and Blackberry? To prove this, I googled and find some articles:
Managing Wireless Builds with Ant, this article from oracle mentioned that J2ME build requires target value to be 1.1:

<javac bootclasspath="${midp_lib}" destdir="build/classes" srcdir="src" target="1.1">

and I found this article in BlackBerry forum:javac -target parameter used by the rapc.exe tool
It shows that you need to use target 1.1 to build blackberry compatible application:

Rapc.exe automatically ensures that these .class files are targeted for BlackBerry® Java® Virtual Machine (BlackBerry JVM) 1.1 or later. This is done to maintain compatibility on the BlackBerry smartphones when the .class files are actually packaged in a .cod file.

If you are using a custom script that calls javac.exe and rapc.exe separately to package your application, you will have to make sure that you are calling javac.exe with -target option so that all generated .class files are compatible. This is a requirement to make sure that the generated .class files are consistent with Connected Limited Device configuration (CLDC) 1.1 BlackBerry JVM implementation.

Conclusion
Based on what I found, I know that Both J2ME and Blackberry Build need target option to 1.1. otherwise the blackberry application might have a weird issue like NoClassDefFoundError.

How I improved my company's build system - build automation

Last 2 weeks I haven't check twitter, did not go to infoQ, all my spare time is focused on one thing: improve the build system, the goal - one button click triggered the build and deploy automation.
This goal is finally achieved last week. Now the build system is scheduled deployment once a day, the automation jobs include following:

1. check out source code from CVS;
2. build client and server artifacts;
3. generate the change log
4. update the wiki and download page
5. upload the artifact into server and downloading repository
6. execute the server deploy script
7. send new release email 

Before only step 1) and 2) are automatic using the ANT script. All other steps are manual, I found this kind of job is boring,painful and quit easy to make mistakes.

I have the idea for build automation for a long time, especially I was inspired by the book "Continuous Delivery" and "Pragmatic Project automation".  My plan to implement the build automation is pretty simple: simulate the human deployment behaviour, implement an ANT target for each manual step. It sound easy, but when I actually did it, it was harder than imagined. But I finally I made it.
Right now everyday we will automatically release a new build, this is a milestone and a great improvement. I feel so happy and so proud of it.

The tools I used
- ANT,  I used scp, filterset, filterchain, sshexec, cvschangelog, I found Ant has lots of gem.
- Groovy, I used AntBuilder to integrate with the existing ANT task, parsing XML, generating text file, quite neat.
- Jenkins,  everyone recommend Hudson/Jenkins CI server, when I used it, it is so easy to use, and has lots of useful plugins. I use Jenkins to schedule and trigger the build.
- ViewVC, a very good tool to browse CVS source code from web, I configure the ViewVC in Jenkins, Jenkins will provide the actually source code link in the changes page.

What I learned

• Pay more attention to your feeling. If you feel boring and painful, don't just complain, try to find a way to improve it. This is a good chance to improve your self, my previous blog describe this.
• A good developer must be able to delegate the dirty jobs to the computers, it would be shame for a good developer to do the repetitive manual jobs over and over.
• I implemented the automatic build script based on the manual process, so I learned: if you can define a repeatable step by step manual process, then you can automate it by program.
• I finished this job usually in down time, I finished one small piece each time, so the improvement for the legacy system has to be emergent.

Next Actions
This is only the first step, there are way more things needs to be done, my next steps will be:
- Add Unit Test and Regression test automation
- Add the code quality analysis report
- Use Ivy for dependency management
- Use Puppet to manage the configuration

Ant mail task issue in Mac

Last week I get stuck again when using the Ant mail task, based on the document from Ant site, the mail task requires the following 2 jar files:
 - mail.jar from JavaMail API 
 - activation.jar from  JAF API

I already downloaded and copied them into /usr/share/ant/lib folder, but still I got the error.
After google a while, I realized that Mac missed another jar file in its ant package, this time it is ant-javamail.jar! So when I just copied the ant-javamail.jar from the downloaded ant package into /usr/share/ant/lib folder, then every thing worked perfectly.

In summary, to make the mail task work in Mac, you need 3 jar files:
 - mail.jar
 - activation.jar
 - ant-javamail.jar

Ant target to send email example (using gmail smtp server):

<target name="sendmail" description="send email">
        <mail mailhost="smtp.gmail.com"
                  mailport="25"
                  user="${your.gmail.account}"
                  password="${your.gmail.password}"
                  ssl="true"
                from="${your.email.address}"
                tolist="${mail.recipient}"
              subject="Results of nightly build">
          <message>This is a test email</message>
       </mail>
</target>