[Process] Distributed Source Code Management and Branching

I am using Mercurial a lot recently (and love it); I really do wonder, why I struggled so long with Subversion. When I first heard the GIT presentation from Linus Torvalds (which is, hm, very entertaining) the whole distributed SCM thing sounded very esotheric for me. However I decided to give it a try, also motivated by the great Chaosradio Express 130 Podcast (German). Yet, I decided to go with Mercurial and not Git; allthough this created some flame-wars within our group, because one of my colleagues is a big Git fan. So be it ;-)

For me Mercurial is a great, easy to install, and pretty easy to understand system. The commandline is really straightforward and the help-texts well written (and interestingly - internationalised). Maybe I follow up another blog post with more details to Mercurial another time.

As easy branching and merging are among the main advantages of the new distributed SCMs, I want to recommend for now the very nice blog-post by Steve Losh "A Guide to Branching in Mercurial". This article provides a good and conclusive introduction to different methods for creating branches with Mercurial and also explains differences to Git and some of it's shortcomings (*g*).

p.s.: For my taste, just one thing is missing: some details on merging.
p.p.s.: Please no comments on my "Git shortcomings" statement, they will be censored out anyway ;-)

[Misc] Clojure & Clojure Book Review

It looks like we are living in a fantastic time concerning programming languages. Creating a new language has never been easier then before. With the two great platforms Java and .Net it's not extremely difficult any more to generate intermediate code from the language you are dreaming of. And even the pragmatic bookshelf has a book in writing on "language patterns" to cover this topic.

Thus I assume that nearly every developer is looking out to bet on A) the best horse or to see B) a language from where you can learn the most. Unfortunately the whole world seems to bet on Scala from the sea of hot languages. And indeed Scala is quite cool and might win the race with good reason. Scala is hot stuff for reason. Nevertheless I never got really warm with Scala and it's hard for me to describe why. It's something in the syntax I really can not explain. My dream language must have a code that looks quite perfect. Years ago I had that feeling coding Ruby (and I love Ruby) because the code simply looked great (although I don't like other things in ruby as e.g the clumsy class definitions (there was a project on rubyforge to fix it. Does someone know the name?)).

So I am still constantly looking out for new languages and of course the pragmatic programmers have brought this book into my view:

Stuart Halloway, "Programming Clojure", 2009

So I might share my thoughts on this book and on the language.

For me still Clojure is more attractive then Scala because:

1. It's the toughest edge for your brain.

Perhaps you belong to the same generation as me: I never really used Lisp, Scheme or the dialects alike. But as I am a Java, D and Ruby Coder => Lisp and thus Clojure ("Lisp Reloaded") is the hardest challenge to learn. And according to the last two book reviews you should always go the hard way to learn the most. This might be even true for you.?! For me it turns out that the amount of round brackets is not the problem.

2. Clojure enforces the use of no variables but immutable data structures (as the successful Erlang does!). The code / structures itself mostly is the container for variables you would define use normally. This has two really strong advantages:

1. No variables mean less errors and less to debug
2. No variables and immutable data means no side effects
Hence it's not so easy to write Clojure code with much side effects.

3. Clojure has the strong concurrency concepts - if you really need mutable data - using STM (Software Transactional Memory) / MVCC, Agents, Atoms, giving you ACID Transactions without the "D" (which is only valid for databases).

4. The Java integration is really smart. Have a look:

(new java.util.Random)

simply generates an object you can easily use to work with.

Java and Clojure can call themselves easily vice versa. This might be a really good argument for you to save investments. Clojure doesn't really try to build up all the java libs from scratch; it reuses them in a clever way.

5. Clojure is indeed fast because it generates pure Java byte code.

And of course all the other stuff that a hot language must have:

• Closures (I still can not believe that they are still discussion this hot feature for Java... it's a shame)
• List Comprehensions
• a workaround for tail recursion and currying and other weird stuff as
• very lazy sequences, trampolining, etc.
  

What also challenged my mind is that Clojure has only three constructs for flow control (as we learned from Oz a simple and nevertheless powerful language doesn't really need much):

• An "if" See an example (if (> num 100) "yes" "no"))
• A "do" which is an iteration of statements - introducing side effects - and thus discouraged in clojure
• And a "loop/recure" that can be used to build everything you need in flow control (Stuart Halloway calls it the Swiss army knife).
  

So before I discuss the downside of the language and the book, let me bring up the two points that I loved most in clojure (even if it takes a lifetime to understand them 100%):

1. It has Metadata incorporated in the language right from the start! So you can tie pre- and post-conditions, tests, doc, arbitrary macros to any kind of data or functions. Whow! Do you remember how long it took Java to introduce Annotations? And to me they are still not 100% part of the language but an add on.

2. Clojure has powerful macros and multimethods. This means DSLs are incorporated. So if you loved the way Ruby can build DLSs (and the pragmatic programmers will bring up a nice book on this Ruby DSL topic!) you will get a step further in Clojure.

The book itself uses all hot Clojure features to create a build system called lancelet (every language since Rubys Rake seems to do this a little bit cooler then Ant does).

What I disliked in the book is that it's still difficult for beginners although it is very well written with beginners in mind. One example: I still havn't found the page where I can read a string from the console. And this is a key feature for beginners to test something. The interactive REPL is not enough here. What I mean is that the book has a thousand brilliant examples like fibonnaci. But fib is a two edged example. It just calculates. It creates no emotions in the reader. A better example is the snake game the book creates on a few pages later (here we get input from a keyListener...).

The language itself has two downsides for me:

1. If you are used to the huge Java Collection Lib you are astonished that Ruby boils down all data structure to just a super powerful Array and Hash (and the others are rarely used). That's cool. So you get the impression that Clojure get's on step further in stating that everything is a set (like Lisp stated that everything is a list). But when working with Clojure you are suddenly confronted with not only sets but vectors, lists, maps and trees. Now the book tells you that you always use the set abstraction to work with this. To me this doesn't really help if e.g. the vector notation differs significantly from a set and list definition. I still don't get used to this. but it must be surely my personal inability.

2. The most important drawback is the key feature at the same time: Clojure has an extreme steep learning curve! To become a true expert in Clojure, i.e. to think and dream in Clojure you need to stress every neuron in your brain. So it all boils down to the question: Is it really worth investing half a year in a hot language like Clojure to be able to produce code that is 3 times more accurate imperative Code?

What do you think?

Regards
Stefan Edlich

Links:

• http://clojure.org
• http://www.pragprog.com/titles/shcloj/programming-clojure
• http://ociweb.com/jnb/jnbMar2009.html
• http://en.wikibooks.org/wiki/Clojure_Programming

And finally have a look at this nice language comparison: Java.next
Make a list. For every topic give points from 0 to 3. What is the most elegant for you?

[Misc] Two Pragprog Books Reviewed

Book Review: The passionate programmer and Pragmatic Thinking & Learning

Recently I am getting more and more attracted by the books from "The Pragmatic Programmers / Bookshelf" (link). So I share my thoughts with a review of three books for you. Here I review:

1. Chad Fowler, "The Passionate Programmer", 2009
2. Andy Hunt, "Pragmatic Thinking and Learning", 2008
   

So let's start:

1) Chad Fowler is well known in the Ruby and Rails scene. So he he shares his visions using 53 chapters. Each with a message for you and explained. These messages look a little like the XP programming message. And sometimes they really read like these XP rules:

• "29. Learn how to fail" (testing)
• "18. Automate yourself into into a Job" (daily integration builds)
• "28. Eight hour burn" (no overtime)
• ...
  

But indeed his stories are nice to read and they go far beyond XP rules. They bind personal experience together with passion and and a possible new perspective for you. So his main point is to step out of daily routine, step back, get better and build up new goals for you.

Chad is quite strong in selling his point and most of the points are really fun to read (for example "20. Mind Reader" or "45 You've already lost your job"). So this book is not for experts who have already found their mission in doing independent consultant work for an apache product of which they are a top committer. It's a book for the employee wishing to get motivated and possible building up new perspectives in his career. And of course for beginners that might be reading an XP book at the same time. Chad includes nice actions for each point so that each point can be validated for yourself.

What distracted me a little is the analogy to other jobs. Many writers today cite that they have played (jazz) music in a band. And the challenges in a jazz band are quite the same to a software developer. Chad elaborates a lot on this topic. Even Andy Hunt (see the next review) draws this analogy and many other books (e.g. Presentation Zen by Garr Reynolds) can not leave this point - e.g. be the worst guy in your team - untouched.

Nevertheless it's a fun read if you want to break out and the book should also be recommended in software engineering / programming lectures.

2) Andy Hunt also wrote a remarkable book in combining cognitive sciences with software development. And indeed this is neither a neuroscience book nore a software development book. It is a wonderful walk through topics like:

• Journey from Novice to Expert
• This is your Brain
• Get in your right mind
• Debug your mind (how your mind works)
• Learn deliberately
• Gain Experience
• Manage Focus
• Beyond Expertise
  

This book could also be named "Your brain - The missing manual for software developers". So it's a wonderful guide to understand your brain and how to improve. The book is full of nice graphics, anecdotes and actions the reader should do. Through the book Andy collects Tips which are grouped together at the end of the book in a nice reference card.

If you read this book you will find some topics not new for you as mind maps or wikis. But Andy puts this in a context, gives many advices and he touches a lot points which will be new for you. For example the intense description or the L- and R-Mode helps a lot on how to reflect and use this modes in daily life. And there are a lot of other great new topics you can experience (as morning notes, SQ3R, etc.).

There were really just a few pages that were uninteresting to meas his elaboration on "expect the unexpected" or his way of categorizing generations.

Nevertheless it's a very practical book covering wide ranges of topics as drawing (there are some drawing exercises for you inside) up to yoga techniques. And everything could be applied for your daily life, job or your software development. So this book is a clear buy recommendation and even better a good present for your hacking friend or partner.

[Arch] UML Tools for Mac OS X

Following up a question I received via Twitter, and the fact, that a significant part of the developer-community is using Macs, I thought this might be a good opportunity to discuss some "UML Options" for the Mac. Now, this article is not meant as a definitive answer, I would hope for some follow-ups by readers in the comments.

Ok lets start: First there is heavy weight stuff, most notable Visual Paradigm. A warning: this is a fat tool. However, among the fat tools it is the one I liked the most. I am not using it any more, but it is generally rather easy to use and very feature rich. However, it is a pretty expensive commercial tool. Yes, they have a "community edition", because it is cool to have a community edition these days. But this one was (when I used it last year) rather a joke. See it as a test-preview.

There are also other commercial tools as well, e.g. Omondo. I have not much idea about this one though. Anyone?

On the other end of the spectrum are tools like UMLet (or Violet), which are also Java-based and work more or less good also on the Mac. These tools are very basic and one should not expect much. They are definitly not suited for "real" projects or commercial application, but can be a nice option e.g. for educational purposes. Sometimes one just needs to create some simple UML diagrams for a presentation, paper or book. For such purposes these tools might be useful. Plus both are Open Source tools.

The probably best free (but not Open Source) UML tool, and the one I would recommend is BOUML and this is sure worth a try. The main issue I have with nearly all free/OS UML tools is, that they are often driven by a single person or just very few developers. Hence the future of the particular tool is always a little unclear. To make things worse, there is no accepted open file-format for UML diagrams, that would allow easy exchangeability of the tool. Hence selecting a UML tool is always sort of a lock-in situation.

Also a consideration could be ArgoUML, which is also an Open Source tool and maybe the oldest one around. Has some issues as all OS tools, but apparently has a functioning community.

Finally there are some more or less general purpose drawing programs, that can be used for technical diagrams like EER or UML models as well (with some limitations) like OmniGraffle or Concept Draw and finally also OpenOffice Draw can be used for general purpose vector-oriented diagrams.

Would be happy about comments, experiences and further suggestions!

[Tech] Monitor your WS calls

If you develop applications, which consumes web services from other applications or integration platforms, debugging can often be very deflating. If you don't use the correct debugging tools, you don't see the generated SOAP messages which are delivered between the parties.

A very useful tool is the Open Source SOAP monitoring tool from predic8. The tool does the same as the TCP monitor from Axis, but provides a more user friendly UI and more settings and features:

• Monitoring of SOAP and HTTP messages
• Rule based SOAP routing
• XML formatting and syntax highlighting for SOAP messages
• Interception and modification of messages
• HTTP chunking
• HTTP 1.1
• Loading and saving of configurations
• Rich graphical User Interface
• Resending of messages

The monitor acts as a proxy. Therefore your client application must send the SOAP/HTTP messages to the proxy monitor, which delegates the messages to the real endpoint. A Quick Starter Guide is also available.

[Pub] Mule Tutorial

In the current issue of the Java Magazin I published a tutorial to develop loose coupled systems with Mule. The tutorial illustrates the usage of an Enterprise Service Bus in an airport domain, where different airport systems communicate with each other over the ESB. In the example I use a set of important Enterprise Integration Patterns and show how these patterns are implemented in Mule. Some patterns I used are:

• Event Driven Consumer
• Content Based Router
• Filter
• Transformation
• Message Splitter
  

The transports and connectors I used from Mule are:

• JMS (Active MQ as message broker)
  
• Quartz Transport
• File Transport
• XMPP transport for instant messaging

The source code of the tutorial can be downloaded here.

Have Fun!

[Misc] Hot deployment with Mule 3 M1

Some interesting news from the Open Source ESB Mule. The first milestone from the third version of Mule is out and comes with a major important feature: Hot Deployment

What is the meaning of hot deployment?

Hot deployment is a process of deploying/redeploying service components without having restart your application container. This is very useful in a production environment when you have multiple applications connected over the enterprise service bus without having to impact users of applications.

Check out the example on the mule homepage.

[Misc] Resilient Services & Software Engineering

I recently read the interesting paper by Brad Allenby and Jonathan Fink "Toward Inherently Secure and Resilient Societies" published in Science August 2005 Vol. 309 and surprisingly enough, free to download. This paper is apparently "inspired" by the attack to the World Trade Center, however discusses resilience of important systems our modern societies are depending on in a more general way. The authors definition of resilience is:

"Resiliency is defined as the capability of a system to maintain its functions and structure in the face of internal and external change and to degrade gracefully when it must."

The further state that:

"[...] the critical infrastructure for many firms is shifting to a substantial degree from their physical assets, such as manufacturing facilities, to knowledge systems and networks and the underlying information and communications technology systems and infrastructure.

[...] the increased reliance on ICT systems and the Internet implied by this process can actually produce vulnerabilities, unless greater emphasis is placed on protecting information infrastructures, especially from deliberate physical or software attack to which they might be most vulnerable given their current structure."

The authors apparently have more physical infrastructure in mind (like physical network backbones and the like), however, I am a little bit more worried on the pace certain type of pretty fragile IT services becomes a foundation for our communication and even business models.

I wrote in a recent blog post about my thoughts on Twitter, which became even more important considering the latest political issues in Iran and the use of this communication infrastructure in the conflict. Twitter is (as we know from the past) not only a rather fragile system, it is additionally proprietary and has in case of failure no fallback solution in place.

But Twitter is not the only example: many of the new "social networks" are proprietary and grow at a very fast speed, and we wonder how stable the underlying software, hardware and data-management strategy is. Resilience is apparently no consideration in a fast changing and highly competitive market. At least not until now.

But not only market forces are troubling these days, also political activities that can effect large numbers of systems. Consider the new "green dam" initiative, where Chinese authorities demand each Windows PC to have a piece of filter software pre-installed that should keep "pornography" away from children. This is of course the next level of Internet censorship, but that is not my point here. My point is, that this software will be installed probably an millions of computers and poses a significant threat to the security of the Internet in case of security holes.

Analysis of the green dam system already reveal a number of serious issues. For example Technology Review writes about potential zombie networks, Wolchok et al. described a serious of vulnerabilities. Now this is not the only attempt in that direction. Germany for example is discussing "official" computer worms that are installed by the authorities on computers of suspects to analyse their activities. France and Germany want to implement internet censorship blocking lists of websites. The list of the blocked websites are not to be revealed and it is questionable who controls the infrastructure. Similar issues can be raised here.

I believe, that also software engineering should start dealing with resilience of ICT services and describe best-practices and test-strategies that help engineers to develop resilient systems, but also to allow to assess the risks that are involved in deployed systems. I am afraid we are more and more building important systems on top of very fragile infrastructure and this poses significant risks for our future society. This infrastructure might be fragile on many levels:

• Usage of proprietary protocols and software that makes migration or graceful degradation very difficult
• Deployment of proprietary systems to a large number of computers that cannot be properly assessed in terms of security vulnerabilities or other potential misuses, instead of providing the option to deploy systems from different vendors for a specific purpose
  
• Single points of failure: many of the new startups operate only very few datacenters, probably even on one single location
• Inter-dependece of services (e.g. one service uses one or multiple potential fragile services)
• Systems that can easily be influenced by pressure groups (e.g. centralised infrastructure vs. p2p systems) e.g. to implement censorship
  
• Weak architecture (e.g. systems are not scaling)
• Missing fallback-scenarios, graceful degradation.

Comments?

[Misc] Technical Debt

Recently I stumbled over a smart blog entry about the 'technical debt' (link).

The idea is quite nice: imagine everyone would have a 'perfect' software system in mind to be build. Well in fact we live in a 'real world' and a 100% perfect project is always a goal but not the current status. But of course we all strive for 100% as we strive for 100% test coverage.

But the fact is that some companies / developers build better code and some build a little worse code. Now imaging if we could measure this 'worsiness'. Of course a 100% accurate and correct measurement is not possible and subjective. But sonar from codehaus tries to go that way.

Their technical debt is shown:

• in $ (!!! ouch this hurts)
• in a spider figure
• in the form of numbers you can drill down
  

What they do is they measure at least:

• The Code coverage
• The Complexity
• The Code Duplication
• The Violations
• The Comments
  

There might be more measurements to be integrated soon. And you will surely agree that code comments should have a different weight then the code complexity. Should they?! But what I suggested in my comment is, that it would be great if this measurable debt would be a standard for all projects.

Software developering companies could use a low debt as a marketing instrument. And they likely sell more! The buyer will check the technical debt of the software they buy. As a usual procedure. If the debt is low, the product might be a good and changeable investment that can grow.

If the debt is high the vendor has a problem. Vendors might think they can lock buyers in because they don't check the technical debt. But I am sure time will change and tools like this will be standard in IDEs in 5 to 10 years. Even to check projects in multiple languages.

So for me it's time to face the boss with hard dollars he has to pay back. Sooner or later. The later the more expensive. Let's fight for a technical debt / good metrics analysis as a common procedure!

Stefan Edlich

[Tech] Cloud Computing

As I think, I already mentioned here, I believe, that Cloud Computing (and Software as a Service, but this is a slightly different topic) are true game changers in our understanding of software infrastructure and development/deployment. Currently things are still quite rough around the edges, but I believe, that in like 3-5 years the default option of application deployment will be in one cloud service or another. Putting iron into the cellar or storage-room will be what it should be in my opinion: mostly a stupid idea ;-)

In the current stream of IT Conversations George Reese talks about practical aspects and experiences with current cloud-services like Amazon S3, Simple DB, virtualisation... Recommended!