[Arch] A Comparative Analysis of State-of-the-Art Component Frameworks

Andreas Pieber and Jakob Spoerk wrote a thorough and very good thesis about software components and Java-based component frameworks. The authors introduce component-based software development, derive criteria to compare frameworks and eventually discuss OSGi, Spring, J2EE, SCA, and JBI on an individual basis and in connection with each other as some problems require the combination of several component frameworks.

Dowload the thesis here.

[Arch] Pattern Based Development of Business Applications

In a recent series Jos Dirksen and Tijs Rademakers describe "pattern based development" on the basis of Open Source Middleware (ESBs). Specifically their first article describes how to implement and integrate applications using Mule and the second article gives a good introduction into the Java Business Integration standard (JBI) and the implementation ServiceMix plus a Message broker.

[Arch] Requirements?!

The Waterfall Requirement

I find this really interesting: In talking with many people who actually do make Software in the last years (not only talk about it *g*) I think I detected that many have a growing problem with the term "requirement". Martin Fowler sums it up greatly in his recent blog post. He makes a strong point in saying that the understanding that many people have about "requirements" is actually still very much driven by a waterfall-like understanding of the software engineering process.

As a matter of fact, requirements seem to be problematic on so many levels: on the level of customer/programmer relationship (do they understand each other), on the level of abstraction, how to manage them, how to test if the implementation follows the requirement that we hopefully have understood correctly, just to name a few. A story often heard is "we spent months with the customer building lists and lists, pages and pages of requirements or bought expensive requirement management software, and in the end the development of the product was very decoupled from these lists; but good to know we have them in the files.".

Now my question is this: the trail from requirements to software seems still very natural to us, and it is stunning for many developers and managers that it actually oftentimes does not work. I would like to add here: sometimes it works very well, but I come to that point later. Now, what could be the replacement of requirements? (In case we agree that they do not work as intended in their traditional sense.)

Observation comes in...

Well, there are modifications of requirements engineering in agile processes like the storytelling in XP. But Martin Fowler makes an interesting other observation: Many successful web-applications actually work out the requirements as they go by providing some base-services, a platform (e.g. for exchanging photos). The very important point in this phase is to build in functionality that allows the management to observe the behaviour of the customers: which functions are they using, what are they annoyed about (forum discussions, email feedback...). Then they add experimental new features and test the acceptance (Or you can as well go the next step: let the customers develop the applications as Yahoo! Pipes shows it).

It's Alive!

The interesting point is, that this procedure was very elaborately described in the book "It's Alive: The Coming Convergence of Information, Biology and Business" written by Christopher Meyer and Stan Davis. A very recommended read. The do not describe so much the process of software engineering but rather strategies of modern enterprises and derive the core principles:

• Seed: bring in a new feature, idea; probably only to a subset of customers, probably in variations for different customers
  
• Select: select the successful variations
  
• Amplify: eventually amplify the successful ideas and bring more of that sort

Meyer and Davis brought exampled from "old economy" but actually in software engineering (particularly in web-applications) we have a very good opportunity to rather easily "seed" new features, observe the behaviour and select and then amplify the good ideas. The key point here is, as also Fowler mentions, to focus more on the "observation framework" (in a technical-, but also in a management-sense!) than on trying to get all requirements right from the beginning.

... and back to the Waterfall

Having said that, I want to come back to a point I mentioned earlier: in the "agility euphoria" some evangelists forget to mention, that there is a broad variety of different software products and engineering efforts. (and I am not speaking of safety critical systems here): In many cases actually a "waterfall inspired" process works pretty well. This can be the case in such projects where a technical guy (a developer at best) has already a lot of experience in a particular domain and is either re-writing a legacy application (a case I am observing right now) or is developing a new application that is actually following a series of similar applications and similar customers he had. And this case is quite a regular one in the industry, and we should not forget this scenario. In such cases the requirements of the new application can often be nailed down quite precisely.

Why is that? Well, actually the "Seed, Select and Amplify" process happened in this technical expert. He or she worked with old applications in that domain, often with a broad range of customers that have experiences with several systems of a sort. So he has developed quite a good understanding of (1) the domain (2) the customer (3) the competitors. In such cases the problem often lies more to get the implementation phase right and not to spoil the project on the last steps e.g., with not so experienced developers (either in the technical sense, or in the domain, or in the worst case in both). Hence agile principles as suggested by Scrum for example can be very helpful for the implementation phase to keep control of the process even though the requirements are quite stable from the beginning.

[Arch] Google AppEngine & Python

Cloud computing is the fashion right now and Google is positioning it's AppEngine against services like Amazon EC2. However similar on the first glance, the two approaches are rather different in detail: Amazon's service is more a virtual server hosting (where you have all freedoms, however are responsible about administration too) plus a set of webservices (like the storage services S3 and SimpleDB or the Queue Service SQS).

AppEngine offers a concrete application development environment in Python plus a simple database that has to be used. So you are limited to Python code and Python frameworks like Django and you cannot install an arbitrary database, on the other hand you do not have to deal with many administration issues and Google deals with the scaling.

Guide van Rossum, the father of Python (who is now employed at Google) gives a very interesting one-hour presentation on YouTube on how to write and configure a Python/Django web-application within the Google AppEngine environment.

[Arch] Trends in Data-Management (aka Databases)

It is interesting for me to observe: relational databases have been attacked several times in the last decades, e.g. with object-oriented databases (gone) or XML databases (gone). Now recently a new trend in data-management seems to appear: databases or better data storage/management mechanisms that follow a much looser paradigm than relational databases often using a lightweigt (often REST or JSON based) access strategy. This demand for new datamanagement strategies seems to have several reasons, some come to my mind:

• Performance: in some cases, complex queries are not required (or can be replaced by simple ones): databases that perform very fast with pure primary key retrieval
  
• Complex datastructures are not needed
• ACID is not needed, i.e. mostly simpel writes are performed but fast reads necessary
  
• Agile development seems to favor rather ad-hoc data-structures vs. carefully planned ones (if this is a good trend is written on a different page)
• Distribution is important and distributed relational databases are a hard thing to do
• Access to rather document-oriented datastructures is required

and probably many more. Already older tools like Apache Lucene (actually designed as full-text search engine) is used in several projects as kind of a database replacement. This is particularly possible when reading is more important then writing data and no particular ACID requirements are in place. But Lucene provides a nice and rich query language for that matter.

Recently Amazons EC2 platform made a lot of waves as a distributed deployment platform to be used for applications that have to scale significantly (there is, btw. an Open Source version implementing part of the interfaces named Eucalyptus). Part of the Amazon toolset are two storage mechanisms: S3 and SimpleDB. For both APIs are available to be used from applications. S3 is a storage mechanism for storing rather larger junks of data (like files, documents) and is organised in "buckets". SimpleDB, currently beeing in beta, is a storage mechanism for more fine-grained issues. With SimpleDB chunks of data can be stored using a primary key (item id) and a set of items that can consist of attribute/value pairs. To access SimpleDB a WSDL interface description is available and a sort of REST-style interface.

The newest kid on the block (as appears to me) is Apaches CouchDB, which is currently in the Apache incubator. CouchDB seems to follow a similar strategy like Amazons SimpleDB but is focuses on REST/JSON style access (here is a nice comparison between SimpleDB and CoudhDB). CouchDB is (unfortunately, in my opinion) written in Erlang which makes installation and usage (at least in the Java environment which most Apache projects share) rather a difficult issue. However, conceptually it seems to be quite interesting and I suppose we will see more projects of that sort soon.

Ah, and speaking of marketing: projects like CouchDB explicitly express that they are not alternatives to relational databases :-) However, the first projects appear that provice RESTful interfaces for relational database...

Btw.: does anyone know other projects in that domain that I have not seen yet?

[Arch] Mock Objects

I stumbled over this article yesterday: A neat and short description of Mock objects and a motivation how Mock objects in general and Mocking frameworks can support (unit) testing particularly with classes that have dependencies. I like this very short introduction because the concept of Mock-objects is actually not so difficult to understand but the need for Mock-frameworks is not so easy to grasp.

If the basic idea is understood the documentation of frameworks like JMock can kick in and do the rest ;-)

Addition: Thanks to the comment of reader Touku who recommended the article from Martin Fowler: Mocks Aren't Stubs.

[Misc] Puppet and Puppetmaster

I am back from Indonesia, and what could be a more worthy topic to write as first blog after the travel? Exactly: Puppet. In Indonesia I listened to the IT Conversations talk with Luke Kanies about his project. Puppet is an open source system-administration framework for Unix-based operating systems. I believe, that puppet shows quite some innovations not easily to be found in other tools and has the potential to be the next step in system administration.

First: the target audience of puppet are system administrators and/or developers that have to roll out and administrate a potential large number of server and client (!) systems. Everyone who has to administrate more than two machines know that doing that manually is for sure not an entertaining business. Now what I believe is puppets strongest idea is, to define an abstraction layer over system administration:

Puppet allowes to define the behaviour of machines in an abstract way by using a language to describe classes of configurations; as in object-oriented languages inheritence is possible. The usual tasks of a sysadmin can be written in the puppet language. More important, puppet tries to abstract from OS details, so it does not matter for ordinary activities like configuring an Apache webserver whether the target OS is Linux, Solaris or BSD. To abstract from concrete resources puppet uses so called resources: a good example are users. As we know, they can be defined and managed in different ways on different platforms and contexts. Puppets resources hence deal with concepts like user, file, cron and so on on different operating systems in the same way.

Essentially puppet can be seen as the missing next step after virtualisation solutions: a virtualisation describes the hardware requirements of a machine, puppet describes the operating system and services requirements. So ideally you define the specifications of your machine (needs Apache Webserver, mysql... version...) and then execute that on the very machine using puppet. If you need a second machine with the same configuration, just reuse the configuration from the first (puppet calls that repeatable configurations).

Puppet is also a tool in the sense, that a so called "puppet-master" can communicate with puppet clients. These clients are under control of the puppet master.

Configurations are idempotent, this means, you do not need to assume a specific context or status on the machine to run a configuration "script". You can simply start a configuration on a specific machine and the configuration definition with puppet brings the machine into the desired state. Actually puppet executed these configurations on a regular interval to keep the machine in the desired state.

As far as I understand puppet so far, it is the next level of system administration (as mentioned above, particularly also in combination with virtualisation) allowing to manage also complex infrastructure. There are apparently already a number of companies and institutions using puppet on a larger scale. Luke Kanies mentiones in his talk that also Google is using puppet so administrate several thousand machines (apparently partly MacOS) but also many other international companies.

Puppet written in Ruby and is provided as Open Source framework, however, one thing that worries me a little bit at the moment is the fact, that there is currently no big community behind puppet. Puppet is the "baby" from Reductive Labs and there essentially from Luke Kanies and I believe few further developers. What I have heard from this project so far is really impressive, and I hope that the project attracts more developers soon and Reductive Labs is open minded enough to open the development to outsiders.

[Pub] JBPM meets ESB

The combination of a process engine and an Enterprise Service Bus (ESB) is one interesting aspect of modern service oriented architectures (SOA). Both, an ESB and process engines provide similar concepts and software architects often have problems to find the right solution. Therefor I and Bernd Rücker wrote an article in the German Java Magazin about it. To have a practical showcase the integration is shown with a small example using JBoss jBPM and two Open Source ESBs: JBoss ESB and Mule.

The easy showcase implements the following example: Some event is generated and saved as a file (This may be an order, some incident, an alert, whatever). This file is picked up by the ESB and a new jbpm process is started. The process contains a human task, where somebody has to review the data of the event and decides, if that event can be ignored (e.g. a false alert) or if it has to be handled. In the latter case, the event is sent to an existing case management system via Web Service (could be Lotus Notes or something like that). The case management systems sends a JMS message as soon as the case is closed. This message is again picked up by the ESB and the right process instance is triggered (called "signaled" in jBPM).

The article covers the following topics:

• The basic combination of a process engine and an ESB
• When makes it sense to combine a process engine with an ESB
• How does JBoss ESB integrate jbpm and which Event Handler the proces designer can use to call ESB services
• How does Mule integrate jbpm and which Event Handlers does Mule provide for the process designer
• Lessons Learned :)
  

To compare the ESB implementatations the show case was implemented with JBoss ESB available here and Mule, available here. Following the links, you will find a detailed description about the two implementation scenarios.

[Misc] (Open Source) Developers and Marketing

In the recent IT-Conversations Steve Yegge from Google talks about developers and their attitude towards marketing. I would say, that many stories he tells are more or less well known, however, I think he raises an issue: the relation between development and marketing (in the "business world") or even more problematic: in Open Source projects.

This starts with project-naming and includes license issues (which customer or manager understands 60 different OSI licenses...) and selling of the product. And in one thing he is definitly right: even when we are working in an Open Source environment and we are (mostly) technicians, we want our project to be used (why else would we put it out there), plus a healthy project needs a proper community. Maybe we should once in a while put code, tests, architecture-discussions and the like aside and try to put on the shoes of our (potential) users. And I am afraid in many Open Source projects we will realise, that these shoes are not fitting all too good ;-)

This brings me btw. to another thought: maybe the way the OS process is structured and organised leads in many cases to excellent code, but not necessary to excellent products (in the sense, that the user understands what the software could do for him and how he could use it efficiently). I think OS projects and their tools actually encourage mostly coders to participate in a project. I think, there are hardly OS projects where some contributors are focusing only on interaction design, documentation, marketing...

Might be worth a second thought?!

[Arch] Pattern Based Development with Mule

I've found an interesting article about Pattern Based Development with Mule. This article illustrates how you can implement the Enteprise Integration Patterns using Mule elements. All code examplex are based on Mule 2.