[Arch] SEDA Model and scalable enterprise applications

Scalability, Performance, Fault Tolerance are classical non functional requirements when building enterprise applications. Setting up the application in a clustered environment is a very popular approach and is often used in practice.

When building process based applications you usually deal with long and short running processes. For example a order process may take one week, whereas a save transaction is a process that takes (should take) only few seconds. Short running processs often implements the backend business logic. In Service Oriented Architectures these short running processes results in a orchestration of provided services from different systems.

Mihai Lucian describes in his article a simple scenario, where different platforms in a backend process are connected by using web service endpoints. One of the platform that is queried has a slow response time. He describes one solution approach to battle to the slow response time by using asynchronous IO in servlet container and the Staged Event Driven Architecture (SEDA) model.

He mentioned that current servlet APIs do not provide methods in order to deliver data to the client in a asynchronous way. AJAX based frameworks currently use one of the modes: Polling, Piggy back and Comet. Apache Tomcat for example provides a implementation to handle asynchronous IO by decouple the requirest and response from the worker thread. So you can prepare your response later.

Now the combination with the SEDA model is very interesting, because you can route your request to the right queue. As Mule is based one the SEDA model, Mihai Lucian illustrates how to implement such a scenario in Mule (see image).

From my point of view the example is very interesting based on the following key points:

• Asynchronous communication in a servlet container
• The most interesting thing is how to correlate the request with the response through different layers
  
• How Mule fits in such a architecture
• Practical example of JMS
• Using Routers in Mule and route the message to the right Queue
• Using Apache CXF in Mule

A full description of the example is provided on the article homepage. At the end of the article he provides some benchmarks in order to provide a clear view of the advanateges using such an architecture.

[Misc] Two new interesting SE books out

Recently two fresh books fell into my hands:

1. The Productive Programmer, Neal Ford, 978-0596519780
2. Clean Code, Robert C. Martin, 978-0132350884
   

The first book is a very ambivalent one.

Pros: It covers a wide range of areas how to be more productive. And so you can find topics like ACCERLERATION, FOCUS, AUTOMATION, Test-Diven Design, Code Analyse, up to very philosophical points. The real strong point of this book is that it has several dozen important points most programmers really forget about in their daily work. Each point might be trivial when analyzed in isolation. But to gather all productive points is a worthy issue. And even some principles as SLAP (already covered by Beck) can not be explained often enough.

Cons: The content really leaves a very diced impression. You can find the same information at several places in the book (like the "magic" unix find) and some points of uninteresting information like his words to Reflection, Exceptions, Array Indices and some flame about EJB 2 (really 2!). And thus it's not astonishing that this book gets only two stars on amazon.de.

But nevertheless: even if the book is a mixed work, if you adhere to 80% of the good rules, you will be a massively better programmer.

2. The Clean Code (subtitle: handbook of agile craftsmanship) is more a real work of craftsmanship itself. The idea is the same as for the book beautiful code I recently reviewed here. Martin has found smart authors for each chapters. And the chapters are:

• Meaningful Names (! which is a horror in big companies with developers from different cultural and technical backgrounds to my opinion...)
• Functions
• Comments (you think you can not learn here?)
• Formatting (sounds boring but is not) and
• Error Handling, Boundaries, Unit Tests, Classes, Systems, Emergence, Concurrency, Successive Refinement, JUnit Internals, and more...
  

It closes with a catalog of smells and heuristics which is a little like the one we had in the Fowler refactoring book but nevertheless of great use.

Robert C. Martins Clean Code Book differs from the rest because there is a lot code in it. And a lot code that migrates from bad to good. You really feel while reading that the authors have invested in strong code examples. This makes it a really valuable resource to read. So it has my strong recommendation.

To conclude: These two books, combined with all the Fowlers and Becks (not the beer...) lead me to create a catalog of all the useful points they have written. It is still small and in beta state. But you are invited to use, contribute or link to this growing list of best practices.

So God bless all (most of?) the software developers!
(who read these books ;-)

[Conf] CEE-SET Conference

Last week I joined the CEE-SET conference in Brno. On the conference I presented a paper written by Robert Thullner, Josef Schiefer and myself: We analyse the application of Open Source frameworks in implementing enterprise integration patterns. For that matter a series of scenarios was implemented with (combinations) of different frameworks like Apache Active MQ, Apache Camel, Apache Service Mix and Mule.

The paper is available for download.

[Arch] Introduction to REST

REST ist an architectural style on how to let distributed applications communicate. It is considered as an alternative approach to XML:RPC or SOAP webservices. I generally like that Google Talks introduction to REST:



I would also recommend additional resources to get a more complete picture on REST, check out this link.

[Arch] OpenSource ESBs

The last couple of years major Open Source ESBs, including MuleSource and ServiceMix, have been expanded and are used in critical business solutions. Tijs Rademakers and Jos Dirksen offer a book which gives an overview about Open Source ESBs and which combination of Open Source technologies with ESBs are used. The main open source solutions covered in this book are Mule and ServiceMix. Therefore most of the examples in the book are based on these two technologies. Other Open Source ESBs that will be covered are Apache Synapse, Open ESB and the new integration framework from Spring, called Spring Integration.

In the TechBrief the authors mentioned that all Open Source ESBs focus on Enterprise Integration Pattersn. If you understand these patterns its very easy to understand the implementation and handling of ESBs.

The book is divided into three parts. The first part concentrates on reader which are not familiar with an ESB:

• Overview about ESB functionality and what Open Source ESBs are available in the Open Source market
• Taking a deep look into the Mule and ServiceMix architecture
• Installation of Mule and ServiceMix and how to run them

The second part focus on ESB core functioanlity which covers some of the Enterprise Integration Patterns. Here the reader becomes some connector examples, like JMS, JDBC, POP3 and Web Services.

The third part covers case studies and also illustrates integration scenarios with BPM engines, like jBPM and Apache ODE.

In the tech brief there was also a short comparison between Mule and Service Mix. When to use which one, is hard to say, it depends on your requirements. But in this interview on of the authors said that in a web service based architecture the JBI approach is often the better choice, but Mule is very often used, because you can also transfer Java objects, which is often very comfortable and faster. They also talk about integration of legacy systems, which is sometimes easier with Mule, because when you use Service Mix all messages must be transformed in XML.

You can download chapter 1 and chapter 4 for the book homepage.

[Arch] A Comparative Analysis of State-of-the-Art Component Frameworks

Andreas Pieber and Jakob Spoerk wrote a thorough and very good thesis about software components and Java-based component frameworks. The authors introduce component-based software development, derive criteria to compare frameworks and eventually discuss OSGi, Spring, J2EE, SCA, and JBI on an individual basis and in connection with each other as some problems require the combination of several component frameworks.

Dowload the thesis here.

[Arch] Pattern Based Development of Business Applications

In a recent series Jos Dirksen and Tijs Rademakers describe "pattern based development" on the basis of Open Source Middleware (ESBs). Specifically their first article describes how to implement and integrate applications using Mule and the second article gives a good introduction into the Java Business Integration standard (JBI) and the implementation ServiceMix plus a Message broker.

[Arch] Requirements?!

The Waterfall Requirement

I find this really interesting: In talking with many people who actually do make Software in the last years (not only talk about it *g*) I think I detected that many have a growing problem with the term "requirement". Martin Fowler sums it up greatly in his recent blog post. He makes a strong point in saying that the understanding that many people have about "requirements" is actually still very much driven by a waterfall-like understanding of the software engineering process.

As a matter of fact, requirements seem to be problematic on so many levels: on the level of customer/programmer relationship (do they understand each other), on the level of abstraction, how to manage them, how to test if the implementation follows the requirement that we hopefully have understood correctly, just to name a few. A story often heard is "we spent months with the customer building lists and lists, pages and pages of requirements or bought expensive requirement management software, and in the end the development of the product was very decoupled from these lists; but good to know we have them in the files.".

Now my question is this: the trail from requirements to software seems still very natural to us, and it is stunning for many developers and managers that it actually oftentimes does not work. I would like to add here: sometimes it works very well, but I come to that point later. Now, what could be the replacement of requirements? (In case we agree that they do not work as intended in their traditional sense.)

Observation comes in...

Well, there are modifications of requirements engineering in agile processes like the storytelling in XP. But Martin Fowler makes an interesting other observation: Many successful web-applications actually work out the requirements as they go by providing some base-services, a platform (e.g. for exchanging photos). The very important point in this phase is to build in functionality that allows the management to observe the behaviour of the customers: which functions are they using, what are they annoyed about (forum discussions, email feedback...). Then they add experimental new features and test the acceptance (Or you can as well go the next step: let the customers develop the applications as Yahoo! Pipes shows it).

It's Alive!

The interesting point is, that this procedure was very elaborately described in the book "It's Alive: The Coming Convergence of Information, Biology and Business" written by Christopher Meyer and Stan Davis. A very recommended read. The do not describe so much the process of software engineering but rather strategies of modern enterprises and derive the core principles:

• Seed: bring in a new feature, idea; probably only to a subset of customers, probably in variations for different customers
  
• Select: select the successful variations
  
• Amplify: eventually amplify the successful ideas and bring more of that sort

Meyer and Davis brought exampled from "old economy" but actually in software engineering (particularly in web-applications) we have a very good opportunity to rather easily "seed" new features, observe the behaviour and select and then amplify the good ideas. The key point here is, as also Fowler mentions, to focus more on the "observation framework" (in a technical-, but also in a management-sense!) than on trying to get all requirements right from the beginning.

... and back to the Waterfall

Having said that, I want to come back to a point I mentioned earlier: in the "agility euphoria" some evangelists forget to mention, that there is a broad variety of different software products and engineering efforts. (and I am not speaking of safety critical systems here): In many cases actually a "waterfall inspired" process works pretty well. This can be the case in such projects where a technical guy (a developer at best) has already a lot of experience in a particular domain and is either re-writing a legacy application (a case I am observing right now) or is developing a new application that is actually following a series of similar applications and similar customers he had. And this case is quite a regular one in the industry, and we should not forget this scenario. In such cases the requirements of the new application can often be nailed down quite precisely.

Why is that? Well, actually the "Seed, Select and Amplify" process happened in this technical expert. He or she worked with old applications in that domain, often with a broad range of customers that have experiences with several systems of a sort. So he has developed quite a good understanding of (1) the domain (2) the customer (3) the competitors. In such cases the problem often lies more to get the implementation phase right and not to spoil the project on the last steps e.g., with not so experienced developers (either in the technical sense, or in the domain, or in the worst case in both). Hence agile principles as suggested by Scrum for example can be very helpful for the implementation phase to keep control of the process even though the requirements are quite stable from the beginning.

[Arch] Google AppEngine & Python

Cloud computing is the fashion right now and Google is positioning it's AppEngine against services like Amazon EC2. However similar on the first glance, the two approaches are rather different in detail: Amazon's service is more a virtual server hosting (where you have all freedoms, however are responsible about administration too) plus a set of webservices (like the storage services S3 and SimpleDB or the Queue Service SQS).

AppEngine offers a concrete application development environment in Python plus a simple database that has to be used. So you are limited to Python code and Python frameworks like Django and you cannot install an arbitrary database, on the other hand you do not have to deal with many administration issues and Google deals with the scaling.

Guide van Rossum, the father of Python (who is now employed at Google) gives a very interesting one-hour presentation on YouTube on how to write and configure a Python/Django web-application within the Google AppEngine environment.

[Arch] Trends in Data-Management (aka Databases)

It is interesting for me to observe: relational databases have been attacked several times in the last decades, e.g. with object-oriented databases (gone) or XML databases (gone). Now recently a new trend in data-management seems to appear: databases or better data storage/management mechanisms that follow a much looser paradigm than relational databases often using a lightweigt (often REST or JSON based) access strategy. This demand for new datamanagement strategies seems to have several reasons, some come to my mind:

• Performance: in some cases, complex queries are not required (or can be replaced by simple ones): databases that perform very fast with pure primary key retrieval
  
• Complex datastructures are not needed
• ACID is not needed, i.e. mostly simpel writes are performed but fast reads necessary
  
• Agile development seems to favor rather ad-hoc data-structures vs. carefully planned ones (if this is a good trend is written on a different page)
• Distribution is important and distributed relational databases are a hard thing to do
• Access to rather document-oriented datastructures is required

and probably many more. Already older tools like Apache Lucene (actually designed as full-text search engine) is used in several projects as kind of a database replacement. This is particularly possible when reading is more important then writing data and no particular ACID requirements are in place. But Lucene provides a nice and rich query language for that matter.

Recently Amazons EC2 platform made a lot of waves as a distributed deployment platform to be used for applications that have to scale significantly (there is, btw. an Open Source version implementing part of the interfaces named Eucalyptus). Part of the Amazon toolset are two storage mechanisms: S3 and SimpleDB. For both APIs are available to be used from applications. S3 is a storage mechanism for storing rather larger junks of data (like files, documents) and is organised in "buckets". SimpleDB, currently beeing in beta, is a storage mechanism for more fine-grained issues. With SimpleDB chunks of data can be stored using a primary key (item id) and a set of items that can consist of attribute/value pairs. To access SimpleDB a WSDL interface description is available and a sort of REST-style interface.

The newest kid on the block (as appears to me) is Apaches CouchDB, which is currently in the Apache incubator. CouchDB seems to follow a similar strategy like Amazons SimpleDB but is focuses on REST/JSON style access (here is a nice comparison between SimpleDB and CoudhDB). CouchDB is (unfortunately, in my opinion) written in Erlang which makes installation and usage (at least in the Java environment which most Apache projects share) rather a difficult issue. However, conceptually it seems to be quite interesting and I suppose we will see more projects of that sort soon.

Ah, and speaking of marketing: projects like CouchDB explicitly express that they are not alternatives to relational databases :-) However, the first projects appear that provice RESTful interfaces for relational database...

Btw.: does anyone know other projects in that domain that I have not seen yet?