Saturday, April 24, 2010

[Process] The Human Factor

In one of his recent blog-posts Martin Fowler explains, why he is not interested in participating in the Software Engineering Method and Theory initiative (SEMAT) by Jacobson, Meyer, and Soley. Now, this is not really big news, however if found the argument interesting. He refers to Alistair Cockburn saying: 
" [...] that since people are the central element in software development, and people are inherently non-linear and unpredictable - such an effort is fundamentally doomed"
This is surprisingly thin reasoning. In my opinion. Don't get me wrong: I am myself not particularly interested in heavy weight processes, yet, the idea to found software engineering on proven concepts not on "fads", "fashion" and I would like to add: "guru speak" sounds good to me. Software engineering still is,  I believe, strongly driven by opinion and authority, not (scientific) evidence. I don't know much about the particular approach though.

However, the "people as a central element" argument is in my opinion flawed to the extreme. People were involved, yes central in most technical enterprises; from flying to the moon, medical treatment to installing a bathroom. And yet, most of these activities have standards, a scientific foundation and well established best practices. This is not to say that every doctor is good, far from it, also not every plumber is doing proper installations. Yet, we would not excuse bad medical treatment or a leaking toilet with "well, there are humans in the center...". Come on.

There is a second misunderstanding at the core of the argument: it is true that individuals are often non-linear and unpredictable. At least this is what we all want to believe. Who does not want to be unique. We are all artists, coding gurus, geeks; everyone is brilliant and irreplaceable. And yet psychology shows us pretty convincingly that we are way more predictable than we would like to believe, even in our irrationality. Particularly in larger numbers and over larger amounts of time. Now, every projects is somewhat unique, yet, there is a lot we can learn about human nature, human/technology interaction. I claim, that it is simply not true that software projects are inherently unpredictable. At least not due to the human factor

We simply have not learned our lessons yet. Agile processes for example have shown significant improvements where applied properly and where management and customers are participating. There might still be a (long) way to go, but human nature sure does not serve as proper excuse.

Wednesday, April 21, 2010

[Arch] Build scalable systems that handle failure withtout losing data

I found a very interesting article on the MSDN Architecture Center illustrates a real life use case about scalable systems. Designing and building scalable systems is one of the major challenges of Software Engineers. A lot of best practices and patterns exist on the web illustrating the problem, but the specific design and the implementation differ in projects. This article tells a real life example of such a system and the essential steps that were done in order to build a scalable system that also handle failure without losing data. The following topics are covered:

  • HTTP and Message Loss
  • Durable Messaging
  • Systems Consistency
  • Transactional Messaging
  • Transient Conditions
  • Deserialization Errors
  • Messages in the Error Queue
  • Time and Message Loss
  • TimetoBeReceived
  • Call Stack Problems
  • Large Messages
  • Small Messages from Large
  • Idempotent Messaging
  • Long-Running Processes
  • Learning from Mistakes