Concurrent development

Background Story

That certain project was in its 4th year and after a few successful deployments in the US/LATAM regions, now the company was trying to push it to EU and APAC. The original developers were all in New York and over the last few months have been frantically working to adapt the product to EU's requirements (Asia was supposed to follow.) The scope was defined as 'whatever it takes', the methodology was a mix between 'trial and error' and bullying the users to contort the requirements to fit the delivered functionality. At the time, the EU team had realized that they would better spend their time staying on top of the changes and making sure the end product meets a minimum standard, so they were not really doing much development.

Time went on, requirements grew, scope shrinked. The project slipped past two deadlines and finally the Asia managers decided they need to take things into their hands and hire a local development team to avoid the communication gap that plagued the EU rollout and regain some control over the schedule. It was first time to have more than one people touching the code the core team was structured in a way that each component has an owner and the owner could do whatever they want. If you have a problem or need a change - ask the owner. The problem was not only that we were in different geographical location, in inconvenient timezone, but we were working on the same code, implementing requirements specified by separate BA teams, chasing schedules devised by separate project-management teams, and it all eventually converged in a common program-steering committee. I could go on, but suffice to say it was quite a mess - the bottom line is that moving from centralized sequential to distributed concurrent development models impose huge burden and the best advice one can give you would be "don't do it!".

Probably the biggest issue was that many people in the core team, just refused to change their way of working in order to accommodate our existence. Every second morning the trunk would not compile, often changes were checked in that prevent servers from starting, our changes were overwritten routinely because somebody's local modifications conflicted and they were unwilling to merge - you name it, we have it. The management layer was protecting them as "due to the years of accumulated experience, the productivity of the core team was much higher that ours, and the productivity hit they would suffer by addressing our petty complaints can not be justified in business terms". Luckily, there were some sensible guys and gradually we got to improve this, still I consider it one of the biggest organizational faults that for a long time the management efforts were focused on suppressing our complaints, rather than backing our suggestions on fixing the environment.

As the first QA delivery was approaching and the trunk was not giving any signs of getting more stable, we tried to think what can we do to stabilize the codebase. Some people said we should branch, others were weary of the cost of merging. The EU team had branched few months ago and all EU implementation were done on the branch and eventually (read 'sometimes') merged to the trunk. When the product was released in EU, they ended up with the problem how do they merge to the trunk. From what I hear it had been a terrible experience, including a lot of functionality rewrites, introduced bugs and regressions.

Knowing the EU problems and knowing that on one hand the trunk was still changing rapidly, on the other hand our requirements were dependent on code that was supposed to be delivered by US, we decided to branch, but keep developing on the trunk. All merges would be in direction trunk-to-branch and this would save us from the dreaded criss-cross merge conflicts. Since most of our problems to that date were with work-in-progress checkins, which we eventually wanted, we decided that we can treat the branch as a stable release line and trunk as unstable bleeding-edge code.

Unstable trunk + Release branch

I was tacitly elected as 'merge-master' and quickly I found myself following the same routine:

1. Every morning I would pull a list of all the unmerged commits and review them in a text editor. Then would move each commit into one of these categories:
• WANTED - changes that are required or prerequisites for implementing our business functionality. These should be always merged.
• BLOCKED - changes that we DO NOT want. These should be always marked as merged (no actual merging, just mark, so they will not appear in the list next time).
• IRRELEVANT - changes that won't hurt us, but we don't strictly need them. We were merging these in the initial stages as keeping the branch close to trunk makes merging easier, as we got closer to the release, we flipped the policy to improve the stability.
2. When I merge or mark as merged the WANTED/IRRELEVANT/BLOCKED groups, I would put the category as a first word int he commit message. This made it easier to pick out the changes that were done directly in the branch (which should be kept to minimum and if necesarry ported manually to trunk). I didn't bother separating the individual changes, since the branch was not meant as a merge-source - this was saving me some time. Overall it was taking between 1 and 3 hours a day.
3. There would be a number of changes that didn't make it to any of the categories. For these I was contacting the comitter and following up. Often it was work in progress, sometimes after clarification they would be categorized on the next day. Usually I would post this communication as a tagged blog-entry in our wiki. There was a page displaying all the entries tagged in this way.

I found out that sorting the changes first by user and then by date simplifies the review significantly. Turned out that TextPad macros can be a very powerful tool for things like this.

The release branch worked well for some time, until a major feature for the next release was implemented on the trunk. We blocked it and ever since then, every commit that touched this dreaded component had to be hand-merged. Often, merging an one-line change resulted in tens of conflicts, so we resorted to rolling back the file in question and manually porting the change. The worst thing is that we tested the trunk extensively, but the change in our release-branch received only cursory examination until it reached QA.

Furthermore, once we reached the second phase of the Asia roll-out, our team split and started to work in parallel on three staged releases, which were supposed to deliver unrelated functionality within 2 months of each other startin 6 months from the date. This meant that we need better mechanism for dealing with divergent codebase and big changes in progress.

Exchange-trunk + Development & Release branches per stream

After taking a step back, we came up with a new branching scheme that satisfied all our requirements. For each pending project phase we would create two parallel branches - development and release (we would call the combination of two branches a 'stream'). In addition, we devised the following policies and procedures:

1. Developers always commit their code changes in the dev-branches.
2. Any code committed to the dev-branch MUST compile. If the branch is broken, people should not commit further unrelated changes until the CI says it's fixed.
3. Each commit in the dev-branch should contain work for a single feature. If there is certain code pertaining to two features, we pick one of them as primary and mark the other one as dependent in the issue-tracking system. Then, all the shared code goes to the primary feature and we know that we can not release the dependent on its own. It is not necesarry that the whole feature is committed in one go or that the dev-branch committed code actually works.
4. When we need some code from a different stream, we would wait until they publish it to trunk and only then we would merge from trunk to the dev-branch. Cross-stream merges are prohibited. We were calling this 'picking-up' the feature. Pick-up changesets should be marked in the commit message.
5. Each time we pick-up a feature, after we do the minimum conflict resolution, so the code works, we would commit the changeset immediately (that's the pickup changeset). This way, any additional enhancements, fixes. etc. will be committed in separate changeset, so it will be easier to merge them back to trunk later.
6. Once a feature is complete and dev-tested on the dev branch, all related changesets for that feature are merged as one consolidated changeset in the release branch. We call this 'feature-promotion'. This practice makes creating release notes relatively easy and allows us to do cool things such as rolling back the whole feature with one command.
7. When we promote a feature that has been picked from trunk, we immediately mark-as-merged this rel-branch commit into trunk to prevent double-merge conflicts. We would look if we have made any fixes on our branch and consolidate them into a single enhancement/bugfix changeset that will be merged directly from dev-branch to trunk (as in the the rel-branch we consolidate the pickup and enhancement changesets).
8. If QA finds that the feature did not work, we would add further bugfix changesets to the rel-branch, but we would strive to keep them to minimum.
9. When a release has passed QA, we would merge each feature-level commit that originated from this stream from the release branch to trunk ('publishing'). There it will be ready for picking up by the other streams (which will merge it in their dev-branch, promote it to release, etc).
10. For each release we would tag the release branch, since it was already stabilized. Bugfix releases were just further tags on that same branch. For urgent production changes, we would create a bugfix branch from the tag (happened only a few times).

Overall it worked well for us. Few months after we addopted this scheme I moved to another company, but I really hope the process is still useful and being improved. An interesting thing is that every time I explain this, the first reactions are along the lines of "does it have to be that complicated?" And while I can agree that complicated it is, I am still to find a simpler streategy that could work on this scale. Any ideas?

The other half of the puzzle

This post on service service granularity from Ross Mason (the original developer of Mule ESB) got me in retrospective mode. I've been thinking about this before and with the time, I have come up with some criteria how to break functionality into services. Unfortunately that turned out to be only half of the puzzle. The other half is the implementation and maintenance strategy that would allow you to achieve your nicely layered boxes-and-arrows vision, while delivering ROI on quarterly intervals and not falling into the JBOWS or stovepipe-system pitfalls.

I personally favor starting top-down, focusing on specific applications and task-level services, and harvesting entity and utility services from working production code. The downside of this is that it is generally more work, often seen as 'tinkering' and in case we consolidate utility code from multiple systems, will likely introduce temporary instabilities.

This makes it a difficult sell to management - what they see is the IT guys trying to get budget to work on some kind of metaphysical inner-beauty thing instead of delivering new features. Saying that this would reduce bugs is not a great argument, because you will be reminded that's why we pay the QA team and by the way, as a professional you are not supposed to write bugs in the first place.

On the other side, the PHBs of the world seems to love the idea of the Centers of Excellence, located in China or Malaysia, full of trained and cheap labour, building the utility services and coding the entity diagrams designed by the Architecture Group in their ivory tower, while the solution engineers use point and click tools to churn 'codeless' composite applications. Unfortunately this breaks at the other end - unexpected interactions are discovered at deployment time, maintenance takes too long as the whole turns out to be more than its parts, exhibiting "emergent behavior" (also known as "underspecified features") left and right.

So, there is no quick and easy way. I guess we need to become IT salesmen and we need to compete with the professional vendor scamsters and the consultancy loan-sharks (that's a topic for another post).

Why ORM?

I like Spring JDBC Template - it is simple, clean and saves a lot from the JDBC drudgery. It provides nice abstraction, simplifying the usage and management of prepared statements and overall if all you want is to do an ocasional query, it doesn't get any simpler. Oh, and you have full control on the underlying JDBC connection, allowing you to exploit any vendor-specific functionality. The provided functionality only makes the data access easier and supports only very basic property to column mapping out of the box (or you can write your own mapper in Java). Because of its lightweight nature, JDBC Template does not allow for transparent caching, horizontal table partitioning, vertical sharding, etc. It also doesn't offer lazy fetching of relations, identity tracking and other instrumentation magic.

Another tool I like is iBATIS. In comparison with JDBC template, it offers more structure, organizing my SQL statements in a logical way and supports most common forms of mapping the result sets to beans (including graphs of objects and joins). Ibatis does support caching (only on a whole query level), still no sharding and partitioning, lazy fetching and identity tracking. IBATIS doesn't try to parse your queries.

The Abator tool can generate iBATIS CRUD mappings, DDL from object model, and object model from a database, but frankly - in its generated form all these suck. The DDL uses only a few datatypes, has no RI or semantic constraints, nor indexes (I use it sometimes, but only as a boilerplate which I extensively tweak afterwards). The generated beans are isomorphic with the underlying tables, which is great if you are doing CRUD, but I find that most of my applications do data analysis, so the queries tend to be quite hairy, half of them encapsulated in stored procedures (for transaction processing I'd rather use Spring or plain JDBC anyway). The bottom line is that Abator handles the trivial cases, but for most of my projects it doesn't help much, so I'm back to manual coding (which is not that bad anyway.)

The first time I looked at ORM was when I checked TopLink for a project around 2000. Back then I got the impression that it introduces a lot of new concepts in order to simplify something that's already simple enough - loading a couple of beans from a table. For that project I used plain JDBC, it took me perhaps 20 mins to write and I can't remember taking much time to maintain that code - the total cost should have been less than 3 man-hours.

The second time I tried Hibernate and Toplink last year (2006) on a prototype with the same dataset and relatively small domain model (for which I finally ended up using iBATIS). The first thing that stroke me was how slow everything was - the startup time was noticeably slower than with iBATIS (about 500-800%), spent in initializing the JPA entity manager and the queries were consistently 30-50% slower. It might have been that I didn't know how to tune it, but I just couldn't achieve satisfactory results. I researched the facilities for mapping to externally defined schema and it stroke me that both Hibernate and TL are way clunkier than iBatis. There were also a number of restrictions that suggested that I need to modify my schema just to enable certain features of the tool... in the end, the identity tracking is a leaky abstraction - the whole story with the attached and detached entities smells to me.

Now the thousand dolar question, given that most applications out there DO NOT use horizontal partitioning or sharding, caching in the application layer is usually more efficient anyway, identity tracking and the transparent (hidden) lazy fetching often causes more problems than it solves (and countless hours lost in app tuning with dubious results), why are so many people obsessed with the idea of cramming an ORM into their application?

The other day I wrote a small application that had four composite entities and a couple of fairly sophisticated queries, the the data access part took me about less than an hour (that's implementation, testing and tuning), so:

• How long would it have taken me if I used JPA? (to get reasonably well performing, tested code)
• If you are using ORM, how many entities do you have? How much time (percentage) do you spend working on your persistence logic? Are you using the database as operational storage or are you trating it as a long-term asset? Do you share the DB with other applications?
• Are the productivity gains realized only when your persistent model is so big that it's unmanageable by other means? Is it to futureproof a product (what is a reasonable timeframe for ROI, any examples?)?
• How important is for you the ability to scale your data access to multiple database servers?
• Have you ever switched an application from one SQL server to another? How much effort was it? What does the application do with the DB?
• What are the benefits of using ORM smart caching and partial queries in comparison with in-memory database like TimesTen or H2?

Please enlighten me...

The Shape of Complexity

Some things are complex and there's no way around it. Still, even if we can't remove the complexity, often we can shape it into different forms. Some applications have shallow and wide complexity - many simple things with relatively few dependencies, but the overall system has mind-boggling emergent behavior (my favorite 25kloc perl scripts example). Other applications have narrow and deep complexity - small code base, but using so much infrastructure amd metaprogramming, that to understand what's going on, you need to be expert in the platform (think about Hello World written with EJB2 or any application using Ruby on Rails).

The horizontal complexity is pretty easy to deal with - find two similar things and create abstraction for them, rinse and repeat. It is important to stop and look every now and then for similar abstractions and for ugly usages. Eliminate the similar abstractions by extracting common functionality in helpers or superclasses; deal with the ugly usages by splitting the abstraction.

By definition, when we introduce abstractions, our complexity becomes 'narrower' and 'taller'. If we put too much stuff into the abstract classes or we nest them too much, we might transform the bunch of simple classes that we couldn't understand when taken together into a somewhat smaller bunch of more complex classes that we can't understand even in isolation. If go overboard in the other direction, e.g. adding too many facades and convenience methods, the API becomes too big without providing enough benefit to learn it (canonical example - I know a project that has a Strings class, containing constants for empty string, single digits, punctoation marks, single letters and other. Apart from being pointless, more verbose and difficult to apply consistently, this also couples all classes in that project to the util package).

So, how do we end up with code base that is neither too tall, neither too wide, but just the right shape?

• Acknowledge that the right shape depends on the individual - some people can cope with more abstraction, while others can remember more facts. A metric for abstraction efficiency can be defined as delta-loc/n^{levels-of-indirection} (where n is a constant bigger than 2).
• Keep in mind the choice of tools - for example IntelliJ IDEA excels at navigating well-factored code, while Vi people often prefer decoupled classes that can be changed with low risk of impacting other areas of code.
• Consider the infrastructure maturity - using XA transactions with JMS is simple, debugging buggy JMS (caugh.. activemq.. caugh) is entirely different issue.
• The experience of the team is important - for some people JMS is the most obvious way to send a piece of data once and only once to another system; others treat JMS as black magic and resort to FTP and cunning rename+move schemes, involving multiple directories and recovery scenarios.
• And finaly, even if different pieces of infrastructure can provide similar functionality, sometimes they vary by the way they do it - in my previous article about component granularity, I mentioned the fat component vs fat Mule configuration scenarios. In this case, we can move the complexity between the Java code and the Mule code. Mule provides out of the box abstractions for threading, routing and transforming, but it is not great as a general-purpose process definition language. Java provides general purpose language, but does not provide high level primitives for threading and routhing. Another example: consider running OLAP against OLTP schema and dedicated warehouse schema - it works, but the difference in speed and CPU utilization can be orders of magnitude.

Pushing Files

In the next couple of posts I'll try to capture a couple of scenarios that I think Mule (and in general ESB) fits in. The target audience are people like my friend who once asked me once 'Well, I got it - it's all about transports, transformers, routers and components. In the end, what would you use it for?'

When I started reading about Mule, the first thing that caught my eye was the huge collection of transports and third party modules. It allows you with a relatively little configuration to automate scenarios like sucking a file, uploading it to FTP and sending an email if it failed or moving it to a audit dir on success. Add a few lines and you can have Mule try to upload, if it fails, try to post to JMS queue and if it still fails automatically create a JIRA issue.

Still, even though Mule is easy to use, it's not simple. Actually even for basic use of Mule, you need to know XML, have idea about Enterprise Integration Patterns and be familiar with Java server applications. Suffice to say that many of my colleagues do not meet these requirements. Compare this with writing a bash or Perl script - it's much easier and quite often it is good enough. I wouldn't be a stretch to say that the TCO of a Perl script for the previous scenario is 10 times less than the TCO of an application built using Mule or any other free or commercial integrated solution.

Things change a bit when you have a big number of integration jobs. If the tasks are similar, a smart scripter would abstract away the processing, capturing it into a parameterized sub-routine or shell script, with multiple simple wrappers with different arguments. Unfortunately, much too often the strategy is copy-and-modify (especially when the guy who wrote the original script has left the company, and the guy who did the first 30 clones also). Cloning is a pragmatic, low-risk strategy since it minimizes the chance that you break stuff that works. Such developers are usually praised for their get-it-done, goal oriented attitude and management likes their low-risk approach. Still, the dirty secret that nobody likes to talk about is that the system grows in complexity until the moment when nobody knows what is it doing anymore (I think Skynet started that way).

Even we assume that the TCO of a single Perl script is 10 times less than the one of a Mule implementation, a single Mule instance with relatively little custom development can easily handle the job of all the 170 scripts that I counted in one directory on a production server I use. If we assume that these scripts were derived from 10 archetypes (actually it's less) this still gives us a cost reduction of 170%. Even more important - the Mule and Spring configurations give you a roadmap, which while not trivial is much easier to comprehend and audit than the 27,227 lines of repetitive Perl code.

To be fair, in this example I am comparing bad scripting code, written by multiple people over the time, without proper governance to an hypothetical Mule solution written by decent developers. Let's assume that you have a team of good scripting developers, writing well structured code and taking care to provide maintenance documentation and guidelines for extending the system. Actually, this would work fine. Effectively these guys will be defining a platform quite similar to what you get from an ESB (in the end it's all Turing-complete languages). The only catch is that in my experience, in a typical enterprise it's much less probable to meet such Perl coders than the above mentioned Java guys. The argument that Perl programs don't have to be messy is quite similar to "guns don't kill people".

Even if we assumed that the mythical Perl programers did exist (and you had all three of them in your IT department), there are still a couple of points that make an Mule an attractive solution.

• Mule gives you a proven architectural framework, so you don't have to invent your own. This includes guidelines, some documentation (getting better) and abstract base classes for many of the EIP patterns.
• The business code written against Mule is usually framework agnostic (POJO).
• Various transports, transformers and out-of-the box components.
• Handles advanced stuff as XA transactions and threading models that are just not possible using Perl.
• Since the bulk of your application will be built on a platform, it will be easier to find people whi have some experience with it, which will shorten the learning curve.
• Extensive JMX monitoring and management.

And here are some good reasons to keep cloning scripts:

• Naturally resilient to regressions.
• Allow you to hire cheaper developers (even if you hire expensive high-qualified ones, they will be just as good as the cheap ones).
• If you plan to scrap the whole thing in the near future (i.e. because the whole system is being replaced)
• If the job is not critical, doesn't change often and forking a script is so easy that there's no point investing in anything better.
• leg·a·cy [ˈle-gə-sē] - code that actually works.

Component Granularity Musings

This article was written based on my experience with Mule ESB, but the general principles should apply for other similar products as well.

Short description of an ESB application

A typical ESB (Enterprise Service Bus) application starts with an endpoint, which is an abstraction for a way to receive some 'event' which would make us do some work. The event carries some information, which could be structured (i.e. JMS message or HTTP request) or simple (the timer fired) - we'll call this information 'payload'. The endpoint passes the event (containing the payload) to a 'component', which does some kind of work and optionally emits a single output event. The output event goes to an outbound endpoint which can send the result back to the client, write it into a database, relay it to an external system, or do something else.

This was a high-level description - the benefits seen from here are that we get to reuse the communication logic between applications and since the endpoints are abstract, we can easily swap the actual transports. Still, a typical ESB provides some additional functionality. In the example above, when we receive an event it was relayed to a predefined component, but that doesn't have to be the case. We can have multiple components configured in the ESB and use 'routers' to determine the component that will process this particular event.

The routers could determine the actual component depending on the payload properties (in which case we call them CBR - content based routers) or based on anything else (e.g. we can have a support call router which routes to different endpoints based on the time of day). When router receives an event, it optionally emits one or many events (unlike a component that optionally emits one). Also, components are supposed to contain exclusively business logic, while routers are mostly concerned with mediating the events in proper manner. Other useful types of routers include aggregating, splitting, resequencing and chaining routers.

For the sake of completeness there are also 'transformers' which can be plugged between endpoint and component to apply some transformation on the event payload, converting the data or adding additional information (enriching); and 'filters' which can be used to discard the messages based on some criteria.

Since the routers, filters and transformers allow us to conditionally specify the components and endpoint, we can choose whether we want to implement our logic inside the component or outside (in the ESB configuration).

Ultra-fine grained routing (anemic components)

When each component contains no control-flow statements (straight-line imperative code), all the application logic is implemented in the ESB configuration. Some would argue that a system implemented in this fashion is more flexible and allows for faster turnaround and better introspection. While these statements are true, we need to consider the other side as well.

First, you are mixing business logic and communication logic at the same level, which is bad since it forces the maintainers to understand both (ideally they should be separated as much as possible). Not only this, but your code becomes platform-dependent (the configuration is code and in this case the ESB becomes more of a platform and less of a glue). This in turn increases the chance that upgrades of the ESB software would break your stuff (especially if you use undocumented or experimental features). We also need to keep in mind that the domain complexity has not changed - we are just expressing it using a different language.

Second, chances are that you have more developers knowing Java than the ESB config language. Also, during development it's much easier to step through the code of a single component rather than trace a message as it goes through multiple queues and thread pools.

From operations point of view, there are more knobs to turn, which increases the chance to turn the wrong ones. The monitoring can take advantage that there are more inspection points, but then you need the consider the usefulness of this information. Do you really care to know how many executions have you got for sell orders and how many for buy orders?

Many vendors (IONA, TIBCO, Progress) offer proprietary process modeling and metadata management facilities, allowing to express your business rules, routing and transformations in graphical notation and enforcing integrity based on metadata. These are expensive products and well worth their money if your project is big enough and you want to accept the vendor lock in. In that case, make sure that you make the most of it and take the time to learn how to use them instead of putting a half-assed simplification layer on top of them (just an example - so far I've seen two wrappers around Spring (in different companies) aiming to make it 'easier' to use. No need to say that none of them had any documentation).

Ultra-coarse grained routing (fat components)

In this scenario we have one component only. Multiple inbound endpoints go in (possibly passing through transformers and filters); and a single stream of events goes out. We use a content-based router to dispatch each event from the output stream, to one of the multiple outbound endpoints.

This is a code-centric approach (you won't make much use from a box-and-arrows editor). It allows one to use Java and standard Java tools and debuggers to implement, unit-test and debug the bulk of the application, while still abstracting the communication code and mundane stuff like transformations. If you want to expose application details for monitoring you have to do it by manually registering your custom MBeans or using the Spring JMX exporter.

One sign that you should consider splitting your component is if you start doing threading. This includes maintaining worker pools, doing half-sync/half-async dispatching, messing with locking, synchronization and notifications.

Sometimes there might be better abstractions for some pieces of code. For example, if your component is implementing a simple generic transformation and you are using Mule, consider extracting a Transformer. On the other hand, if the transformation is simple, but you feel that it is not generic enough then you have a choice to a) move it to a transformer and have your component focus on the business logic; b) leave it there and reduce the number of classes you maintain.

The Fine Line between Fat and Voluptuous... Components

...this time it is not the J Lo's booty. Actually, as much as I've looked I haven't found a good set of recommendations about how to structure components in an ESB. There's some stuff from the SOA guys, but they have some ill gotten assumptions that all the invocation between services has to be remote and marshaled through XML, which is not necessarily the case for a single application using ESB as a platform.

I am in no way authority on EAI, but here is my attempt at defining some guidelines (take them with a big lump of salt):

• Multiple components make your application complex. A good starting point for a new application is sticking everything in one component and extract components as necessary using the following guidelines. In general I've found it easier to split components than to merge components (Java's method-invocation semantics is less expressive than ESB routing).
• If you need to checkpoint your processing, consider splitting the stages into different components and separating them with durable queues. The queues could be either JMS or Mule VM queues with persistence enabled. With proper use of transactions, this would allow you to survive application crash or do a failover (you still need to take care of any non-transactional endpoints). Alternative is to keep using one component and checkpoint using a distributed transactional cache configured with redundancy.
• If you need to control the resources allocated to certain part of the processing, you can extract it in a separate component and change the number of workers using a SEDA approach (check also this article; the Scatter-Gather and Composed Message Processor patterns). Another valid approach is to use a compute grid like GridGain, giving you more functionality, but increasing the total complexity of the application. We should also mention GigaSpaces, providing a platform based on JINI and JavaSpaces with good integration capabilities.
• When parts of the processing have different state lifecycle. It is best illustrated by example: we receive a stream of events on an inbound endpoint (let's call it 'control stream'). Each event has a payload consisting of condition and a 'data stream' endpoint address . On certain conditions, we want to start receiving and processing the events on the data stream in a way which requires independent state. The implementation would be to have one component that would monitor the control stream and keep track of the registered processors, creating new ones as necessary. The processors are concerned only with their data stream of events and each of them has separate state. A clumsy alternative (antipattern?) is to have a single component subscribed for all data streams and use a cache to lookup the state based on some key derived from the incoming event. A possibly better alternative for this specific case is to embed an ESP processor like Esper
• If you want to extract a component to reuse it, consider extracting it as a domain object first. If it encapsulates processing and is ESB aware, then perhaps it's better off as transformer, router or agent (using Mule terminology here). So far I haven't had the need to reuse a business-logic component between projects.
• If two components are tightly coupled and do not take advantage of any ESB-supplied functionality on the connection between them, consider hiding them behind a single Java facade. It's easier to unit test a POJO than to integration-test a component.

As always, any opinions are welcome.