Friday, January 15, 2010

Setting up PuTTY to use public keys for authentication

I've looked on the internet for a quick step-by-step guide how to get PuTTY to use public key authentication with OpenSSH daemon and it took me some time to figure. I'm posting these instructions in case anybody else has the same needs.

Prerequisites

Make sure that your OpenSSH configuration (usually /etc/ssh/sshd_config) contains the following line:

PubkeyAuthentication yes

In my case (CentOS 5.4) it was disabled by default.

Also, you would need the full PuTTY suite which can be downloaded form here (get putty.zip).

Generating the key

This is a way to generate the key with Putty. Alternatively you can generate it with OpenSSH's ssh-keygen tool and convert it to PuTTY format.

  1. Start PUTTYGEN.EXE
  2. In the parameters box at the bottom of the window, choose type of key 'SSH-2 RSA', set the bit size to 2048.
  3. Click the Generate button, move the mouse over the blank area until the progres bar fills up.
  4. Enter your notes in the comment line (this is displayed to you when you use the key, you can change later).
  5. Enter key-phrase, make it long and complex, write it down in a secure place or print it and hide it somewhere in your freezer.
  6. Save the private key (*.ppk) in a reasonably secure filesystem location. Even if somebody gets access to your private key, they will still need your passphrase to use it.
  7. Copy the text from the text box under the 'Public key for pasting into OpenSSH authorized_keys file:' and paste it on one line in a new file called authorized_keys (we'll use that later). The file should contain a single line terminated by Unix-style new-line and there shall not be an empty line after it.
  8. Close PUTTYGEN.EXE

Associating the key with your Unix account

  1. Login to your unix account
  2. Create a .ssh directory under your home if it does not exist
  3. Copy the authorized_keys file there
  4. Do chmod 700 ~/.ssh ; chmod 600 ~/.ssh/authorized_keys

This needs to be done for each machine you are connecting to. In this case it helps if your homw is NFS mounted.

Using the key directly

  1. Start PuTTY
  2. Specify user@host in the 'Session > Host Name' field.
  3. Specify the path to your private key file in the 'Connection > SSH > Auth > Private key file' box.
  4. Click the 'Open' button at the bottom of the PuTTY settings dialog.
  5. When prompted, enter your private-key pass-phrase and you will be logged in without entering your Unix password

Setting up Pageant to cache the decrypted private key

Let's look what we have done. The good thing is that our password does not travel over the wire and is not susceptible to man-in-the-middle attacks. The bad thing is that we used to enter the short and easy password of our Unix account, while now we have to enter the long and difficult pass-phrase of our key every time we establish a new Unix connection. In order to avoid this, we can use PuTTY Pageant which is SSH authentication agent (Unix equivalent is ssh-agent)

  1. Start PAGEANT.EXE
  2. Click on the computer-with-hat icon in your system tray.
  3. Choose the Add Key option and pick your private key (*.ppk)
  4. Enter your pass-phrase
  5. Close the pageant dialog

From now on, when establishing SSH session Putty will try to use the decrypted key from Pageant first and then fall back to password auth if none of the keys match.

You can create a shortcut, starting Pageant and passing the paths to your keys as arguments. This will start Pageant and load the keys in one step, but you will still need to specify the pass-phrase every time you do this (typically after system restart).

Keep in mind that Pageant holds the private key in memory unencrypted. If anybody captures a heap dump of the process, they can get access to your private key without knowing the pass-phrase. That's why, you might want to stop the Pageant if you are not using it for a long time or if you shae the machine in multi-user environment.

If using Pageant, you might also check the Putty option 'Connection > SSH > Auth agent forwarding', which will allow you to use your key from the remote machine on which you are logged on.

How fast an SSD drive do you need

If you need Intel X25-E 32G SSD for 70% of the cheapest listed price (shipping extra), please let me know.

For a long time I thaught that the bottleneck of all builds was the HDD, so when I got my new notebook, the first thing I did was to add a spiffy extra Intel X25-E SSD hard drive to it. As I expected, the builds went much faster. To my surprise, the HDD throughput stayed fairly low during the build, which suggested that the benefit of SSD drives kicks in early and buying higher grade drives doesn't make much difference as the bottleneck moves to the CPU quite quickly. All this makes sense, considering that a typical application has hundreds to thousands of files and the HDD spends a lot of its time seeking rather than reading.

When I cleared the IntelliJ IDEA caches and opened the IDEA Community project (total 900MB, 66k files), during the indexing, one of the cores stayed pegged, the HDD read throughput did not exceed 4mb/s for 4 mins then for 1 minute it rose to top 25mb/s, avg I guess around 15mb/s. The write never exceeded 10mb/s, and for most part it was bellow 4mb/s, the last minute was between 5 and 7mb/s.

During initial compilation, the 2 cores of the CPU (2.53GHz T9400) were quite busy, staying above 80% all the time, the disk utilization during compilation stayed less than 4mb/sec with the ocasional peaks at 6mb/sec. The write peaks were 5mb/sec, for the most time bellow 1mb/sec.

At the end of the compilation, the index update took 50 seconds, with average read throughput ~10mb/s, peaking at 30.5mb/s, the write peaked at 7mb/sec. During that time the CPU utilization dropped around 50%, which only shows that IDEA's indexing is not using both CPUs.

The bottomline is - it's not worth byuing expensive SSDs for consumer usage - cheaper ones are just as good for the average home and software development workflow. Most of the applications do not involve transferring huge volumes of data and the slowness of the spinning-platter HDDs is mostly because of seek times and fragmentation. Expensive SDDs are warranted if you are working extensively with media files or are processing huge amounts of data on the disk.

Saturday, November 14, 2009

Adding custom handler for JMX service URLs

Recently, I've been creating some wiki pages describing the JMX management capabilities of the system I work on and I found it quite annoying to use the MX4J HTTP Connector or copy and paste the huge-ass service:jmx:rmi:///jndi/rmi://host:port/connector URLs, so I dug in MSDN, waisted some time and without further ado, I present you:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\SERVICE]
@="URL:Service Location Protocol"
"URL Protocol"=""

[HKEY_CLASSES_ROOT\SERVICE\shell]
@="open"

[HKEY_CLASSES_ROOT\SERVICE\shell\open]

[HKEY_CLASSES_ROOT\SERVICE\shell\open\command]
@="\"C:\\Program Files\\Java\\jdk1.6.0_06\\bin\\jconsole.exe\" %1"

Copy the text in the box above, paste it in Notepad, edit the blue text to point to your latest JDK, save as service-url-handler.reg and doubleclick to merge in the registry. If you make a mistake, rinse and repeat.

Once you are done, your can use HTML like this:

<a title="Static Data Server" href="service:jmx:rmi:///jndi/rmi://prodhost0123:9002/jmxrmi">manage SDS</a>

On clicking the link, the browser will open JConsole for the specified service URL.

Monday, January 12, 2009

The other half of the puzzle

This post on service service granularity from Ross Mason (the original developer of Mule ESB) got me in retrospective mode. I've been thinking about this before and with the time, I have come up with some criteria how to break functionality into services. Unfortunately that turned out to be only half of the puzzle. The other half is the implementation and maintenance strategy that would allow you to achieve your nicely layered boxes-and-arrows vision, while delivering ROI on quarterly intervals and not falling into the JBOWS or stovepipe-system pitfalls.

I personally favor starting top-down, focusing on specific applications and task-level services, and harvesting entity and utility services from working production code. The downside of this is that it is generally more work, often seen as 'tinkering' and in case we consolidate utility code from multiple systems, will likely introduce temporary instabilities.

This makes it a difficult sell to management - what they see is the IT guys trying to get budget to work on some kind of metaphysical inner-beauty thing instead of delivering new features. Saying that this would reduce bugs is not a great argument, because you will be reminded that's why we pay the QA team and by the way, as a professional you are not supposed to write bugs in the first place.

On the other side, the PHBs of the world seems to love the idea of the Centers of Excellence, located in China or Malaysia, full of trained and cheap labour, building the utility services and coding the entity diagrams designed by the Architecture Group in their ivory tower, while the solution engineers use point and click tools to churn 'codeless' composite applications. Unfortunately this breaks at the other end - unexpected interactions are discovered at deployment time, maintenance takes too long as the whole turns out to be more than its parts, exhibiting "emergent behavior" (also known as "underspecified features") left and right.

So, there is no quick and easy way. I guess we need to become IT salesmen and we need to compete with the professional vendor scamsters and the consultancy loan-sharks (that's a topic for another post).

Thursday, October 30, 2008

Mavenizaton - A real life scenario

The other day I posted a message on a certain vendor’s support forum asking about the possibility to include minimal POMs in their jars (so they will be easier to mass-import into local repository) and I got the following reply:

We want to move from ant to maven for a long time but never got a chance to do it. Our ant script is pretty big and complex so we are afraid it will take us a while to migrate to maven. Since we have other important things to work on, we keep delaying this kind of infrastructure things that have relatively less impact than bugs. If you know any way to migrate from ant to maven quickly, I would love to hear.

So I asked if I can get access to their Ant script and they kindly sent me that 3000-lines of a beast.

The Challenge

The product is comprised of several modules which have well-defined dependencies on one another and can be used in various configurations. Each configuration is described in a build.properties file by xxx-source-path, xxx-properties-path and xxx-output-path properties. The source-path contains both Java files and misc. resources (there are no tests). The xxx-properties-path contains property files only.

In the end, for each module the build produces the following artifacts:

  • JAR files – depending on properties, these can be:
    • Obfuscated with Zelix KlassMaster
    • Debug builds (still obfuscated, but with line-numbers)
    • Postprocessed by Retroweaver for Java 1.4 compatibility
  • Zipped sources
  • Zipped source stubs – public and protected method definitions and javadoc (no method bodies)
  • Zipped Javadocs
  • User manual in PDF format

In addition to these primary artifacts, the project:

  • Generates a couple of demo applications packaged as single-click JARs and JNLP.
  • Filters all property and some source files, substituting placeholders with values defined in the build.properties.
  • Compiles on the fly a doclet and then uses it to generate the source stubs.
  • Creates a bunch of bundles, containing different combinations of the aforementioned artifacts.

The vendor has a policy of releasing new demo-build every day which expires after 28 days.

The First Step – Fake It

In fact, we, the customers don’t care whether you are using Maven internally. All we care is to receive a POM with correct dependencies and (if possible) to be able to download the libraries from the official Maven repository (a.k.a. Repo1) or if not – your private repository (i.e. http://repo.your-company.com/demo or http://repo.your-company.com/licensed ).

In order to achieve this:

  1. For each artifact, create a pom.properties file containing 3 properties: version, groupId and artifactId.
  2. For each artifact, create a minimal POM containg the artifact group, name, version and dependencies information. Alternatively you can put placeholders and use Ant filtering to resolve the group/name/version from the pom.properties
  3. Integrate the new files into your build process, treating them as yet another resource file and placing them under the META-INF/maven/your/artifact/group in each artifact JAR.
  4. If desired use artifact:deploy task to push the artifacts to a Maven repository.

This is all you need to do for the ‘regular’ JARs. For the various other files, you would want to use the same artifact name and a classifier I would suggest using:

  • sources – standard Maven convention, since the full sources are not released to regular customers, it would make sense to have them either under a different classifier (e.g. fullsources) or same classifier, but from a private repo
  • javadocs – standard Maven convention
  • jdk14 – for retrozapped binaries
  • debug – for obfuscated binaries with line numbers

Then you Migrate

If we step back, the current build proces is something like this:

  1. Preparation Phase
    1. Set variables
    2. Compile custom doclet
  2. Generate Sources
    1. Filter particular files to resolve values from properties
  3. Compile
  4. Copy resources
  5. Package binaries (create jar file)
  6. Obfuscate
  7. Retrozap
  8. Javadocs
  9. Source Stubs
  10. Assemble distribution archives
  11. Build the demos

Here is how it would look after we convert the build to Maven:

  1. All the path variables go away, replaced by standard Maven directory layout (you should be able to reorganize the source trees for a few hours on a Friday afternoon.) As a last resort you can define custom directories in the build section.
  2. The company-name, version, etc. props go away replaced by the respective POM elements.
  3. The rest of the settings are specified in profiles defined on project or user level.
  4. The resource filtering is a built in feature for maven, but if you need to pre-process Java files, you would need to attach your own plugin to the grnerate-sources lifecycle phase.
  5. Compile and Package are standard Maven targets and you don’t need to do anything.
  6. The ZKM obfuscation would be tricky because there is no Maven plug in (or at least I couldn’t find any). I suggest that you use the AntRun plugin as a temporary hack.
  7. The Javadoc plugin would automatically generate your Javadocs, you need to explicitly specify a second execution for your custom doclet
  8. The assembly plugin should help you organize your different distributions.

This leaves out a lot of details, but ultimately some of the decisions are yours to make. I would suggest using a script to reorganize your sources, so you can do a number of dry-runs and verify that the result is as expected before you change the script to perform the move in the version control system.

Project Layout Considerations

Each Maven POM has exactly one parent, but can be part of multiple reactors. Both parent and reactors are pom-packaged artifacts, but their role is different. Most tutorials use the same pom for both purposes, but there is benefit in separating the functions (more about this in a bit).

POM Inheritance

The role of the parent is to provide defaults for the child POMs that inherit from it. The POMs are resolved by the normal Maven dependency resolution mechanism, which means that if you want the child projects to reflect a change in a parent-pom, you need to install/deploy the parent first. Alternatively, you can use the project/parent/relativePath element to explicitly specify the parent-pom’s path in the file layout (in this case, the changes are picked from the file-system.)

At the top-level parent you should use dependencyManagement to specify versions and transitive-dependency-exclusions for all artifacts used by the project. No child-artifacts should explicitly declare versions and exclusions.

You should use pluginManagement to fix the plugin versions and (optionally) specify common options for plugins. It’s ok (although not recommended) to override the plugin options in the children-pom’s, but you should not specify versions there. If you need the same overrides in multiple children, consider extracting them into intermediary-parent pom.

Finally, you should try to specify most of your profiles on a parent-pom level. This, combined with the pluginManagement allows you to define the bulk your build in the parent and leave the children DRY.

The children have to explicitly specify the master-pom version which makes it a bit of a pain to increment the version (on the other side, the master-pom usually changes much slower.)

Building in Dependency Order

The normal way for Maven to resolve dependencies between projects is through the local repository. This means that if project A depends on project B and we make a change in B, in order to test A against the new change, we need to first install or deploy B and then build A, so it would pick up the new changes (this is assuming we’re using SNAPSHOT versions).

The reactor projects take care about such closely related artifacts and automatically rebuild them in the correct order. Also they act as a grouping for the sub-projects (useful when you build assemblies). The reactors themselves are artifacts using pom-packaging (same as parents), but they explicitly list the relative path to each constituting module, and this is the reason that I recommend separation between parents and reactors:

Creating Distributions

The standard way of packaging distributions in Maven is using the Assembly plugin . It allows you to describe your distribution package in terms of project resources, artifactsIds and classifiers. A typical assembly descriptor would be:

  • Create a tar and zip files with classifier ‘distro’.
    • Add all files from /src/main/assembly
      • excluding **.xml, and **.sh.
    • Add all files from /src/main/assembly
      • that match **.sh
      • and set the unix permissions to 551.
    • Add the maven artifact
      • with group-id=com.company.product.modules and artifact-id=module1 and classifier=obfuscated
      • put it under lib.
    • Add all the transitive dependencies of this project (assuming this descriptor is in a reactor project)
      • and put them under lib/ext

You can also unpack and consolidate JARs, specify manifest-entries and signing for single-JAR distributions and so on.

Each artifact can produce more than one assemblies. If you want your assemblies to go into the repo, make sure you use the ‘attached’ goal. Most of the times, I use the reactor poms for distributions, but sometimes I just define the assembly in the JAR module’s POM.

Parents and Children vs. Reactors and Modules
Category Parent/Child Reactor/Module Merged Parent+Reactor
Direction of Coupling Children know the names of their parent, parents don’t know their children Reactors knows the locations of their modules, modules don’t even know whether they are being built as a part of a reactor Creates circular dependency between children/modules and parent/reactor artifacts.
Multiplicity Each child has exactly one parent, specified in the child POM. A module can be part of more than one reactors. You must keep it to one reactor. If you define alternative reactor, the parent will still be resolved to the main parent-POM (as specified in the children POM’s), which can lead to a great deal of confusion (and undefined behaviour for some plugins).
Location The parent is usually resolved through the repo (optionally through relative path) The modules are always resolved through relative path You need to physically layout your project, so the parent POM is accessible by relative path (which is not needed otherwise).
Change Drivers The parent changes when your build process changes. The reactor changes when you add a new module By the artificial coupling, your POM is subject to more change. This is particularly bad for the parent role, because you need to manually update the parent-versions of the children.

Given this table, I’d recommend a project layout along these lines:

-+-+- project-root/
 |
 +-+- modules/
 | |
 | +-+- module1/ 
 | | |
 | | +--- pom.xml       | inherit from modules parent; specify dependencies 
 | | |                  | and module details (name, version, etc.)
 | | +-+- src/
 | |   +-+- main/
 | |   | +--- java/
 | |   | +--- resources/
 | |   |
 | |   +-+- test/
 | |     +--- java/
 | |     +--- resources/
 | |
 | | 
 | +-+- module2/ 
 | | |
 | | ...
 | |
 | +-+- module3/
 | | |
 | | ...
 | |
 | +-+- reactor1/       | e.g. all modules - used for the CI build and pre-commit
 | | |
 | | +--- pom.xml       | no parent; specify the modules as ../module1, ../module2, etc.
 | |
 | +-+- reactor2/       | e.g. only module2+module3 - used by the module3 developers only
 | | |
 | | +--- pom.xml   
 | |
 | +--- pom.xml         | modules parent - inherits from the main parent pom and specifies 
 |                      | module-specific profiles and plugin settings
 |
 +-+- applications/
 | |
 | +-+- application1/ 
 | | |
 | | +--- pom.xml       | inherit from apps parent; specify dependencies, module details 
 | | |                  | and whatever custom steps necesarry for the application
 | | ...
 | | 
 | +-+- application2/ 
 | | |
 | | ...
 | |
 | +-+- reactor/        | e.g. all applications
 | | |
 | | +--- pom.xml       | no parent; specify the modules as ../application1, ../application2, etc.
 | |  
 | |
 | +--- pom.xml         | apps parent - inherits from the main parent pom and specifies 
 |                      | module-specific profiles and plugin settings
 |
 +--- main-reactor      | specify as modules ../applications/reactor and ../modules/reactor1
 |
 +--- pom.xml           | the main parent pom

A few Tips

When You getStuck with a Bad Plugin

For all operations that you can’t easily achieve with the existing Maven plugins, you can fall back to the Antrun plugin and reuse snippets of your current build script. Yes, it’s a hack, but it gets the job done and you can fix it when it breaks.

Try to keep these hacks in the parent pom.

Dependency Management

When you define dependencies, take care to specify proper scopes (default is ‘compile’, also you should consider ‘provided’ and ‘test’). You should also mark all optional dependencies like POI HSSF. For some artifacts you might need to specify dependency-excludes, so Maven doesn’t bundle half of the Apache Commons with you.

Evaluation Builds

You might want to release the daily evaluation builds as snapshots. Maven has specific semantic for snapshots that it always refreshes them based on timestamp. For example, if you release an artifact my.group:module:1.0.0, Maven will download it once to the local repository and never look for another version unless you delete it manually (that's why you should NEVER re-cut a release once it's been pushed out - just declare the version broken and increment).

If you use a snapshot version (i.e. my.group:module:1.0.0-eval-SNAPSHOT), Maven will check once a day (unless forced with '-u') and automatically any new version base on timestamp. To achieve this you might want to cut an eval-branch from the tag of each release and change the version using a shell script or something.

Saturday, October 25, 2008

Why ORM?

I like Spring JDBC Template - it is simple, clean and saves a lot from the JDBC drudgery. It provides nice abstraction, simplifying the usage and management of prepared statements and overall if all you want is to do an ocasional query, it doesn't get any simpler. Oh, and you have full control on the underlying JDBC connection, allowing you to exploit any vendor-specific functionality. The provided functionality only makes the data access easier and supports only very basic property to column mapping out of the box (or you can write your own mapper in Java). Because of its lightweight nature, JDBC Template does not allow for transparent caching, horizontal table partitioning, vertical sharding, etc. It also doesn't offer lazy fetching of relations, identity tracking and other instrumentation magic.

Another tool I like is iBATIS. In comparison with JDBC template, it offers more structure, organizing my SQL statements in a logical way and supports most common forms of mapping the result sets to beans (including graphs of objects and joins). Ibatis does support caching (only on a whole query level), still no sharding and partitioning, lazy fetching and identity tracking. IBATIS doesn't try to parse your queries.

The Abator tool can generate iBATIS CRUD mappings, DDL from object model, and object model from a database, but frankly - in its generated form all these suck. The DDL uses only a few datatypes, has no RI or semantic constraints, nor indexes (I use it sometimes, but only as a boilerplate which I extensively tweak afterwards). The generated beans are isomorphic with the underlying tables, which is great if you are doing CRUD, but I find that most of my applications do data analysis, so the queries tend to be quite hairy, half of them encapsulated in stored procedures (for transaction processing I'd rather use Spring or plain JDBC anyway). The bottom line is that Abator handles the trivial cases, but for most of my projects it doesn't help much, so I'm back to manual coding (which is not that bad anyway.)

The first time I looked at ORM was when I checked TopLink for a project around 2000. Back then I got the impression that it introduces a lot of new concepts in order to simplify something that's already simple enough - loading a couple of beans from a table. For that project I used plain JDBC, it took me perhaps 20 mins to write and I can't remember taking much time to maintain that code - the total cost should have been less than 3 man-hours.

The second time I tried Hibernate and Toplink last year (2006) on a prototype with the same dataset and relatively small domain model (for which I finally ended up using iBATIS). The first thing that stroke me was how slow everything was - the startup time was noticeably slower than with iBATIS (about 500-800%), spent in initializing the JPA entity manager and the queries were consistently 30-50% slower. It might have been that I didn't know how to tune it, but I just couldn't achieve satisfactory results. I researched the facilities for mapping to externally defined schema and it stroke me that both Hibernate and TL are way clunkier than iBatis. There were also a number of restrictions that suggested that I need to modify my schema just to enable certain features of the tool... in the end, the identity tracking is a leaky abstraction - the whole story with the attached and detached entities smells to me.

Now the thousand dolar question, given that most applications out there DO NOT use horizontal partitioning or sharding, caching in the application layer is usually more efficient anyway, identity tracking and the transparent (hidden) lazy fetching often causes more problems than it solves (and countless hours lost in app tuning with dubious results), why are so many people obsessed with the idea of cramming an ORM into their application?

The other day I wrote a small application that had four composite entities and a couple of fairly sophisticated queries, the the data access part took me about less than an hour (that's implementation, testing and tuning), so:

  • How long would it have taken me if I used JPA? (to get reasonably well performing, tested code)
  • If you are using ORM, how many entities do you have? How much time (percentage) do you spend working on your persistence logic? Are you using the database as operational storage or are you trating it as a long-term asset? Do you share the DB with other applications?
  • Are the productivity gains realized only when your persistent model is so big that it's unmanageable by other means? Is it to futureproof a product (what is a reasonable timeframe for ROI, any examples?)?
  • How important is for you the ability to scale your data access to multiple database servers?
  • Have you ever switched an application from one SQL server to another? How much effort was it? What does the application do with the DB?
  • What are the benefits of using ORM smart caching and partial queries in comparison with in-memory database like TimesTen or H2?

Please enlighten me...

Wednesday, September 10, 2008

On Scopes

Let's define component as a programming abstraction that exposes a number of interfaces and is instantiated and managed by a container. A component has implementation that sits behind the interface and provides the functionality, which could be a custom Java class, a proxy, a BPEL engine, a mock object, or even a mechanical turk solving captchas.

In the case of Java class components, they are usually composed of smaller Java classes like the ones in java.lang.* and java.util.* packages. All but the most simple Java objects are actually object graphs which are instantiated, wired togerher and orchestrated by the Java language and the JVM. The Java language specifies the composition structure and behavior, while JVM provides the means to invoke operations and share data. In contrast, the components are instantiated by a container and the container is responsible to wire them together (usually based on a blueprint in the form of a configuration or class annotations). The container can also provide a number of declarative services to its components: transaction management, access control, performance statistics and transparent distribution. Some popular containers are OSGi, Spring, EJB, ActiveX, MTS/COM+.

Even if a container can expose a local component over remote interface without requiring any code changes, practice has shown that taking a single-server application and distributing it in such fashion usually results in a disaster. This doesn't mean that the non-invasive distribution support is useless though. Given responsible usage, it can enable you to build distributed applications without having to deal with the transport API, using clean and reusable code (POJO). There are a number of new frameworks that are built around the concept and try to provide the right level of transparency and control (Mule ESB, Spring Remoting, Spring Integration are some popular choices).

Once you get too many components in the same contaner, autowiring starts doing most unexpected things, debugging becomes difficult and you start loosing track of your system. The solution is to add scoping. Spring supports limited scoping facilities in the form of parent-child application contexts. Although it solves the problem with configuration visibility, it does not provide facilities for controlling the public interface of the scope (explicitly exported objects). OSGi tackles this problem from a different direction using controlled classloaders, Spring OSGi is trying to be the best of both worlds, giving you scoped configuration and runtime autodiscoverable scoped components.

<interlude/>

I just finished reading Distributed Event Based Systems and they have an excellent chapter on utilizing scopes as means to structure a distributed event-driven application (chapter 6). They suggest a scoped architecture, where each component can belong to multiple scopes, scopes are linked in an arbitrary graph and notifications are governed by the different scope and component properties. The scopes described can declaratively provide a number of services (think AOP for messages). Some of these services are:

  • Containment - logical grouping of components and other scopes, allowing easy addressing for event dissemination and policy enforcement.
  • Scope Interface - defining which messages can cross the scope boundary and enforcing this policy
  • Transmission policy - determine which scope components will receive a certain message and/or which components have a right to publish/receive messages that cross the scope boundary.
  • Mappings - when a message crosses the scope boundary, enrich or transform it based on the source and destination scopes.

Scopes could be used to delineate groups with different QoS requirements, or to partition the application in geographical or security domains (or even both at the same time) and apply the appropriate policies. Integrating an external systems is also easier, because they are treated as yet another scope (as opposed to "enemy in the bee's nest") and you can provide rich access to the messages that the third party is entitled to without having to bolt on yet another authentication and authorization mechanism.

The book focuses in great detail and implementation strategies, analyzing different approaches ranging from simplistic (flooding) to very complex (integrated routing). Right now, the described architecture can not be implemented directly in any eventing system that I know of, though we can see some of the ideas in Jini (peer to peer with cooperative filtering), JMS (in implementation supporting topic hierarchies), OSGi (intra-JVM) or SCA (a lot of talk about scopes and composites, but I don't see them doing anything useful with them). The authors also propose a configuration language for managing scoped systems with the specified features. I can see AMQP as a flexible low-level specification that would allow to build something like this by implementing custom exchanges. We can use the scope configuration language as an intermediate representation, from which we can generate the custom exchange code.

Ultimately, I believe that in 10 years from now this scoping and routing facilities will become standard part of the messaging middleware (and perhaps hardware?) I'm looking in the general direction of AMQP, possibly SCA (despite their WS-* fetish) and new high level languages directly expressing architectural concerns like ACME and Einstein.

About Me: check my blogger profile for details.

About You: you've been tracked by Google Analytics and Google Feed Burner and Statcounter. If you feel this violates your privacy, feel free to disable your JavaScript for this domain.

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 Unported License.