Yesterday, I blathered on about how great DataPower would be for usage in WebSphere Process Server. Your developer would be able to stay completely inside the Generic Business Object (GBO) data view while leveraging DataPower’s awesome ability to transform efficiently. Very neat indeed.

An interesting is scenario where DataPower IS the ESB. No WebSphere Process Server, No WebSphere Enterprise Service Bus. Do it all inside the appliance. How can this be done? Well DataPower possesses all the major services that are required in order to implement an integration: Routing, Transformation/Normalization, Validation, Authentication, Logging and protocol bridging.

• Routing: A single request can cause multiple endpoints invocations
  • DataPower provides variables that span the context of a Request/Response
  • Currently datapower does them in serial but I hear that parallel invocations may be supported.
• Transformation/Normalization: As we all know, the XSLT abilities of the product
• Validation: Again, a built-in feature of Datapower to validate SOAP envelops and also against XSD schema
• Authentication: DataPower can integrate with Tivoli Access Manager (TAM) and Tivoli Federated Identity Manager (TFIM)
• Logging: Datapower can write events to it’s own logs, or even emit Common BaseEvents (CBE)
• Protocol Bridging: Datapower supports MQ, SIBus (WebSphere Default Messaging Provider) and Web Services (along with some pure TCP protocols)

If you gave me a system that can just do this, I can provide you an Enterprise Service Bus that’s pretty damn good. 99% of the time in your system will be spent waiting on web service responses, rather than bottlenecked inside of your application service. With the appliance, I get both the ‘server hardware’ and the ‘application server’ in a single box for a cost thats substantially less than a license for ESB, server hardware to support it and administrators to maintain it.

Now, if I look at the functionality of WebSphere Enterprise Service Bus, I notice a lot of the same constructs.

• Routing: Mediation Flow Component*, Imports/Exports
• Transformation/Normalization: Also done via XSLT (in v6.0.2) + various primitives
• Validation: Built into some primitives.
• Authentication: Leverages all the capabilities of WebSphere
• Logging: WebSphere tracing, CBE Emittal
• Protocol Bridging: MQ, MQ/JMS, SOAP/HTTP, SOAP/JMS, HTTP (v6.1)

* Mediation Flow Components are great for mapping a single request to a single response, but they fall apart when you want to invoke multiple endpoints and compose a new service from them.

I’m hard pressed to find a ton of scenarios where I can recommend WESB to a customer with no plans to ‘move up’ to Process Server. Scenarios exist, but I’m trying to take the view of the most common constructs in an integration. Process Server is relatively safe, you’ll still need it for Human Tasks, Long running BPEL, State machines etc. So I see the market for that product alive and well.

Of course, there are trade offs to using DataPower as your ESB. It doesn’t integrate with WebSphere Adapters. It doesn’t participate in a dual phase (XA) transaction. You have to be an XSLT guru. You don’t have a nice candy-coated UI like WebSphere Integration Developer. If you are a mid-size company looking to get your bang for a buck, you may not even miss those features.

In summary: I still like what I see with DataPower

Related Posts

• WB552: Random DataPower Thoughts Part 12
• DataPower – Proxy & Co-Processing
• Make SOA real with IBM WebSphere Enterprise Service Bus and IBM WebSphere DataPower SOA Appliances
• What’s new in IBM WebSphere Enterprise Service Bus V6.1
• WB552: Random DataPower Thoughts Part 10

In a previous post, I discussed the DeveloperWorks Process Server Endurance article. In that article, they ran into a situation where the number of maxMessages was breached (100 by default). At my client site, we had a situation where messages in an Event Sequenced queue would remain locked forever, causing the maxMessages to be breached.

Strangely, It only occurred in our production cluster. We worked with IBM Support and it was found that this is a known issue that was resolved in WPS 6.0.2.2. Our cluster’s messaging engines are configured as active-passive which allows for fail-over processing.

Background explanation of Event Sequencing

By default, messages are processed in the order that they arrive in a queue. While this is a true statement, you have to remember that Message Driven Beans (MDBs) by default allow 10 instances to be receiving messages. So while they get picked up in order, they may be processed in parallel. A problem arrives when one message is dependent on the processing of a message earlier in the queue. Think of a typical database insert or update choice. If the row doesn’t exist, I want to insert it. If the row exists, then update it with the new info.

In a single threaded scenario, the first message is processed, creates the row. The second message arrives and updates the row. In a multi-threaded scenario, both messages are picked up at the same time and non-deterministically both may land up determining they need to insert the row, resulting in duplicate rows.

Event sequencing allows us to say “Hey, message1 and message2 are actually related. They have an id that means they work on the same data. Process server, if you see a message with an id, please ensure that you don’t process the next message with the same id until the first message has completed processing”. This results in a system where messages with the same id are processes sequentially while messages with unique ids are processed in parallel.

Background explanation of Event Sequencings Implementation in Process Server

Event Sequencing has it’s own database inside of the WPSDB that is created via the WPS install procedures. (FYI, Event Sequencing and any tables or tools that it uses are normally prefixed with ‘es’ somewhere. ‘esAdmin’ allows you to interact with the event sequencing runtime). In this table, event sequencing stores the UID of the message on the queue, the value of the user-defined key, and various module related information about where the message is supposed to go.

When a message arrives on a queue with event sequencing turned on, the first thing that happens is the sequencing mdb will pick up the message, query it’s table to determine if the key is currently ‘locked’ (in use). If it’s in use then the message remains locked on the queue, waiting for the lock to be released. If it’s not in use, the key becomes locked and then Event Sequencing passes the message off to the module/component which begins it’s normal processing. When the invocation is complete, event sequencing removes the semaphore from the database and the next message with that key is processed.

The Problem Report

Event sequencing works fine on a single server, but what happened on a clustered server is that the UID of the message actually changed depending on which messaging engine picked up the message. In a clustered fail-over scenario, the UID of the message became out of sync with the UID in the event sequencing database. The “new” UID message would complete, but event sequencing would display an error about “Unable to find UID xxxxxxxxx” when trying to unlock. It just left all the locks in place.

My Rant-y Feature Request

One thing I didn’t quite understand is why a lock wouldn’t time-out after X minutes. Event sequencing is bound by the global WebSphere transaction timeout value of 180 seconds. You have 180 seconds to complete your synchonous call or WebSphere will time it out and rollback. No lock in the database could possibly exist longer than this timeout. The user could then define the action to take (stop processing the next message, or just continue and assume the risk). Ceasing the processing of a certain key is a huge value to my client. We deal primarily in migration projects where the old system used to shut down the listener port on any error, rendering the application unusable. In today’s WPS architecture, we get Failed events but we lose the sequencing of the messages. I’d love to be able to say “key XYZ is in failed state, please just accumulate the subsequent messages while support performs problem determination”.

Today

We’re in the process of upgrading our versions but support was able to reproduce and verify upgrading resolved the problem.

Related Posts

• WebSphere Process Server: Messages sitting on queues unprocessed
• WAS: Resolution to Messages in the SIB locked forever
• WebSphere Process Server Endurance
• BPEL: Beware the use of nested loops in a short running process
• WebSphere Enterprise Service Bus does not support Event Sequencing

If you’ve worked developing an application for more than five minutes, chances are that you’ve encounter a big long exception trace. The best thing to do is try to look at the very top for a non-generic exception that relates to your problem. This article will attempt to describe exactly what happened via the information contained in the stack trace. Hopefully it will provide a better understanding of the system and all the wonderful stuff that WebSphere does under the covers (but then exposes to you in dirty stack dump form).

This is an extremely long post, so click for the whole thing
Read the rest of this entry »

Related Posts

• WebSphere Admin Console SRVE0190E: FileNotFound Errors when clicking certain hyperlinks
• WebSphere Adapter for JDBC Export – ‘InvalidVerbException’
• WebSphere Adapter for JDBC: SetObjectKeys NullPointer Exception
• Do not use forward slashes in SIBus Destination names with JMS
• Deconstructing CWWBA0135E: Class ‘x’ does not have a feature named ‘y’

I stumbled across a great article on developerWorks that outlines what’s new in WPS v6.1 and a second article on whats new in WID v6.1. It’s nice to see IBM come out with a detailed article of whats changed between versions aside from the usual useless marketing banter.

WebSphere ESB improvements

New mediation primitives let you extend the reach and capabilities of these products. One new primitive enables mediations to use business object maps in the mediation flow. Fan-in and Fan-out scenarios are supported through new primitives for splitting one message into multiple parts, or aggregating several messages into one. ServiceInvoke is a new primitive to invoke a service within a mediation flow.

If you used WESB in the v6.0.x stack, it was a crippled solution. It was designed strictly for mediations of a single request to a single response. You were restricted to using an old RAD based XSLT editor that was great at losing your mapping and frustrating to use for large data objects. You couldn’t access the nice ‘candy’ UI of the Business Object Mapping runtime and you had to write custom snippets to invoke multiple services in a single mediation flow.

With these features in v6.1, it narrows the gap between when to choose a ‘Module’ and when to choose a “Mediation Module”. Mediation modules can now perform a solid degree of choreography. It also provides smaller businesses with the ability to leverage the cheaper license cost of WebSphere Enterprise Service Bus to get their foot into the door with IBM software.

WSDL and XSD support

WebSphere Integration Developer V6.1 has several improvements for the handling of WSDL and XSD. Additional XSD constructs are now supported, such as xsd:choice, the xsd:any family, and xsd:union. Validation is improved as well, using the same WSDL validator as IBM Rational Application Developer, enabling better performance.

XSD support in WID/WPS has always been a hot topic with me. I spent a lot of time working with these APIs when I was a member of the Test Client team at the IBM Toronto Lab and I know far too much about XSD and limitations. I’m happy to see that we’ve not got support for the more complicated types and I hope that the xsd:any support is actually a first class citizen in this release.

Integrated Test Client
New capabilities enable the automation of testing. You can author test cases manually or from a test client invocation trace. You can organize individual test cases into test suites that belong to a test project. Tests can then be deployed to the integrated test server and executed. In addition, you can use ANT scripts for automatic invocation. You can create a component test project that holds test suites, execution traces, configurations and emulators. The test suite is made up of test cases, based on one of two available test patterns: operation-level testing or scenario based testing.

Of course, I have to mention the new test suite support from my old team. This was a feature request I heard every day from every customer over the last three years, so I’m ecstatic to see it finally there. The test client can now transform from your ‘one-off’ development test environment to a fully fledged testing solution.

Unfortunately, I don’t get the chance to use of these features at my current client. We’re a 6.0.2.x shop with no plans on upgrading in the near or distant future.

Related Posts

• Business Object Maps and ‘Required’ Attributes
• What’s new in WebSphere Integration Developer V6.2
• WebSphere Integration Developer & WebSphere Process Server v6.1.2 released
• WSDL Anti-Pattern: xsd:Any and xsd:AnyType To Encapsulate Future Changes
• WebSphere Integration Developer: Struggling with the Business Object Mapper

Developerworks has a fantastic article written about Endurance Testing with WebSphere Process Server. Taken from the abstract:

Endurance testing is an important aspect of reliability. This article provides insight into the various problems and solutions encountered by the WebSphere Process Server Validation team as they performed an endurance run on WebSphere Process Server V6.0.2. Upon completing this article, you will be able to tune up your own WebSphere Process Server environment for optimum performance and stability.

What they’ve done is setup clustered, production quality WPS instances through a typical scenario of WS invocation to BPEL to Business Rule lookups and more web services calls. Then they hammered the server with 30 business objects of 20KB in size per minute (1800 an hour or 43,000 a day) for a week and documented the issues they ran into. It also includes all the spots where they made performance tunes to the server and required databases.

I think the most valuable part of the article is the “Problems and solutions” section. Here, they describe all the problems that were encountered and the resolution. I’d like to point our three of them specifically, because I personally encountered them with my client. We actually do a smaller number of transactions per hour (700-1000) and still hit problems.

Observed out of memory errors after running for three days. We noticed that it was due to memory fragmentation.

We added the following JVM args:
-Dibm.dg.trc.print=st_verify -Xgcpolicy:optavgpause -Xcompactgc -Xnopartialcompactgc

My client had an issue where the JVM size was turned up too high and we ran out of native memory (if your server ever just “POOF” disappears, you ran out of native memory) and added the same lines.

Problems with event sequencing messages. WebSphere Process Server release comes with controlled max active message size defaulted to 100 (/opt/WebSphere Process Server/ properties/EventSequencing.properties). In our ND7 seven-day run for the PE scenario, endurance created problems in event sequencing some messages.

This problem was attributed to limited active message size. Making it 0 (which means infinite) solved the issue, and we saw the messages getting event sequenced properly.

In our project, we use Event Sequencing. We also experienced a situation where the queue stopped consuming messages because the ‘maxActiveMessages’ was blown. I believe in my heart of hearts that this is a PMR (Event Sequencing doesn’t unlock the messages properly, but that’s for another post). We changed it to 0 (unlimited) and our queue processed successfully*.

Messages do not process completely. The default value for Work managers > es-workmanager > Maximum number of threads = 20. During ND7: seven-day run for the PE scenario for endurance, we observed that certain messages were never processed completely. Some of them were queued up in the SCA queues and therefore were not processed.

Setting activation specs for BPEL defines how many instances of BPEL can be created. Depending upon the throughput, we set the number of concurrency for BPEL instances created at a given time, we changed the value to match BPEApiActivationSpec(JMS activation specification => BPEApiActivationSpec and BPEInternalActivationSpec, the max concurrent value is 40). Increasing the concurrency to 40, helped solve the issue.

My client currently has an issue where messages are sitting in internal BPEL SCA queues in the ‘LOCKED’ state and not getting processed. We haven’t yet followed this recommendation but I suspect we will need to.

The only negative thing I have to say about this article is that they focused too much on ‘working around’ the problems and not delving deeper within IBM to get them fixed. The article was published on 18 Jul 2007 but here it is 27 Feb 2008 and the problems still exist.

Related Posts

• Event Sequencing Message Locking – Resolution!
• WebSphere Process Server: Messages sitting on queues unprocessed
• BPEL: Beware the use of nested loops in a short running process
• WebSphere Process Server Discussions
• WebSphere Default Messaging (SIBus) and WebService Error Handling

The com.ibm.websphere.bo.BOFactory interface defines two methods:

• DataObject create(namespace, name)
• DataObject createByElement(namespace, name)

Both methods take the same two parameters and both return DataObjects. Most of the time, create() is used and returns the DataObject as expected. Sometimes, though, you’ll find that it returns null when the namespace,name pair is correct.

The reason for this peculiarity is based on the structure of the XSD that backs the Business Object. If your XSD looks like:

<xs:complextype name="Person">
  <xs:sequence>
    <xs:element name="firstname" type="xs:string">
    <xs:element name="lastname" type="xs:string">
  </xs:element>
</xs:element></xs:sequence></xs:complextype>

Here, the type “Person” is defined with xs:complexType, so we use the create(..) method. If you created your business objects in WID, they are in this format.

If however, our definition uses an ‘xs:element’ tag such as:

<xs:element name="Person">
  <xs:complextype>
    <xs:sequence>
      <xs:element name="firstname" type="xs:string">
      <xs:element name="lastname" type="xs:string">
    </xs:element>
  </xs:element>
</xs:sequence></xs:complextype></xs:element>

Then we have to use createByElement(..) to get our DataObject as the complexType attribute is ‘anonymous’ (lacking a name attribute) and the name is only provided by the ‘element’ element.

You may be asking yourself why the BOFactory doesn’t just look for one and then the other by default? It would be a very good question. The problem with that solution arises when you have an Element type declaration and a complexType declaration in the same namespace with the same name but unique attributes. You’d never be able to get an instance of the DataObject for the ‘element defined’ structure.

There’s one interesting thing to note when it comes to using the ‘Create Specific BO’ Visual Snippet. This snippet is actually implemented by calling create(..), checking if the value is null, and then calling createByElement(..). I guess they didn’t have the same concerns that the BOFactory team did

Related Posts

• Interop between BOXMLSerializer and JMSDataBindingImplXML
• Visual Snippet Editor: Forgets Java Syntax
• WebSphere Process Server and an RPC/Literal WSDL : Trench War
• WebSphere Transformation Extender – “One or More Inputs was Invalid”
• WebSphere Integration Developer – Namespaces Cannot Span Projects

I’m a large supporter of the IBM DeveloperWorks Forums for WebSphere Integration Developer and WebSphere Process Server (including WebSphere Enterprise Service Bus). Having the wealth of experience on this platform that I have, I tend to respond to 75%-80% of the questions. Sometimes, a question appears that sparks my brain with a rant or deserves a deeper explanation. I created this site to capture that information.

In addition, I’m going to blog about my own experiences with IBM’s Business Integration suite and SOA in general. This’ll include the times when things are great, and (more likely) the times when they aren’t so great.

I’m currently Perficient-ly titled a ‘Technical Architect’ which means I spend a lot of my work day writing architecture and internal implementation documents. I’m also going to tackle a lot of the design decisions that I make in a project that leads to the systems my clients create.

I’m also a large proponent of idea sharing. I’ve learned most of what I know through the intellectual generosity of my professional colleagues through the years and would like to take this opportunity to give back to the community. So I encourage any readers to feel free to send me emails with suggestions of topics you’d like to know more about.

In summary, this blog will contain:

• My own take on Business Integration and SOA
• In depth answers to questions I encounter
• Design Decisions
• Reviews of Articles I find on the internet on BI and SOA
• Quirks/Questions/Issues that occur with my engagements
• Information on new products that I get a chance to play with (ex. WebSphere Business Services Fabric)
• Whatever else evolves in this space down the road

Cheers.

Related Posts

• Reader Introductions – Who are you? What you up to? What would you like to know?
• I’m now an Independent Consultant looking for Contracts
• Request for Questions & Comments
• The lack of a non-IBM WebSphere Community
• Seven Reasons Why Corporate Blogs Suck