I’ve started reading up on DataPower in anticipation of writing the certification test in May (At the WebSphere Services Technical Conference 2008). Two of the usage scenarios that have caught my interest are ‘Proxy’ and ‘Co-processor’.

In Proxy, DataPower acts as proxy to your web service . The scenario given is a website that uses Cascading Style Sheets to create a website. In the non-DataPower scenario, your application server creates the XML, applies the style sheet to it and then sends it back to the client. Apparently, this is a very CPU-intensive task. When the box is introduced, the application server sends the XML content back to DataPower, and it applies the style sheet and sends the response to the client. The article described that DataPower was able to perform the transform seven times faster (7s to 1s) than the application server. The application server’s CPU usage already decreased substantially. Seems like a shoe-in product for style sheet heavy web sites. You decrease the load on the app server, making it able to do more business work and the load on the datapower box is also negligible. You could send transforms at it at ‘wire-speed’ (100Mb/s in and 100Mb/s out). Wow.

In the second scenario, the DataPower box is called by the application server to perform transformations on demand. Say a business process requires the invocation and transformation of three web service calls. Without the box, the app server does the conversion from Generic Business Object (GBO) to Application Specific Business Object (ASBO). It then invokes the endpoints, gets the resultant data and applies more transformations to create the final composed output. Depending on the size of the objects, your server could be spending a ton of time transforming data. With the DataPower box, you make WebServices calls to it to convert from GBO to ASBO, invoke the service and convert the ASBO response into the GBO response.

The very neat part of the co-processing solution is that your integration developer never needs to leave the ‘GBO’ view of the data. This makes the actual integration solution much easier to understand, and keeps the data at a generic level that most people in the company could understand.

The DataPower core business case is that they view data transformations as a commodity. It’s a commodity that shouldn’t be running on your expensive application server, mixing business logic with transformation logic. They think that your application server should spend it’s time providing business services and less time compiling stylesheets. It makes sense to me. You buy a single datapower box and it will probably be able to serve all your transformations from all your systems. You lower the cost of ownership and server maintenance (less servers) while increasing your transaction throughput (via the CPU time not spent transforming) and overall response times (DataPower is just faster than an app server).

Win-Win. Cool product.

We have a box that I can access in Perficient, so I should be able to provide some hands-on numbers once I get to that stage.

Related Posts

• WebSphere DataPower vs WebSphere Enterprise Service Bus
• WB552: Random DataPower Thoughts Part 12
• Make SOA real with IBM WebSphere Enterprise Service Bus and IBM WebSphere DataPower SOA Appliances
• IBM WebSphere DataPower SOA Appliances Part I: Overview and Getting Started
• WB552: Random DataPower Thoughts Part 9

From DeveloperWorks: Demystifying WebSphere Business Services Fabric End Point Selection:

Learn how the WebSphere Business Services Fabric Dynamic Assembler uses
content, context and contract to dynamically select service endpoints. You’ll learn
how policies are used to select candidate endpoints, and how the Dynamic Assembler
handles policy conflicts and policy resolution.

On the heels of my opening discussion about Fabric and how its a product ahead of it’s time, comes an article from developerWorks that delves into the “how” of the product. This article provides an introduction to the terms that the fabric team likes to toss around: “Policies”, “Assertions”, “Dynamic Assembler” etc.

Related Posts

• WebSphere Business Services Fabric
• WebSphere Business Services Fabric, Part 2: Extending the ontology models
• Getting Started with IBM WebSphere Business Services Fabric V6.1
• Adding custom roles in WebSphere Business Services Fabric
• Introduction to application server clustering with WebSphere Business Services Fabric

Usually, there aren’t a lot of books published on specific products as the software release cycle tends to be faster than the publishing industry can keep up with and information gets out of date quickly. I was surprised when I saw that a new book about Business Integration with IBM software was released: WebSphere Business Integration Primer: Process Server, BPEL, SCA, and SOA (The developerWorks Series).

I haven’t had a chance to get a copy of the book for review. However, I’m highly confident that it is a must read for anyone working on this product stack. Why? Because I’ve had the opportunity in my career to work with Vinod Jessani, Ashok Iyengar and Michele Chilanti. They are amongst the very top IBM consultants on this technology. A quick search of DeveloperWorks will show that they’ve published many articles (Michele has a great article on setting up a cluster that I’ll review later) and are proven authors.

I expect that this book will be a great resource for both new and experienced Business Integration specialists.

Related Posts

• IBM Redbooks: WebSphere Process Server and WebSphere Integration Developer
• Exploring IBM SOA Technology and Practice reviewed by Don Ferguson
• WebSphere Business Integration V6.1 Performance Tuning
• Goals
• The lack of a non-IBM WebSphere Community

I would be remiss if I didn’t mention my first article published to developerworks (Co-Authored with Richard Gregory). It’s about the WebSphere Integration Test Client and learning about the features that are beyond “enter data, press continue, watch module run”.

Here’s a short list of the things that you’ll discover:

* Testing modules: selecting one or more components in a module to test and testing multiple modules.
* How to reuse common values.
* Attaching to a module when an external client calls components.
* Emulating broken or missing parts of an assembly diagram.
* Saving and rerunning tests

I, obviously, recommend that WID developers read it. Hopefully you can shave a few seconds or minutes off every test and save a substantial amount of time in your development cycle.

Related Posts

• Taking component testing to the next level in WebSphere Integration Developer
• Exploring The Contents of a WID Module Project
• Article about how to write Articles for WebSphere DeveloperWorks
• Adding JMS or MQ/JMS Custom Header Attributes
• IBM Certification Tests

I stumbled across a great article on developerWorks that outlines what’s new in WPS v6.1 and a second article on whats new in WID v6.1. It’s nice to see IBM come out with a detailed article of whats changed between versions aside from the usual useless marketing banter.

WebSphere ESB improvements

New mediation primitives let you extend the reach and capabilities of these products. One new primitive enables mediations to use business object maps in the mediation flow. Fan-in and Fan-out scenarios are supported through new primitives for splitting one message into multiple parts, or aggregating several messages into one. ServiceInvoke is a new primitive to invoke a service within a mediation flow.

If you used WESB in the v6.0.x stack, it was a crippled solution. It was designed strictly for mediations of a single request to a single response. You were restricted to using an old RAD based XSLT editor that was great at losing your mapping and frustrating to use for large data objects. You couldn’t access the nice ‘candy’ UI of the Business Object Mapping runtime and you had to write custom snippets to invoke multiple services in a single mediation flow.

With these features in v6.1, it narrows the gap between when to choose a ‘Module’ and when to choose a “Mediation Module”. Mediation modules can now perform a solid degree of choreography. It also provides smaller businesses with the ability to leverage the cheaper license cost of WebSphere Enterprise Service Bus to get their foot into the door with IBM software.

WSDL and XSD support

WebSphere Integration Developer V6.1 has several improvements for the handling of WSDL and XSD. Additional XSD constructs are now supported, such as xsd:choice, the xsd:any family, and xsd:union. Validation is improved as well, using the same WSDL validator as IBM Rational Application Developer, enabling better performance.

XSD support in WID/WPS has always been a hot topic with me. I spent a lot of time working with these APIs when I was a member of the Test Client team at the IBM Toronto Lab and I know far too much about XSD and limitations. I’m happy to see that we’ve not got support for the more complicated types and I hope that the xsd:any support is actually a first class citizen in this release.

Integrated Test Client
New capabilities enable the automation of testing. You can author test cases manually or from a test client invocation trace. You can organize individual test cases into test suites that belong to a test project. Tests can then be deployed to the integrated test server and executed. In addition, you can use ANT scripts for automatic invocation. You can create a component test project that holds test suites, execution traces, configurations and emulators. The test suite is made up of test cases, based on one of two available test patterns: operation-level testing or scenario based testing.

Of course, I have to mention the new test suite support from my old team. This was a feature request I heard every day from every customer over the last three years, so I’m ecstatic to see it finally there. The test client can now transform from your ‘one-off’ development test environment to a fully fledged testing solution.

Unfortunately, I don’t get the chance to use of these features at my current client. We’re a 6.0.2.x shop with no plans on upgrading in the near or distant future.

Related Posts

• Business Object Maps and ‘Required’ Attributes
• What’s new in WebSphere Integration Developer V6.2
• WebSphere Integration Developer & WebSphere Process Server v6.1.2 released
• WSDL Anti-Pattern: xsd:Any and xsd:AnyType To Encapsulate Future Changes
• WebSphere Integration Developer: Struggling with the Business Object Mapper

Developerworks has a fantastic article written about Endurance Testing with WebSphere Process Server. Taken from the abstract:

Endurance testing is an important aspect of reliability. This article provides insight into the various problems and solutions encountered by the WebSphere Process Server Validation team as they performed an endurance run on WebSphere Process Server V6.0.2. Upon completing this article, you will be able to tune up your own WebSphere Process Server environment for optimum performance and stability.

What they’ve done is setup clustered, production quality WPS instances through a typical scenario of WS invocation to BPEL to Business Rule lookups and more web services calls. Then they hammered the server with 30 business objects of 20KB in size per minute (1800 an hour or 43,000 a day) for a week and documented the issues they ran into. It also includes all the spots where they made performance tunes to the server and required databases.

I think the most valuable part of the article is the “Problems and solutions” section. Here, they describe all the problems that were encountered and the resolution. I’d like to point our three of them specifically, because I personally encountered them with my client. We actually do a smaller number of transactions per hour (700-1000) and still hit problems.

Observed out of memory errors after running for three days. We noticed that it was due to memory fragmentation.

We added the following JVM args:
-Dibm.dg.trc.print=st_verify -Xgcpolicy:optavgpause -Xcompactgc -Xnopartialcompactgc

My client had an issue where the JVM size was turned up too high and we ran out of native memory (if your server ever just “POOF” disappears, you ran out of native memory) and added the same lines.

Problems with event sequencing messages. WebSphere Process Server release comes with controlled max active message size defaulted to 100 (/opt/WebSphere Process Server/ properties/EventSequencing.properties). In our ND7 seven-day run for the PE scenario, endurance created problems in event sequencing some messages.

This problem was attributed to limited active message size. Making it 0 (which means infinite) solved the issue, and we saw the messages getting event sequenced properly.

In our project, we use Event Sequencing. We also experienced a situation where the queue stopped consuming messages because the ‘maxActiveMessages’ was blown. I believe in my heart of hearts that this is a PMR (Event Sequencing doesn’t unlock the messages properly, but that’s for another post). We changed it to 0 (unlimited) and our queue processed successfully*.

Messages do not process completely. The default value for Work managers > es-workmanager > Maximum number of threads = 20. During ND7: seven-day run for the PE scenario for endurance, we observed that certain messages were never processed completely. Some of them were queued up in the SCA queues and therefore were not processed.

Setting activation specs for BPEL defines how many instances of BPEL can be created. Depending upon the throughput, we set the number of concurrency for BPEL instances created at a given time, we changed the value to match BPEApiActivationSpec(JMS activation specification => BPEApiActivationSpec and BPEInternalActivationSpec, the max concurrent value is 40). Increasing the concurrency to 40, helped solve the issue.

My client currently has an issue where messages are sitting in internal BPEL SCA queues in the ‘LOCKED’ state and not getting processed. We haven’t yet followed this recommendation but I suspect we will need to.

The only negative thing I have to say about this article is that they focused too much on ‘working around’ the problems and not delving deeper within IBM to get them fixed. The article was published on 18 Jul 2007 but here it is 27 Feb 2008 and the problems still exist.

Related Posts

• Event Sequencing Message Locking – Resolution!
• WebSphere Process Server: Messages sitting on queues unprocessed
• BPEL: Beware the use of nested loops in a short running process
• WebSphere Process Server Discussions
• WebSphere Default Messaging (SIBus) and WebService Error Handling