DataPower Architectural Design Patterns: Integrating and Securing Services Across Domains

August 26th, 2008 syndication Posted in DataPower, Design Decisions No Comments »

From DeveloperWorks, DataPower Architectural Design Patterns: Integrating and Securing Services Across Domains
Draft Redbook, last updated: Tue, 26 Aug 2008 - Introduction to DataPower Services - Integration Services - Security Services IBM® WebSphere® DataPower® SOA Appliances are purpose-built network devices that offer a wide variety of functionality such as the securing and management of SOA Applications, Enterprise Service Bus Integration, and high speed XSL execution.
I'm happy to see the emergence of design patterns for datapower.

Related Posts

AddThis Social Bookmark Button

BPEL: Beware the use of nested loops in a short running process

July 3rd, 2008 dan Posted in Business Integration Tips, Design Decisions, WebSphere Business Events, WebSphere Integration Developer, WebSphere Process Server No Comments »

One of the restrictions when dealing with a short running process is that it must always run within a single transaction. The implication is that the process must complete within the default transaction timeout window. On an application server, this is 120s.

We had an issue where our Process Server was creating a large number of locks to rows on the SQL Server table. Eventually, SQL Server escalation would kick in and the entire table would lock. This would preclude other users of the database from being able to read/insert data.

Upon inspection of the offending process, we found that a short running process contained three nested loops which performed multiple invocations against the SQL server database. On the original test data, there weren’t enough nested objects created to determine if a large object could complete within 120s.

In addition, by default in WebSphere Application Server there are 10 activation specs that can process messages from a queue in parallel. This means that you need to ensure thete are enough database connections to service the requests.

Our solution to the problem was to break the transaction down into smaller parts and transmit the parts via JMS queues to new modules. A one-way invocation does not require the source bpel to block for a response, so it is able to now complete within the transaction timeout boundary.

Related Posts

AddThis Social Bookmark Button

WebSphere Business Integration V6.1 Performance Tuning

June 9th, 2008 syndication Posted in Design Decisions, DeveloperWorks, WebSphere Enterprise Service Bus, WebSphere Integration Developer, WebSphere Process Server No Comments »

From DeveloperWorks, WebSphere Business Integration V6.1 Performance Tuning
Draft Redpaper, last updated: Mon, 9 Jun 2008 - Learn valuable tips for tuning - Get the latest best practices - Try the example settings This IBM® Redpaper was produced by the IBM WebSphere® Process Server, WebSphere Enterprise Service Bus, WebSphere Adapters, and WebSphere Business Monitor performance teams in Austin Texas, Böblingen Germany, and Hursley England.
This book attacks performance on three levels, Architecture, Developerment and Performance Tuning. All key issues that need to be considered to create systems that can meet Service Level Agreements (SLA). I'm happy to see this kind of information get published so close to the public release of a version.

Related Posts

AddThis Social Bookmark Button

WSDL Anti-Pattern: xsd:Any and xsd:AnyType To Encapsulate Future Changes

April 24th, 2008 dan Posted in Best Of DZ.com, Design Decisions, WebSphere Integration Developer, WebSphere Process Server 2 Comments »

The cousin to yesterday’s WSDL Anti-Pattern: The ‘Single XML String’ Service is the use of the XSD specification’s ‘any’.

‘Any’ literally means “any well-formed XML contained in this section is valid”. ‘anyType’ means “any valid XSD Type”. We can start to see the correlation between the Single XML String service and the use of these generic types. They are used to dilute the definition contained in the WSDL to attempt to account for unforeseen changes in the future.

The benefit is that the WSDL doesn’t need to change as the service evolves, even though the services themselves will have to be modified to support the new arbitrary types.

The drawbacks are numerous. Tooling support for xsd:any’s is also low. You lose message validation. You lose strong definitions of your service. All the points that were made in the previous post apply to this style as well.

This is not to say that there aren’t valid reasons to use ‘any’ and ‘anyType’. Say, for a service that dynamically routes messages based on header contents. It’s just that most of the times they are used, it’s for the wrong reasons. Yes, I’m talking about you Industry Standard Schema Definitions (easily the worst offenders).

Just in case you didn’t like any of my arguments, here is a very well written DeveloperWorks Tip on using generic types with WSDL and the pitfalls that can occur:

Summary

Although it may seem like a good idea to use xsd:any in your Web service’s XML schema, there are pitfalls to doing so: you are deviating from the intent of WSDL, and the language mapping of xsd:any may not be easy to use. Although I won’t go so far as to say using xsd:any is a bad practice, I do hope this tip has given you a better awareness of problems you may encounter before you encounter them.

Related Posts

AddThis Social Bookmark Button

WSDL Anti-Pattern: The ‘Single XML String’ Service

April 23rd, 2008 dan Posted in Best Of DZ.com, Design Decisions, WebSphere Integration Developer, WebSphere Service Registry and Repository 2 Comments »

WSDL is a very useful technology. It allows service providers and consumers to agree on namespaces, operation names, the data to be transmitted in a request and the data to expect in a response. All very good things to know, all in a platform neutral way.

Now, WSDL tells us what all the elements and attributes are for a service invocation. What can we do at the WSDL level to account for future changes to the service? This is where some implementations get it wrong.

When you are creating a service, you can only account for the changes that are expected. Maybe a new product will be added or a field may become depreciated. These are easy changes to account for via the way you structure your XSD definitions. In this case, I would say that the service takes a List of Products thats unbounded.

If the change that is being requested created a structurally different message (via additional fields), then you just have to suck it up and modify the WSDL (either marking the old fields as optional if you want compatibility with current clients) or just making all your clients upgrade. To make my point, I’m going to ignore multiple version service offerings or use of WebSphere Service Registry and Repository.

What I’ve seen though, are services where the providers attempt to reduce ANY potential modification to the WSDL interface. This usually results in a service where the operation takes a single string parameter called ‘xml’. The client is expected to serialize the message itself and include it as a string payload in the request.

What this anti-pattern does is remove the entire point of WSDL and XSD. The ability to know what data to send to a service call and what data to expect back in a response. This is no longer captured at the level it should be. If I am a consumer of this service, how do I know what the XML to be transmitted should look like? I have to go through alternative social channels to get this information. That’s the very problem that WSDL was created to solve!

Other problems with”services as strings” deal with the added complexity that the service implementor and consumer need to perform themselves. Today, you can generate a SOAP/HTTP client for a wsdl automatically. The client will serialize your platform specific objects into SOAP and transmit them. In the anti-pattern, the developers themselves need to call the code to serialize the real content and then pass it on to the generate proxy which serializes it again.

What do you gain through this anti-pattern? Well you are no longer bound to making WSDL changes as the service matures. If a new product is created then the service provider and service consumer’s implementations must be updated, but the WSDL is left alone. This pattern is normally used to allow for ‘dynamicity, to account for changes in the data sent between the two services. The gotcha is that you are probably going to have to modify the implementation code anyway to handle the new data. You didn’t really gain anything, but you gave up strict typing and validation.

I’ve found that when the anti-pattern is used, it’s with good intentions but you are far better off just specifying the data explicitly in the WSDL and dealing with changes through your regular change management process for both WSDL and implementation.

Related Posts

AddThis Social Bookmark Button

Tip: Agree To Concrete WSDL Definitions before Development Begins

April 16th, 2008 dan Posted in Best Of DZ.com, Business Integration Tips, Design Decisions No Comments »

If you are a provider or a consumer about to enter development on a web service, you should have an understanding with your counterpart of the WSDL that defines the service. It should be very well ‘baked’ and changes once development begins should be minimial.

Major things like name spaces and object names should rarely change once development begins. Name spaces and object names are treated as a key by mapping tools. If either of them change, expect to waste time refactoring your development artifacts to use the new definitions.

Your WSDL file is a contract. A contract that describes what data I can expect from an invocation. How can you expect to perform a development exercise without a strongly defined contract?

Also, the runtime that you are going to use to implement your service is irrelevant. I received three WSDLS that were completely different because the developer was switching between runtimes and I assume just autogenerating the WSDL from the tooling.

Good anti-pattern and a solid reason why these kinds of projects miss deadlines.

Related Posts

AddThis Social Bookmark Button

Impact 2008: Impressions of Day 2

April 9th, 2008 dan Posted in Design Decisions, Impact 2008, Project Zero, WebSphere Process Server No Comments »

Day two is in the bag, the B52′s are B-50-done. Perficient and MTS Allstream gave a session about SOA Patterns that I thought was today but was actually yesterday. Oops. Here’s the abstract if you have a time machine:

TSP-2518 – Effectively selecting integration patterns
Brent Legris, Perficient, Inc., Senior Technical Architect, Olivera Zatezalo, MTS Allstream Inc., Sr Manager, Integration, BT&IT

The foundation of knowledge that rests beneath service oriented architecture (SOA) encompasses a myriad of integration patterns not only messaging, enterprise service bus and orchestration, but shared databases, ETL, operational data stores, and others. Effectively selecting between integration patterns is a critical capability to develop as your SOA evolves, in order to avoid running around with a hammer thinking everything is a nail. This session interactively develops various integration scenarios, discussing pattern alternatives and their effective implementation. We will discuss the tradeoffs of different approaches, but close with guidelines that can be consistently followed to exploit the capabilities of an enterprise infrastructure.

Level: Intermediate

What I like about this presentation is that it’s about patterns and decisions and trade-offs, but primarily focuses on WebSphere Integration Developer and WebSphere Process Server. I like it when patterns come ‘out from the mist’ and you actually see them implemented in a system. I’m a developer at heart. Hopefully the presentation slides will be made available on the Impact site.

The other thing I heard a lot about was WebSphere sMash the IBM commercialized, supported version of Project Zero.

As for other impressions of Day 2, I don’t really have any. It seemed like a normal day at a conference in Vegas to me.

Related Posts

AddThis Social Bookmark Button

The VETO SOA Pattern

April 1st, 2008 dan Posted in Design Decisions, WebSphere Integration Developer, WebSphere Process Server No Comments »

I’ve never been much of a patterns guy. I kind of always have the ability to apply common sense to a system and turn up something that is straight forward and makes sense. Periodically I get asked about what set of patterns I used to decide how a system should behave. To answer this question I tend to laugh maniacally, throw a flash-bang on the floor and run out of the room.  So I was glad to see that the pattern I’ve been using recently is called the VETO pattern.

Allow me to describe the requirements of the system that led me to the VETO pattern.

I have a source system message obtained from a staging table. This message is incomplete and needs to run additional queries against the target database in order to complete it. Once these additional queries are complete, the data needs to be inserted into the target database.

The original system performed all these tasks in a single java proxy, mangling the logic for enriching the source message with the SQL for inserting into the target database. I was tasked to migrate this to WebSphere Process Server with the knowledge that additional clients would need to use this service.

My solution was contained in four modules:

The first module received the message from the source system, validated it and determined what fields needed to be enriched. After enrichment, it transforms it into a target database message.  A second module provided the methods for enrichment from the Target Database. The third module processes a fully enriched source object for insertion into the target database. The fourth module just held the WebSphere Adapter for JDBC. I like keeping it in on it’s own incase independent adapter fixes are required.

Anyway, turns out that this is the VETO pattern. A pattern that I won’t be forgetting anytime soon.

Related Posts

AddThis Social Bookmark Button

JDBC Adapter Export with events that comprise multiple rows

February 28th, 2008 dan Posted in Design Decisions, WebSphere Process Server No Comments »

At my client, we had a unique situation occur when integrating a database and an export for the WebSphere Adapter for JDBC. In a ‘typical’ scenario, a single row is written into a staging table, an SQL trigger writes this single row into the adapter’s event table, the adapter polls, converts the data into a business object and starts the module.

In this migration scenario at my client, a single transaction was actually comprised of multiple rows written into the staging table in a single transaction. The previous implementation had a built-in concept of table scanning and aggregation which we need to reproduce in process server. Each row of data was a ‘delta’ and logic is then used to ‘flatten’ these rows into a single transaction. We need a way of telling the adapter that “these n rows are not unique transactions, but actually a single transaction”.

Out of the box, the adapter doesn’t have any feature to combine multiple rows like this. A proposal was to use a long-running BPEL process using correlation to aggregate the data, but that would induce a ton of overhead and complexity onto our servers. There’s a slightly more elegant solution.

The WebSphere Adapter can represent foreign key relationships between tables as containment business objects (Parent-Child). It also supports lists of children. So what we did at a high level was create a table representation of a wrapper type that defined a primary key. Then, all the children that relate to a single WPS transaction are given foreign key relationships to the wrapper row primary key. When the wrapper row is written to the event table, the adapter picks up all the related children and we get our desired behavior: A single WPS transaction containing multiple rows of data.

The drawback is that we had to add a little bit more logic to our trigger, but the benefits are that we aren’t overloading our server trying to do the event correlation ourselves. We’ve pawned it off on a system (database) that was designed for that kind of work.

Related Posts

AddThis Social Bookmark Button