Service Versioning and Schema Evolution

This is a topic that comes up quite often when talking to clients. Based on some of the successful implementations that I have been lucky to be involved in, the one demoninator is how important XML Schema is in that success. In addition the existence of a Schema Custodian (Thomas Erl term) or a individual who guards the Schema with his/her life. The more difficult a Schema is to change, the more likely that it will succeed in an enterprise SOA. Sounds kind of strange but the foundational aspect of XML Schema in its use in WSDL, BPEL, XSLT, XPath/XQuery, auto-generated skeletons/stubs.... means that minor changes could seriously impact a deployment.

One of the clients I work with has successfully employed a XML Schema definition for 3+ years now. The schemas have been allowed to evolve via the use of optional elements but those evolutions have been through a small group of Schema Owners and it is not modifyable via the developer community. Because of the inherent difficulty in changing the schema, it has prevented Schema creep and the potential interoperability damage quick changes could cause. The benefit of this has been interoperability across a wide range of technologies starting with four different app containers and two different JMS providers.

One other area of discussion is on versioning of Schemas/Services. One approach that I have seen is the use of the namespace and ESB technology to route based on unique namespaces. Namespace versioning can be something like:

http://www.mycompany.com/2007/12/ProvisioningService

This service would then have been developed in December 2007. People are familiar with this approach with some of the WS-* standards like WS-RM. The good part of this is that for me to invoke a service using this as the target namespace, requires that my SOAP Request call it out. The bad news is that changing this will break any clients that were developed against it. A quick solution is to support several services if the namespace changes by infrastructure like an ESB and XPath Routing. A simple XPath statement focusing on the SOAP Envelope could route this request to the correct destination. There are several other versioning solutions, check out developer.capeclear.com for a whitepaper on Service Versioning.

Fine Grained vs Coarse Grained BPEL

At a customer this past week, the team was examining the WSDL that had been defined for a new project. The WSDL was broken into three definitions but was going to provide aggregation capabilities for a single consumer. Since the consumer portion was already under development, there was a strong desire to limit the number of changes made to the interfaces. The first discussion was whether to use the several interfaces as separate services or to boil them down into one interface. If this was done, do we just map the current operations onto the WSDL interface or do we replace them. The current services were pretty fine grained providing specific capabilities for search. The results returned were represented as different elements with no consistency. Two options were then explored:

1) Create a single WSDL that had N number of operations matching the current operations that were currently spread across several WSDL.

2) Blow up the WSDLs and start from scratch creating a coarse grained interface that provided a single search operation. For the Web Service implementation, it would need to examine the query data to determine which type of search was being requested or a new element with an enumeration of search types could be added.

In both cases, revisiting the schema and collapsing the numerous elements that existed previously into a query input and search result was agreed upon.

To demonstrate the impact of solution 1 or 2, we then turned to BPEL and developed the service implementation as a BPEL Service. Why BPEL, well this service was meant to aggregate data from various sources depending on the input.

If we examine solution 1, the WSDL would have 8 operations. One option to support this was the use of a the Pick Activity and OnMessage Activity. One OnMessage per operation could be employed creating processing pipe creating an Event Driven design.

If we examine solution 2, the WSDL would have a single operation and thus would support a more traditional Receive/Reply Activity.

So in essence, the use of a Fine Grained WSDL impacted/reduced the options for BPEL development, whereas Coarse Grained WSDL did not.

Simplification via the Service Interface

I was recently working with a client who had developed a solid demonstration vehicle showing many moving parts. As I was working with them, we identified that the service interfaces were developed with varying degrees of granularity. The data granularity for some of the interfaces was fine grained expecting small sets of data whereas other expected a full document or were coarse grained.

To simplify the demonstration and complete the round-tripping exercise they were shooting for, we standardized the WSDL definitions to use the same Schema and to push the entire document across the wire between the services. In essence we standardized on a coarse definition. This then untethered the controller service they had defined in BPEL to just copy the top level element between services.

As changes occurred in the demo, the refactoring demonstrated that maintenance or updates were much simpler or non-existent.

From a business perspective using the business documents in the interfaces provides a SOA centric approach towards development. This approach doesn't always make sense as with very large documents, there may technology or complexity reasons to limit the use of such an approach. In those cases fine grained access make sense albeit with more work on the orchestration side.

Thomas Erl SOA Training

Last week I was able to participate in training provided by Thomas Erl and his SOA Systems consulting organization. Being a fan of his books and how they crystallize the SOA concepts and associate them to the Web Services Standards, I was very excited to participate. Unfortunately I was only able to partake in 2 of the 4 days. These days focused on the SOA Methodology he has been working on namely SOA Modeling, SOA Analysis and the beginnings of SOA Design.

Through his various books such as SOA Principles of Service Design and Service-Oriented Architecture a process of examining a current business process was defined. During the process, services are logically grouped into task, entity and utility. Each of these denotes a certain level of capability or interaction. The Modeling/Analysis phases help identify which bucket they belong in and confirm they follow his 8 SOA Principals such as Autonomy, Stateleness etc.

What did I get out of the session? Simply put a methodology that allowed the team I was working with to break a current process into its core services and model the message exchanges that would occur if implemented. This was done at a high level but demonstrated a simple approach that allowed both developers and a business analyst to communicate effectively.

Operations is my customer

So I got a chance to partake in a Franklin Covey Leadership Training exercise several weeks ago. I was not sure what I would take out of the training and whether it would be beneficial. I was extremely impressed by its content, focus and most importantly the instructor. Long story short the class tried to identify what is a leader and how to become an effective one. As part of the sessions a discussion around identifying your customer came up. For the current customer I am working for, it is a little interesting in that with middleware there are many customers. Specifically though once in production, operations becomes a major customer.

What I learned is that to make an effective middleware platform, quality comes in the flavors of Reliability, Scalability, Manageability. When these quality attributes skip a beat or fail to occur, Operations has to swoop into action. The benefit of the training underscored the need for me to develop simpler and more capable tools to handle customer issues. Additionally, the increasing complexity of our SOA has begun to exacerbate the monitoring and management once in production. Log4J logs, JMX and Management Services become the virtual console for operations and development.

While these solutions may not occur immediately, I am working with them to define best practices, provide training and eventually automate solutions. This has resulted in reduced operational issues as well as increased self-reliance. Most importantly, this process never ends and must continue to be executed to provide consistency of results.

SOA Training, what is it?

Based on my current role as an independent consultant I have run into several clients looking to develop SOA technical skills inside the organization. Due to my Web Services lineage, I of course look at the SOA solution set via the W3C and OASIS toolbox. From a training perspective I highly recommend the Cape Clear Developer site for two training facilities:

1) XML Schema / SOAP/ WSDL

Check out  http://developer.capeclear.com/?q=justenough

2) BPEL

Check out http://developer.capeclear.com/?q=bpeltraining

I am looking for some decent tutorials on XSLT/XPath. Currently I have been using the www.w3c.org and O'Reilly. If you have any suggestions on alternative tutorials/locations let me know. 

3) SOA Specific Articles / Books

Thomas Erl and his series of publications are a must. http://www.soamagazine.com and http://www.soabooks.com

Finally although not specifically for SOA but rather all the other technologies like  AJAX, OOAD, HTML etc the Head First series from O'Reilly is excellent.  

To Spring or not to Spring

I am trying to get a sense from people as to which framework people are selecting, Spring or EE5. Several different projects I am privy to at different companies have employed Spring with good success. Another colleague just published a book on JBoss SEAM with its EE5 underpinnings and is very high on its capabilities. Long story short it seems like this is going to be a Ford versus Chevy debate. Maybe I am wrong but would love to hear any feedback. 

Due to the use of Spring configuration and Spring Bean support in Cape Clear 7.5, I see my path has been pre-determined. 

The Web Service Standards Soup / BigSOAGrid.com

Over the past several weeks, a former colleague and I have put together a grid of the current Web Service Standards starting with the core and extending to the WS* features. You can find this at:

http://www.soamag.com/I11/1007-1.asp

or at its home:

http://www.bigsoagrid.com

The purpose of this is to highlight the many options / directions the standards have gone as well as the large amount of vendor participation in these specifications. Obviously having vendors write standards is a double edged sword as they most often will have an offering following the standard release and will be pushing it on you and I. With the open source movement however, you will find a bunch of these standards available in Apache projects, Codehaus etc and the time/effort spent by other technologists can save time for us all.

One example of this was a project I worked on recently for a State Government contractor. There was a need to support basic authentication and integrate this with LDAP. Within a half an hour discussion, the decision was made to use WS-Security UsernameToken. The implementation took advantage of the WSSJ package from Apache and in a few hours this feature was ready to test. The alternative might have been a custom SOAP Header and the resulting WSDL / Interceptor work to build out the infrastructure. Not worth it when deadlines are tight!

So take a look at the BigSOAGrid and get up to speed on the latest and greatest. Maybe you will find a standards based option for your project needs!

SCA Like Features in Cape Clear 7.5

Building large scale SOA has several pain points. One of them is in the assembly and deployment to a SOA cluster. Cape Clear 7.5 is a providing a starter to graphically assembling SOA using Web Service standards and Spring. I got a chance to participate in the Cape Clear 7.5 webinar this past week and saw a flexible mediation development tool which in turn developed a spring configuration. Within the Cape Clear Studio (Eclipse plugin) environment, transports, transformation, logging, exception handling, routing and Spring Beans can be combined into an assembly to perform mediation tasks. The diverse transport selection and simplicity by which this is defined was impressive. 

A next step with this environment will be to apply policy between the steps/services. The final step would be in then being able to apply all these pieces to a diverse set of deployment platforms (both Cape Clear and non-Cape Clear).  Why would I want to do this?

Let me provide an example. One customer I work with has open source J2EE container, a COTS J2EE container, Tomcat, Apache Axis.... and the list goes on and on. Trying to develop a SOA in this environment is one issue, deploying to all these entities and managing this in an automated fashion is another. A single deployment description will help lead towards deployment automation as well potentially a monitoring description. 

How does UML apply to SOA?

I am currently participating in a UML / OOAD training class learning about the UML language and what it provides the Object Oriented development space. A question that I had asked early in the class to the instructor was how SOA can take advantage of UML. Unfortunately the instructor did not have an answer with his background being in application development versus service/integration development.

So here are the diagrams we have covered:

• class diagram
• sequence diagram
• state machine diagram
• activity diagram

Still to go are:

• component diagram
• use case diagram

Based on the above it appears that BPEL could be described in a non-BPEL fashion using the activity diagram. An activity diagram could simply be described as a flow chart. It supports the definition of activities, looping, conditions etc. Sounds kinda like BPEL doesn't it.

Another diagram that appears to have some use is the component diagram. I am not 100% sure but the component diagram typically shows components in a system and how they interact with one another. Looking at Martin Fowler's publication UML Distilled (Third Edition), pg 141:

"Use component diagrams when you are dividing your system into components and want to show their inter-relationships through interfaces..."

Additional details in the publication and diagram examples appear to provide a way to visually describe a SOA where Web Services are the components.

I haven't researched this with other organizations but will trying to apply these prinicipals/approaches for some upcoming projects to help in the detailed design documentation phase.

Apache CXF, alternative to Axis?

Working on a small project with a startup and another consultant turned me onto Apache CXF (Celtix XFire). Both of us began looking at open source, specifically Axis 2 to provide a capable Web Service layer for an existing J2EE hosted application. After about a couple of weeks we gave up due to the poor documentation and issues that we ran into. We then decided to look into alternative containers such as Glassfish, JBoss etc and came upon Apache CXF. Some of the factors we decided on were a JAX-WS implementation with the ability to plugin into various implementations and support for the basic WS-* standards such as WS-Security.

Today I used the Apache CXF wsdl2java features to simply generate a client stub to a .Net service. In addition I needed to enable SSL communcation. From install to executing the client, it took 3-4 hours with a bit of debugging on the CXF client configuration file.

Current impressions are good documentation, clean generated code with no major issues.

Check it out at http://incubator.apache.org/cxf

Dynamic Endpoints in Cape Clear BPEL

Creating dynamic endpoints in BPEL using Cape Clear is pretty easy. Just simply overwrite the PartnerLink using an Assign/Copy activity where the From value is the URL and the To is the partnerlink. Having done this in a previous implementation I statically defined these URLs in the BPEL itself. I could have developed a WS that returned the dynamic endpoint but thought that was too much for what was needed at the time. The URLs were static and would not change. Well 6 months later, the URLs are changing as new destinations are being added to the BPEL, in fact double the destinations.

I am now refactoring the current BPEL to use a new feature in Cape Clear BPEL, the XPath function getInitValueParam(). The getInitValueParam() function allows me to send in a property name as a string and get back its value. These properties are defined in the initParams section of the Cape Clear deployment descriptor (ws-application.xml).

I am also employing a while loop to examine the inbound XML for destination data and in combination with the above function, parameterize the destinations externally to the BPEL.

Here is the scenario, an inbound XML messages defines the names of the various intermediaries in the overall process. This is represented in an element called system as shown below:

<system>systema<system>
<system>systemb<system>

The system element is unbounded but is an enumeration of valid destinations. Within the new BPEL, a counter is initialized and a while loop determines if the counter is less than or equal to the number of recipients. For each system I wanted to extract the system element value (the SOAP intermediary name) and then use the getInitValueParam() function to return the destinations URL . I then overwrite the PartnerLink value with this destination and I have performed dynamic endpoint assignment.

Sounds good right! Well I finally got it working through the help of a fellow BPEL blogger. The problem that I had was accessing the system element via array notation. My array value was defined as a variable called counter. I tried using a combination of XPath that would get me down to the system node and then combine this with bpws:getVariableData() of the counter value. No luck. I found this link which got me through the issue quickly:

http://covarm.tvu.ac.uk/blog/?p=4

Basically, it was necessary to use the concat function in a Copy activity to create a xpath query that builds the actual expression I was interested in, in this case something like:

concat('.../.../ns1:system[', bpws:getVariableData('counter'),']' )

then, this value was assigned to a variable called xpath. Then in the next Copy activity I refer to the new variable xpath:

bpws:getVariableData('...', 'parameters', bpws:getVariableData('xpath') )

A lot of work to do something so simple in Java. Oh well, the combination of these features is helping dramatically simplify my original BPEL and providing unlimited scalability for new systems via configuration instead of coding.

Applying JUnit/XMLUnit to Web Services

As part of developing an agile SOA environment for a couple of clients, testing becomes a critical component. So perusing through cyberspace, I came upon an excellent and simple approach towards this goal. Check out http://www.ibm.com/developerworks/webservices/edu/ws-dw-ws-soa-autotest2.html . This article brings an Apache Commons HTTP Client, JUnit and XMLUnit to bear. The result is a reusable client that posts a SOAP request and writes the SOAP response received to the file system. Then XMLUnit methods are used to provide comparisons between the actual response and the desired response (stored in another file). This approach works well for Request/Response services.

The challenge I have is that my most important client does not have this style of service. Rather they are using One-Way/JMS style of communications. My thoughts here are centering around the use of a centrally available file system or database. Each of the SOAP Intermediaries would need to store the inbound and outbound requests to this storage point. Once the overall cycle is complete, the same XMLUnit features could be used to compare the entire request lifecycle.

Rules Engines / Rule Based Services

How many people have actually implemented Rules Engines? I first ran into Expert Learning Systems in the 90's when working in the Mechanical Computer Aided Engineering (MCAE) space. They were being used to electronically document engineers/designers knowledge for the development of mechanical components. The idea at the time was to use this knowledge to optimize part development and reduce lead time. The infrastructure behind this was a Rules Engine.

I don't know the answer on the current adoption of Rules Engines but for the right problem, it appears that it provides a compelling solution. I am currently investigating this sector to help identify if the technology is pluggable in a SOA and what type of business problems are best served. It appears that logic that is dynamic is the best candidate. The ability to update an external configuration without having to go through a typical release cycle if the logic was embedded in a bean is definitely attractive. The languages are pretty different between vendors, some using Natural Language approaches while others use a proprietary scripting language. The only constant is support for the JSR-094 specification.

Most of the vendors generate a java implementation or even web services so inclusion into a Web Services infrastructure is pretty straight forward.

So far I have looked at Jess and OpenRules and found the open source community to provide a very good starting point.

Evolutionary SOA / Agile Development

About two years ago, my former company undertook a major development effort to implement an agile development process based on SCRUM. With a team of 20 developers, the company slowly but surely made the effort to transform the development process and use incremental / agile techniques to create a better quality product. The result after 2+ years is evident in the stability and consistency in releases of new features.

One of my current customers is also striving to implement a similar process. Currently development occurs in one major release a year and focuses on a monolothic EJB infrastructure. Unlike the previous company, the switch from a traditional waterfall process to a more agile/incremental environment has several challenges:

• History (the current organization was built around a single major release and has done so for many years)
• Platforms (the current infrastructure is comprised of very heavyweight EJB applications that are not flexible/agile)

The organization is tackling this on several fronts:

• Executive Support (most important is to get executive support on the use of a incremental/agile development approach which in turn will directly correlate to performance factors for the business)
• Organizational Changes (re-organizing the development team to focus on services)
• Simplification/Refactoring/Replacement (simplifying the exiting infrastructure for immediate gains and replacing the monolith with a series of simpler services for long term benefit)
• Putting out to pasture (placing the existing monolith into a legacy category thus limiting its scope/usage)
• Training (bringing in external consultants to guide the existing development staff on the best practices and fast track the simplification/replacement strategies).

Overall, I have been trying to do the same with the current ESB infrastructure. Through consistent interaction with operations and my desire to continue to refactor/simplify the production solution, improvements in reliability and customer support have been made. Incrementally I am removing / improving the production solution to make it extensible and more reliable for the large volumes of customer that interact with it. This process will never end and thus like the Toyota commercials, will hopefully result in the Corolla of ESBs (400,000 miles and still going!).

A Bad Foundation makes a Bad House

If you have worked with BPEL, you know how dependent it is on the foundation. In this case the foundation is XML Schema and the sub-floor is WSDL. Challenges introduced in the XML Schema or WSDL, ripple there way into the BPEL definition. Here are a few novice and advanced issues that have bitten me over the past several years when dealing with BPEL variables:

- Unique namespaces are critical/mandatory in XML Schema/WSDL as they are used to uniquely identify variables and their content.

- Schemas with imports should utilize xsd:ref otherwise bpel:assign activities become a nightmare. BPEL Variables that are generated from XML Schema/WSDL that import other schemas will each have their own namespaces. This makes copying root elements between two variables with different namespaces impossible using the bpel:copy activity. The option is to copy each element individually and see the BPEL script grow exponentially. BPEL 2.0 and its support of inline XSLT will eliminate this issue.

- Some Web Services are RPC Encoded versus Document Literal. bpws:getVariableData syntax is different when extracting values from a RPC Encoded BPEL Variable versus a Document Literal BPEL Variable. BPEL Editors eliminate this issue, however when editing by hand the later will include the variable, part and XPath to the element. RPC only includes the variable and part.

- One of the most simple but common mistakes is to not initialize the BPEL Variables with a literal assignment when using Document Literal services. This will cause run-time failures or other depending on the BPEL engine you are using.

WS-* Bloat, what is actually being used

Below is a list of WS/XML Specifications that I have implemented with customers that are in production or in pilot mode. The first generation specifications (XML Schema, XSLT/XPath, SOAP, WSDL) are used widely. The second/third generation specifications would be BPEL and the WS-* standards.

• SOAP
• WSDL
• XML Schema - Pretty much the standard for data definition and more powerful than DTDs (nothing new here).
• XSLT/XPath - Standard for data transformation (haven't used XQuery yet).
• BPEL - Standard for Orchestration Web Services
• WS-RM - only have used it at one client but it provides reliability as advertised.
• WS-Security - not used in production but have several customers moving to it for up-coming releases.
• WS-Addressing - examining this and other specifications for a routing road map. Not currently used in production work that I have performed.

So as you can see there are still a lot of missing specifications that are floating in the WS-* ether as well as UDDI (an unpopular specification in project based SOA). The above is based on a sampling of 200+ customers that I have worked with from 2002.

JMS Queues versus JMS Topics

Today I was working with a customer on demonstrating integration of a JMS Topic into their ESB. For years now I have spent all my JMS Integration work specifically using JMS Queues. For this integration, however, the customer was utilizing a topic. Based on my research the following rule of thumb appears to delineate when one is used over the other:

• Topics are utilized when the information is time senstive (i.e. stock quote).
• Topics are utilized to broadcast of information to a wide audience (or subscribers).
• Queues are utilized when performing transactions.

There may be several other reasons to use one over the other, but this applied to this specific customer and helped address its usage.

Personal Services show the way

Although this is not directly related to my main discussion about SOA/Web Services, I noticed a trend in the past 3 days that impacted me and shows the power of what SOA/Web Services/Internet technologies are bringing and will bring .

So on Friday I checked my gmail account and noticed that LinkedIn (http://www.linkedin.com/) had notified me of several contacts who were interested in touching base and joining my personal network. One was a fomer colleague and another was a former friend from college. Within a couple of hours I had caught up on 6 years with my colleague and 15 years with my college buddy. The part that is interesting is all I needed to do was post my LinkedIn profile (my personal WSDL). This included my destination (email address) and content. LinkedIn hosts some general operations and away you go.

So a single personal service in a short time frame brought me information relevant to me and simplified my networking life dramatically. Just imagine all the phone calls, mixers, conferences etc that you would have had to attend. Who's got the time!

Personal Integration is here and helping me and many others.

Transport or Message Level Security?

This is a question that has been posed to me at a lot of state government pilots over the past few years with regard to encryption/privacy of messages. The following is the simple rule of thumb that I have used based on the best practices ( http://www.ws-i.org ) and discussions with security/web service professionals.

• If securing two points (a client and a service for instance) and the assumption is that security is not required once it is received on either end, then transport level security is sufficient (HTTPS, SFTP etc.)
• If it is unknown how many systems/intermediaries will be touching the message then message level security should be utilized (WS-Security / XML-Encryption).

An interesting point by the security gurus is that transport level security has the potential of being at various strengths (i.e. 128 bit). As a message travels between more than one point, there is the potential for the privacy of the message to vary. This is not the case with message level security as the strength is specified within the SOAP Headers of the message and thus remains constant. This is an important consideration as a lot of state agencies begin integrating across departments/agencies which in turn may be across networks.

One simple reason for using ESB in a SOA

Well here is a real world example of why an ESB is a nice piece of infrastructure when things fail. I am working with a company which had a failed transaction. An ESB is being used to broker communications across multiple applications. The failed transaction had originally been created by one of those applications and placed on a JMS Queue for processing. Because there is an ESB, the destination service is not only available over JMS but also HTTP (the default transport for the waiting BPEL Service) or other transports that you can choose from.

We simply resolved this issue by loading the SOAP request into a SOAP Client in Eclipse and doing an HTTP Post to the service.

There are many other reasons to using an ESB but just the flexibility in supporting multiple transports can help out in a pinch.

Simple SOA Client

Several months ago I needed to demonstrate a workflow task in a BPEL Service. Searching the web, I came across an old colleague. In his spare time he founded http://www.magoosoft.com/. Using it reminded me of an email client like Outlook Express. It supports multiple protocols and more importantly renders the SOAP message as a HTML Form. So for the scenario that I was working on, it was simply defined by importing the WSDL for the BPEL Service to define the call back and messages. Once defined a few properties were specified and my magoo client would receive messages, allow me to view and then respond back to the waiting BPEL Service. All the work was done within a day and demonstrated a human workflow component for SOA Pilot.

SOA for the masses?

Most of the blogs I have read don't seem to talk about the developers viewpoint ("the masses") of actually implementing an enterprise SOA using ESB or other technologies. Maybe I have missed some good independent literature out there.

Currently I am working with a variety of clients that are large and small using third party ESB, MOM or Open Source. In my travels I have run into the good and bad of SOA and when done right, the benefits it can provide.

As I navigate my customers problems and learn/implement the WS-* standards, I'll post what I find. Maybe this will help you or at a minimum draw some debate.