SOA Sidewalk

Getting soapUI AMF example on MacOS Running

2011-08-07T07:21:00.000-07:00

If you are trying to get soapUI 3.6.1 AMF example running with BlazeDS samples, make sure you do the following:

- Place the flexsample.jar file under /Applicatons/eviware/soapUI-Pro-3.6.1.app/Contents/Resources/app/lib
- Restart the application and create a TestSuite, TestCase, AMF Test Step
- Specify the endpoint of: http://localhost:8400/samples/messagebroker/amf
- Specify the AMF Call as: product.getProducts

All is good!

SEVERE: WSSERVLET11: failed to parse runtime descriptor: exception during WSDL parsing

2011-06-28T08:43:00.000-07:00

Have been doing development of JAX-WS service in Metro/Tomcat 6 and ran across the following failure when deploying the WAR file. Examination of google results didn't provide the exact answer I needed but resolved it by doing the following:

- Under the WEB-INF a directory called com existed that I had noticed before.
- Examined the contents of it and found the package and java classes was reproduced in this location.
- Deleted the directory, recreated the WAR and redeployed.

All was good!

Musings about Oracle SOA Suite 11G

2010-12-11T17:34:00.000-08:00

A little over a year ago I undertook a project to migrate a customer of mine from Cape Clear to a new solution. The solution chosen was Oracle Fusion Middleware and more specifically we settled on the Oracle SOA Suite 11G. With no training, great assistance by Oracle Product Management and lot of trial and error we were able to get SCA composites with BPEL components developed within a 1.5-2 month time frame. Like most SOA Platforms, the secret was not so much in the development but more so in the runtime and tuning. Like most BPEL engines, this one was no different in that the Database and its proper configuration required great effort on the part of Oracle support and my clients DBA and WLS experts. The migration took approximately 10 months. So what did I learn?

- Development with Web Service standards is a significant time saver.
- We struggled with where the rubber meets the road.
-- JMS Adapter configuration within the container to integrate to a non-certified JMS Provider.
-- Oracle RAC Tuning and Configuration for SOA Suite specifically
-- WLS Tuning and Configuration for SOA Suite specifically

So in the end migration if planned properly can be done, but always leave extra time for the tuning and performance testing.

Oracle vs. Open Source

2010-12-11T10:37:00.000-08:00

With all the clients we work with, HKM has to be flexible to meet their budgetary needs. Some of our clients will have enterprise agreements and others are just starting. To this end, open source is the defacto starting point for many. Over the next several months I will be diving into the Apache solutions to a take another peak at the most recent Apache CXF, Camel and Ode. I am specifically interested in Ode to see how it has progressed from when I tried it 2 years ago and how close these solutions are to Oracle SOA Suite and other traditional vendors.

Oracle SOA Suite 11G, JmsAdapter and Sonic MQ Topics

2010-11-22T07:13:00.000-08:00

Recently we were working with a customer who needed to integrate SOA Suite to Progress Sonic MQ and specifically topics that were hosted. The first step with Oracle is to edit the JmsAdapter.rar configuration (specifically weblogic-ra.xml) to point at Sonic MQ. Sonic

<connection-instance>
<jndi-name>eis/sonic/queue</jndi-name>
<connection-properties>
<properties>
<property>
<name>ConnectionFactoryLocation</name>
<value>MyCF</value>
</property>
<property>
<name>FactoryProperties</name>
<value>java.naming.factory.initial=com.sonicsw.jndi.mfcontext.MFContextFactory;java.naming.provider.url=tcp://sonicmq1:2506;com.sonicsw.jndi.mfcontext.domain=Domain1</value>
</property>
<property>
<name>AcknowledgeMode</name>
<value>AUTO_ACKNOWLEDGE</value>
</property>
<property>
<name>IsTopic</name>
<value>true</value>
</property>
<property>
<name>IsTransacted</name>
<value>false</value>
</property>
<property>
<name>Username</name>
<value></value>
</property>
<property>
<name>Password</name>
<value></value>
</property>
</properties>
</connection-properties>
</connection-instance>

Having integrated to Sonic Queue(s) in the past, we were running into an issue where the SOA Suite cluster was picking up the message for each node within the cluster and was unaware of initially how to solve this. A simple solution was identified and specifically it was within the jDeveloper JmsAdapter Wizard. The following image shows the "Durable Subscriber Id". By simply specifying a value here, the cluster is only allowed to consume one message from the topic.

Oracle 11.1.1.2 XPath issues

2010-03-21T18:43:00.000-07:00

Currently I have been migrating BPEL for a client using Oracle SOA Suite and ran into incompatibilities between 11.1.1.1.0 and 11.1.1.2.0. A simple example that worked in 11.1.1.1.0 and not in the latest patched released is a simple evaluation where I am trying to confirm the existence of an element in a message.

<bpws:getVariableData("message","part","/element/element")> = true()

The same statement also works in Cape Clear ESB and its BPEL implementation.

One workaround that I used was to count the nodes using the Oracle XPath Advanced Functions:

ora:countNodes("message","part","/element/element") > 0

I am working with Oracle support services to see how this can be resolved in the code base.

Open Cloud Manifesto and Standardization

2010-03-21T18:24:00.000-07:00

The Open Cloud Manifesto provides a solid definition of the state of cloud computing. Just like other industry movements in distributed computing, enterprise integration or other, cloud computing is evolving in many different directions but the core patterns and architecture of outsourced infrastructure, platforms and software supporting elasticity, multi-tenancy, metering etc.

The hope of the manifesto and other cloud computing standards/organization, standardization in the areas of deployment, execution and management are desirable. When standards exist, it allows us to focus on generating business logic and the value it provides versus the technical wiring and potential workarounds necessary. Just like Web Services and the standards they developed over the past 10 years, I envision cloud computing will take a similar period of time to clarify and flush out the details that can be standardized.

The one area that has traction is support for REST/SOAP based APIs. In fact as companies develop private clouds and potentially expose them as to the public, other architectures such as SOA become important in providing concise and clear interfaces to business logic.

Wiseclouds: Eating the Dog Food

2010-02-15T17:23:00.000-08:00

As part of our development of WiseClouds, it is important that we demonstrate our agnostic nature in evolving the company and its technologies. The site itself is a Google Apps Engine based with integrations into Salesforce.com and Amazon EC2. We will also include Azure in the future along with WordPress for blogging services. This simple example allows us to demonstrate the use of PaaS, IaaS and Saas.

Wiseclouds is out

2010-02-15T17:10:00.000-08:00

As part of my own transformation to work with customers in new and interesting ways, I am collaborating with a team at WiseClouds to bring cloud computing training and consulting to a wider audience. Similar and yet different then SOA, Cloud means many things but is already proving its value in a few customers I work with and is on many a tongue in IT.

Anyway, check out what we are working towards which is generating very useful training content along with a group of experienced practitioners in Service and Cloud Computing technologies.

Cloud Computing begins but beware of the bill!

2010-01-18T11:39:00.000-08:00

Per Gartner and other research organizations, customers have gotten familiar to the Adoption curve slide as part of state of the union for a technology segment. Cloud Computing in its various flavors (SaaS, IaaS and PaaS) has hit main street or in Gartner parlance is with the Early Adopters. Organizations that adopted SaaS (i.e. Salesforce.com, Workday) realized the financial benefits of the pay per subscriber model. Now as the space has rolled out Infrastructure and Platform services, customers are dipping their toe in the water or jumping right in. Some teams are placing functions in the cloud to take advantage of the elasticity features due to the processing volume necessary. Others are rapidly developing applications to fill voids in their technology stack sometimes by the Sales organization or other non-traditional technology. This wild-west period eventually settles down when the rest of the marketplace catches up and Cloud becomes the standard technology medium.

The piece that is daunting right now is the cost that can creep up on you if you are not careful. With billing being an aggregation of the data stored, data shuttled, number of instances, number of connections or other metrics, it gets easy to forget the costs especially if elasticity is enabled in the environment. Similar to mobile data plans, triple play packages, the end of the month bill can be staggering. For now, make sure you not only architect a solution but also the cost models associated with that solution. A good rule of thumb is when developing and testing within the cloud, the costs are quite reasonable. If developing a large scale application that will be running 24x7, be clear on the aggregate costs and talk with the vendor about pricing flexibility.

The following examples provide details from two popular vendors on pricing:

Microsoft Azure Pricing

Amazon Web Services Pricing

Cloud to Cloud Integration

2009-12-03T18:53:00.000-08:00

After having hosted a Cloud Computing event in November and hearing how companies are dipping their feet in the cloud, the next logical step we discussed was inter cloud communication. As a small company I may buy into Salesforce.com for CRM, Workday for HR/Financials and Google AppEngine for custom apps. How do I tie these solutions together for data synchronization? What if the data to be exchanged is extremely large? What solutions can I use to do the heavy lifting?

Service Bus solutions either deployed locally or in the cloud provide a composition medium to support such interactions. Cloud platforms expose public APIs via REST or SOAP for near-real time communication.

Large data sets and exchange of this for processing is being investigated by a interesting consortium called Open Cloud Consortium.

Another interesting area may be the creation of middleware in the cloud. Grand Central tried to support this in the early 2000 timeframe. It was a little ahead of its time but now we are seeing Amazon SQS, Microsoft Azure w/Biztalk and other similar platforms.

QCon, Security and Musings

2009-12-03T18:34:00.000-08:00

I was fortunate to speak at QCon San Francisco, CA on November 20 discussing Service Security and my own journey on understanding security but more importantly how services can be hacked. It was interesting when examining the audience to see a mixture of participants but the lack of questions was a little disconcerning. I could take three things from that:

1) Everyone in the audience was familiar with service hacking / security.
2) People are not very familiar and were afraid to ask questions or didn't understand the content.
3) People were not interested.

Since the audience stayed for the entire presentation and questions were basic, I think the majority of the audience was in category 2.

With the increasing discussion on Cloud Computing (QCON was loaded with Cloud presentations as was SOA / Cloud Symposium 2.0/1.0) security had very little presence. In understanding security, my own education was due to a client requirement. With just a couple months of effort, I was able to better understand the security technology. The harder part has been in understanding the hacking culture, finding helpful material and approaches and how that impacts services I create. Here is the link to the QCon presentation.

ECAUG 1.0: Architecture User Group

2009-11-05T17:23:00.000-08:00

Wednesday this week I hosted a user group in Waltham, MA focusing on Cloud Computing experiences. I was fortunate to have Steve Robbins of Modus21 and Matson Wade representing HKM at the event. In their discussions, they broke down their experiences with Amazon Web Services specifically EC2, S3, SQS and a few other features. If you are interested in participating in the future, let us know. We can extend an invite to the Ning forum that we have setup and you can check out the presentations. A couple of criteria at the moment are you are based in the US and you are willing to proactively participate. No vendors are allowed which in turn provides a more frank and interesting discussion.

SMB not fit for SOA?

2009-10-29T11:33:00.000-07:00

During a training class I was leading today, a very interesting question came up which is "All this SOA stuff we have been talking about today is great for a large company but seems to be overkill for the SMB (Small / Medium Business). Since SMB makes up a much larger percentage of the business world then traditional business, why should we care?".

Well one way to look at the problem is that services are permeating the cloud computing space, in fact last week's SOA Symposium co-hosted a Cloud Computer Conference. Cloud computing is becoming a game-changer for SMB in providing a variety of tooling from productivity (documents, email, etc.) to CRM (i.e. Saleforce.com) to HR (i.e. Workday) to Infrastructure (i.e. Amazon/Azure). The one common aspect in these is the ability to interact using service APIs and thus the need for SMB to be adept at service consumption and composition.

Versioning Podcasts

2009-10-14T19:05:00.001-07:00

As part of my work with SOA Systems / SOABooks.com, I was asked to support Podcasts on versioning based on my participating with David Orchard (formerly BEA). Check the two podcasts out and fire over some questions if you have any.

Flexible Contracts?

2009-10-14T18:54:00.000-07:00

After having delivered SOA Training to various clients this summer and fall many have asked how to deal with change. The contract is the center piece of services and with an effective versioning strategy it can be difficult to alter due to coupling that occurs with service consumers. David Orchard, James Pasley and others have been documenting versioning strategies. From my work with David on the SOA Patterns book(Erl 2008) and my consulting with organizations employing Agile Methodology, contract refactoring/change is a mandatory requirement.

Identifying a versioning strategy such as Strict, Flexible or Loose along with Versioning Identification and Compatibility design patterns provides a foundation for indicating change and providing consistency in managing the consumer impact.

For further details check out a presentation related to mixing Agile along with SOA Design Patterns and Meet in the Middle Strategy.

SOA Symposium 2.0: Rotterdam, Netherlands

2009-10-14T18:15:00.000-07:00

It is amazing another year has passed and SOA Symposium 2.0 is around the corner. As part of the presentation work, two areas that I have focused on in blog postings and customer work is in Agile Methodology and Service Hacking. Check out the presentations and feel free to post any questions.

East Coast Architecture Group: Cloud Computing Seminar

2009-10-14T18:07:00.000-07:00

The past couple years has seen an increase in the discussion and interest in Cloud Computing. In my own experience it started with Salesforce.com in 2003/4 and then Workday which acquired Cape Clear in 2008. More recently I have been tracking the use of Amazon EC2/S3 etc. and the impacts that IaaS is doing to the IT community. To help customers and architects that I have interacted with in the past 10 years, my company is hosting / starting an Architecture User Group. The first topic to be discussed is Cloud Computing and people that have experiences with this medium. The following URL provides specifics on the topic and dates:

East Coast User Group: November 4th, 2009, Waltham, MA

Justification for constraining your XML Types

2009-07-11T13:19:00.000-07:00

As part of work for a client and some training I need to perform in the future, I have been spending time on hacking of web services. A simple mechanism that is often unchecked is the use of types that have no restrictions within XML Schema and WSDL. An example is simple schema below where the Social Security Number is not restricted in length and type:

<xsd:element name="SSN" type="xsd:string"/>

If this element is used as a part of a SQL Query in a Web Services, there is the potential for SQL Injection attacks. SQL Injection is where hackers look for elements that are not constrained and thus can take advantage of the un-restricted size to insert additional sql. A simple mechanism to reduce this is to restrict the element via simpleType.

<xsd:element name="SSN" type="SSNType"/>

<simpleType name="SSNType">
<restriction base="string">
<minLength value="9"/>>
<maxLength value="11"/>
</restriction>
</simpleType>

This then reduces the overall buffer available for the hacker between 9 and 11 characters.

Agile in the trenches (Part 1)

2009-05-21T18:37:00.001-07:00

As part of my research on how Agile Methodologies can interface with Service-Oriented Architecture, I am working with a client performing a 2 week iteration. The iteration is focusing on automating a process via a Service Composition. The goal of the iteration is for an architect to provide a reference implementation for a development team. The development team will then use the reference implementation as the basis for the final service composition.

The reference implementation employs design patterns for version identification, process abstraction/centralization as well as design standards and industry standards. This implementation reduces the requirements on the development team to create the entire service composition and more on modification of a framework. The overall idea of having architecture teams support development teams came about in discussions with Kevin Davis. We discussed how the enterprise architecture team could become a much more effective participant in development cycles by performing co-development.

During execution of the architecture iteration the following was performed to keep the development team/architecture team interactions simple and concise:

1) Stand up meetings are replaced with email stand up meetings. The customer is the development team and to reduce the impact on this team, a simple summary of the current tasks and blocking issues are identified. The development team manager is responsible for getting blocking issues resolved.

2) An iteration planning meeting still occurs in this model to identify the user story(s) and development tasks. These are then entered into a Agile Management Tool for tracking and visibility.

3) A retrospection meeting occurs to demonstrate the user story solution and delivery results for the identified development tasks. As part of the delivery, a step by step documentation set is provided to help the development team utilize the reference framework.

This same process is being executed by other Architecture team members for future features/technology required by other development teams. The execution of these iterations is meant to help validate/alter the process towards a consistent delivery approach.

SOA in the DOD / Government

2009-04-07T17:38:00.000-07:00

I just had the opportunity to participate at the latest SOA Symposium focusing on the Department of Defense and other government agencies. The theme was on the people aspects of SOA and how success is directly measured by the amount of collaboration and teamwork in the project.

Vendor presentations by IBM, Microsoft, Oracle, Red Hat and Amberpoint talked to Cloud Computing, SAAS, Event Processing, Open-Source and Governance.

Separate to this I have had the opportunity to do some training and consulting in the area exposing me to the contracting base and additional government agencies. It is clear that data sharing between agencies appears to be increasing and currently service APIs are seen as the goal.

SOA Architecture and Agile Methodology

2009-03-19T19:52:00.001-07:00

Over the past couple of months I have been exposed to colleagues, professional acquaintances, and training that has discussed the merits of Agile Methodologies in creating a loose but improved development organization. One client that I work for has several teams that are employing short iterations and have improved their overall deliver of software functionality. The company has also spent time training the teams via experienced Agilists and employed collaboration software to support epics, mrf, user stories, product backlog etc.

Working in the architecture group, there is currently no process in place but we are considering applying Agile and short 2 week iterations to deliver value to the product teams. In essence have the product teams help define challenging issues to be solved to free them up and allow them to focus on their tactical user stories/development tasks. In addition having the architecture team sit in on the product teams iterations to identify / guide the development teams towards existing services in the domain/enterprise to reduce/eliminate rebuilding existing logic. This can be done by participating in the Iteration Planning Meeting at the beginning of each iteration.

From these experience I will be trying some role/organizational approaches towards SOA in an Agile environment. They include setting up clear versioning policies for the service contracts and using design patterns to decouple the contract from the logic. This will allow the contract to more easily change from its potential tactical definition to a more domain/strategic one.

How easy is SOA?

2009-03-12T22:01:00.000-07:00

When reading blogs on SOA or technology in general, what strikes me is the amount of misleading information that exists in the space or even worse, the lack of information. When tackling new technologies in open source or standards, the lack of documentation or practical information is quite often missing. Once in awhile, I find a clear and concise document that is able to help me learn a technology rapidly. An example of this was some work I was doing debugging a Hibernate issue. To really understand the process/facilities that Hibernate provides, the base documentation provided a solid basis and was able to get me to completion within a couple of hours.

Compare this with work I was doing with some WS Standards, where the only information I could find was the specification and a poorly documented reference implementation.

As part of my business diversification towards providing training in the SOA and Open-Source marketplace, I will be trying to distill how SOA, the available tools and design standards can be applied to people that have little to no experience. An example of this is a group of QA personnel who I will be training in SOAPUI.

Another group I am training is around building a SOA Application that is front-ended by FLEX and back-ended by MySQL.

My current experiences is always focused on the KISS principle. Don't assume the audience is familiar with XSD, WSDL, Service-Orientation etc.

To date these students are getting the examples and are helping evolve these heterogenous scenarios into potentially valuable training offerings. So I can definitely answer that SOA is not easy however if approached properly, a wide audience can participate and incorporate the core principles and technologies.

"SOA is Dead" Questions

2009-02-03T10:28:00.001-08:00

Anne Thomas Manes from Burton Group recently posted an article related to the Death of SOA. Many customers that I work with who are in the middle of SOA Projects had questions on the article. Anne's presentation on SOA Fatigue at the SOA Symposium was also a prelude to this posting. If one examines the article, you can draw the following points:

- SOA Technologies from various vendors have become obtuse, costly and difficult to implement
- Enterprise SOA success in the marketplace is limited to a small collection of customers and is not widely seen due to the amount of effort to establish the entire Inventory.
- Service-orientation principles are still valid and important in providing value to the customer.
- Services should be the focus not SOA.

If one is attempting to apply SOA to a domain or Enterprise, we should revisit the goals of SOA:

- Intrinsic Interoperability
- Vendor Independence
- Increased Federation
- Business and Technology Domain Alignment
- Increased ROI
- Organization Agility
- Reduced IT Burden

These goals are then supported by the following principles of service-orientation

- Standardized Contract
- Loose Coupling
- Reusability
- Abstraction
- Autonomy
- Statelessness
- Discoverability
- Composability

The application of service-orientation helps guide business analysts and architects towards resuable, agnostic services. By following these principles, we in affect are directly supporting interoperability, vendor independence and federation. Service development also benefits from business and technology domain alignment when analyzing existing processes in an Inventory Analysis.

The success of customers I work with has been on supporting a domain versus the entire Enterprise. Success of the domain efforts in several cases has caused the business side of the organization to consolidate functionality in other divisions to leverage the domain services and their demonstrated benefits.

The key in Anne Thomas Manes blog is that the fundamental piece of SOA, the service, is still important. A service is logic that has applied service orientation to a meaningful extent. This in combination with SOA Design Patterns provides the architect/developer with a solid foundation for realizing the benefits. How you the develop, register, host, monitor the service can be done in a heterogeneous way with a wide range of standards, technologies and patterns.

SOA Design Patterns Online

2009-01-06T19:26:00.001-08:00

In a previous post I talked about the first SOA Symposium held in Amsterdam last October. One of the book releases from Thomas Erl was SOA Design Patterns which is now available in an updated form online. Check out www.soapatterns.org. In addition to the published patterns are candidate patterns around REST, binary data transmission and others.

WOA versus SOA

2008-12-31T12:02:00.000-08:00

My discussion point in my last entry discussed how SOA is agnostic in which technology should be used to employ it. Joe McKendrick and Dion Hinchcliffe at ZDNet along with research I did for a client identified a growing interest in using REST to support SOA. ZDNet refers to it as Web Oriented Architecture, a subset of SOA.

Dion Hinchcliffe's discussion on WOA

Some points that are referred to in these articles identify these characteristics:

- HTTP(S)
- POX
- JSON

Effectively from the runup of Web 2.0 technologies, the ever increasing amount of public APIs, and the increasing amount of mashups that are being performed, Web Resources are more available then Web Services. WOA is being identified as the necessary amount of complexity to get the job done.

If we compare WOA to SOA, there are some distinct differences with one distinct one being the Service or Resource Contract. WSDL/XSD/WS-Policy are front in center for describing a Web Service. Within REST, the contract is implicit and is based on HTTP verbs and XML Schema. If developing a web service, the contract analysis, design and development effort is the main focus whereby the service foundation layer and data representation layer are identified. In WOA, the data representation layer is the focus instead.

Web Services != SOA

2008-11-21T07:16:00.000-08:00

During a recent trip to a company technical summit a discussion came about related to how SOA is just Web Services. As a SOA Trainer, Architect and Implementer I live and breath it every day. Most peoples perspectives probably fall in this category and currently most of the SOA implementations that I have worked with have used Web Services. The key is that Web Services is just an implementation technology not a SOA.

Books from Thomas Erl, Intel Press and others have identified that SOA is a design paradigm that focuses on a series of principles that tie back into key business goals and benefits. Examples of these are presented at SOA Systems but include:

- Interoperability
- Federation
- Cooperative Development between Business Analysts and Technical Architects
- Vendor/Technology Choice
- ROI
- Reduced IT Effort
- Business Agility

Now a SOA could be constructed using CORBA, Java, SCA/SDO, Web Services, REST etc. The implementation mechanism that one chooses should support interoperability, federation etc. Currently XML Based technologies provide the best interoperability both in communication/exchange but in vendor support, standards, tooling but this may change over time and probably will.

So if a person asks me what SOA is, I simple identify it as a design paradigm for developing and evolutionary distributed architecture. It has its basis in OO, AOP, EAI, BPM and other styles. It is like the Bruce Lee of architectures having evolved from various fighting styles.

SOA Design Patterns

2008-10-10T14:01:00.000-07:00

As part of the SOA Symposium I was excited to watch the release of the latest Prentice Hall title, SOA Design Patterns by Thomas Erl. This is a large effort and I was fortunate to be asked by Thomas to participate in a contributing author capacity. Over 80+ patterns are outlined and include many of the speakers at the SOA Symposium. It is good to see such cooperation and the development of more independent material for the international audience. Some of the patterns are available at www.soapatterns.org which will be updated to match the book as well as candidate design patterns.

Another title that I am very interested in reviewing is also another Prentice Hall title, Web Service Contract Design & Versioning for SOA.

SOA Symposium 1.0

2008-10-10T13:50:00.000-07:00

Well it has been quite awhile since I have posted but now that I have a few minutes I figure I would discuss my experience at SOA Symposium 1.0 in Netherlands, Amsterdam. I am working for SOA Systems delivering SOA training this week and was fortunate to speak at three sessions and participate in a panel discussion with my peers. I was very impressed with the quality of sessions and speakers that were able to take time out and share their thoughts on the future of SOA. Check out www.soasymposium.com and look for publicly available content sometime in the future.

There was a lot of discussion on how SOA Governance is becoming more important along with more efficient platforms and alternative consumers. One common thread in my discussion with attendees was how important the people quotient is in creating a successful SOA.

Cape Clear ESB (Workday) Assemblies Context Options

2008-08-08T18:55:00.000-07:00

One client I am assisting is the Cape Clear ESB and it Assembly functionality. This provides mediation support via a Spring Bean configuration file which is generated via graphical drag/drop tools. Because the Assembly Service is stateless and the project we were working on required some context sharing, we had to open up the covers and learn about org.springframework.context.ApplicationContext and org.springframework.beans.factory.InitializingBean. A single bean with a scope of singleton was developed to provide context for all instances of the assembly and when the assembly was loaded this information was initialized as part of the afterPropertiesSet() method.

Within each instance of the assembly, Cape Clear provides another context called com.capeclear.mediation.MediationContext which provides a setProperty() and getProperty() method. This provides simple storage of variables within an instance.

XACML: An option for authorization

2008-08-08T18:45:00.001-07:00

I am currently taking the next step in security development learning about XACML from www.oasis-open.org. In the scenario that I am working on with the client, there Identity Management Solution will be acting as the Policy Authoring Point whereas the ESB that we have deployed will act as a Policy Enforcement Point and Policy Decision Point. Due to the fact that we have a clustered ESB a decision that we are mulling over is whether to have a central/shared directory for XACML Policies or use the support for XACML Request/Responses to do it real time. Our concern is the latency especially in a high volume environment and we are currently steering towards a shared directory. This will allow each ESB to take a portion of the load and keep the processing local.

As part of a SOA Solution we will be using Service Agents or Intermediaries to support the authentication (SAML) and authorization (XACML). The ESB provides out of the box interceptors for both of these and only requires that the Service Consumer be able to interact with the SAML Provider.

SAML

2008-06-18T22:24:00.000-07:00

Well it's been awhile since my last post but work and life have been quite hectic. Currently I have a client that is attempting to integrate a Identity Management/Policy Decision application with their ESB. As part of this effort examination of the many options has presented itself and we have agreed to examine the Security Assertion Markup Language as a potential solution. The overall use case is to have internal/external clients use an SSO approach towards invoking available services.

To demonstrate this capability a SAML Assertion will be inserted into the SOAP Header of a request. The deployed Web Service in the ESB will include Message Processing Logic in the form of an interceptor. The interceptor will examine the SOAP Header and SAML Assertion. At this time, it will be absent of a Digital Signature which adds an additional layer of integrity for the Assertion. The other challenge is the use of SAML 1.0 versus the current supported version of 2.0. Fortunately, the PDP has support for SAML 1.0 eliminating the immediate need to custom code 2.0 support.

For a sample SAML Assertion, check out the Sun Web Site at:

Sun Sample

and OASIS which manages the specification.

OASIS Site

Service Scalability

2008-04-22T19:59:00.000-07:00

Over the past few weeks I have spoken with several customers who are implementing SOA in various stages and have decided to employ Web Services as the foundation. One of the key concerns is scalability and the ability to handle large volume for public / private services. An approach that evolved from these discussions was the following:

- For new services, specify governance in the Service Analysis / Modeling phase. This should specify the SLA for the Service and help identify the potential reuse and usage pattern that may be experienced.

- Services typically begin with a single instance using protocols such as HTTP

- Service scalability should be tested via Performance tools to determine how it degrades with load, supportable performance etc.

- Services should be monitored using JMX or similar API to determine runtime usage and usage patterns.

- Make services stateless so that they can be easily clustered.

- If service usage is higher than anticipated, consider configuring a cluster of service instances and use Hardware or Software load balancing.

- If throttling is desired, have service instances read requests from a JMS Queue as this places an abstraction between the service consumer / service provider. Queues allow for asynchronous communication and persistence point for inbound requests. Services can then throttle how quickly requests are processed without losing messages.

Deployment / Test of simple BPEL Process in Apache ODE

2008-04-04T00:38:00.000-07:00

In my previous post I spoke of the configuration / startup of Apache ODE. This morning I deployed one of the examples and tested it using the provided sendsoap.sh utility and SOAPUI. A couple of things were necessary to do this:

1) New processes are stored in a folder with the process name and include:
- BPEL File
- WSDL(s) File
- deploy.xml (deployment descriptor)
- other necessary files
2) The folder is stored under the directory.
3) On startup the server will load the processes and report on it within the server log file as shown below:

4) The Apache ODE User Guide then describes a way to test services using a script called sendsoap.sh under the Apache ODE Distribution bin directory. To get this to run I needed to open up permissions using chmod. One this was complete the syntax for executing this is:

/bin/sendsoap.sh (URL to the web service) (SOAP file sample)

or in the case of what is provided in the distro:

/bin/sendsoap.sh http://localhost:8080/ode/processes/helloWorld examples/HelloWorld2/testRequest.soap

When executed the following is returned in the client window:

The tomcat/catalina server window shows the following:

5) Testing in SOAPUI, the following was executed and the results are shown in the following image:
- Created new project in SOAPUI
- Selected WSDL
- Generated a message
- Invoked Service

Setting up Apache ODE within Tomcat / Axis2

2008-04-02T20:52:00.000-07:00

So it has been a busy week and I have had little time to work on the Open Source BPEL investigation. I have decided for simplicty sake that I start with Apache Axis2 versus Apache ServiceMix. This will allow me to quickly fire in SOAP requests and not have to become familiar with the ServiceMix infrastructure. I will tackle this next and apply what is learned here to that.

So here are the steps that I followed to get Apache ODE installed within Axis2. I made the mistake of initially downloading the Axis2 binary distro versus Tomcat. The Apache ODE instructions work with Tomcat/Axis but not Axis2 standalone.

1) Download Apache Tomcat 6.0.16
2) Download Apache Axis2 1.3 WAR distribution
3) Download Apache ODE 1.1.1 WAR distribution
4) Copy axis2.war to tomcat/webapps
5) Copy ode.war to tomcat/webapps
6) Set JAVA_HOME
7) Run Tomcat using tomcat/bin/catalina.sh run

The console then shows the following on startup:

Success!

I then accessed the Axis2 Console and could see the Apache ODE Services by going to http://localhost:8080/ode.

Next will be deployment of a BPEL sample into Apache ODE and invoking using SOAPUI.

Cheers!

Eclipse BPEL Project

2008-03-26T11:10:00.000-07:00

Just wanted to give a rundown of the Eclipse (Europa) BPEL Project and my impressions at the moment. The eclipse bpel project is located at:

Eclipse BPEL Project

Having experienced most of my BPEL development in either TextPad (BPEL 1.0) or in the Cape Clear Studio solution (BPEL 1.1), the Eclipse BPEL Project was not too difficult to slide into. Here are the highlights:

- BPEL Project is only focused on BPEL 2.0. No BPEL 1.1 at this time.
- Overall GUI is nicely organized and employs a very similar look and feel to the BPEL Editor in Cape Clear.

-Assignment Editor in the current release M3, does not provide a graphical drill down to the schema element level. You can select your messages but no further.

-Activities are easily selected and placed in the panel. The Properties window needs to be visible to then edit those value.
-No tab to edit the BPEL directly with context sensitive help. I opened the BPEL in an XML Editor and flipped between the windows.
-XPath is done manually rather than created via the selection tool.
-PartnerLink defintion process is not as easy as in Cape Clear but it does generate an artifact WSDL file that includes the partnerlink definition so you do not need to muck up your WSDLs with the elements.

Issues that I have experienced:

1) Currently the Web Service Tools that is bundled with the BPEL Project appears to have a bug with imported schema. This is obviously unrelated to the BPEL Project but the full-lifecycle of developing the Service Contract and using this for the BPEL is not working properly. Currently have to use inline schema instead of importing.

What I haven't tried:

1) Deploying generated BPEL 2.0 into a container. Will try Apache ODE in either Axis or ServiceMix to confirm execution.

SOAPUI Benefits

2008-03-18T22:13:00.000-07:00

As part of the previously mentioned Continuous Integration effort I am undergoing, an effective/simple approach to Unit Testing needs to be identified. JUnit works well when performing assertions on Java. When testing Web Services, I decided to look for alternatives. One alternative is to use XMLUnit as I described in an earlier post. In order to test for the current client, a different approach was in order. Their current QA Staff are not familiar with Web Services so a Web Services centric Testing tool is an additional requirement. SOAPUI available at www.soapui.org provides a very simple and capable Test environment. GUI, command line and JUnit integration are provided in the freely available version.

So currently my Continuous Integration looks like:

- CruiseControl invoking Ant Script
- Ant Script invoking:
- Build / Validation Process
- Deploy Utility
- Junit TestCase
- SOAPUI Project

More to come....

OSS to the rescue for BPEL/ESB

2008-03-06T22:37:00.000-08:00

Several of my customers are small independent start-ups that are heavily involved with OSS. Just recently a Web Service integration highlighted the amount of Java code necessary to broker interactions with external partners. The decision was made to examine the OSS marketplace for an ESB and BPEL development tools.

I will be examining the BPEL Project from Eclipse, Apache ODE and Apache ServiceMix. I was very suprised at the quality of Apache CXF and its inclusion in the ServiceMix container was one of the factors in proceeding with this evaluation.

The goal of the evaluation is to show the ability to compose a task service that will create users in two separate services (Apache CXF and Microsoft .Net). In addition, some of the mediation facilities will be examined for simplicity and flexibility.

Does anyone have thoughts on ServiceMix?

I may also attend a session at Sun related to their ESB/WS Tools next week. If I attend I will post my thoughts and how it rates with the niche Cape Clear solution and Apache solutons.

Continuous Builds with Web Services

2008-03-06T22:20:00.000-08:00

In two of my current projects, I am working with clients that are moving to continuous builds from complete manual processes. What is a continuous build you may ask? Well anytime a change in the source code repository occurs, a build could be triggered and deployed to a development server. As part of the build process a suite of Unit Tests could be executed to confirm there is no regression in capabilities. Companies like Cape Clear employ this process to develop daily builds which are much improved in quality over the previous semi-automated processes.

The ingredients to this always falls into the same basic technologies and like most people OSS provides the answers:

- CruiseControl for the continuous build management
- Apache Ant for the build scripts
- JUnit for Unit Testing
- Some sort of Version Control Repository like SVN or SCM
- Web Service Container / ESB (Cape Clear or the like)

So is it difficult? Well I was suprised to see how quickly CruiseControl could be downloaded, started and configured through two configuration files (config.xml and build-.xml). In just a couple of hours the software was downloaded and a build was completed.

The necessary components to execute the build are the configurations described above but also the main build.xml that in the case of Cape Clear, invoked the Eclipse Buckminster Framework. This effectively creates a headless Eclipse Workspace and allows the ant script to run the validators, package the content, deploy it and test it. Since the Web Service/BPEL developer is not writing a compiled language, the validators are the key component in the build process.

Other things that I need to complete revolve around the rewriting of URLs or namespaces depending on which environment the build is being created for. Ant provides a soluton there via the replace feature which allows for the search/replace of tokens. This functionality is extremely helpful in complex BPEL where many partner services equates to many WSDL/endpoints. The other feature not complete is the execution of a JUnit test case using the Buckminster/Cape Clear launch feature.

Event Driven versus Orchestration

2008-02-21T16:34:00.000-08:00

I had an interesting conversation the other day at a client who was very interested in an event driven approach towards integration versus the traditional centralized orchestration model. The discussion identified that within the event driven model, the state of the process would be maintained in the message versus in the singular centralized orchestration (BPEL). Both solutions have their strengths and drawbacks. One of the big drawbacks that was discussed was in monitoring/management. Operations likes to have a centralized, process oriented depiction of what is occurring. The event model we discussed would spread out the process across a wide range of intermediaries and the message would be updated along the way. Not an ideal situation if viewing from this perspective. Whereas, the BPEL Service provides a more centralized view/approach.

An option for Event Driven would be to utilize an auditing interceptor that would provide a centralized view/update of the process execution. Operations could then monitor this service to get the depiction they preferred.

A strength of the BPEL approach was in the composing of task services or business processes. An event approach everyone thought would not be as easy to compose, however the scalability/performance benefits is very intriguing. This discussion will continue and hopefully draw out distinct patterns on when one approach is better than another.

Cape Clear and the bigger picture

2008-02-21T15:52:00.000-08:00

Well over a week ago Cape Clear was consumed by the SAAS marketplace. When I first heard the news I scratched my head but as SAAS gains steam, the need to share data via automation becomes more important. The increasing communication of data outside organizations has not decreased but only increased making simplified integration an imperative. Currently I am working with a several clients that are sharing data with business partners to provide a more complete solution. REST and SOAP interfaces are the standard and from experience are easier then the RosettaNet and EDI standards of the past. These standards spent a lot of time identifying a domain, event model, protocol etc that provided an out of the box roadmap for B2B.

These days the protocol and policy's are standardized however the event model and domain are custom to each. Most of the customers I am doing work for are implementing simple B2B models that share core data and simple event models. This is currently comprised of a WSDL on each end with customers invoking each others interface.

How does this tie into SAAS? Well if you are familiar with the first evolution such as Salesforce.com, there was a backlash from large organizations around storing their data offsite with poor APIs to extract and manipulate the data. After several iterations, Salesforce.com has improved their APIs and you see the second generation like Google, Workday and others providing compelling solutions for GIS, ERP etc. These services have taken off not only in small niche like applications but in the Fortune 1000 and Local/State/Federal Government.

What SAAS has exposed is the need for integration never decreases, it just sort of shifts from internal to external or if you look from a SOA perspective, the interoperability is baked in from the start.

The acquisition of Cape Clear is testament to this fact and identifies the need to use standards to simplify adoption.

BPEL Caching / Persistence

2008-02-03T22:05:00.001-08:00

I recently attended a vendors presentation on some caching technology whereby Java Objects could be stored in an in-memory DB or caching layer. The architecture appeared to be quite sound in that it utilized a peer to peer network to manage copies of the information for reliability purposes. The demonstration sparked my interest in its application to stateful BPEL as persisting to a database is a costly exercise when executing large amount of transactions. Prior to / Following this I read subsequent articles at www.soamagazine.com which discussed BPEL Reliabilty using such a concept. A reliable in-memory grid caching store was discussed:

The original article:

Article 1

A current article:

Article 2

Application of technologies such as these will provide dramatic impacts to stateful BPEL Services that are more and more being utilized for Service Composition.

Apache CXF and SOAP Headers

2008-02-03T21:47:00.000-08:00

Here is a link that helped me with the Apache CXF client capabilities. The requirement was to send a request to a web service which required a SOAP Header expressing authentication information. The WSDL for the Web Service did not express the request in the abstract portion but rather in the binding/concrete portion of the WSDL. Apache CXF generated a class to support the setting of the Authentication Header but no methods to add it to the request. Using Google and a decent portion of scanning/reading, I found that Apache CXF will not generate the appropriate methods to attach a SOAP Header. Java Annotations provided the answer as described at:

tapestryofthoughts.blogspot.com

By adding an annotation @WebParam along with the header=true flag and other appropriate values in the generated <####>SOAP.java as well as instantiating the Authentication Header via the generated class inside the <####>SOAP_Client.java, the client was sending a SOAP Header to the destination service.

Versioning Part 2

2008-01-14T14:55:00.001-08:00

In my readings, I found the topic of versioning discussed in Thomas Erl's SOA Principals of Service Design. I wanted to provide a brief mention of Thomas Erl's vantage as it provides another view of the topic.

As I mentioned in the previous post, changes to the WSDL definition causes service consumers to break. Thomas' take on this is for core entity services and their associated contract to remain constant. Extension to those services can be performed which will not affect the service consumer. For Services that need to be dynamic and cannot be developed with longevity in mind, then SOA Governance is applied to manage versioning of services.

SOA Governance solutions that I have heard marketed in the past provided XML Routing solutions which would inspect requests and then use a lookup mechanism to route to the eventual destination. In this case, multiple versions of a service could exist and it would be up to the Governance solution to manage the Addressing aspects.

Another perspective of Service Versioning is James Pasley of Cape Clear and his examination of the current environment. His description of versioning best practices is available at:

http://developer.capeclear.com/versioning

and discusses the options for deploying various service versions concurrently.

Service Versioning and Schema Evolution

2007-12-19T21:27:00.000-08:00

This is a topic that comes up quite often when talking to clients. Based on some of the successful implementations that I have been lucky to be involved in, the one demoninator is how important XML Schema is in that success. In addition the existence of a Schema Custodian (Thomas Erl term) or a individual who guards the Schema with his/her life. The more difficult a Schema is to change, the more likely that it will succeed in an enterprise SOA. Sounds kind of strange but the foundational aspect of XML Schema in its use in WSDL, BPEL, XSLT, XPath/XQuery, auto-generated skeletons/stubs.... means that minor changes could seriously impact a deployment.

One of the clients I work with has successfully employed a XML Schema definition for 3+ years now. The schemas have been allowed to evolve via the use of optional elements but those evolutions have been through a small group of Schema Owners and it is not modifyable via the developer community. Because of the inherent difficulty in changing the schema, it has prevented Schema creep and the potential interoperability damage quick changes could cause. The benefit of this has been interoperability across a wide range of technologies starting with four different app containers and two different JMS providers.

One other area of discussion is on versioning of Schemas/Services. One approach that I have seen is the use of the namespace and ESB technology to route based on unique namespaces. Namespace versioning can be something like:

http://www.mycompany.com/2007/12/ProvisioningService

This service would then have been developed in December 2007. People are familiar with this approach with some of the WS-* standards like WS-RM. The good part of this is that for me to invoke a service using this as the target namespace, requires that my SOAP Request call it out. The bad news is that changing this will break any clients that were developed against it. A quick solution is to support several services if the namespace changes by infrastructure like an ESB and XPath Routing. A simple XPath statement focusing on the SOAP Envelope could route this request to the correct destination. There are several other versioning solutions, check out developer.capeclear.com for a whitepaper on Service Versioning.

Fine Grained vs Coarse Grained BPEL

2007-12-19T21:13:00.000-08:00

At a customer this past week, the team was examining the WSDL that had been defined for a new project. The WSDL was broken into three definitions but was going to provide aggregation capabilities for a single consumer. Since the consumer portion was already under development, there was a strong desire to limit the number of changes made to the interfaces. The first discussion was whether to use the several interfaces as separate services or to boil them down into one interface. If this was done, do we just map the current operations onto the WSDL interface or do we replace them. The current services were pretty fine grained providing specific capabilities for search. The results returned were represented as different elements with no consistency. Two options were then explored:

1) Create a single WSDL that had N number of operations matching the current operations that were currently spread across several WSDL.

2) Blow up the WSDLs and start from scratch creating a coarse grained interface that provided a single search operation. For the Web Service implementation, it would need to examine the query data to determine which type of search was being requested or a new element with an enumeration of search types could be added.

In both cases, revisiting the schema and collapsing the numerous elements that existed previously into a query input and search result was agreed upon.

To demonstrate the impact of solution 1 or 2, we then turned to BPEL and developed the service implementation as a BPEL Service. Why BPEL, well this service was meant to aggregate data from various sources depending on the input.

If we examine solution 1, the WSDL would have 8 operations. One option to support this was the use of a the Pick Activity and OnMessage Activity. One OnMessage per operation could be employed creating processing pipe creating an Event Driven design.

If we examine solution 2, the WSDL would have a single operation and thus would support a more traditional Receive/Reply Activity.

So in essence, the use of a Fine Grained WSDL impacted/reduced the options for BPEL development, whereas Coarse Grained WSDL did not.

Simplification via the Service Interface

2007-12-10T13:16:00.000-08:00

I was recently working with a client who had developed a solid demonstration vehicle showing many moving parts. As I was working with them, we identified that the service interfaces were developed with varying degrees of granularity. The data granularity for some of the interfaces was fine grained expecting small sets of data whereas other expected a full document or were coarse grained.

To simplify the demonstration and complete the round-tripping exercise they were shooting for, we standardized the WSDL definitions to use the same Schema and to push the entire document across the wire between the services. In essence we standardized on a coarse definition. This then untethered the controller service they had defined in BPEL to just copy the top level element between services.

As changes occurred in the demo, the refactoring demonstrated that maintenance or updates were much simpler or non-existent.

From a business perspective using the business documents in the interfaces provides a SOA centric approach towards development. This approach doesn't always make sense as with very large documents, there may technology or complexity reasons to limit the use of such an approach. In those cases fine grained access make sense albeit with more work on the orchestration side.

Thomas Erl SOA Training

2007-12-10T13:05:00.000-08:00

Last week I was able to participate in training provided by Thomas Erl and his SOA Systems consulting organization. Being a fan of his books and how they crystallize the SOA concepts and associate them to the Web Services Standards, I was very excited to participate. Unfortunately I was only able to partake in 2 of the 4 days. These days focused on the SOA Methodology he has been working on namely SOA Modeling, SOA Analysis and the beginnings of SOA Design.

Through his various books such as SOA Principles of Service Design and Service-Oriented Architecture a process of examining a current business process was defined. During the process, services are logically grouped into task, entity and utility. Each of these denotes a certain level of capability or interaction. The Modeling/Analysis phases help identify which bucket they belong in and confirm they follow his 8 SOA Principals such as Autonomy, Stateleness etc.

What did I get out of the session? Simply put a methodology that allowed the team I was working with to break a current process into its core services and model the message exchanges that would occur if implemented. This was done at a high level but demonstrated a simple approach that allowed both developers and a business analyst to communicate effectively.

Operations is my customer

2007-11-07T13:16:00.000-08:00

So I got a chance to partake in a Franklin Covey Leadership Training exercise several weeks ago. I was not sure what I would take out of the training and whether it would be beneficial. I was extremely impressed by its content, focus and most importantly the instructor. Long story short the class tried to identify what is a leader and how to become an effective one. As part of the sessions a discussion around identifying your customer came up. For the current customer I am working for, it is a little interesting in that with middleware there are many customers. Specifically though once in production, operations becomes a major customer.

What I learned is that to make an effective middleware platform, quality comes in the flavors of Reliability, Scalability, Manageability. When these quality attributes skip a beat or fail to occur, Operations has to swoop into action. The benefit of the training underscored the need for me to develop simpler and more capable tools to handle customer issues. Additionally, the increasing complexity of our SOA has begun to exacerbate the monitoring and management once in production. Log4J logs, JMX and Management Services become the virtual console for operations and development.

While these solutions may not occur immediately, I am working with them to define best practices, provide training and eventually automate solutions. This has resulted in reduced operational issues as well as increased self-reliance. Most importantly, this process never ends and must continue to be executed to provide consistency of results.

SOA Training, what is it?

2007-11-07T13:02:00.000-08:00

Based on my current role as an independent consultant I have run into several clients looking to develop SOA technical skills inside the organization. Due to my Web Services lineage, I of course look at the SOA solution set via the W3C and OASIS toolbox. From a training perspective I highly recommend the Cape Clear Developer site for two training facilities:

1) XML Schema / SOAP/ WSDL

Check out http://developer.capeclear.com/?q=justenough

2) BPEL

Check out http://developer.capeclear.com/?q=bpeltraining

I am looking for some decent tutorials on XSLT/XPath. Currently I have been using the www.w3c.org and O'Reilly. If you have any suggestions on alternative tutorials/locations let me know.

3) SOA Specific Articles / Books

Thomas Erl and his series of publications are a must. http://www.soamagazine.com and http://www.soabooks.com

Finally although not specifically for SOA but rather all the other technologies like AJAX, OOAD, HTML etc the Head First series from O'Reilly is excellent.

To Spring or not to Spring

2007-11-07T12:54:00.000-08:00

I am trying to get a sense from people as to which framework people are selecting, Spring or EE5. Several different projects I am privy to at different companies have employed Spring with good success. Another colleague just published a book on JBoss SEAM with its EE5 underpinnings and is very high on its capabilities. Long story short it seems like this is going to be a Ford versus Chevy debate. Maybe I am wrong but would love to hear any feedback.

Due to the use of Spring configuration and Spring Bean support in Cape Clear 7.5, I see my path has been pre-determined.

The Web Service Standards Soup / BigSOAGrid.com

2007-10-06T18:45:00.000-07:00

Over the past several weeks, a former colleague and I have put together a grid of the current Web Service Standards starting with the core and extending to the WS* features. You can find this at:

http://www.soamag.com/I11/1007-1.asp

or at its home:

http://www.bigsoagrid.com

The purpose of this is to highlight the many options / directions the standards have gone as well as the large amount of vendor participation in these specifications. Obviously having vendors write standards is a double edged sword as they most often will have an offering following the standard release and will be pushing it on you and I. With the open source movement however, you will find a bunch of these standards available in Apache projects, Codehaus etc and the time/effort spent by other technologists can save time for us all.

One example of this was a project I worked on recently for a State Government contractor. There was a need to support basic authentication and integrate this with LDAP. Within a half an hour discussion, the decision was made to use WS-Security UsernameToken. The implementation took advantage of the WSSJ package from Apache and in a few hours this feature was ready to test. The alternative might have been a custom SOAP Header and the resulting WSDL / Interceptor work to build out the infrastructure. Not worth it when deadlines are tight!

So take a look at the BigSOAGrid and get up to speed on the latest and greatest. Maybe you will find a standards based option for your project needs!

SCA Like Features in Cape Clear 7.5

2007-09-26T18:23:00.000-07:00

Building large scale SOA has several pain points. One of them is in the assembly and deployment to a SOA cluster. Cape Clear 7.5 is a providing a starter to graphically assembling SOA using Web Service standards and Spring. I got a chance to participate in the Cape Clear 7.5 webinar this past week and saw a flexible mediation development tool which in turn developed a spring configuration. Within the Cape Clear Studio (Eclipse plugin) environment, transports, transformation, logging, exception handling, routing and Spring Beans can be combined into an assembly to perform mediation tasks. The diverse transport selection and simplicity by which this is defined was impressive.

A next step with this environment will be to apply policy between the steps/services. The final step would be in then being able to apply all these pieces to a diverse set of deployment platforms (both Cape Clear and non-Cape Clear). Why would I want to do this?

Let me provide an example. One customer I work with has open source J2EE container, a COTS J2EE container, Tomcat, Apache Axis.... and the list goes on and on. Trying to develop a SOA in this environment is one issue, deploying to all these entities and managing this in an automated fashion is another. A single deployment description will help lead towards deployment automation as well potentially a monitoring description.

How does UML apply to SOA?

2007-09-11T18:29:00.000-07:00

I am currently participating in a UML / OOAD training class learning about the UML language and what it provides the Object Oriented development space. A question that I had asked early in the class to the instructor was how SOA can take advantage of UML. Unfortunately the instructor did not have an answer with his background being in application development versus service/integration development.

So here are the diagrams we have covered:

class diagram
sequence diagram
state machine diagram
activity diagram

Still to go are:

component diagram
use case diagram

Based on the above it appears that BPEL could be described in a non-BPEL fashion using the activity diagram. An activity diagram could simply be described as a flow chart. It supports the definition of activities, looping, conditions etc. Sounds kinda like BPEL doesn't it.

Another diagram that appears to have some use is the component diagram. I am not 100% sure but the component diagram typically shows components in a system and how they interact with one another. Looking at Martin Fowler's publication UML Distilled (Third Edition), pg 141:

"Use component diagrams when you are dividing your system into components and want to show their inter-relationships through interfaces..."

Additional details in the publication and diagram examples appear to provide a way to visually describe a SOA where Web Services are the components.

I haven't researched this with other organizations but will trying to apply these prinicipals/approaches for some upcoming projects to help in the detailed design documentation phase.

Apache CXF, alternative to Axis?

2007-09-07T21:24:00.000-07:00

Working on a small project with a startup and another consultant turned me onto Apache CXF (Celtix XFire). Both of us began looking at open source, specifically Axis 2 to provide a capable Web Service layer for an existing J2EE hosted application. After about a couple of weeks we gave up due to the poor documentation and issues that we ran into. We then decided to look into alternative containers such as Glassfish, JBoss etc and came upon Apache CXF. Some of the factors we decided on were a JAX-WS implementation with the ability to plugin into various implementations and support for the basic WS-* standards such as WS-Security.

Today I used the Apache CXF wsdl2java features to simply generate a client stub to a .Net service. In addition I needed to enable SSL communcation. From install to executing the client, it took 3-4 hours with a bit of debugging on the CXF client configuration file.

Current impressions are good documentation, clean generated code with no major issues.

Check it out at http://incubator.apache.org/cxf

Dynamic Endpoints in Cape Clear BPEL

2007-08-31T18:30:00.000-07:00

Creating dynamic endpoints in BPEL using Cape Clear is pretty easy. Just simply overwrite the PartnerLink using an Assign/Copy activity where the From value is the URL and the To is the partnerlink. Having done this in a previous implementation I statically defined these URLs in the BPEL itself. I could have developed a WS that returned the dynamic endpoint but thought that was too much for what was needed at the time. The URLs were static and would not change. Well 6 months later, the URLs are changing as new destinations are being added to the BPEL, in fact double the destinations.

I am now refactoring the current BPEL to use a new feature in Cape Clear BPEL, the XPath function getInitValueParam(). The getInitValueParam() function allows me to send in a property name as a string and get back its value. These properties are defined in the initParams section of the Cape Clear deployment descriptor (ws-application.xml).

I am also employing a while loop to examine the inbound XML for destination data and in combination with the above function, parameterize the destinations externally to the BPEL.

Here is the scenario, an inbound XML messages defines the names of the various intermediaries in the overall process. This is represented in an element called system as shown below:

<system>systema<system>
<system>systemb<system>

The system element is unbounded but is an enumeration of valid destinations. Within the new BPEL, a counter is initialized and a while loop determines if the counter is less than or equal to the number of recipients. For each system I wanted to extract the system element value (the SOAP intermediary name) and then use the getInitValueParam() function to return the destinations URL . I then overwrite the PartnerLink value with this destination and I have performed dynamic endpoint assignment.

Sounds good right! Well I finally got it working through the help of a fellow BPEL blogger. The problem that I had was accessing the system element via array notation. My array value was defined as a variable called counter. I tried using a combination of XPath that would get me down to the system node and then combine this with bpws:getVariableData() of the counter value. No luck. I found this link which got me through the issue quickly:

http://covarm.tvu.ac.uk/blog/?p=4

Basically, it was necessary to use the concat function in a Copy activity to create a xpath query that builds the actual expression I was interested in, in this case something like:

concat('.../.../ns1:system[', bpws:getVariableData('counter'),']' )

then, this value was assigned to a variable called xpath. Then in the next Copy activity I refer to the new variable xpath:

bpws:getVariableData('...', 'parameters', bpws:getVariableData('xpath') )

A lot of work to do something so simple in Java. Oh well, the combination of these features is helping dramatically simplify my original BPEL and providing unlimited scalability for new systems via configuration instead of coding.

Applying JUnit/XMLUnit to Web Services

2007-08-25T19:12:00.000-07:00

As part of developing an agile SOA environment for a couple of clients, testing becomes a critical component. So perusing through cyberspace, I came upon an excellent and simple approach towards this goal. Check out http://www.ibm.com/developerworks/webservices/edu/ws-dw-ws-soa-autotest2.html . This article brings an Apache Commons HTTP Client, JUnit and XMLUnit to bear. The result is a reusable client that posts a SOAP request and writes the SOAP response received to the file system. Then XMLUnit methods are used to provide comparisons between the actual response and the desired response (stored in another file). This approach works well for Request/Response services.

The challenge I have is that my most important client does not have this style of service. Rather they are using One-Way/JMS style of communications. My thoughts here are centering around the use of a centrally available file system or database. Each of the SOAP Intermediaries would need to store the inbound and outbound requests to this storage point. Once the overall cycle is complete, the same XMLUnit features could be used to compare the entire request lifecycle.

Rules Engines / Rule Based Services

2007-08-16T09:49:00.000-07:00

How many people have actually implemented Rules Engines? I first ran into Expert Learning Systems in the 90's when working in the Mechanical Computer Aided Engineering (MCAE) space. They were being used to electronically document engineers/designers knowledge for the development of mechanical components. The idea at the time was to use this knowledge to optimize part development and reduce lead time. The infrastructure behind this was a Rules Engine.

I don't know the answer on the current adoption of Rules Engines but for the right problem, it appears that it provides a compelling solution. I am currently investigating this sector to help identify if the technology is pluggable in a SOA and what type of business problems are best served. It appears that logic that is dynamic is the best candidate. The ability to update an external configuration without having to go through a typical release cycle if the logic was embedded in a bean is definitely attractive. The languages are pretty different between vendors, some using Natural Language approaches while others use a proprietary scripting language. The only constant is support for the JSR-094 specification.

Most of the vendors generate a java implementation or even web services so inclusion into a Web Services infrastructure is pretty straight forward.

So far I have looked at Jess and OpenRules and found the open source community to provide a very good starting point.

Evolutionary SOA / Agile Development

2007-08-13T18:40:00.000-07:00

About two years ago, my former company undertook a major development effort to implement an agile development process based on SCRUM. With a team of 20 developers, the company slowly but surely made the effort to transform the development process and use incremental / agile techniques to create a better quality product. The result after 2+ years is evident in the stability and consistency in releases of new features.

One of my current customers is also striving to implement a similar process. Currently development occurs in one major release a year and focuses on a monolothic EJB infrastructure. Unlike the previous company, the switch from a traditional waterfall process to a more agile/incremental environment has several challenges:

History (the current organization was built around a single major release and has done so for many years)
Platforms (the current infrastructure is comprised of very heavyweight EJB applications that are not flexible/agile)

The organization is tackling this on several fronts:

Executive Support (most important is to get executive support on the use of a incremental/agile development approach which in turn will directly correlate to performance factors for the business)
Organizational Changes (re-organizing the development team to focus on services)
Simplification/Refactoring/Replacement (simplifying the exiting infrastructure for immediate gains and replacing the monolith with a series of simpler services for long term benefit)
Putting out to pasture (placing the existing monolith into a legacy category thus limiting its scope/usage)
Training (bringing in external consultants to guide the existing development staff on the best practices and fast track the simplification/replacement strategies).

Overall, I have been trying to do the same with the current ESB infrastructure. Through consistent interaction with operations and my desire to continue to refactor/simplify the production solution, improvements in reliability and customer support have been made. Incrementally I am removing / improving the production solution to make it extensible and more reliable for the large volumes of customer that interact with it. This process will never end and thus like the Toyota commercials, will hopefully result in the Corolla of ESBs (400,000 miles and still going!).

A Bad Foundation makes a Bad House

2007-08-08T20:49:00.000-07:00

If you have worked with BPEL, you know how dependent it is on the foundation. In this case the foundation is XML Schema and the sub-floor is WSDL. Challenges introduced in the XML Schema or WSDL, ripple there way into the BPEL definition. Here are a few novice and advanced issues that have bitten me over the past several years when dealing with BPEL variables:

- Unique namespaces are critical/mandatory in XML Schema/WSDL as they are used to uniquely identify variables and their content.

- Schemas with imports should utilize xsd:ref otherwise bpel:assign activities become a nightmare. BPEL Variables that are generated from XML Schema/WSDL that import other schemas will each have their own namespaces. This makes copying root elements between two variables with different namespaces impossible using the bpel:copy activity. The option is to copy each element individually and see the BPEL script grow exponentially. BPEL 2.0 and its support of inline XSLT will eliminate this issue.

- Some Web Services are RPC Encoded versus Document Literal. bpws:getVariableData syntax is different when extracting values from a RPC Encoded BPEL Variable versus a Document Literal BPEL Variable. BPEL Editors eliminate this issue, however when editing by hand the later will include the variable, part and XPath to the element. RPC only includes the variable and part.

- One of the most simple but common mistakes is to not initialize the BPEL Variables with a literal assignment when using Document Literal services. This will cause run-time failures or other depending on the BPEL engine you are using.

WS-* Bloat, what is actually being used

2007-07-31T19:45:00.000-07:00

Below is a list of WS/XML Specifications that I have implemented with customers that are in production or in pilot mode. The first generation specifications (XML Schema, XSLT/XPath, SOAP, WSDL) are used widely. The second/third generation specifications would be BPEL and the WS-* standards.

SOAP
WSDL
XML Schema - Pretty much the standard for data definition and more powerful than DTDs (nothing new here).
XSLT/XPath - Standard for data transformation (haven't used XQuery yet).
BPEL - Standard for Orchestration Web Services
WS-RM - only have used it at one client but it provides reliability as advertised.
WS-Security - not used in production but have several customers moving to it for up-coming releases.
WS-Addressing - examining this and other specifications for a routing road map. Not currently used in production work that I have performed.

So as you can see there are still a lot of missing specifications that are floating in the WS-* ether as well as UDDI (an unpopular specification in project based SOA). The above is based on a sampling of 200+ customers that I have worked with from 2002.

JMS Queues versus JMS Topics

2007-07-30T19:06:00.000-07:00

Today I was working with a customer on demonstrating integration of a JMS Topic into their ESB. For years now I have spent all my JMS Integration work specifically using JMS Queues. For this integration, however, the customer was utilizing a topic. Based on my research the following rule of thumb appears to delineate when one is used over the other:

Topics are utilized when the information is time senstive (i.e. stock quote).
Topics are utilized to broadcast of information to a wide audience (or subscribers).
Queues are utilized when performing transactions.

There may be several other reasons to use one over the other, but this applied to this specific customer and helped address its usage.

Personal Services show the way

2007-07-29T20:23:00.000-07:00

Although this is not directly related to my main discussion about SOA/Web Services, I noticed a trend in the past 3 days that impacted me and shows the power of what SOA/Web Services/Internet technologies are bringing and will bring .

So on Friday I checked my gmail account and noticed that LinkedIn (http://www.linkedin.com/) had notified me of several contacts who were interested in touching base and joining my personal network. One was a fomer colleague and another was a former friend from college. Within a couple of hours I had caught up on 6 years with my colleague and 15 years with my college buddy. The part that is interesting is all I needed to do was post my LinkedIn profile (my personal WSDL). This included my destination (email address) and content. LinkedIn hosts some general operations and away you go.

So a single personal service in a short time frame brought me information relevant to me and simplified my networking life dramatically. Just imagine all the phone calls, mixers, conferences etc that you would have had to attend. Who's got the time!

Personal Integration is here and helping me and many others.

Transport or Message Level Security?

2007-07-28T07:16:00.000-07:00

This is a question that has been posed to me at a lot of state government pilots over the past few years with regard to encryption/privacy of messages. The following is the simple rule of thumb that I have used based on the best practices ( http://www.ws-i.org ) and discussions with security/web service professionals.

If securing two points (a client and a service for instance) and the assumption is that security is not required once it is received on either end, then transport level security is sufficient (HTTPS, SFTP etc.)
If it is unknown how many systems/intermediaries will be touching the message then message level security should be utilized (WS-Security / XML-Encryption).

An interesting point by the security gurus is that transport level security has the potential of being at various strengths (i.e. 128 bit). As a message travels between more than one point, there is the potential for the privacy of the message to vary. This is not the case with message level security as the strength is specified within the SOAP Headers of the message and thus remains constant. This is an important consideration as a lot of state agencies begin integrating across departments/agencies which in turn may be across networks.

One simple reason for using ESB in a SOA

2007-07-27T11:44:00.001-07:00

Well here is a real world example of why an ESB is a nice piece of infrastructure when things fail. I am working with a company which had a failed transaction. An ESB is being used to broker communications across multiple applications. The failed transaction had originally been created by one of those applications and placed on a JMS Queue for processing. Because there is an ESB, the destination service is not only available over JMS but also HTTP (the default transport for the waiting BPEL Service) or other transports that you can choose from.

We simply resolved this issue by loading the SOAP request into a SOAP Client in Eclipse and doing an HTTP Post to the service.

There are many other reasons to using an ESB but just the flexibility in supporting multiple transports can help out in a pinch.

Simple SOA Client

2007-07-26T20:23:00.001-07:00

Several months ago I needed to demonstrate a workflow task in a BPEL Service. Searching the web, I came across an old colleague. In his spare time he founded http://www.magoosoft.com/. Using it reminded me of an email client like Outlook Express. It supports multiple protocols and more importantly renders the SOAP message as a HTML Form. So for the scenario that I was working on, it was simply defined by importing the WSDL for the BPEL Service to define the call back and messages. Once defined a few properties were specified and my magoo client would receive messages, allow me to view and then respond back to the waiting BPEL Service. All the work was done within a day and demonstrated a human workflow component for SOA Pilot.

SOA for the masses?

2007-07-26T19:52:00.000-07:00

Most of the blogs I have read don't seem to talk about the developers viewpoint ("the masses") of actually implementing an enterprise SOA using ESB or other technologies. Maybe I have missed some good independent literature out there.

Currently I am working with a variety of clients that are large and small using third party ESB, MOM or Open Source. In my travels I have run into the good and bad of SOA and when done right, the benefits it can provide.

As I navigate my customers problems and learn/implement the WS-* standards, I'll post what I find. Maybe this will help you or at a minimum draw some debate.