After twenty years of enterprise systems experience, and after an equal number of years of trying to SIMPLIFY the seamless flow of data and transactions, it’s clear to me. Despite all of the hype around SOA and ESB’s, BAPI’s, API’s, MDM, ETL, data warehouses and open architectures, SYSTEMS INTEGRATION remains a major cause of project overruns and systems implementation disappointments. In fact, I’d have to say that as regards COMMERCE systems and Customer Relationship Management (CRM), the complexities and challenges of system integration are perhaps the single most common failure point in these implementations. Why? It’s almost always underestimated. I mean, given today’s technologies, surely, “We can make anything talk to anything, right?”
Talk? Certainly. Seamlessly, intelligently and accurately communicate and work harmoniously, well…not so much. And that is where our commerce to ERP integration efforts still run into trouble. I’m thinking through several recent large-scale, global Commerce Platform projects where the failure to address the need for transaction and data synchronization undermined what would have been an otherwise GREAT solution. Like most failures, there are multiple root causes. Here are my top three:
- The Magic of Middleware – how can I go wrong if I use ‘middleware’ designed specifically for cross-system integration? Software to Extract/Transform/Load is a starting point; but it’s not the end point.
- Data Mapping Complexities – Or, you can’t get there from here. It would be one thing if data mapping was always field to field or object to object, but the reality is often that mapping takes on a one-to-many relationship or requires enhancing one or both of the integrated system structures. This is where things get messy – trying to build the logic to fully map/synchronize these without recreating the host systems.
- Out of Sight/Out of Mind – Believe it or not, THIS was actually the state of mind in a few recent implementations. Integration was assumed to be manageable, so the major budget and attention went toward the ‘hard’ parts of the implementation – the NEW system. In the end, the outcomes fell short due to the inconsistencies and gaps left by the absence of a well-planned and executed integration of key data/transactions.
Are we connecting???
Here’s an SAP example. The concept of Sales Area includes the combination of sales organization, distribution channel, and division. It’s commonly used in SAP ECC sales and distribution structures. However, as with many aspects of SAP, there are few non-SAP systems that have a fully equivalent data structure to handle these ‘real-world’ complexities. So, in the rare cases where only a single sales area has been implemented in SAP ECC, the integration to other systems remains relatively simple. But, in the most common cases, as soon as you add a second sales area, then the integration becomes a lot more complicated and typically requires the creation of custom entities and fields in the non-SAP systems. This isn’t an impossible problem per se, but it needs to be anticipated, adequately estimated, and properly designed. So, this example actually hits all three of the above common ‘misses’.
Here’s another set of issues that we see over and over. Product configuration and pricing data pose some of THE most complex, even intractable in some cases, integration challenges. The problem is that they model a lot of data and logic in a way that must be kept in sync between integrated systems. Making even a small change in one system’s logic without making corresponding changes in the integrated system can lead to runtime errors. This is by far the highest risk integration area and it must carefully designed and maintained in order to function reliably.
What’s the ANSWER? The devil still lives in the details. Having a complete understanding of both sides of the integration remains the key to success. Although there’s no ‘silver bullet’ for Systems Integration, at least give yourself a chance by addressing the key integration points as one of the critical success factors. Experience counts. Personally, my company has a long history of developing comprehensive integration within a wide range of systems landscapes. This integration has been a key enabler in some very sophisticated commerce and quoting solutions that then also create orders in backend ERP systems. By bringing extensive experience and understanding of various source and target systems enables us to anticipate and address many common integration challenges. Over the years we have integrated nearly every major line of business application in every major category to each other. The most common integrations have been:
- CRM and ERP
- CPQ (Configure, Price, Quote) and CRM
- CPQ and ERP
- CPQ and EDI
- CPQ and CAD
Next comes the choice of middleware. Which is the best? We are asked by almost every customer to help choose the “BEST” middleware. It really depends on a number of considerations. One of the first is to see which, if any, middleware applications are already in use within a given enterprise. In most cases, any middleware that provides a publish / subscribe and guaranteed delivery mechanism can adequately satisfy an integration requirement. There are of course various usability considerations, especially for modeling the mapping between systems. The key is to determine how many of the specific API connections are already leveraged in middleware. Following is a sample of the middleware applications that we commonly use: WebMethods, Tibco, BizTalk, Scribe, SAP PI, and of course there’s always “Custom” for some very specific needs. Each of these options has its place; but knowing the trade-offs is key.
Finally comes the exception handling. It is not a matter of whether an integration will fail but what happens when it DOES fail (and it will fail). Communication disruptions between systems is probably the most common failure (or even one system being offline). Upgrades to one or the other system are probably the next most common cause of failure. No matter the cause of the failure, it is most important to KNOW that a failure occurred, to know WHY it occurred, and then be able to recover from the failure. Well-designed integration anticipates such errors and gracefully responds ensuring consistent high performance with minimal human intervention.
So why write this blog about integration? Why is system integration more important than ever? We live in a very connected world, more so every day. Integration has always been the lifeblood that connects business systems to form business solutions. And today’s best integrated business solutions are engaging end users like never before by bringing together an array of systems into a comprehensive connected experience. eLogic is currently building such a solution that turns classic thinking on its ear. Instead of talking about “systems of record”, we are focusing on “systems of engagement!” We are less interested in what systems “owns” the data and rather what system “owns” the user experience. And enterprise class integration is the key to making it all work. Stay tuned for my upcoming blog where I will discuss the details of this exciting new concept.