Integration Architecture for Salesforce.com

This article was originally published for DeveloperForce on November 26, 2014.  See the following link: http://sforce.co/1AWIC3h

As a Salesforce.com Architect it is your role to lead your company in the evolution of it’s Integration Architecture. A good architect must understand both integration architecture and integration patterns.  The difference between the two is analogous to designing the highway vs driving cars on the highway.  The Salesforce1 Platform offers architects and developers a wide array of integration technologies and recommended patterns (the cars); however without the correct Integration Architecture and technology infrastructure (the highway) your projects and solutions will be at risk for performance, scalability, data integrity, and many other problems.  This article will introduce you to the components of an effective Integration Architecture as well as walk you through a reference design similar to many of my Enterprise clients.  Hopefully this article will be used together with the official Salesforce Integration Patterns guide when architecting your Salesforce.com solutions.

What are the components of a good Salesforce.com Integration Architecture?

The Integration Architecture aligns the Business Strategy with Technical Capabilities

The best Salesforce Architecture’s are not based upon incumbent technology, singular architecture approaches, or corporate politics.  The best Salesforce Architecture’s are based upon DELIVERING BUSINESS VALUE.  What this means for the architect is to focus on what are the business’s requirements, roadmap, and needs for which you will offer technical capabilities.  In other words – you need to see where the business wants to drive, and figure out which highways and roads are necessary to support the amount of traffic.  Idealistic architecture (for example 100% Services Oriented Architecture) may cripple your ability to provide the capabilities needed by your business when they need them.

The Integration Architecture supports a mix of batch processing and real-time services middleware

Good Salesforce.com architects have learned that the best integration designs supports both batch and Service-based patterns.  This means you have multiple types of middleware at work.  I have had clients that had 3-4 different integration platforms in their Salesforce.com architectural landscape.  This is because not one solution ever can effectivly meet ALL your requirements, and once again the idealistic architecture’s are not as important as supporting the business’s needs.

The Integration Architecture is based upon Business Service Level Agreements (SLAs)

A mature organization and architect will attempt to define SLAs for data and process integrations.  These SLAs have an important role on Salesforce.com projects as they may radically affect the chosen technology and integration pattern.  The SLAs should be based upon real business needs (sorry – not everything in life needs to be real time) that help define the non-functional requirements.  If you only need to drive a few miles you do not NEED a highway.  However if you are going on a road-trip I hope you aren’t taking side-roads!  Define your solutions based upon your business’s service level requirements.

The Integration Architecture has a clearly defined standard for applying different Integration Use Cases

As your landscape evolves and your Salesforce.com expertise matures, the goal is to define a set of capabilities and standards for all Salesforce.com integrations at your company.  Each project should not have to define when and where to use what technologies, how and when to authenticate, etc.  These architectually significant designs should be standardized for your enteprise.  This is where a Center of Excellence or Architecture Review Board comes into play.  Each project should be subservient to a higher integration architecture authority.

A Typical Enteprise Salesforce.com Integration Architecture

Let’s take a look at a reference Salesforce.com Integration Architecture.  This may or may not look like your existing landscape – however this reference is based upon years of work at many Fortune 500 companies.  The reference design also does not recommend one technology vendor or solution over another – rather the goal is to understand the technical capabilities that you can (and probably should) consider as your Salesforce.com landscape matures.

A Salesforce.com Reference Integration Architecture

Let’s take a look at the most common integration use-cases and how they apply to your Salesforce.com Integration Architecture.  The direction of the arrows in the reference model is not necessarily the way the data is moving, but rather the way the integration connection is being established.  This is a critical aspect of Integration Architecture as it pertains to your security and any real-time requirements.

Cloud-to-Ground (Salesforce.com Originated)

In Cloud-to-Ground use cases you are attempting to push a transaction (message or data) from Salesforce into your On-Premise infrastructure.

Capability #1 – The Salesforce.com originated message is relayed to a DMZ (demilitarized zone) service end-point.  This could be a firewall, a services gateway appliance, or reverse proxy.  You must work closely with your security team to define this layer as opening the corporate firewall to inbound web traffic is a high security risk.  This is where much (if not all) of your security authentication from Salesforce.com occurs.  Whitelisted IPs, two-way SSL, and basic HTTP authentication are some of the ways to authenticate Salesforce into the DMZ layer.

Capability #2- The message is relayed from the DMZ security zone into the trusted On-Premise infrastructure.  The message is usually destined for a Enterprise Service Bus (ESB) and durable message queue.  The ESB also would handle any transformation, mediation, and orchestration services required by the detailed integration requirements.

Capability #3- Depending on your Enterprise Architecture the ESB maybe pushing the message into the SOA infrastructure.  These web-services are providing consumer agnostic data and business process services to the Enterprise.  Salesforce.com can become a consumer (and later a producer) of these SOA services.  By re-using existing SOA web-services you can save your project a lot of time and money as oppossed to integrating directly into the source system.  If you do not have a SOA layer your project maybe responsible for integrating directly into the legacy application.

Capability #4- Another key capability for mature Salesforce.com Integration Architectures is for some sort of On-Premise database access.  This maybe a standalone database or part of a more formal Enterprise Data Warehouse (including an ODS – an operational data store).  Most commonly (but not always) in a Cloud-to-Ground scenario this transaction would be a database READ.  Salesforce.com can read data from the database in real (or near-real) time.

Ground-to-Cloud (On-Premise Originated)

In Ground-to-Cloud use cases you are attempting to push AND pull data from Salesforce from your On-Premise infrastructure.

Capability #5 – A mature Integration Architecture should be handling all of the real-time calls into Salesforce from the ESB.  However if you do NOT have an ESB, this step would occur from each separate application requiring access to Salesforce.  From a security stand-point it is much better to handle all of the calls to Salesforce from a centralized integration middleware.  You can use oAuth or user/pw session based authentication to Salesforce.  The middleware may already have a session with Salesforce so that you don’t need to log-in again for every transaction.

Capability #6 – Many integrations can be accomplished in a batch design.  This is often the cheapest and fastest way to get data in and out of Salesforce.com.  I would recommend a robust ETL solution is necessary for all Salesforce environments.  (This maybe as simple as Salesforce’s Data Loader Command Line Interface).  The role of the ETL is to move large data volumes using the Bulk API where possible.

Capability #7 – As a Salesforce.com architect you have a responsibility to your company or client to off-load your Salesforce data into a replicated copy.  My argument for this is that Salesforce’s database is not likely to have outages or lose data, however you and your team are VERY likely to break your own data via user error, bad-code, or run-away processes.  By replicating your data off-line you now have the power to restore data to an earlier state without engaging Salesforce (who may or may not be able to restore it exactly as necessary).

Capability #8 – The ETL is also responsible for moving data in and out of your database infrastructure.  Often data is necessary to be staged in Salesforce (Accounts for example) from the EDW.  Also pulling data down from Salesforce into your EDW maybe much easier when done using batch processing patterns.

Cloud-to-Cloud

Capability #9 – If you have multiple orgs (see my article on multi-org strategy) you will often have the need to integrate between the Orgs.  Salesforce makes this (sometimes too) easy via Salesforce2Salesforce.  You can also directly contact another Org via RESTful web services integration.  Salesforce.com’s road-map includes the ability to consume other org’s data via OData which may also be a good way of providing read-only access across your org landscape.

Capability #10 – Salesforce’s robust integration technology makes it very easy to integrate point-to-point with other systems.  While this would be recommended for some solutions (Google Maps mashups, etc) I would recommend staying away from this design in large enterprises.  The more that Salesforce is made to be the hub of integration activity, the more time you will spend building, maintaining, and troubleshooting integrations as opposed to building new business value.  This is a trap I have seen many companies fall into.

Capability #11 – Rather than using Salesforce.com to be your hub of cloud-to-cloud integration activity many companies have moved towards Cloud based Integration-as-a-Service packages.  While not true ESB’s per se, many integration vendors have started providing cloud based solutions for managing your cloud-to-cloud use cases.   Because these solutions are specifically tailored for Salesforce.com (and other popular SaaS vendors), the time to build and deploy an integration can be radically reduced as opposed to using the ESB.

Cabability #12 – The cloud service bus’s can handle service mediation, transformation, routing, error handling, etc to your other cloud based end-points.  Having to build durable and resilient integration solutions inside of Salesforce can be expensive and very complicated.  Middleware should be used where and when possible.

Cabability #13 – Some companies prefer to broker all integrations through their ESB, including Cloud-to-Cloud use cases.  My warnings here are this: the cost of highly resilient ESB’s can be EXTREMELY high.  If the service levels between Salesforce.com and Workday, for example, must go through your on-premise technology, you maybe shooting yourself in the foot.  Now your “Cloud” solution is piggy-backing on the same technical infrastructure, cost, service levels, and release timeline of your On-Premise solutions.  Tread lightly and make sure to design your Integration Architecture first and foremost about delivering BUSINESS VALUE.

In Summary

I was previously an Enterprise Architect working with Service Oriented Architectures before becoming a Salesforce.com Certified Technical Architect.  When I was first introduced to Salesforce I was shocked to see either a 100% dependency on batch integration technology or a 100% reluctance to use anything but Real-Time services design.  However one of the reasons I enjoy what I do so much is that I have learned that there is NO GLASS SLIPPER in Salesforce Integration Architecture.  One size does not fit all and no one solution can be the best for all or your requirements.  It is your responsibility as the architect to analyze, recommend, and implement a variety of integration capabilities that will enable your team, clients, and company to realize the powerful transformation of moving to the Salesforce1 platform.