What is a Workflow Engine for Process Orchestration?

A workflow engine is a software application that orchestrates human and automated activities. It creates, manages, and monitors the state of activities in a business process, such as the processing and approval of a claim in an insurance case, and determines the extra activity it should transition to according to defined business processes. Sometimes, such software is called an orchestration engine instead.

Workflow engines have distinct faces. Some people mainly focus on BPM suites. Others see visual coding integration tools like Dell Boomi as a workflow and orchestration engine. For data streaming with Apache Kafka, the Confluent Stream Designer enables building pipelines with a low code/no code approach.

And Business Process Management (BPM)?

A workflow engine is a core business process management (BPM) component. BPM is a broad topic with many definitions. From a technical perspective, when people talk about workflow engines or orchestration and automation of human and automated tasks, they usually mean BPM tools.

The Association of Business Process Management Professionals defines BPM as:

“Business process management (BPM) is a disciplined approach to identify, design, execute, document, measure, monitor, and control both automated and non-automated business processes to achieve consistent, targeted results aligned with an organization’s strategic goals. BPM involves the deliberate, collaborative and increasingly technology-aided definition, improvement, innovation, and management of end-to-end business processes that drive business results, create value, and enable an organization to meet its business objectives with more agility. BPM enables an enterprise to align its business processes to its business strategy, leading to effective overall company performance through improvements of specific work activities either within a specific department, across the enterprise, or between organizations.“

Tools for Workflow Automation: BPMS, ETL and iPaaS

A BPM suite (BPMS) is a technological suite of tools designed to help BPM professionals accomplish their goals. I had a talk ~10 years ago at ECSA 2014 in Vienna: “The Next-Generation BPM for a Big Data World: Intelligent Business Process Management Suites (iBPMS)“. Hence, you see how long this discussion has existed already.

I worked for TIBCO and used products like ActiveMatrix BPM and TIBCO BusinessWorks. Today, most people use open-source engines like Camunda or SaaS from cloud service providers. Most proprietary legacy BPMS I saw ten years ago do not have much presence in the enterprises anymore. And many relevant vendors today don’t use the term BPM anymore for tactical reasons

Some BPM vendors like Camunda also moved forward with fully managed cloud services or new (more cloud-native and scalable) engines like Zeebe. As a side note: I wish Camunda had built Zeebe on top of Kafka instead of building its own engine with similar characteristics. They must invest so much into scalability and reliability instead of focusing on a great workflow automation tool. Not sure if this pays off.

Traditional ETL and data integration tools fit into this category, too. Their core function is to automate processes (from a different angle). Cloud-native platforms like MuleSoft or Dell Boomi are called Integration Platform as a Service (iPaaS). I explored the differences between Kafka and iPaaS in a separate article.

This is NOT Robotic Process Automation (RPA)!

Before I look at the relationship between data streaming with Kafka and BPM workflow engines, it is essential to separate another group of automation tools: Robotic process automation (RPA).

RPA is another form of business process automation that relies on software robots. This automation technology is often seen as artificial intelligence (AI), even though it is usually a more uncomplicated automation of human tasks.

In traditional workflow automation tools, a software developer produces a list of actions to automate a task and interface to the backend system using internal APIs or dedicated scripting language.

In contrast, RPA systems develop the action list by watching the user perform that task in the application’s graphical user interface (GUI) and then perform the automation by repeating those tasks directly in the GUI. This can lower the barrier to using automation in products that might not otherwise feature APIs for this purpose.

RPA is excellent for legacy workflow automation that requires repeating human actions with a GUI. This is a different business problem. Of course, overlaps exist. For instance, Gartner’s Magic Quadrant for RPA does not just include RPA vendors like UiPath but also traditional BPM or integration vendors like Pegasystems or MuleSoft (Salesforce) that move into this business. RPA tools integrate well with Kafka. Besides that, they are out of the scope of this article as they solve a different problem than Kafka or BPM workflow engines.

Data Streaming with Kafka and BPM Workflow Engines: Friends or Enemies?

A workflow engine, respectively a BPM suite, has different goals and sweet spots compared to data streaming technologies like Apache Kafka. Hence, these technologies are complementary. No surprise that most BPM tools added support for Apache Kafka (the de facto standard for data streaming) instead of just supporting request-response integration with web service interfaces like HTTP and SOAP/WSDL, similar to any ETL tool and Enterprise Service Bus (ESB) has Kafka connectors today.

This article explores specific case studies that leverage Kafka for stateful business process orchestration. I no longer see BPM and workflow engines kicking off many new projects. Many automation tasks are done with data streaming instead of adding another engine to the stack “just for business process automation”. Still, while the techniques overlap, they are complementary. Hence, I would call it frenemies.

When to use Apache Kafka instead of BPM Suites/Workflow Engines?

To be very clear initially: Apache Kafka cannot and does not replace BPM engines! Many nuances must be evaluated to choose the right tool or combination. And I still see customers using BPM tools and integrating them with Kafka.

Camunda did a great job similar to Confluent, bringing the open source core into an enterprise solution and, finally, a fully managed cloud service. Hence, this is the number one BPM engine I see in the field; but it is not like I see it every week in customer meetings.

Kafka as the stateful data hub and BPM for human interaction

So, from my 10+ years of experience with integration and BPM engines, here is my guide for choosing the right tool for the job:

• Data streaming with Kafka became the scalable real-time data hub in most enterprises to process data and decouple applications. So, this is often the starting point for looking at a new business process’s automation, whether it is a small or high volume of data, no matter if transactional or analytical data.
• If your business process is complex and cannot easily be defined or modeled without BPMN, go for it. BPM engines provide visual coding and deployment/monitoring of the automated process. But don’t model every single business process in the enterprise. In a very agile world, time-to-market and changing business strategies all the time is not easier if too much time is spent on (outdated) diagrams. That’s not the goal of BPM, of course. But I have seen too many old, inaccurate BPMN or UML visualizations in customer meetings.
• If you don’t have Kafka yet and need just some automation of a few processes, you might start with a BPM workflow engine. However, both data streaming and BPM are strategic platforms. A simple tool like IFTTT (“If This Then That”) might do the job well for automating a few workflows with human interaction in a single domain.
• The sweet spot of a process orchestration tool is a business process that requires automated and manual tasks. The BPM workflow engine orchestrates complex human interaction in conjunction with automated processes (whether streaming/messaging interfaces, API calls, or file processing). Process orchestration tools support developers with pleasant features for generating UIs for human forms in mobile apps or web browsers.

Data Streaming as the stateful workflow engine or with another BPM tool?

TL;DR: Via Kafka as the data hub, you already access all the data on other platforms. Evaluate per use case and project if tools like Kafka Streams can implement and automate the business process or if a dedicated workflow engine is better. Both integrate very well. Often, a combination of both is the right choice.

Saying Kafka is too complex does not count anymore (in the cloud), as it is just one click away via fully managed and supported pay-as-you-go with SaaS offerings like Confluent Cloud.

In this blog, I want to share a few stories where the stateful workflows were automated natively with data streaming using Kafka and its ecosystem, like Kafka Connect and Kafka Streams.

To learn more about BPM case studies, look at Camunda and similar vendors. They provide excellent success case studies, and some use cases combine the workflow engine with Kafka.

Apache Kafka as Workflow and Orchestration Engine

Apache Kafka focuses on data streaming, i.e., real-time data integration and stream processing. However, the core functionality of Kafka includes database characteristics like persistence, high availability, guaranteed ordering, transaction APIs, and (limited) query capabilities. Kafka is some kind of database but does not try to replace others like Oracle, MongoDB, Elasticsearch, et al.

Apache Kafka as stateful and reliable persistence layer

Some of Kafka’s functionalities enable implementing the automation of workflows. This includes stateful processing:

• The distributed commit log provides persistence, reliability, and guaranteed ordering. Tiered Storage (only available via some vendors or cloud services today) makes long-term storage of big data cost-efficient and scalable.
• The replayability of historical data is crucial to rewind events in case of a failure or other issues within the business process. Timestamps of events, in combination with a clever design of the Kafka Topics and related Partitions, allow the separation of concerns.
• While a Kafka topic is the source of truth for the entire history with guaranteed ordering, a compacted topic can be used for quick lookups of the current, updated state. This combination enables the storage of information forever and updates to existing information with the updated state. Plus, querying the data via key/value request.
• Stream processing with Kafka Streams or KSQL allows keeping the current state of a business process in the application, even long-term, over months or years. Interactive queries on top of the streaming applications allow API queries against the current state in microservices (as an application-centric alternative to using compacted topics).
• Kafka Connect with connectors to cloud-native SaaS and legacy systems, clients for many programming languages (including Java, Python, C++, or Go), and other API interfaces like the REST Proxy integrate with any data source or data sink required for business process automation.
• The Schema Registry ensures the enforcement of API contracts (= schemas) in all event producers and consumers.
• Some Kafka distributions and cloud services add data governance, policy enforcement, and advanced security features to control data flow and privacy/compliance concerns.

The Saga Design Pattern for Stateful Orchestration with Kafka

Apache Kafka supports exactly-once semantics and transactions in Kafka. Transactions are often required for automating business processes. However, a scalable cloud-native infrastructure like Kafka does not use XA transactions with the two-phase commit protocol (like you might know from Oracle and IBM MQ) if separate transactions in different databases need to be executed. This does not scale well. Therefore, this is no option.

Learn more about transactions in Kafka (including the trade-offs) in a dedicated blog post: “Apache Kafka for Big Data Analytics AND Transactions“.

Sometimes, another alternative is needed for workflow automation with transaction requirements. Welcome to a merging design pattern that originated with microservice architectures: The SAGA design pattern is a crucial concept for implementing stateful orchestration if one transaction in the business process spans multiple applications or service calls. SAGA pattern enables an application to maintain data consistency across multiple services without using distributed transactions.

Instead, the SAGA pattern uses a controller that starts one or more “transactions” (= regular service calls without transaction guarantee) and only continues after all expected service calls return the expected response:

1. The Order Service receives the POST /orders request and creates the Create Order saga orchestrator
2. The saga orchestrator creates an Order in the PENDING state
3. It then sends a Reserve Credit command to the Customer Service
4. The Customer Service attempts to reserve credit
5. It then sends back a reply message indicating the outcome
6. The saga orchestrator either approves or rejects the Order

Find more details from the above example at microservices.io’s detailed explanation of the SAGA pattern.

Case studies for workflow automation and process orchestration with Apache Kafka

Reading the above sections, you should better understand Kafka’s persistence capabilities and the SAGA design pattern. Many business processes can be automated at scale in real-time with the Kafka ecosystem. There is no need for another workflow engine or BPM suite.

Learn from interesting real-world examples of different industries:

• Digital Native: Salesforce
• Financial Services: Swisscom
• Insurance: Mobiliar
• Open Source Tool: Kestra

Salesforce: Web-scale Kafka Workflow Engine

Salesforce built a scalable real-time workflow engine powered by Kafka.

Why? The company recommends to “use Kafka to make workflow engines more reliable”.

At Current 2022, Salesforce presented their project for workflow and process automation with a stateful Kafka engine. I highly recommend checking out the entire talk, or at least the slide deck.

Salesforce introduced a workflow engine concept that only uses Kafka to persist state transitions and execution results. The system banks on Kafka’s high reliability, transactionality, and high scale to keep setup and operating costs low. The first target use case was the Continuous Integration (CI) systems.

A demo of the system presented:

• Parallel and nested CI workflow definitions of varying declaration formats
• Real-time visualization of progress with the help of Kafka.
• Chaos and load generation to showcase how retries and scale-ups work.
• Extension points
• Contrasting the implementation to other popular workflow engines

Here is an example of the workflow architecture:

A few notes on the advantages Salesforce pointed out for using Kafka instead of other databases or CI/CD tools:

• The only stateful component is Kafka -> Dominating reliability setup
• Kafka was chosen as the persistence layer -> SLAs / reliability better than with a database/sharded Jenkings/NoSQL -> Four nines (+ horizontal scaling) instead of three nines (see slide “reliability contrast”)
• Restart K8S clusters and CI workflows again easily
• Kafka State Topic -> Store the full workflow graph in each message (workflow Protobuf with defined schemas)
• Compacted Topic updates states: not started, running, complete -> Track, manage, and count state machine transitions

The Salesforce implementation is open source and available on Github: “Junction Workflow is an upcoming workflow engine that uses compacted Kafka topics to manage state transitions of workflows that users pass into it. It is designed to take multiple workflow definition formats.”

Swisscom Custodigit: Secure crypto investments with stateful data streaming and orchestration

Custodigit is a modern banking platform for digital assets and cryptocurrencies. It provides crucial features and guarantees for seriously regulated crypto investments:

• Secure storage of wallets
• Sending and receiving on the blockchain
• Trading via brokers and exchanges
• Regulated environment (a key aspect and no surprise as this product is coming from the Swiss; a very regulated market)

The following architecture diagrams are only available in Germany, unfortunately. But I think you get the points. With the SAGA pattern, Custodigit leverages Kafka Streams for stateful orchestration.

The Custodigit microservice architecture uses microservices to integrate with financial brokers, stock markets, cryptocurrency blockchains like Bitcoin, and crypto exchanges:

Custodigit implements the SAGA pattern for stateful orchestration. Stateless business logic is truly decoupled, while the saga orchestrator keeps the state for choreography between the other services:

Swiss Mobiliar: Decoupling and workflow orchestration

Swiss Mobiliar (Schweizerische Mobiliar, aka Die Mobiliar) is the oldest private insurer in Switzerland. Event Streaming powered by Apache Kafka with Kafka supports various use cases at Swiss Mobiliar:

• Orchestrator application to track the state of a billing process
• Kafka as database and Kafka Streams for data processing
• Complex stateful aggregations across contracts and re-calculations
• Continuous monitoring in real-time

Mobiliar’s architecture shows the decoupling of applications and orchestration of events:

Here you can see the data structures and states defined in API contracts and enforced via the Schema Registry:

Also, check out the on-demand webinar with Mobiliar and Spoud to learn more about their Kafka usage.

Kestra: Open-source Kafka workflow engine and scheduling platform

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines. The project is open-source and available on GitHub:

• Any kind of workflow: Workflows can start simple and progress to more complex systems with branching, parallel, dynamic tasks, flow dependencies
• ‍ Easy to learn: Flows are in simple, descriptive language defined in YAML—you don’t need to be a developer to create a new flow.
• Easy to extend: Plugins are everywhere in Kestra, many are available from the Kestra core team, but you can create one easily.
• Any triggers: Kestra is event-based at heart—you can trigger an execution from API, schedule, detection, events
• A rich user interface: The built-in web interface allows you to create, run, and monitor all your flows—no need to deploy your flows, just edit them.
• Enjoy infinite scalability: Kestra is built around top cloud native technologies—scale to millions of executions stress-free.

This is an excellent project with a cool drag & drop UI:

I copied the description from the project. Please check out the open-source project for more details.

What about Apache Flink for streaming workflow orchestration and BPM?

This post covered various case studies for the Kafka ecosystem as a stateful workflow orchestration engine for business process automation. Apache Flink is a stream processing framework that complements Kafka and sees significant adoption.

The above case studies used Kafka Streams as the stateful stream processing engine. It is a brilliant choice if you want to embed the workflow logic into its own application/microservice/container.

Apache Flink has other sweet spots: Powerful stateful analytics, unified batch and real-time processing, ANSI SQL support, and more. In a detailed blog post, I explored the differences between Kafka Streams and Apache Flink for stream processing.

Apache Flink for Complex Event Processing (CEP) and cross-cluster queries

The following differentiating features make Flink an excellent choice for some workflows:

• Complex Event Processing (CEP): CEP generates new events to trigger action based on situations it detects across multiple event streams with events of different types (situations that build up over time and space). Event Stream Processing (ESP) detects patterns over event streams with homogenous events (i.e., patterns over time). The powerful pattern API of FlinkCEP allows you to define complex pattern sequences you want to extract from your input stream to detect potential matches.
• “Read once, write many”: Flink allows different analytics without readying the Kafka topic several times. For instance, two queries on the same Kafka Topic like “CTAS .. * FROM mytopic WHERE eventType=1” and “CTAS .. * FROM mytopic WHERE eventType=2” can be grouped together. The query plan will only do one read. This fundamentally differs from Kafka-native stream processing engines like Kafka Streams or KQL.
• Cross-cluster queries: Data processing across different Kafka topics of independent Kafka clusters of different business units is a new kind of opportunity to optimize and automate an existing end-to-end business process. Be careful when using this feature wisely, though. It can become an anti-pattern in the enterprise architecture and create complex and unmanageable “spaghetti integrations”.

Should you use Flink as a workflow automation engine? It depends. Flink is great for stateful calculations and queries. The domain-driven design enabled by data streaming enables choosing the right technology for each business problem. Evaluate Kafka Streams, Flink, BPMS, and RPA. Combine them as it makes the most sense from a business perspective, cost, time to market, and enterprise architecture. A modern data mesh abstracts the technology behind each data product. Kafka as the heart decouples the microservices and provides data sharing in real-time.

Kafka as Workflow Engine or with other Orchestration Tools

BPMN or similar flowcharts are great for modeling business processes. Business and technical teams easily understand the visualization. It documents the business process for later changes and refactoring. Various workflow engines solve the automation: BPMS, RPA tools, ETL and iPaaS data integration platforms, or data streaming.

The blog post explored several case studies where Kafka is used as a scalable and reliable workflow engine to automate business processes. This is not always the best option. Human interaction, long-running processes, or complex workflow logic are signs to choose a dedicated tool may be better. Ensure understanding of the underlying design pattern like SAGA, evaluate dedicated orchestration and BPM engines like Camunda, and choose the right tool for the job.

Combining data streaming and a separate workflow engine is sometimes best. However, the impressive example of Salesforce proved Kafka could and should be used as a scalable, reliable workflow and stateful orchestration engine for the proper use cases. Fortunately, modern enterprise architecture concepts like microservices, data mesh, and domain-driven design allow you to choose the right tool for each problem. For instance, in some projects, Apache Flink might be a terrific add-on for challenges that require Complex Event Processing to automate the stateful workflow.

How do you implement business process automation? What tools do you use with or without Kafka? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka as Workflow and Orchestration Engine appeared first on Kai Waehner.

Key takeaways of the talk:

• A cloud-native architecture enables flexible and agile development
• Modern middleware leverages containers, Microservices and a cloud-native architecture
• Just using Jenkins and Docker is not enough

A Cloud-Native Architecture for Middleware

Cloud Native development and deployment enables:

• Scalable services
• Resiliency
• Automatic load balancing and failover
• DevOps
• Usage of public cloud platforms, but also private or hybrid
• Vendor-agnostic deployment
• Shorter time to results and increased flexibility

With all this, you can focus on solving your business problems instead of spending your time with plenty of technical issues in ”static and inflexible legacy architectures”. The following requirements are relevant for building a cloud-native architecture:

Microservices and Containers

• Independent Services
• Flexible Deployment

Continuous Integration and Continuous Delivery

• Scripting / Automatic Test and Deployment
• Service Discovery
• Dynamic Distributed Configuration Management

Scalability and Failover

• Cluster Management (Scheduling and Orchestration)
• Load Balancing (Server-side and Client-side)
• Resilience Design Patterns

Cloud Native Platform

• Self-Service Agile Infrastructure
• Private / Public / Hybrid PaaS

Slide Deck from JPoint Moscow 2016

The slide deck discusses requirements for a cloud-native architecture, available platforms such as Docker, Kubernetes, Apache Mesos or Cloud Foundry, and how all these components are related to Middleware:

As always, feedback appreciated!

By the way: This slide deck is also an “add-on” and update to my well discussed article of last year: “Does a good Microservices architecture spell the death of the Enterprise Service Bus?”

The post Microservices, Containers, Docker and a Cloud-Native Architecture in the Middleware World appeared first on Kai Waehner.

Making the data “actionable” is the real challenge! Seeing the information that helps to make a decision on a composite dashboard using business intelligence (BI) and big data analytics is just the first step and where too many companies stop. An enterprise must be able to fire off the business process to execute the decision made regarding the data. That’s where the buzzword “Intelligent Business Process Management Suite (iBPMS)” comes into play.

iBPMS = BPM + Big Data / Fast Data Analytics + Social Integration

iBPMS is a term introduced by Gartner – an information technology research analyst – to indicate the evolution of the classic BPMS into the next-generation BPM, which includes integration of big data analytics, social media and mobile devices into organization’s business processes support.

Some other companies and analysts use other names, for example “Operational Business Process” or “Intelligent Business Operation (IBO)”. Many people abbreviate this topic with iBPM instead of iBPMS. However, in the end, everybody is talking about an intelligent business processes.

An intelligent business process combines big data, analytics and business process management (BPM) – including case management! This enables applications and humans to make data-driven decisions based on big data analytics. Two flavors exist: „Process starts big data analytics“ (e.g. recommendation engine) and „big data analytics starts process“ (e.g. prevention of flu epidemic).

Let’s look at a real world use case to show why realizing intelligent business processes makes a lot of sense, and how to actually build such a solution.

Use Case: Improved Customer Relationship Management (CRM)

A casino wants to increase customer satisfaction. Therefore, the casino leverages big data analytics and gives customers a digital identity (including hotel preferences, gambling behavior, etc.), so that customers can get personalized offers in real time. For instance, the casino can offer a 30% coupon for a show ticket (which is not sold out yet) or a free steak tonight (which would be perishable until tomorrow anyway, besides many seats are available in the restaurant currently). Besides increasing customer satisfaction, the casino creates further benefits such as cost reduction or increased revenue. For instance, a customer visiting a show or eating a steak will also spend money for drinks.

Products to Implement an Intelligent Business Process

It does not matter if you use Salesforce CRM or any other product for your customer management. Also, many different BPM and integration tools are available on the market: TIBCO, IBM, Oracle, Software AG, to name a few leaders…

The following implementation uses different TIBCO products to implement the use case. Even if you have never seen these products before, you will understand easily how these tools work together to realize an intelligent business process.

All these products are loosely coupled, but highly integrated. Each TIBCO product has a specific task to solve. Nevertheless, they connect to each other very well via specific adapters (e.g. the Tibbr plugin of BusinessWorks) or standards for interoperability such as SOAP / REST Web Services or JMS messaging.

Let’s discuss the steps that are necessary to realize the described use case.

Step 1: Integration of Siebel CRM, SAP ERP and CICS Mainframe with TIBCO BusinessWorks ESB

The first step is integration of different systems and interfaces. Complex transformations have to be realized to format and process all required information correctly. TIBCO Business Works is used as integration platform (ESB).

Tasks:

• Integrate customer data from Siebel CRM.
• Integrate casino data from SAP ERP.
• Integrate payment information from CICS mainframe.
• Process incoming gambling information from slot machines via EDI.
• Push transformed streaming events in real time to output connector.

Depending on the use case, you can integrate any other technology,  e.g. Hadoop or a Data Warehouse (DWH) such as Teradata or HP Vertica.

Step 2: Real Time Streaming Analytics with TIBCO StreamBase

The pre-processed data is pushed into a stream processing engine for doing real time streaming analytics. TIBCO StreamBase is a mature product with awesome tooling for these tasks.

Tasks:

• Filter and analyze all kinds of events.
• Correlate relevant events.
• If possible, act in real time automatically.
• Otherwise, start a business process for human interaction.

For example, you correlate events such as “customer lost a lot of money gambling”, “complaint via Twitter”, “good weather” and “seats available at pool bar” to send a 50% coupon for a cocktail at the pool bar. If no free seats are available, a human has to make a decision how to improve customer satisfaction. If a customer complains about his current situation another more complex business process has to be initiated using case management features of the BPM tool.

Step 3: Automatic Reaction or Human Interaction with TIBCO ActiveMatrix BPM

If a task cannot be automated completely, a business process instance is started to react appropriately to an event. This can be a relative simple process with human interaction and automated steps, or a more complex situation requiring flexible case management. TIBCO ActiveMatrix BPM is the right tool for this job. The current release already includes several case management features!

Tasks:

• Do something to make customer happy again, e.g. check if the steak restaurant has a lot of steaks in stock. Call customer on his mobile phone and offer a steak coupon.
• Send steak coupon via SMS or email.
• Or escalate to your manager if customer does not appreciate the offer. Case management features can be used here to enable humans to act in a more flexible way to unexpected events.

Step 4: Work Distribution to Mobile Apps with TIBCO Tibbr (Social Enterprise Platform)

TIBCO Tibbr is a social enterprise platform similar to Facebook, but for Enterprises; including several additional and advanced features such as security, customization or integration with other TIBCO and non-TIBCO products and applications. Tibbr’s process notifications are used for work distribution to occasional users. These can interact via their iPhone respectively Android smartphone apps or other mobile clients.

In the above example, the manager would receive a push message about the escalation on his iPhone while walking around in the casino. He can react immediately by sending a Tibbr message to a colleague or starting another business process.

Step 5: TIBCO BPM Analytics – A Picture is Worth a Thousand Processes

After the implementation is deployed and running, you can investigate and improve your processes by using explorative data analytics. TIBCO BPM Analytics provides end-to-end process visibility including self-service, interactive, drag and drop reports for business users. TIBCO Spotfire – a Business Intelligence tool for explorative data analytics – is integrated into TIBCO ActiveMatrix BPM (and many other TIBCO products) for that reason.

The Realization of Intelligent Business Processes (iBPMS) is no Rocket Science

The above sections showed an implementation of an intelligent business process using iBPMS tooling. A BPM solution is not the only thing you need to realize intelligent business processes. You need to integrate different enterprise applications and big data / fast data analytics. Integration and separation of concerns is key for success of such a project. Integration of social enterprise platforms becomes prevalent for supporting occasional users. iBPMS sounds like a very complex topic first. However, it is easy to implement iBPMS if you can use loosely coupled, but integrated tooling that can solve your requirements.

Kai Wähner works as Technical Lead at TIBCO. All opinions are his own and do not necessarily represent his employer. Kai’s main area of expertise lies within the fields of Application Integration, Big Data, SOA, BPM, Cloud Computing, Java EE and Enterprise Architecture Management. He is speaker at international IT conferences such as JavaOne, ApacheCon or OOP, writes articles for professional journals, and shares his experiences with new technologies on his blog. Contact: LinkedIn, @KaiWaehner or kontakt@kai-waehner.de.

The post Intelligent BPM Suite (iBPMS): Implementation of a CRM Use Case appeared first on Kai Waehner.

I want to share the slides with you. As always, I appreciate every comment or feedback…

Abstract: iBPMS / iBPM

Here is the abstract of my session about iBPMS:

Business Process Management (BPM) is established, tools are stable, and many companies use it successfully. However, today’s business processes are based just on “dumb” data from relational databases or web services. Humans make decisions based on this information. Instead, the value of big data analytics should be integrated into business processes, too. Besides, user interfaces are inflexible. Modern concepts such as mobile devices or social media are not integrated into business processes. That is status quo. Companies miss a huge opportunity here!
This session explains the idea behind next-generation BPM (also called Intelligent Business Process Management, iBPMS, iBPM), which includes big data analytics, social media, and mobile device support. The talk will focus on real world use cases. The audience will learn how to realize intelligent business processes technically by combining BPM, integration, big data and analytics.

Use Case: TIBCO AMX BPM + BusinessWorks + StreamBase + Tibbr

The content of the slides is vendor-independent. It will help you to understand the concepts of iBPMS and how different parts such as BPM, Big Data Analytics or Integration are related. It does not matter if you want to / have to use IBM, Oracle, TIBCO, or any other software for realizing iBPMS.

To demonstrate the implementation of a real world sue case, the slides also include an example of how to implement iBPMS with the TIBCO middleware stack. The solution uses:

• TIBCO ActiveMatrix BPM for business process management to combine human interaction and automatic tasks
• TIBCO ActiveMatrix BusinessWorks – an Enterprise Service Bus (ESB) – for integration  of applications (SAP, Salesforce, Mainframe, EDI, etc.) and technologies (SOAP Web Services, REST APIs, JMS, TCP, etc.)
• TIBCO StreamBase for stream processing (fast data processing and streaming analytics)
• TIBCO Tibbr as social enterprise network for work distribution to occasional users

A huge benefit of the TIBCO stack is that the products are loosely coupled, but integrated. Thus, it is easy to implement iBPMS.

Slides: iBPMS at ECSA 2014

Here are the slides:

The post Intelligent Business Process Management Suites (iBPMS) – The Next-Generation BPM for a Big Data World appeared first on Kai Waehner.

Content

Slides

The post Slides from OOP 2014 Online: Next-Generation BPM – How to create intelligent Business Processes thanks to Big Data appeared first on Kai Waehner.

Content

BPM is established, tools are stable, many companies use it successfully. However, today’s business processes are based on data from relational databases or web services. Humans make decisions due to this information. Companies also use business intelligence and other tools to analyze their data.  Though, business processes are executed without access to this important information because technical challenges occur when trying to integrate big masses of data from many different sources into the BPM engine. Additionally, bad data quality due to duplication, incompleteness and inconsistency prevents humans from making good decisions. That is status quo. Companies miss a huge opportunity here!

This session explains how to achieve intelligent business processes, which use big data to improve performance and outcomes. A live demo shows how big data can be integrated into business processes easily – just with open source tooling. In the end, the audience will understand why BPM needs big data to achieve intelligent business processes.

Slides

As always, I appreciate any kind of feedback via comment, email, Twitter, or social networks (LinkedIn, Xing). Thanks.

Best regards,

Kai Wähner (Twitter: @KaiWaehner)

The post Slides online: How to create intelligent Business Processes thanks to Big Data (BPM, Apache Hadoop, Talend, Open Source) appeared first on Kai Waehner.

Abstract

Data exchanges in and between companies increase a lot. The number of applications which must be integrated increases, too. As solution, an Enterprise Service Bus (ESB) can be used in almost every integration project – no matter which technologies, transport protocols, data formats, or environments such as Java or .NET are used. All integration projects can be realized in a consistent way without redundant boilerplate code. However, an ESB offers many further features, such as business process management (BPM), master data management, business activity monitoring, or big data. Plenty of ESB products are on the market which differ a lot regarding concepts, programming models, tooling, and open source vs. proprietary. Really one is spoilt for choice.

Learnings

This session shows and compares ESB alternatives and discusses their pros and cons. Besides, a recommendation will be given when to use just a lightweight framework for a specific problem such as systems integration or BPM instead of a (too) powerful ESB. You will also learn what an integration suite is, how it differs from an ESB, and when to use which alternative.

Slides

Here are the slides of my session:

Feel free to tell me your opinions in the comments. I appreciate every feedback!

Best regards,

Kai Wähner (Twitter: @KaiWaehner)

The post Spoilt for Choice: How to Choose the Right Enterprise Service Bus (ESB)? appeared first on Kai Waehner.

Great Article: “What is the TCO difference between WebSphere and JBoss?”

I have spotted a really great article about comparing prices of open source and proprietary products: “What is the TCO difference between WebSphere and JBoss?“. The interesting aspect is, that this article is written by an IBM-biased company (Prolifics). Usually, only open source vendors write such comparisons. I really like this article, seriously! It is good to see comparisons not only by open source vendors, but also by vendors such as IBM (in this case, Prolifics cannot be considered unbiased, it is an IBM consulting company – but that is fine). I just want to give my two cents to this article in the following…

Features, Performance, Availability?

I definitely agree that proprietary vendors have more features, best performance, and highest availability. So, if I have got a 100 million dollar project, where I need all of these features, and where I have to deploy to 1000s of servers, then IBM (or Oracle or SAG or XYZ) might be a good choice! However, in probably 95 percent of use cases, you do not need ALL features, BEST performance, and HIGHEST availability. You just need to solve your problem! Think about this before deciding for a whole stack of proprietary products.

Regarding some other aspects: I disagree that proprietary products have got better manageability and ease of use.

Manageability?

Production-ready installations for open source products can be done within one day – without a lot of expensive consulting efforts. You cannot install the production-ready Websphere stack in one or two days! It is much more complex. Maybe you can make it to install the development edition in one or two days on your laptop. Maybe…

And what about manageability if there is a missing feature. In an open source product, you are very flexible. You can add or change features as you want. Just change the code. That’s it. As these products usually base on open source projects (e.g. Eclipse or Apache), you find all information you need, including documentation and a large community (which helps for free). If you do not want to change it by yourself, the commercial support will help you quickly and just charge the “consulting days” of implementing the new feature. You will get a new feature in a few days or weeks (depending on its size). Try to get a new feature or a change request from proprietary vendors. Good luck. You have to pay a lot of money and / or wait a very long time!

Ease of Use?

As a developer, you can just download an open source product, use an one-click-installer, and use it. Usually, the product is an unified platform, i.e. you can do everything within the development environment intuitively. You are learning by doing. If you want to use the IBM Websphere stack, you have to install several different products. Yes, they are all “Websphere”, but nevertheless, they are different products with different tooling, based on different technologies and code bases (as many of the products come from acquisitions).

Conclusion?

So again, I really like this comparison from an IBM perspective. Every decision maker should consider both views (in the case of this article JBoss and IBM), and then decide by himself. Both solutions are good, but they differ a lot – not just in pricing! Look at all products deeply, do a proof of concept, then make a decision!

What’s your opinion? Feel free to give a comment…

Best regards,

Kai Wähner (Twitter: @KaiWaehner)

The post What is the TCO difference between IBM WebSphere and Open Source JBoss? – Just my two cents… appeared first on Kai Waehner.

Here is a short summary for introducing BPM succesfully (for people who are short of time and not reading the whole article):

• Rule 1: If you want to do BPM correctly, then be aware that your primary goal is to improve Business-IT-Alignment!
• Rule 2: If you want to do BPM correctly, then do not try to realize it without a service-oriented Architecture!
• Rule 3: If you want to do BPM correctly, then be aware that BPM is not just tooling, but an engineering process with a huge, iterative lifecycle!
• Rule 4: If you want to do BPM correctly, then use it only when you really gain benefits!
• Rule 5: If you want to do BPM correctly, then do NOT believe in BPM myths!
• Rule 6: If you want to do BPM correctly, then use BPMN 2.0!
• Rule 7: If you want to do BPM correctly, then choose the right BPM tool for the job!
• Rule 8: If you want to do BPM correctly, then do NOT use it for systems integration!

Please leave feedback, if you have any other opinions or experiences.

Best regards,

Kai Wähner (Twitter: @KaiWaehner)

The post Lessons Learned: Best Practices for a Successful Introduction of Business Process Management (BPM) appeared first on Kai Waehner.