Blog Series – Kafka in Healthcare

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

• Overview – Data Streaming Use Cases and Architectures for Healthcare (including Slide Deck)
• Legacy Modernization and Hybrid Cloud (Optum / UnitedHealth Group, Centene, Bayer)
• THIS POST: Streaming ETL (Bayer, Babylon Health)
• Real-time Analytics (Cerner, Celmatix, CDC/Centers for Disease Control and Prevention)
• Machine Learning and Data Science (Recursion, Humana)
• Open API and Omnichannel (Care.com, Invitae)

Stay tuned for a dedicated blog post for each of these topics as part of this blog series. I will link the blogs here as soon as they are available (in the next few weeks). Subscribe to my newsletter to get an email after each publication (no spam or ads).

Streaming ETL with Apache Kafka

Streaming ETL is similar to concepts you might know from traditional ETL tools. I have already explored how data streaming with Kafka differs from data integration tools and iPaaS cloud services. The critical difference is that you leverage a single platform for data integration and processing at scale in real-time. There is no need to combine several platforms to achieve this. The result is a Kappa architecture that enables real-time but also batch workloads with a single integration architecture.

Streaming ETL with Kafka combines different components and features:

• Kafka Connect as Kafka-native integration framework
• Kafka Connect source and sink connectors to consume and produce data from/to any other database, application, or API
• Single Message Transform (SMT) – an optional Kafka Connect feature – to process (filter, change, remove, etc.) incoming or outgoing messages within the connector deployment
• Kafka Streams or ksqlDB for continuous data processing in real-time at scale for stateless or stateful ETL jobs
• Data governance via schema management, enforcement and versioning using the Schema Registry
• Security and access control using features like role-based access control, audit logs, and end-to-end encryption

In the cloud, you can leverage a serverless Kafka offering for the whole Streaming ETL pipeline. Confluent Cloud fully manages Kafka’s end-to-end infrastructure, including connectors, ksqlDB workloads, data governance, and security.

One last general note: Don’t Design for Data at Rest to Reverse it! Learn more here: “When to Use Reverse ETL and when it is an Anti-Pattern“. Instead, use real-time Streaming ETL for Data in Motion and the Kappa architecture from scratch.

Let’s look at a few real-world deployments in the healthcare sector.

Babylon Health – PII and GDRP compliant Security

Babylon Health is a digital-first health service provider and value-based care company that combines an artificial intelligence-powered platform with virtual clinical operations for patients. Patients are connected with health care professionals through its web and mobile application.

Babylon’s mission is to put an accessible and affordable health service in the hands of every person on earth. For that mission, Babylon built an agile microservice architecture with the Kafka ecosystem:

Here are the “wonders of working” in Healthcare for Babylon (= reasons to choose Kafka):

• Real-time data processing
• Replayability of historical information
• Order matters and is ensured with guaranteed ordering
• GDPR and data ownership for PII compliant security
• Data governance via the schema registry to provide true decoupling and access via many programming languages like Java, Python, and Ruby

Bayer – Data Integration and Processing in R&D

Bayer AG is a German multinational pharmaceutical and life sciences company and one of the largest pharmaceutical companies in the world. They leverage Kafka in various use cases and business domains.

The following scenario is from the research and development department of the pharma business unit. Their focus areas are cardiovascular diseases, oncology, and women’s health. The division employs over 7,500 R&D people and expenses over 2.75 billion euros for R&D.

The use case Bayer presented at a recent Kafka Summit is about analyzing clinical trials, patents, reports, news, and literature leveraging the Kafka ecosystem. The R&D team processes 250 Million documents from 30+ individual data sources. The data includes 7 TB of raw text-rich data with daily updates, additions, and deletions. Algorithms and data evolve. Bayer needs to completely reprocess the data regularly. Various document streams with different formats and schemas flow through several text processing and enrichment steps.

Scalable, reliable Kafka pipelines with Kafka Streams (Java) and Faust (Python) replaced custom, error-prone, non-scalable scripts. Schemas are used as the data interface to ensure data governance. Avro is the first-class citizen data format to enable compression and better throughput.

The true decoupling of Kafka in conjunction with the Schema Registry guarantees interoperability among different components and technologies (java, python, commercial tools, open-source, scientific, proprietary).

Streaming ETL with Kafka for Real-Time Data Integration at any Scale

Think about IoT sensor analytics, cybersecurity, patient communication, insurance, research, and many other domains. Real-time data beats slow data in the healthcare supply chain almost everywhere.

This blog post explored the capabilities of the Apache Kafka Ecosystem for Streaming ETL. Real-world deployments from Babylon Health and Bayer showed how enterprises successfully deploy Kafka for different enterprise architecture use cases.

How do you leverage data streaming with Apache Kafka in the healthcare industry? What architecture does your platform use? Which products do you combine with data streaming? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Streaming ETL with Apache Kafka in the Healthcare Industry appeared first on Kai Waehner.

Blog series: Apache Kafka for Cybersecurity

This blog series explores why security features such as RBAC, encryption, and audit logs are only the foundation of a secure event streaming infrastructure. Learn about use cases,  architectures, and reference deployments for Kafka in the cybersecurity space:

• Part 1: Data in Motion as cybersecurity backbone
• Part 2: Situational awareness
• Part 3 (THIS POST): Threat intelligence
• Part 4: Forensics
• Part 5: Air-gapped and zero trust environments
• Part 6: SIEM / SOAR modernization

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Cyber Threat Intelligence

Threat intelligence, or cyber threat intelligence, reduces harm by improving decision-making before, during, and after cybersecurity incidents reducing operational mean time to recovery, and reducing adversary dwell time for information technology environments.

Threat intelligence is evidence-based knowledge, including context, mechanisms, indicators, implications, and action-oriented advice about an existing or emerging menace or hazard to assets. This intelligence can be used to inform decisions regarding the subject’s response to that menace or hazard.

Threat intelligence solutions gather raw data about emerging or existing threat actors & threats from various sources. This data is then analyzed and filtered to produce threat intel feeds and management reports that contain information that automated security control solutions can use.

Threat intelligence keeps organizations informed of the risks of advanced persistent threats, zero-day threats and exploits, and how to protect against them.

Situational Awareness is Not Enough…

… but the foundation to collect and pre-process data in real-time at scale. Only real-time situational awareness enables real-time threat intelligence to provide huge benefits to the enterprise:

• Mitigate harmful events in cyberspace
• Proactive cybersecurity posture that is predictive, not just reactive
• Bolster overall risk management policies
• Improved detection of threats
• Better decision-making during and following the detection of a cyber intrusion

In summary, threat intelligence allows to:

• See the whole board. And see it more quickly.
• See around corners.
• See the enemy before they see you.

Threat Intelligence for Prevention or Mitigation across the Cyber Kill Chain

Threat intelligence is the knowledge that allows you to prevent or mitigate cyberattacks. It covers all the phases of the so-called “Cyber Kill Chain“:

Threat intelligence provides several benefits:

• Empowers organizations to develop a proactive cybersecurity posture and to bolster overall risk management policies
• Drives momentum toward a cybersecurity posture that is predictive, not just reactive
• Enables improved detection of threats
• Informs better decision-making during and following the detection of a cyber intrusion

Transactional Data vs. Analytics Data

Most use cases around data-in-motion are about all the data. This is true for all transactional use cases and even for many analytical use cases. Each event is valuable: A sale, an order, a payment, an alert from a machine, etc.

However, data is often full of noise. As I discussed earlier in this blog series, the goal in the cybersecurity space is to find the needle in the haystack and to reduce false-positive alerts.

SIEM, SOAR, OT, and ICS are almost always analytic processing regimes, BUT knowing when they are not is important. Kafka can configure topics to be tuned for transactions or analytics.  That is unprecedented in the history of data processing. Threat intelligence (= awareness-in-motion) assumes the PATTERN is valuable, not the data.

Analytics in Motion powered by Kafka Streams / ksqlDB

As you can hopefully imagine from the above requirements and characteristics, event streaming with Apache Kafka and its streaming analytics ecosystem is a perfect fit for the technical infrastructure for threat intelligence.

Threat detection makes sense of the signal and the noise of the data by continuously processing signatures. This enables to detect, contain and neutralize threats proactively:

Analytics can be many things in such a scenario:

• Simple business logic such as stateless filtering or stateful aggregations
• Complex business rules with custom code or with an integrated separate rules engine like Drools
• Machine Learning / Deep Learning by embedding any analytic model (TensorFlow, H2O, DataRobot, whatever) into the Kafka streaming application
• Information from external systems applied in real-time in the right context

On a high level, the advantages of using Kafka Streams or ksqlDB for threat intelligence can be described as follows:

• A single scalable and reliable real-time infrastructure for end-to-end data integration and data processing
• Flexibility to write custom rules and embed other rules engines, frameworks, or trained models
• Integration with other threat detection systems like IDS, SIEM, SOAR

The business logic for cyber threat detection looks different for every use case. Known attack patterns like MITRE ATT&ACK help with the implementation. However, situational awareness and threat detection also need to detect unknown anomalies.

Let’s now take a look at a concrete example.

Intel’s Cyber Intelligence Platform

Let me quote Intel themselves:

“As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. At Intel, we’ve addressed this need by implementing a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Confluent. We believe that CIP positions us for the best defense against cyber threats well into the future.

Our CIP ingests tens of terabytes of data each day and transforms it into actionable insights through streams processing, context-smart applications, and advanced analytics techniques. Kafka serves as a massive data pipeline within the platform. It provides us the ability to operate on data in-stream, enabling us to reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately leads to better prevention.”

Let’s explore Intel’s CIP for threat intelligence in more detail.

Detecting Vulnerabilities with Stream Processing

Intel’s CIP leverages the whole Kafka ecosystem provided by Confluent:

• Ingestion: Kafka producer clients for various sources such as databases, scanning engines, IP address management, asset management inventory, etc.
• Streaming analytics: Kafka Streams for filtering vulnerabilities by business unit, joining asset ownership with vulnerable assets, etc.
• Egress: Kafka Connect sink connectors for data lakes, IT partners, other business units, SIEM, SOAR, etc.
• High availability: Multi-Region Clusters (MRC) for high availability across regions
• And much more…

Here is a high-level architecture:

Intel’s Kafka Maturity Timeline

Building a cybersecurity infrastructure is not a big bang. A step-by-step approach starts with integrating the first sources and sinks, some simple stream processing, and deployment as a pilot project. Over time, more and more data sources and sinks are added, the business logic gets more powerful, and the scale increases.

Intel’s Kafka maturity timeline shows their learning curve:

Kafka Benefits to Intel

Intel describes their benefits for leveraging event streaming as follows:

• Economies of scale
• Operate on data in the stream
• Reduce technical debt and downstream costs
• Generates contextually rich data
• Global scale and reach
• Always on
• Modern architecture with a thriving community
• Kafka leadership through Confluent expertise

That’s pretty much the same reasons I use in many of my other blog posts to explain the rise of data in motion powered by Apache Kafka across industries and use cases…

For more intel on Intel’s Cyber Intelligence Platform powered by Confluent and Splunk, check out their whitepaper and Kafka Summit talk.

Scalable Real-time Cyber Threat Intelligence with Kafka

Kafka is not just used as a backbone and source of truth for data. It also monitors, correlates, and proactively acts on events from various real-time and batch data sources to implement cyber threat intelligence.

The Cyber Intelligence Platform from Intel is a great example of a Kafka-powered cybersecurity solution. It leverages the whole Kafka ecosystem to build a scalable and reliable real-time integration and processing layer. The streaming analytics logic depends on the use case. It can cover simple business logic but also external rules engines or analytic models.

How do you fight against cybersecurity risks? What technologies and architectures do you use to implement cyber threat intelligence? How does Kafka complement other tools? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Kafka for Cybersecurity (Part 3 of 6) – Cyber Threat Intelligence appeared first on Kai Waehner.

Blog series: Apache Kafka for Cybersecurity

This blog series explores why security features such as RBAC, encryption, and audit logs are only the foundation of a secure event streaming infrastructure. Learn about use cases,  architectures, and reference deployments for Kafka in the cybersecurity space:

• Part 1: Data in Motion as cybersecurity backbone
• Part 2 (THIS POST): Situational awareness
• Part 3: Threat intelligence
• Part 4: Forensics
• Part 5: Air-gapped and zero trust environments
• Part 6: SIEM / SOAR modernization

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

The Four Stages of an Adaptive Security Architecture

Gartner defines four stages of adaptive security architecture to prevent, detect, respond and predict cybersecurity incidents:

Continuous monitoring and analytics are the keys to building a successful proactive cybersecurity solution. It should be obvious: Continuous monitoring and analytics require a scalable real-time infrastructure. Data at rest, i.e., stored in a database, data warehouse, or data lake, cannot continuously monitor data in real-time with batch processes.

Situational Awareness

“Situation awareness is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.” Source: Endsley, M. R. SAGAT: A methodology for the measurement of situation awareness (NOR DOC 87-83). Hawthorne, CA: Northrop Corp.

Here is a theoretical view on situational awareness and the relation between humans and computers:

Cyber Situational Awareness = Continuous Real-Time Analytics

Cyber Situational Awareness is the subset of all situation awareness necessary to support taking action in cyber. It is the mandatory key concept to defend against cybersecurity attacks.

Automation and analytics in real-time are key characteristics:

No matter how good the threat detection algorithms and security platforms are. Prevention or at least detection of attacks need to happen in real-time. And predictions with cutting-edge machine learning models do not help if they are executed in a batch process over might.

Situational awareness covers various levels beyond the raw network events. It includes all environments, including application data, logs, people, and processes.

I covered the challenges in the first post of this blog series. In summary, cybersecurity experts’ key challenge is finding the needle(s) in the haystack. The haystack is typically huge, i.e., massive volumes of data. Often, it is not just one haystack but many. Hence, a key task is to reduce false positives.

Situational awareness is not just about viewing the dashboard but understand what’s going on in real-time. Situational awareness finds the relevant data to create critical (rare) alerts automatically. No human can handle the huge volumes of data.

Situational Awareness in Motion with Kafka

The Kafka ecosystem provides the components to correlate massive volumes of data in real-time to provide situational awareness across the enterprise find all needles in the haystack:

Event streaming powered by the Kafka ecosystem delivers contextually rich data to reduce false positives:

• Collect all events from data sources with Kafka Connect
• Filter event streams with Kafka Connect’s Single Message Transforms (SMT) so that only relevant data gets into the Kafka topic
• Empower real-time streaming applications with Kafka Streams or ksqlDB to correlate events across various source interfaces
• Forward priority events to other systems such as the SIEM/SOAR with Kafka Connect or any other Kafka client (Java, C, C++, .NET, Go, JavaScript, HTTP via REST Proxy, etc.)

Example: Situational Awareness with Kafka and SIEM/SOAR

SIEM/SOAR modernization is its own blog post of this series. But the following picture depicts how Kafka enables true decoupling between applications in a domain-driven design (DDD):

A traditional data store like a data lake is NOT the right spot for implementing situational awareness as it is data at rest. Data at rest is not a bad thing. Several use cases such as reporting (business intelligence), analytics (batch processing), and model training (machine learning) require this approach. Real-time beats slow data. Hence, event streaming with the de facto standard Apache Kafka is the right fit for situational awareness. 

Event streaming and data lake technologies are complementary, not competitive. The blog post “Serverless Kafka in a Cloud-native Data Lake Architecture” explores this discussion in much more detail by looking at AWS’ lake house strategy and its relation to event streaming.

The Data

Situational awareness requires data. A lot of data. Across various interfaces and communication paradigms. A few examples:

• Text Files TXT
• Firewalls and network devices
• Binary files
• Antivirus
• Databases
• Access
• APIs
• Audit logs
• Network flows
• Intrusion detection
• Syslog
• And many more…

Let’s look at the three steps of implementing situational awareness: Data producers, data normalization and enrichment, and data consumers.

Data Producers

Data comes from various sources. This includes real-time systems, batch systems, files, and much more.  The data includes high-volume logs (including Netflow and indirectly PCAP) and low volume transactional events:

Data Normalization and Enrichment

The key success factor to implementing situational awareness is data correlation in real-time at scale. This includes data normalization and processing such as filter, aggregate, transform, etc.:

With Kafka, end-to-end data integration and continuous stream processing are possible with a single scalable and reliable platform. This is very different from the traditional MQ/ETL/ESB approach. Data governance concepts for enforcing data structures and ensuring data quality are crucial on the client-side and server-side. For this reason, the Schema Registry is a mandatory component in most Kafka architectures.

Data Consumers

A single system cannot implement cyber situational awareness. Different questions, challenges, and problems require different tools. Hence, most Kafka deployments run various Kafka consumers using different communication paradigms and different speeds:

Some workloads require data correlation in real-time to detect anomalies or even prevent threats as soon as possible. Kafka-native applications come into play. The client technology is flexible depending on the infrastructure, use case, and developer experience. Java, C, C++, Go are some coding options. Kafka Streams or ksqlDB provide out-of-the-box stream processing capabilities. The latter is the recommended option as it provides many features built-in such as sliding windows to build stateful aggregations.

A SIEM, SOAR, or data lake is complementary to run other analytics use cases for threat detection, intrusion detection, or reporting. The SIEM/SOAR modernization blog post of this series explores this combination in more detail.

Situational Awareness with Kafka and Sigma

Let’s take a look at a concrete example. A few of my colleagues built a great implementation for cyber situational awareness: Confluent Sigma. So, what is it?

Sigma – An Open Signature Format for Cyber Detection

Sigma is a generic and open signature format that allows you to describe relevant log events straightforwardly. The rule format is very flexible, easy to write, and applicable to any log file. The main purpose of this project is to provide a structured form in which cybersecurity engineers can describe their developed detection methods and make them shareable with others – either within the company or even share with the community.

A few characteristics that describe Sigma:

• Open-source framework
• A domain-specific language (DSL)
• Specify patterns in cyber data
• Sigma is for log files what Snort is for network traffic, and YARA is for files

Sigma provides integration with various tools such as ArcSight, QRadar, Splunk, Elasticsearch, Sumologic, and many more. However, as you learned in this post, many scenarios for cyber situational awareness require real-time data correlation at scale. That’s where Kafka comes into play. Having said this, a huge benefit is that you can specify a Sigma signature once and then use all the mentioned tools.

Confluent Sigma

Confluent Sigma is an open-source project implemented by a few of my colleagues. Kudos to Michael Peacock, William LaForest, and a few more. The project integrates Sigma into Kafka by embedding the Sigma rules into stream processing applications powered by Kafka Streams and ksqlDB:

Situational Awareness with Zeek, Kafka Streams, KSQL, and Sigma

Here is a concrete event streaming architecture for situational awareness:

A few notes on the above picture:

• Sigma defines the signature rules
• Zeek provides incoming IDS log events at high volume in real-time
• Confluent Platform processes and correlates the data in real-time
• The stream processing part built with Kafka Streams and ksqlDB includes stateless functions such as filtering and stateful functions such as aggregations
• The calculated detections get ingested into a Zeek dashboard and other Kafka consumers

Here is an example of a Sigma Rule for windowing and aggregation of logs:

The Kafka Streams topology of the example looks like this:

My colleagues will do a webinar to demonstrate Confluent Sigma in more detail, including a live demo. I will update and share the on-demand link here as soon as available. Some demo code is available on Github.

Cisco ThousandEyes Endpoint Agents

Let’s take a look at a concrete Kafka-native example to implement situational awareness at scale in real-time. Cisco ThousandEyes Endpoint Agents is a monitoring tool to gain visibility from any employee to any application over any network. It provides proactive and real-time monitoring of application experience and network connectivity.

The platform leverage the whole Kafka ecosystem for data integration and stream processing:

• Kafka Streams for stateful network tests
• Interactive queries for fetching results
• Kafka Streams for windowed aggregations for alerting use cases
• Kafka Connect for integration with backend systems such as MySQL, Elastic, MongoDB
  

ThousandEyes’ tech blog is a great resource to understand the implementation in more detail.

Kafka is a Key Piece to Implement Cyber Situational Awareness

Cyber situational awareness is mandatory to defend against cybersecurity attacks. A successful implementation requires continuous real-time analytics at scale. This is why the Kafka ecosystem is a perfect fit.

The Confluent Sigma implementation shows how to build a flexible but scalable and reliable architecture for realizing situational awareness. Event streaming is a key piece of the puzzle.

However, it is not a replacement for other tools such as Zeek for network analysis or Splunk as SIEM. Instead, event streaming complements these technologies and provides a central nervous system that connects and truly decouples these other systems. Additionally, the Kafka ecosystem provides the right tools for real-time stream processing.

How did you implement cyber situational awareness? Is it already real-time and scalable? What technologies and architectures do you use? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Kafka for Cybersecurity (Part 2 of 6) – Cyber Situational Awareness appeared first on Kai Waehner.

Confluent Platform 5.4 (with Apache Kafka 2.4)

CP 5.4 (based on AK 2.4) brings some awesome new features to build scalable, reliable and secure event streaming deployments:

• Security: Role-Based Access Control (RBAC) and Structured Audit Logs
• Resilience: Multi-Region Clusters (MRC)
• Data Compatibility: Server-side Schema Validation for Schema Registry
• Management & Monitoring: Control Center enhancements, RBAC management, Replicator monitoring
• Performance & Elasticity: Tiered Storage (preview)
• Stream Processing: New ksqlDB features like Pull Queries and Kafka Connect Integration (preview)

Rapid Pace of Innovation for Mission-Critical Event Streaming

Confluent Platform (CP), the commercial offering from #1 Kafka vendor Confluent, gets more powerful and mature with every release (no surprise). CP releases are tightly coupled to the latest Apache Kafka release to leverage bug fixes and new features from the open source community.

Let’s take a looks at my favorite new features: RBAC, Multi-Region Stretched Clusters and Tiered Storage for Apache Kafka.

Role Based Access Control (RBAC) for Apache Kafka

Kafka provides Access Control Lists (ACL). This is a low level feature and misses higher level, scalable configuration options.

Role-Based Access Control (RBAC) provides platform-wide security with fine-tuned granularity:

• Granular control of access permissions
• Clusters, topics, consumer groups, connectors
• Efficient management at large scale
• Delegate authorization management to true resource owners
• Platform-wide standardization
• Enforced via GUI, CLI and APIs
• Enforced across all CP components: Connect, KSQL, Schema Registry, REST Proxy, Control Center and MQTT Proxy
• Integration with LDAP and Active Directory

Multi-Region Replication (MRC) for Reliable Stretched Deployments

Stretched Clusters are a common deployment option for Kafka. However, this setup is hard to operate and cannot spread over different regions.

Multi-Region Clusters (MRC) change the game for disaster recovery for Kafka:

• Zero downtime and zero data loss for critical Kafka Topics
• Automated client fail-over
• Streamlined DR operations
• Leverages Kafka’s internal replication
• No separate Connect clusters
• Single multi-region cluster with high write throughput
• Asynchronous replication using “Observer” replicas
• Low bandwidth costs and high read throughput
• Remote consumers read data locally, directly from Observers

Check out “Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments” to understand the trade-offs between a “normal stretched cluster” and Confluent’s Multi-Region Clusters (MRC)”.

MRC is really a game-changer for mission-critical Kafka deployments to guarantee zero downtime and zero data loss (also known as RPO=0 and RTO=0).

Tiered Storage for Better Scalability and Long Term Storage

Tiered Storage (preview release) enables Kafka with infinite retention cost-effectively and  elastic scalability / data rebalancing:

• Infinite retention
• Older data is offloaded to inexpensive object storage, accessible at any time
• Reduced storage costs
• Storage limitations, like capacity and duration, are effectively uncapped
• Elastic scalability
• “Lighter” Kafka brokers enable instantaneous load balancing when scaling up

“Streaming Machine Learning with Tiered Storage and Without a Data Lake” shows a concrete example to leverage Tiered Storage for real time analytics and machine learning using TensorFlow.

Confluent Platform 5.4 (Slides + Video)

Here are the slides with an overview about the new features:

Here is a link to the video recording. Please note that this was delivered in German language.

Confluent Cloud – The Fully-Managed Alternative

Oh, and if you prefer a cloud service instead of self-managed Kafka deployments, please check out Confluent Cloud – the only really fully-managed Kafka-as-a-Service offering on the market with consumption-based pricing and mission critical SLAs.

Available on all major cloud providers (AWS, GCP and Azure). Leveraging the latest release of Kafka and providing tons of additional features (like fully managed connectors, Schema Registry and ksqlDB).

The post Confluent Platform 5.4 + Apache Kafka 2.4 (RBAC, Tiered Storage, Multi-Region Clusters, Audit Logs, and more) appeared first on Kai Waehner.

The Evolution of the Telecommunications Industry

The telecommunications industries within the sector of information and communication technology is made up of all telecommunications / telephone companies and internet service providers. It plays the crucial role in the evolution of mobile communications and the information society.

Traditional telephone calls continue to be the industry’s biggest revenue generator in Telecommunications Industry. But thanks to advances in network technology, Telecom today is less about voice and increasingly about text (messaging, email) and images (e.g. video streaming).

High-speed internet access for computer-based data applications such as broadband information services and interactive entertainment, is pervasive. Digital subscriber line (DSL) is the main broadband telecom technology. The fastest growth comes from (value-added) services delivered over mobile networks.

Telco Architecture – OSS, BSS and Cloud

The traditional telco landscape is separated into OSS (Operations Support Systems), BSS (Business Support Systems) and OSS-BSS-Integration.

OSS track network inventory, assets and provisioning of services. BSS deal with customer relationship management (CRM) and processes such as taking orders, processing bills, and collecting payments. Here is an example of such a Telecommunications Industry landscape:

(Overview about OSS / BSS Landscape)

Modern architectures (have to) include one more aspect: Cloud. Like in almost all other traditional industries, telcos move towards cloud. The consequence is a hybrid architecture. Telco have to stay hybrid forever because a core part of their business is infrastructure. A combination of on premises data centers, edge processing and multi cloud architectures is the new normal in Telecommunications Industry.

Telco Business – CDR (Call Detail Record), Customer 360, Media and more

The business in Telecommunications Industry did not change that much from an end user perspective yet. Many basic concepts are still the same. OSS and BSS work together. Call Detail Records (CDR) metadata from phone calls is still processed and analysed to monitor the infrastructure and sell services to customers. Smartphone contracts and media offerings are  still pretty similar like ten years ago (you just get higher data limits per month).

However, the telecom business for the traditional telecommunications industry is changing significantly these days. It has to change. Otherwise, competitors like Apple or Netflix take over most of the business.

I like how this picture from Huawei describes these changes of the digital transformation in the telco industry:

The trend goes towards an open ecosystem, partnering, context-specific personalized offerings, multichannel, improved customer service and new innovative products.

These trends create a lot of challenges for the technical implementation. Traditional technologies which were used in the last 20 years do not work to realize this digital transformation. Requirements include real time processing at scale, decoupled and flexible applications and infrastructure, and still a reliable system with zero down time and zero data loss.

The combination of these requirements makes clear why more and more Telco projects and infrastructure are built with event streaming in mind and leveraging the de facto standard Apache Kafka and its ecosystem.

Use Cases for Apache Kafka in Telco

Apache Kafka is used in various very different telco projects in OSS, BSS, OSS-BSS-Integration. Many different architectures are used, included edge, data center, hybrid and multi cloud. Let’s take a look at some examples for use cases and architectures.

Middleware / Central Event Streaming Platform

• Central Event Streaming Platform as core infrastructure for integration and new applications
• Integration on all levels: OSS, BSS-OSS-Integration, BSS
• OSS – Separate clusters are used in each country / OpCo for integration with syslog etc.
• Integration with legacy and modern systems (including integration to existing legacy middleware)
• Move Data from OSS Fixed, Mobile, Cable into various systems
• Spanning different business units (TV, Analytics, Video Platform, Sales and Services)
• Integration with external partners (today via APIs, future: Kafka-native)
• Merger after acquisitions

Monitoring (Hardware, Performance, Security)

• Real time monitoring of their network devices (routers, switches, other network devices)
• Outside infrastructure: Million miles of fiber and coax, and over 40 million in home devices (severe weather to power grid outages to construction-related disruptions)
• Proactive monitoring: Real time monitoring, problem analysis, metrics reporting and action response
• Nationwide distributed IT infrastructure for all districts -> Fully-managed Security Information and Event Management
• Smarter network planning and deployment, as well as more precise ROI-based investment decisions, taking into account the profitability of each radio site based on customers’ actual and predicted profitability
• SIEM / SOAR: Mission critical cyber defence (typically a combination with other products like Elastic, Splunk, etc.)

Data Distribution

• Event Hub across the whole telco organisation for automation and virtualization of transports to enable data-driven infrastructure decisions
• Abstracted customer layer for all customer facing apps (Mobile App, Web Site, other CRM Systems)
• Next generation media distribution network. Provides alarm correlation to allow identifying network faults and reroute traffic in real time.
• Share information about network performance back to external customers
• API Gateway for network providers
• Accelerate the simplification and automation of standard processes, in both operational and support areas. These include IT and network operations, customer management back office functions and all other administrative activities
• Message archiving service for customers (traders specifically) that want fully searchable archive of all messages (delivered through all channels).

Data Processing

• CDR (Call Detail Record) processing: Technical ETL and business applications (range of scenarios like including upsell, churn analysis)
• Data consolidation issues with many data sources, late arriving events, data out of sync and high SLA guarantees
• Text messaging service to let you send and receive text and multimedia messages seamlessly on your smartphone, tablet, computer or the web using your mobile number; including rate limiting of SMS, MMS and RCS (Rich Communication Services) requests specifically for seasonal spikes; including redirect of RCS via SMS when delivery ability for the RCS message is not present
• Personalized messages and recommendations are built and then delivered to customer inboxes

Business Applications

• Microservices: Infrastructure for agile and decoupled app development
• BSS: Data engine, customer engagement engine, access broker, dynamic business engine, information backbone, back office systems, customer 360, master data management, …
• Customer 360: Track every interaction that the customer has with any channel
• Supply Chain Management (SCM): Real Time Inventory. Support real time information down to the store level
• Billing, Accounts, Lines of Service Data. Relieve Ops Burden
• Make the mobile app and digital marketing channels over time become the main customer acquisition and management platform
• B2B products like Inventory / Asset Tracking, Fleet Management
• Connected Car solutions to retailers for aftersales, including vehicle analytics for drivers, Infotainment, vehicle diagnostics, personal assistance, SOS, remote access, convenience features
• Connected Car solution platform for auto manufacturers to automate vehicle analytics for insurance companies and fleet management, tracking sales; vehicle performance; consumer demographics, etc.

Hybrid Infrastructure and Cloud Migration

• Consume data from on premises and monitor in cloud
• Aggregation cluster: Hub-and-spoke architecture with several regional data centers capturing data to local Kafka clusters and then sending metrics info to the main cluster
• Edge processing (on different levels: OpCo, data center, retail store, …)

Media

• Pass all data through a single source of truth (real time, highly scalable) for decoupling and quick innovation of building new services
• IP-to-MAC: Handle all incoming MAC address registration requests from set-top boxes (to check account entitlements to make sure they are a valid customer)
• Managing content metadata / epg
• Processing pay per view bookings
• Fraud use cases (MAC address cloning, sharing of streaming account details, SMS Spam detection, etc); leveraging machine learning in real time at scale for predictions
• Digital natives (like Netflix) built completely new businesses that were born digital. Everything in a digitally-native business is an event
• Context-specific advertising in real time based on user data (including product upsell, coupons, and other revenue creation with ad partners)

10 Reasons for Event Streaming (i.e. Apache Kafka) in Telco Industry

10 Reasons why telco projects leverage Event Streaming and Apache Kafka:

1. Real Time
2. Scalable
3. Cost Reduction
4. 24/7 – Zero downtime, zero data loss
5. Decoupling – Storage, Domain Driven Design (DDD), Reprocessing of events
6. Data processing and stateful client applications
7. Integration
8. Hybrid Architecture – On Premises, multi cloud, edge computing
9. Fully managed cloud
10. No vendor locking

These characteristics are not just for telco business but relevant in general, of course. However, in the video below, I cover how they map to the telco industry and OSS / BSS use cases.

Obviously, Apache Kafka alone has a lot of strong characteristics. But to realize the digital transformation successfully, you also need expertise (services and support) and tools (development, operations and monitoring) to set up, operate and scale such an infrastructure 24/7 mission-critical.

Slide Deck

The following slide deck covers the evolution of Apache Kafka in the telecom sector, including use cases, architectures and technologies (OSS, BSS, OTT, IMS, NFV, Middleware, Mainframe, etc.):

Live Whiteboard

The following whiteboard explains why and how different companies in the telco industry leverage Event Streaming and Apache Kafka in OSS, BSS, OSS-BSS-Integration and Cloud / Hybrid Architectures. I cover in detail how the above 10 characteristics can used in your next telco project:

Video Recording – Telco Use Cases and Architectures

Here is a webinar where I talk about use cases, architectures and technologies around Apache Kafka in the telecom sector:

What are your experiences with modernizing the infrastructure and applications in the telco industry? Did you or do you plan to use Apache Kafka and its ecosystem? What is your strategy? Let’s connect on LinkedIn and discuss it!

The post Event Streaming and Apache Kafka in Telco Industry appeared first on Kai Waehner.

Abstract of the Keynote Presentation

Business digitalization covers trends like microservices, the Internet of Things or Machine Learning. This is driving the need to process events at a whole new scale, speed and efficiency. Traditional solutions like ETL / data integration or messaging are not build to serve these needs.

Today, the open source project Apache Kafka is being used by thousands of companies including over 60% of the Fortune 100. These companies power and innovate their businesses by focusing their data strategies around event-driven architectures leveraging event streaming.

We will discuss the market and technology changes that have given rise to Kafka and to event-driven architectures. The audience learns the key aspects of building a platform for stream processing with Kafka. Examples of productive use cases from the automotive, manufacturing and transportation sector will showcase the power of event streaming.

Event Streaming Whiteboard – A Fantastic Live Drawing

My session was live-drawn during my presentation by so called “graphic recorders” from remarker. Here is the impressive outcome:

Slide Deck

Here is the slide deck of my presentation. It covers

• the history of Apache Kafka and Event Streaming
• core design principles and architecture of Apache Kafka
• use cases from Lyft, Audi, Deutsche Bahn, Bosch, EON, and more

A specific focus of the presentation was on “stream processing”; a core feature and design concept as part of event streaming platforms. It allows to continuously process massive volumes of data in stateless or stateful applications in real time:

You can checkout my presentation and video recording about KSQL at Big Data Spain 2018 (and many other resources on the web) for more details about stream processing with the Apache Kafka ecosystem.

The post The Rise Of Event Streaming – Why Apache Kafka Changes Everything appeared first on Kai Waehner.

In November 2019, I attended the SPS Conference in Nuremberg. This is one of the most important events about Industrial IoT (IIoT). Vendors and attendees from all over the world fly in to make business and discuss new products. Hotel prices in this region go up from usually 80-100€ to over 300€ per night. Germany is still known for its excellent engineering and manufacturing industry. German companies drive a lot of innovation and standardization around Internet of Things (IoT) and Industry 4.0.

The article discusses:

• The relation between Operational Technology (OT) and Information Technology (IT)
• The advantages and architecture of an open event streaming platform for edge and global IIoT infrastructures
• Examples and use cases for Digital Twin infrastructures leveraging an event streaming platform for storage and processing in real time at scale

SPS – A Trade Show in Germany for Global Industrial IoT Vendors

“SPS covers the entire spectrum of smart and digital automation – from simple sensors to intelligent solutions, from what is feasible today to the vision of a fully digitalized industrial world“. No surprise almost all vendors show software and hardware solutions for innovative use cases like

• Condition monitoring in real time
• Predictive maintenance using artificial intelligence (AI) – I prefer the more realistic term “machine learning”, a subset of AI
• Integration of legacy machines and proprietary protocols in the shop floors
• Robotics
• New digital services
• Other related buzzwords.

Digital Twin as Huge Value Proposition

New software and hardware needs to improve business processes, increase revenue or cut costs. Many booths showed solution for a digital twin as part of this buzzword bingo and value proposition. Most vendors exhibited complete solutions as hardware or software products. Nobody talked about the underlying implementation. Not all vendors could explain in detail how the infrastructure really scales and performs under the hood.

This post starts from a different direction. It begins with definition and use cases of a digital twin infrastructure. The challenges and requirements are discussed in detail. Afterwards, possible architectures and combinations of solutions show the benefits of an open and scalable event streaming platform as part of the puzzle.

The value proposition of a digital twin always discusses the combination of OT and IT:

• OT: Operation Technology; dealing with machines and devices
• IT: Information Technology, dealing with information

Excursus: SPS == PLC —> A Core Component in each IoT Infrastructure

A funny, but relevant marginal note for non-German people: The event name of the trade fair and acronym “SPS” stands for “smart production solutions”. However, in Germany “SPS” actually stands for “SpeicherProgrammierbare Steuerung”. The English translation might be very familiar for you: “Programmable Logic Controller” or shortened “PLC“.

PLC is a core component in any industrial infrastructure. This industrial digital computer has been ruggedized and adapted for the control of manufacturing processes, such as:

• Assembly lines
• Robotic devices
• Any activity that requires high reliability control and ease of programming and process fault diagnosis

PLCs are built to withstand extreme temperatures, strong vibrations, high humidity, and more. Furthermore, since they are not reliant on a PC or network, a PLC will continue to independently function without any connectivity. PLC is OT. This is very different from the IT hardware a software engineers knows from developing and deploying “normal” Java, .NET or Golang applications.

Digital Twin – Merging the Physical and the Digital World

A digital twin is a digital replica of a living or non-living physical entity. By bridging the physical and the virtual world, data is transmitted seamlessly allowing the virtual entity to exist simultaneously with the physical entity. The digital twin therefore interconnects OT and IT.

Digital Replica of Potential and Actual Physical Assets

Digital twin refers to a digital replica of potential and actual physical assets (physical twin), processes, people, places, systems and devices. The digital replica can be used for various purposes. The digital representation provides both the elements and the dynamics of how an IoT device operates and lives throughout its life cycle.

Definitions of digital twin technology used in prior research emphasize two important characteristics. Firstly, each definition emphasizes the connection between the physical model and the corresponding virtual model or virtual counterpart. Secondly, this connection is established by generating real time data using sensors.

Digital Twins Link Internet of Things and Artificial Intelligence

Digital twins connect internet of things, artificial intelligence, machine learning and software analytics to create living digital simulation models. These models update and change as their physical counterparts change. A digital twin continuously learns and updates itself from multiple sources to represent its near real-time status, working condition or position.

This learning system learns from

• itself, using sensor data that conveys various aspects of its operating condition.
• human experts, such as engineers with deep and relevant industry domain knowledge.
• other similar machines
• other similar fleets of machines
• the larger systems and environment in which it may be a part of.

A digital twin also integrates historical data from past machine usage to factor into its digital model.

Use Cases for a Digital Twin in Various Industries

In various industrial sectors, digital twins are used to optimize the operation and maintenance of physical assets, systems and manufacturing processes. They are a formative technology for the IIoT. A digital twin in the workplace is often considered part of Robotic Process Automation (RPA). Per Industry-analyst firm Gartner, a digital twin is part of the broader and emerging hyperautomation category.

Some industries that can leverage digital twins include:

• Manufacturing Industry: Physical manufacturing objects are virtualized and represented as digital twin models seamlessly and closely integrated in both the physical and cyber spaces. Physical objects and twin models interact in a mutually beneficial manner. Therefore, the IT infrastructure also sends control commands back to the actuators of the machines (OT).
• Automotive Industry: Digital twins in the automobile industry use existing data in order to facilitate processes and reduce marginal costs. They can also suggest incorporating new features in the car that can reduce car accidents on the road.
• Healthcare Industry: Lives can be improved in terms of medical health, sports and education by taking a more data-driven approach to healthcare. The biggest benefit of the digital twin on the healthcare industry is the fact that healthcare can be tailored to anticipate on the responses of individual patients.

Examples for Digital Twins in the Industrial IoT

Digital twins are used in various scenarios. For example, they enable the optimization of the maintenance of power generation equipment such as power generation turbines, jet engines and locomotives. Further examples of industry applications are aircraft engines, wind turbines, large structures (e.g. offshore platforms, offshore vessels), heating, ventilation, and air conditioning (HVAC) control systems, locomotives, buildings, utilities (electric, gas, water, waste water networks).

Let’s take a look at some use cases in more detail:

Monitoring, Diagnostics and Prognostics

A digital twin can be used for monitoring, diagnostics and prognostics to optimize asset performance and utilization. In this field, sensory data can be combined with historical data, human expertise and fleet and simulation learning to improve the outcome of prognostics. Therefore, complex prognostics and intelligent maintenance system platforms can use digital twins in finding the root cause of issues and improve productivity.

Digital twins of autonomous vehicles and their sensor suite embedded in a traffic and environment simulation have also been proposed as a means to overcome the significant development, testing and validation challenges for the automotive application. In particular when the related algorithms are based on artificial intelligence approaches that require extensive training data and validation data sets.

3D Modeling for the Creation of Digital Companions

Digital twins are used for 3D modeling to create digital companions for the physical objects. It can be used to view the status of the actual physical object. This provides a way to project physical objects into the digital world. For instance, when sensors collect data from a connected device, the sensor data can be used to update a “digital twin” copy of the device’s state in real time.

The term “device shadow” is also used for the concept of a digital twin. The digital twin is meant to be an up-to-date and accurate copy of the physical object’s properties and states. This includes information such as shape, position, gesture, status and motion.

Embedded Digital Twin

Some manufacturers embed a digital twin into their device. This improves quality, allows earlier fault detection and gives better feedback on product usage to the product designers.

How to Build a Digital Twin Infrastructure?

TL;DR: You need to have the correct information in real time at the right location to be able to analyze the data and act properly. Otherwise you will have conversations like the following:

Challenges and Requirements for Building a Scalable, Reliable Digital Twin

The following challenges have to be solved to implement a successful digital twin infrastructure:

• Connectivity: Different machines and sensors typically do not provide the same interface. Some use a modern standard like MQTT or OPC-UA. Though, many (older) machines use proprietary interfaces. Furthermore, you also need to integrate to the rest of the enterprise.
• Ingestion and Processing: End-to-end pipelines and correlation of all relevant information from all machines, devices, MES, ERP, SCM, and any other related enterprise software in the factories, data center or cloud.
• Real Time: Ingestion of all information in real time (this is a tough term; in this case “real time” typically means milliseconds, sometimes even seconds or minutes are fine).
• Long-term Storage: Storage of the data for reporting, batch analytics, correlations of data from different data sources, and other use cases you did not think at the time when the data was created.
• Security: Trusted data pipelines using authentication, authorization and encryption.
• Scalability: Ingestion and processing of sensor data from one or more shop floors creates a lot of data.
• Decoupling: Sensors produce data continuously. They don’t ask if the consumers are available and if they can keep up with the input. The handling of backpressure and decoupling if producers and consumers is mandatory.
• Multi-Region or Global Deployment: Analysis and correlation of data in one plant is great. But it is even better if you can correlate and analyze the data from all your plants. Maybe even plants deployed all over the world.
• High Availability: Building a pipeline from one shop floor to the cloud for analytics is great. However, even if you just integrate a single plant, the pipeline typically has to run 24/7. In some cases without data loss and with order guarantees of the sensor events.
• Role-Based Access Control (RBAC): Integration of tens or hundreds of machines requires capabilities to manage the relation between the hardware and its digital twin. This includes the technical integration, role-based access control, and more.

The above list covers technical requirements. On top, business services have to be provided. This includes device management, analytics, web UI / mobile app, and other capabilities depending on your use cases.

So, how do you get there? Let’s discuss three alternatives in the following sections.

Solution #1 => IoT / IIoT COTS Product

COTS (commercial off-the-shelf) is software or hardware products that are ready-made and available for sale. IIoT COTS products are built for exactly one problem: Development, Deployment and Operations of IoT use cases.

Many IoT COTS solutions are available on the market. This includes products like Siemens MindSphere, PTC IoT, GE Predix, Hitachi Lumada or Cisco Kinetic for deployments in the data center or in different clouds.

Major cloud providers like AWS, GCP, Azure and Alibaba provide IoT-specific services. Cloud providers are a potential alternative if you are fine with the trade-offs of vendor lock-in. Main advantage: Native integration into the ecosystem of the cloud provider. Main disadvantage: Lock-in into the ecosystem of the cloud provider.

At the SPS trade show, 100+ vendors presented their IoT solution to connect shop floors, ingest data into the data center or cloud, and do analytics. Often, many different tools, products and cloud services are combined to solve a specific problem. This can either be obvious or just under the hood with OEM partners. I already covered the discussion around using one single integration infrastructure versus many different middleware components in much more detail.

Read Gartner’s Magic Quadrant for Industrial IoT Platforms 2019. The analyst report describes the mess under the hood of most IIoT products. Plenty of acquisitions and different code bases are the foundation of many commercial products. Scalability and automated rollouts are key challenges. Download of the report is possible with a paid Gartner account or via free download from any vendor with distribution rights, like PTC.

Proprietary and Vendor-specific Features and Products

Automation industry typically uses proprietary and expensive products. All of them are marketed as flexible cloud products. The truth is that most of them are not really cloud-native. This means they are not scalable and not extendible like you would expect it from a cloud-native service. Some problems I have heard  in the past from end users:

• Scalability is not given. Many products don’t use a scalable architecture. Instead, additional black boxes or monoliths are added if you need to scale. This challenges uptime SLAs, burden of operations and cost.
• Some vendors just deploy their legacy on premise solution into AWS EC2 instances and call it a cloud product. This is far away from being cloud native.
• Even IoT solutions from some global cloud providers do not scale good enough to implement large IoT projects (e.g. to connect to one million cars). I saw several companies which evaluate a cloud-native MQTT solution. They then went to a specific MQTT provider afterwards (but still used the cloud provider for its other great services, like computing resources, analytics services, etc).
• Most complete IoT solutions use many different components and code bases under the hood. Even if you buy one single product, the infrastructure usually uses various technologies and (OEM) vendors. This makes development, testing, roll-out and 24/7 end-to-end operations much harder and more costly.

Long Product Lifecycles with Proprietary Protocols

In automation industry, product lifecycles are very long (tens of years). Simple changes or upgrades are not possible.

IIoT usually uses incompatible protocols. Typically, not just the products, but also these protocols are proprietary. They are just built for hardware and machines of one specific vendor. Siemens S7, Modbus, Allen Bradley, Beckhoff ADS, to name a few of these protocols and “standards”.

OPC-UA is supported more and more. This is a real standard. However, it has all the pros and cons of an open standard. It is often poorly implemented by the vendors and requires an app server on top of the PLC.

In most scenarios I have seen, connectivity to machines and sensors from different vendors is required, too. Even in one single plant, you typically have many different technologies and communication paradigms to integrate.

There is More than just the Machines and PLCs…

No matter how you integrate to the shop floor machines; this is just the first step to get data correlated with other systems from your enterprise. Connectivity to all the machines in the shop floors is not sufficient. Integration and combination of the IoT data with other enterprise applications is crucial. You also want to provide and sell additional services in the data center or cloud. In addition, you might even want to integrate with partners and suppliers. This adds additional value to your product. You can provide features you don’t sell by yourself.

Therefore, an IoT COTS solution does not solve all the challenges. There is huge demand to build an open, flexible, scalable platform. This is the reason why Apache Kafka comes into play in many projects as event streaming platform.

Solution #2 => Apache Kafka as Event Streaming Platform for the Digital Twin Infrastructure

Apache Kafka provides many business and technical characteristics out-of-the-box:

• Cost reduction due to open core principle
• No vendor lock-in
• Flexibility and Extensibility
• Scalability
• Standards-based Integration
• Infrastructure-, vendor- and technology-independent
• Decoupling of applications and machines

Apache Kafka – An Immutable, Distributed Log for Real Time Processing and Long Term Storage

I assume you already know Apache Kafka: The de-facto standard for real-time event streaming. Apache Kafka provides the following characteristics:

• Open-source (Apache 2.0 License)
• Global-scale
• High volume messaging
• Real-time
• Persistent storage for backup and decoupling of sources and sinks
• Connectivity (via Kafka Connect)
• Continuous Stream processing (via Kafka Streams)

This is all included within the Apache Kafka framework. Fully independent of any vendor. Vibrant community with thousands of contributors from many hundreds of companies. Adoption all over the world in any industry.

If you need more details about Apache Kafka, check out the Kafka website, the extensive Confluent documentation, or hundreds of free video recordings and slide decks from all Kafka Summit events to learn about the technology and use cases.

How is Kafka related to Industrial IoT, shop floors and building digital twins? Let’s take a quick look at Kafka’s capabilities for integration and continuous data processing at scale.

Connectivity to Industrial Control Systems (ICS)

Kafka can connect to different functional levels of a manufacturing control operation:

• Integrate directly to Level 1 (PLC / DCS): Kafka Connect or any other Kafka Clients (Java, Python, .NET, Go, JavaScript, etc.) can connect directly to PLCs or an OPC-UA  server. The the data gets directly ingested from the machines and devices. Check out “Apache Kafka, KSQL and Apache PLC4X for IIoT Data Integration and Processing” for an example to integrate with different PLC protocols like Siemens S7 and Modbus in real time at scale.
• Integrate on level 2 (Plant Supervisory) or even above on 3 (Production Control) or 4 (Production Scheduling): For instance, you integrate with MES / ERP / SCM systems via REST / HTTP interfaces or any MQTT-supported plant supervisory or production control infrastructure. The blog post “IoT and Event Streaming at Scale with MQTT” discusses a few different options.
• What about level 0 (Field level)? Today, IT typically only connects to PLCs respectively DCS. IT does not directly integrate to sensors and actuators in the field bus, switch or end node. This will probably change in the future. Sensors get smarter and more powerful. And new standards for the “last mile” of the network emerge, like 10BASE-T1L.

We discussed different integration levels between IT and OT infrastructure. However, connecting from the IT to the PLC and shop floor is just half of the story…

Connectivity and Data Ingestion to MES, ERP, SCM, Big Data, Cloud and the Rest of the Enterprise

Kafka Connect enables reliable and scalable integration of Kafka with any other system. This is important as you need to integrate and correlate sensor data from PLCs with applications and databases from the rest of the enterprise. Kafka Connect provides sources and sinks to

• Databases like Oracle or SAP Hana
• Big Data Analytics like Hadoop, Spark or Google Cloud Machine Learning
• Enterprise Applications like ERP, CRM, SCM
• Any other application using custom connectors or Kafka clients

Continuous Stream Processing and Streaming Analytics in Real Time on the Digital Twin with Kafka

A pipeline from machines to other systems in real time at scale is just part of the full story. You also need to continuously process the data. For instance, you implement streaming ETL, real time predictions with analytic models, or execute any other business logic.

Kafka Streams allows to write standard Java apps and microservices to continuously process your data in real-time with a lightweight stream processing Java API. You could alos use ksqlDB to do stream processing using SQL semantics. Both frameworks use Kafka natively under the hood. Therefore, you leverage all Kafka-native features like high throughput, high availability and zero downtime out-of-the box.

The integration with other streaming solutions like Apache Flink, Spark Streaming, AWS Kinesis or other commercial products like TIBCO StreamBase, IBM Streams or Software AG’s Apama is possible, of course.

Kafka is so great and widely adopted because it decouples all sources and sinks from each other leveraging its distributed messaging and storage infrastructure. Smart endpoints and dumb pipes is a key design principle applied with Kafka automatically to decouple services through a Domain-Driven Design (DDD):

Kafka + Machine Learning / Deep Learning = Real Time Predictions in Industrial IoT

The combination of Kafka and Machine Learning for digital twins is not different from any other industry. However, IoT projects usually generate big data sets throught real time streaming sensor data. Therefore, Kafka + Machine Learning makes even more sense in IoT projects for building digital twins than in many other projects where you lack big data sets.

Analytic models need to be applied at scale. Predictions often need to happen in real time. Data preprocessing, feature engineering and model training also need to happen as fast as possible. Monitoring the whole infrastructure in real time at scale is a key requirement, too. This includes not just the infrastructure for the machines and sensors in the shop floor, but also the overall end-to-end edge-to-cloud infrastructure.

With this in mind, you quickly understand that Machine Learning / Deep Learning  and Apache Kafka are very complementary. I have covered this in detail in many other blog posts and presentations. Get started here for more details:

• Blog Post: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka
• Slide Deck: Apache Kafka + Machine Learning => Intelligent Real Time Applications
• Video Recording: Deep Learning in Mission Critical and Scalable Real Time Applications with Open Source Frameworks

Example for Kafka + ML + IoT: Embedded System at the Edge using an Analytic Model in the Firmware

Let’s discuss a quick example for Kafka + ML + IoT Edge: Embedded systems are very inflexible. It is hard to change code or business rules. The code in the firmware applies between input and out of the hardware. The code is embedded which implements business rules. However,  new code or business rules cannot be simply deployed with a script or continuous delivery like you know it from your favorite Java / .NET / Go / Python application and tools like Maven and Jenkins. Instead, each change requires a new, long development lifecycle including tests, certifications and a manufacturing process.

There is another option Instead of writing and embedding business rules in code with a complex and costly process: Analytic models can be trained on historical data. This can happen anywhere. For instance, you can ingest sensor data into a data lake in the cloud via Kafka. The model is trained  in the elastic and flexible cloud infrastructure. Finally, this model is deployed to an embedded system respectively a new firmware version is created with this model (using the long, expensive process). However, updating (i.e. improving) the model (which is already deployed on the embedded system) gets much easier because no code has to be changed. The mode is “just” re-trained and improved.

This way, business rules can be updated and improved by improving the already deployed model in the embedded system. No new development lifecycle, testing and certification and manufacturing process are required. DNP/AISS1 from SSV Software is one example of a hardware starter kit with pre-installed ML algorithms.

Solution #3 => Kafka + IIoT COTS Product as Complementary Solutions for the Digital Twin

The above sections described how to use either an IoT COTS product or an event streaming platform like Apache Kafka and its ecosystem for building a digital twin infrastructure. Interestingly, Kafka and IoT COTS are actually combined in most deployments I have seen so far.

Most IoT COTS products provide out-of-the-box Kafka connectors because end users are asking for this feature all the time. Let’s discuss the combination of Kafka and other IoT products in more detail.

Kafka is Complementary to Industry Solutions such as Siemens MindSphere or Cisco Kinetic

Kafka can be used in very different scenarios. It is s best for building a scalable, reliable and open infrastructure to integrate IoT at the edge and the rest of the enterprise.

However, Kafka is not the silver bullet for every problem. A specific IoT product might be the better choice for integration and processing of IoT interfaces and shop floors. This depends on the specific requirements, existing ecosystem and already used software. Complexity and cost of solutions need to be evaluated. “Build vs. buy” is always a valid question. Often, the best choice and solution is a mix of building an open, flexible, self-built central streaming infrastructure and buying COTS for specific integration and processing scenarios.

Different Combinations of Kafka and IoT Solutions

The combination of an even streaming platform like Kafka with one or more other IoT products or frameworks is very common:

Sometimes, the shop floor is connecting to an IoT solution. The IoT solution is used as gateway or proxy. This can be a broad, powerful (but also more complex and expensive) solution like Siemens MindSphere. Or the choice is to deploy “just” a specific solution, more lightweight solution to solve one problem. For instance, HiveMQ could be deployed as scalable MQTT cluster to connect to machines and devices. This IoT gateway or proxy connects to Kafka. Kafka is either deployed in the same infrastructure or in another data center or cloud. The Kafka cluster connects to the rest of the enterprise.

In other scenarios, Kafka is used as IoT gateway or proxy to connect to the PLCs or Distributed Control System (DCS) directly. Kafka then connects to an IoT Solution like AWS IoT or Google Cloud’s MQTT Bridge where further processing and analytics happen.

Communication is often bidirectional. No matter what architecture you choose. This means the data is ingested from the shop floor, processed and correlated, and finally control events are sent back to the machines. For instance, in predictive analytics, you first train analytic models with tools like TensorFlow in the cloud. Then you deploy the analytic model at the edge for real time predictions.

Eclipse ditto – An Open Source Framework Dedicated to the Digital Twin; with Kafka Integration

There is not just commercial IoT solutions on the market, of course. Kafka is complementary to COTS IoT solutions and to open source IoT frameworks. Eclipse IoT alone provides various different IoT frameworks. Let’s take a look at one of them, which fits perfectly into this blog post:

Eclipse ditto – an open source framework for building a digital twin. With the decoupled principle of Kafka, it is straightforward to leverage other frameworks for specific tasks.

ditto was created to help realizing digital twins with an open source API. It provides features like like Device-as-a-Service, state management for digital twins, and APIs to organize your set of digital twins. Kafka integration is built-in into ditto out-of-the-box. Therefore, ditto and Kafka complement each other very well to build a digital twin infrastructure.

The World is Hybrid and Polyglot

The world is hybrid and polyglot in terms of technologies and standards. Different machines use different technologies and protocols. Each plant uses its own frameworks, products and standards. Business units use different analytics tools and not always the same clouds provider. And so on…

Global Kafka Architecture for Edge / On Premises / Hybrid / Multi Cloud Deployments of the Digital Twin

Kafka is often not just the backbone to integrate IoT devices and the rest of the enterprise. More and more companies deploy multiple Kafka clusters in different regions, facilities and cloud providers.

The right architecture for Kafka deployments depends on the use cases, SLAs and many other requirements. If you want to build a digital twin architecture, you typically have to think about edge AND data centers / cloud infrastructure:

Apache Kafka as Global Nervous System for Streaming Data

Using Kafka as global nervous system for streaming data typically means you spin up different Kafka clusters. The following scenarios are very common:

• Local edge Kafka clusters in the shop floors: Each factory has its own Kafka cluster to integrate with the machines, sensors and assembly lines. But also with ERP systems, SCADA monitoring tools, and mobile devices from the workers. This is typically a very small Kafka cluster with e.g. three Brokers (which still can process ~100+ MB/sec). Sometimes, one single Kafka broker is deployed. This is fine if you do not need high availability and prefer low cost and very simple operations.
• Central regional Kafka clusters: Kafka clusters are deployed in different regions. Each Kafka cluster is used to ingest, process and aggregate data from different factories in that region or from all cars within a region. These Kafka clusters are bigger than the local Kafka clusters as they need to integrate data from several edge Kafka clusters. The integration can be realized easily and reliable with Confluent Replicator or in the future maybe with MirrorMaker 2 (if it matures over time). Don’t use Mirrormaker 1 at all – you can find many good reasons on the web. Another option is to directly integrate Kafka clients deployed at the edge to a central regional Kafka cluster. Either with a Kafka Client using Java, C, C++, Python, Go or another programming language. Or using a proxy in the middle, like Confluent REST Proxy, Confluent MQTT Proxy, or any MQTT Broker outside the Kafka environment. Find out more details about comparing different MQTT and HTTP-based IoT integration options for Kafka here.
• Multi-region or global Kafka clusters: You can deploy one Kafka cluster in each region or continent. Then replicate the data between each other (one- or bidirectional) in real time using Confluent Replicator. Or you can leverage the multi-data center replication Kafka feature from Confluent Platform to spin up one logical cluster over different regions. The latter provides automatic fail-over, zero data loss and much easier operations of server and client side.

This is just a quick summary of deployment options for Kafka clusters at the edge, on premises or in the cloud. You typically combine different options to deploy a hybrid and global Kafka infrastructure. Often, you start small with one pipeline and a single Kafka cluster. Scaling up and rolling out the global expansion should be included into the planning from the beginning. I will speak in more detail about different “architecture patterns and best practices for distributed, hybrid and global Apache Kafka deployments” at DevNexus in Atlanta in February 2020. This is a good topic for another blog post in 2020.

Polyglot Infrastructure – There is no AWS, GCP or Azure Cloud in China and Russia!

For global deployments, you need to choose the right cloud providers or build your own data centers in some different regions.

For example, you might leverage Confluent Cloud. This is a fully managed Kafka service with usage-based pricing and enterprise-ready SLAs in Europe and the US on Azure, AWS or GCP. Confluent Cloud is a real serverless approach. No need to think about Kafka Brokers, operations, scalability, rebalancing, security, fine tuning, upgrades.

US cloud providers do not provide cloud services in China. Alibaba is the leading cloud provider. Kafka can be deployed on Alibaba cloud. Or choose a generic cloud-native infrastructure like Kubernetes. Confluent Operator, a Kubernetes Operator including CRD and Helm charts, is a tool to support and automate provisioning and operations on any Kubernetes infrastructure.

No public cloud is available in Russia at all. The reasons is mainly legal restrictions. Kafka has to be deployed on premises.

In some scenarios, the data from different Kafka clusters in different regions is replicated and aggregated. Some anonymous sensor data from all continents can be correlated to find new insights. But some specific user data might always just stay in the country of origin and local region.

Standardized Infrastructure Templates and Automation

Many companies build one general infrastructure template on a specific abstraction level. This template can then be deployed to different data centers and cloud providers the same way. This standardizes and eases operations in global Kafka deployments.

Cloud-Native Kubernetes for Robust IoT and Self-Healing, Scalable Kafka Cluster

Today, Kubernetes is often choosen as the abstraction layer. Kubernetes is deployed and managed by the cloud provider (e.g. GKE on GCP) or an operations team on premises. All required infrastructure on top of Kubernetes is scripted and automated with a template framework (e.g. Terraform). This can then be rolled out to different regions and infrastructures in the same standardized and automated way.

The same is applicable for the Kafka infrastructure on top of Kubernetes. Either you leverage existing tools like Confluent Operator or build your own scripts and custom resource definitions. The Kafka Operator for Kubernetes has several features built-in, like automated handling of fail-over, rolling upgrades and security configuration.

Find the right abstraction level for your Digital Twin infrastructure:

You can also use tools like the open source Kafka Ansible scripts to deploy and operate the Kafka ecosystem (including components like Schema Registry or Kafka Connect), of course.

However, the beauty of a cloud-native infrastructure like Kubernetes is its self-healing and robust characteristics. Failure is expected. This means that your infrastructure continues to run without downtime or data loss in case of node or network failures.

This is quite important if you deploy a digital twin infrastructure and roll it out to different regions, countries and business units. Many failures are handled automatically (in terms of continuous operations without downtime or data loss).

Not every failure requires a P1 and call to the support hotline. The system continues to run while an ops team can replace defect infrastructure in the background without time pressure. This is exactly what you need to deploy robust IoT solutions to production at scale.

Apache Kafka as the Digital Twin for 100000 Connected Cars

Let’s conclude with a specific example to build a Digital Twin infrastructure with the Kafka ecosystem. We use an implementation from the automotive industry. But this is applicable to any scenario where you want to build and leverage digital twins. Read more about “Use Cases for Apache Kafka in Automotive Industry” here.

Honestly, this demo was not built with the idea of creating a digital twin infrastructure in mind. However, think about it (and take a look at some definitions, architectures and solutions on the web): Digital Twin is just a concept and software design pattern. Remember our definition from the beginning of the article: A digital twin is a digital replica of a living or non-living physical entity. Therefore, the decision of the right architecture and technology depends on the use case(s).

We built a demo which shows how you can integrate with tens or hundreds of thousands IoT devices and process the data in real time. The demo use case is predictive maintenance (i.e. anomaly detection) in a connected car infrastructure to predict motor engine failures: “Building a Digital Twin for Streaming Machine Learning at Scale from 100000 IoT Devices with HiveMQ, Apache Kafka and TensorFlow“.

In this example, the data from 100000 cars is ingested and stored in the Kafka cluster, i.e. the digital twin, for further processing and real time analytics.

Kafka client applications consume the data for different use cases and in different speed:

1. Real time data pre-processing and data engineering using the data from the digital twin with Kafka Streams and KSQL / ksqlDB.
2. Streaming model training (i.e. without a data lake in the middle) with the Maschine Learning / Deep Learning framework TensorFlow and its Kafka plugin (part of TensorFlow I/O). In our example, we train two neural networks: An unsupervised Autoencoder for anomaly detection and a supervised LSTM.
3. Model deployment for inference in real time on new car sensor events to predict potential failures in the motor engine.
4. Ingestion of the data into another batch system, database or data lake (Oracle, HDFS, Elastic, AWS S3, Google Cloud Storage, whatever).
5. Another consumer could be a real time time series database like InfluxDB or TimescaleDB.

Build your own Digital Twin Infrastructure with Kafka and its Open Ecosystem!

I hope this post gave you some insights and ideas. I described three options to build the infrastructure for a digital twin: 1) IoT COTS, 2) Kafka, 3) Kafka + Iot COTS. Many companies leverage Apache Kafka as central nervous system for their IoT infrastructure in one way or the other.

Digital Twin is just one of many possible IoT use cases for an event streaming platform. Often, Kafka is “just” part of the solution. Pick and choose the right tools for your use cases. Evaluate the Kafka ecosystem and different IoT frameworks / solutions to find the best combination for your project. Don’t forget to include the vision and long-term planning into your decisions! If you plan it right from the beginning, it is (relative) straightforward to start with a pilot or MVP. Then roll it out to the first plant into production. Over time, deploy a global Digital Twin infrastructure…

The post Apache Kafka as Digital Twin for Open, Scalable, Reliable Industrial IoT (IIoT) appeared first on Kai Waehner.

I am really excited about the role of Apache Kafka and its ecosystem in the automotive industry. Kafka became the central nervous system of many applications in various different areas related to automotive industry. Machine Learning also got more and more impact on these use cases.

A Long Journey – From Car Production and Sales to Digital Services

My trip to Detroit started with some private events. First, I watched the amazing football college game of the Michigan Wolverines hosting the Michigan State Spartans in front of 111000 fans in Ann Arbor. The next day I went to the Ford Piquette Plant in Detroit where Henry Ford and his team manufactured the first cars. The Henry Ford Museum was another great visit to learn more about the mastermind Henry Ford and the history of car manufacturing in the US.

The Car Industry is Changing…

I talked to many local people. The main excitement was around three topics in the state of Michigan: Universities, sports and cars. I have to admit that talking about car companies created a much more frustrating and negative mood for many people. They work in the factories and offices of car companies. They were more happy about discussions around sports and universities. Many people are worried about the future of the automotive industry.

Thus, let’s think in more detail about the car business… A car company produces and sells cars. A supplier produces parts for the car. A vendor shop or independent company provides maintenance and repair services.

That was the situation for many decades. Welcome to a new era where car companies have to do much more than just manufacturing and selling cars to stay competitive.

I think people should not be worried, though. This is very exciting. While many old jobs will be done my machines and robots in the future, new tasks and job roles will be created for everybody.

Customer Behaviour has Changed…

Customer behaviour has changed significantly:

• Customers expect digital services and integration with their own IT ecosystem (smartphone, music streaming service, smart home, and many other apps)
• Customers look for cheap repair services away from the car vendors because it is much cheaper but still a good service
• Repair shops use 3rd party suppliers for car parts because the original supplier  is too expensive and does not add any value or better product quality
• Tesla, Uber, Volkswagen, Daimler and many other automotive and non-automotive companies work on autonomous driving (in several steps and releases: Automatic distance measurement  on the highway —> Automatic parking in the parking lot —> Driving back to the entrance of the supermarket when you call your call via your smartphone —> Real autonomous driving to the final destination)
• Zipcar, car2go, drivenow and other companies provide car sharing services where car usage is paid by minutes and miles: You use your smartphone to locate the next car at the airport, drive to your home in the city, and leave the car in front of your house so that the next customer can pick it up
• Uber, Lyft, Grab, Free Now (the most ridiculous company name you could have for ride sharing) and other services provide cheap ride sharing with fantastic user experience: You order a taxi via mobile app, select guest services (like choice of music, wish of conversations with the driver or not), see expected estimated time of arrival and expected cost, monitor the status of the taxi location in real time, get transported, pay automatically by the integrated payment service, collect points at the integrated and partnered loyalty systems (like Hilton in case of Lyft or MilesAndMore like Free Now to name a few examples)

In short, companies in the automotive industry have to change their business model fundamentally to stay competitive. Otherwise, they will become the next “hardware supplier” for a tech giant. This would reduce profit, establish tough dependencies and create market pressure. Even the existence of the company is at risk. No matter if you are on the market for 10 or 100 years, already.

Digital Use Cases in Automotive Industry

The background about the history and new requirements for the automotive industry is important to understand. Every car-related company can add more value and innovation to the own business model.

The use cases can be separated in infrastructure, manufacturing and customer interaction (while there are some overlaps, of course).

Infrastructure for Connected Cars

A fully connected infrastructure with real time communication is required to build innovative use cases such as:

• Connected cars: The “Hello World” example for discussing cutting edge use cases in automotive industry. It includes more or less all challenges you have to solve in Internet of Things (IoT) scenarios, including connectivity to millions of devices, large scale throughput, reliable communication with bad and low network connectivity, real time processing requirements. Ingest and process sensor information from millions of cars in real time to correlate events and. Provide bi-directional communication between cars and other applications. Do big data analytics in the backend to get new insights. Build additional services to the manufacturer, customer and partners.
• Fleet Management: A specific example of real time correlation of events from various different systems like mobile apps, hardware integrated into cars and trucks, and backend systems like CRMs and payment systems. Various examples exist in different industries, including logistics of truck fleets, track&trace of package delivery, ridesharing, food services, and many more.
• Emergency system: If your car shows abnormal behaviour  or crashed into a tree, the car automatically sends a crash notification (including details about the crash scenario) to the police and hospital next to you to send help immediately.
• Smart City and Smart Driving: Connect and correlate data of cars with other devices like traffic lights. Automotive companies partner with cities and other data providers to offer better security and more comfortable driving experience. Collect and share basic safety messages. Monitor cars and send speed / slow-down warnings, wrong-way detection, congestion. Provide crowdsourced data with applications such as Waze, Google Maps, Twitter, Nextdoor.

Manufacturing and Industrial IoT (IIoT)

Producing cars is a very complex task. This includes the production of the car itself (the “hardware”) and the intelligent logic (the “software”) of the car. Both integrate deeply with each other to implement use cases such as:

• Industrial IoT (IIoT) and smart manufacturing: Predictive maintenance, early part scrapping and improved product quality are some examples to improve the quality of the manufactured parts, reduce costs and risks. The analysis of car sensor data also allows to implement additional digital services on top of the plain car parts.
• Autonomous Driving: One of the most interesting use cases. Real time processing of big data sets and using cutting edge technologies such as Neural Networks allows car companies to build car with autonomous driving features. This is still in early stage and will probably take many years before it is available in Europe. But some US states and China are pressing ahead with self-driving cars on public streets, already. This will take some more years in most places. In the meantime, you can already use less powerful but still impressive features like Tesla’s “summon car” feature. This is not working perfectly today, but this is just a matter of time.

Customer Interactions and Data Analytics

Connected infrastructures and self-driving cars are awesome. Another added value comes through the possibility to communicate directly with the drivers to provide additional services, for instance:

• Remote Diagnostics: Analytics can happen at the edge (e.g. in the car)  or in a data center / cloud. Autonomous driving needs to detect persons in real time. Thus, this needs to be applied in the car. Cross selling is an example where you correlate data between different backend systems and remote users or apps. Therefore, this should happen in the backend. You just send the result of the correlation back to the car or mobile app of the driver. Predictive maintenance can happen directly in the car or machine. Or you correlate the events in the backend and just send the alerts to the car or machine to stop it before the engine breaks. No matter where you deploy the analytics logic, it typically has to happen in real time. Usually within milliseconds or seconds, but sometimes minutes or even hours are sufficient.
• Identity Management: The days are over where you use a key (a piece of hardware) to open your car and start your engine. Car sharing services provide a mobile app. This is used for registration, payments and handing over the key (for the time of usage). Security (Authentication, authorization and encryption of data) are getting more and more important in IoT. This is even true in Industrial IoT where security discussions did not exist at all in the past. Car sharing is a great example where this is a key requirement to deploy an infrastructure and service successfully. The next step several companies are working on (long live China!) is image recognition instead of using a physical key, or smartphone to open your car and start the engine. Other use cases include improving the customer experience in the car. For instance, default configuration of seat, radio and air conditioning can be based on the driver’s historical behaviour (stored forever in your favorite database) and the outside ecosystem (e.g. integration with a weather service to know how warm it is). You will probably also not need to carry your driver’s license with you in the future because police will also just scan your face. I was pretty impressed when I just had to scan my face at Global Entry entering the US border coming from Germany last week, instead of scanning my passport and entering all the annoying details about my travel and private information.
• Aftersales: Car companies and manufacturers need to sell additional services to make more money and / or to make customers happy and loyal. This includes use cases like upselling (provide 100 extra horsepower via digital download for 24 hours to have some fun on German Autobahn) and cross-selling (provide a 20% coupon for the partnered steak house coming at the next stop 10 miles always from your current driving location).
• Payment Integration: In the US, it is common to pay everything by credit card. In Europe, many shops still just accept cash. In the future, payments will be done automatically. You just go to the gas station or electric charging unit and refuel respectively load your car and leave. Similar to existing an Uber or Lyft car today. The payment is done via the integrated partner payment system of your choice. Once again, loyalty systems are integrated, too. Authentication and authorization are done via cameras and neural networks doing image recognition.
• Data Monetization: Connected car infrastructures produce a lot of data. Depending on law, privacy options and other regulations, automotive companies can and will do its own analytics, but also sell the data or parts of the data to partners (hopefully anonymized properly). Like Google or Facebook today, the automotive industry will be able to make a fortune with the data of the cars and all the connected partner systems. Most customers will probably agree to this “feature” because they get added value out of this, too (like discounts or “free” additional services).

Wow, this is a lot of use cases for digitalization of the automotive business, isn’t it? And there are many more to come…

Challenges and Requirements to Build Automotive Infrastructure at Scale

Let’s now quickly discuss the challenges and requirements to solve with all those exciting use cases:

• Integration between many different consumer apps and backend systems (including many legacy and proprietary interfaces, and different communication paradigms like real time streaming, request-response, batch processing)
• Big Data – sensors and millions of mobile apps from users produce a large set of (structured and unstructured) data continuously
• Hybrid integration and deployments at the edge (e.g. cars), local proxies in different regions, backend applications in data centers or cloud, and SaaS applications
• Global scale with zero downtime, rolling out the applications on different infrastructures (there is no AWS / Azure / GCP) in China but only Alibaba, and there is no cloud at all in Russia (just private clouds on premises).
• Real time information is required in most scenarios to provide a good digital experience and added business value

This is a lot of challenges to solve. If you know Apache Kafka already well (and maybe also other traditional middleware), then you can probably guess why Kafka is such a good fit here.

Apache Kafka in the Automotive Industry

After we understood the various innovative use cases and its related challenges in the automotive industry, let’s now discuss (quickly) why so many automotive companies and suppliers use Apache Kafka as central nervous system for the discussed use cases.

Kafka for Real Time Streaming at Scale

Car sensors produce data continuously. The more cars you connect, the more data you get. Kafka provides messaging and streaming at scale, providing a reliable, highly scalable infrastructure with zero downtime. Rolling upgrades, backwards compatibility, up- and downscale at runtime, and other features are built-in.

Kafka for Handling Backpressure and Decoupling of Services

Cars are decoupled from the backed systems and mobile apps. Kafka provides storage for backpressure and decoupling of services. It is not just a messaging system! Kafka also stores data as long as you want. For instance, a Kafka topic is stored for a few days for log analytics. Another Kafka topic is stored for years to analyze customer and payment transactions of the past. Order guarantees, timestamps and high availability are provided out of-the-box. Connected applications and data stores can build a materialized view to leverage the data.

Kafka for Integration with Legacy and Modern Technologies

Car data needs to be integrated. Kafka is the backbone for backend systems (legacy like ERP, Mainframe + modern SaaS like Cloud CRM, big data analytics). Connectivity via Kafka Connect provides Kafka-native integration at scale to any legacy or modern technology and communication paradigm. Either directly from Kafka clients or via other technologies as proxy like MQTT to cars, Websockets to mobile apps, via proprietary protocols like Siemens S7 or open standards like OPC-UA to machines in plants). Check out my comparison between Kafka and Middleware (MQ, ETL, ESB) for more details.

Kafka for Continuous Stream Processing with Kafka Streams and ksqlDB

Car data needs to be processed and correlated in real time at scale with other backend databases and applications. Kafka Streams or ksqlDB provide Kafka-native capabilities to do process data continuously, for use cases like Streaming ETL, stateful aggregations (sliding windows) and building business applications with their own state (leveraging rocksdb under the hood – no need for another database.

Machine Learning and Kafka in Automotive Use Cases

Kafka is used more and more in Machine Learning infrastructures. Some examples from tech giants are Uber, Netflix or Paypal. The big challenge about Machine Learning is the deploy at scale in a reliable way (for both model training and predictions). I covered this topic in details in various blog posts, videos and demos. Check out this blog post to get started: “How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka“.

Kafka for Data Engineering, Model Training + Deployment and Monitoring

In the Automotive Industry, Machine Learning needs to be applied at scale. Predictions often need to happen in real time. Therefore, more and more automotive-related use cases discussed in the above section leverage Kafka together with Machine Learning for different aspects like:

• Data ingestion and processing of data with Kafka from cars and other systems for both model training of historical data and real time predictions on new events – ideally using the same ingestion and pre-processing pipeline for both.
• Streaming model training with Kafka without an additional database in the middle (e.g. leveraging TensorFlow I/O and its native Kafka integration) or the storage in a data lake (e.g. HDFS, AWS S3) for model training.
• Model deployment with Kafka at the edge (e.g. in autonomous driving for stopping the car via image recognition because a person is on the street in front of the car) or in the data center / cloud (e.g. predictions for cross selling by correlation of real time behaviour together with correlated information from the loyalty and CRM systems in backend). Check out the video and slides discussing “Event-driven Model Serving: Stream Processing vs. RPC” from Kafka Summit for more details.
• Monitoring with Kafka (e.g. real time alerting, or distributed tracing to detect anomalies and the root cause of problems).

One huge advantage of using Kafka for Machine Learning infrastructures is solving the impedance mismatch between the data scientist (loving rapid prototyping with Python and Jyipter) and the software developer / production engineer (loving scalable and reliable Java applications).

Kafka Architecture – From Edge Deployments to Global Replication

The architecture for Kafka deployments depends on the use cases, SLAs and many other requirements. Automotive companies and suppliers typically have to think globally:

Using Kafka as global nervous system for streaming data typically means you spin up different Kafka clusters. The following scenarios are very common in the automotive industry:

• Local edge Kafka clusters in the factories: Each factory has its own Kafka cluster to integrate with the machines, sensors and assembly lines. But also with ERP systems, SCADA monitoring tools, and mobile devices from the workers. This is typically a very small Kafka cluster with e.g. three Brokers (which still can process ~100+ MB/sec). Sometimes, just one single Kafka broker is deployed. This is fine if you do not need high availability and prefer low cost and very simple operations.
• Central regional Kafka clusters: Kafka clusters are deployed in different regions. Each Kafka cluster is used to ingest, process and aggregate data from different factories in that region or from all cars within a region. These Kafka clusters are bigger than the local Kafka clusters as they need to integrate data from several edge Kafka clusters. The integration can be realized easily and reliable with Confluent Replicator or in the future maybe with MirrorMaker 2. Don’t use Mirrormaker 1 at all – you can find many good reasons on the web). Another option is to directly integrate Kafka clients deployed at the edge to a central regional Kafka cluster. Either with a Kafka Client using Java, C, C++, Python, Go or another programming language. Or using a proxy in the middle, like Confluent REST Proxy, Confluent MQTT Proxy, or any MQTT Broker outside the Kafka environment. Find out more details about comparing different MQTT and HTTP-based IoT integration options for Kafka here.
• Global Kafka clusters: You can deploy one Kafka cluster in each region or continent. Then replicate the data between each other (one- or bidirectional) in real time using Confluent Replicator. Or you can leverage the multi-data center replication Kafka feature from Confluent to spin up one logical cluster over different regions.

This is just a quick summary of deployment options for Kafka clusters at the edge, on premises or in the cloud. You typically combine different options to deploy a hybrid and global Kafka infrastructure. I will speak about different “architecture patterns for distributed, hybrid and global Apache Kafka deployments” at DevNexus in Atlanta in February 2020.

For example, you might leverage Confluent Cloud, a fully managed Kafka service with usage-based pricing and enterprise-ready SLAs in Europe and the US on Azure, AWS or GCP. But you need to use Alibaba cloud in China and used self-managed Kafka there deployed to Kubernetes and operated by Confluent Operator. In Russia, no public cloud is available at all. You need to deploy on premises and manage the cluster by yourself. Then you replicate and aggregate some anonymous sensor data from all continents to get new aggregated insights. But some specific user data might always just stay in the country of origin and local region.

Live Demo – 100000 Connected Cars

We built a live demo (available on Github) to show how easy it is to set up a connected car infrastructure at scale (including a cutting edge streaming machine learning example for predictive maintenance in real time): Streaming Machine Learning at Scale from 100000 IoT Devices with HiveMQ, Apache Kafka and TensorfLow:

Such a setup can be the foundation for most of the discussed use cases in this blog post.

You can also take a look at the following link to see the slide deck and video recording. They discuss the demo architecture and walking you through the live demo:

Please let me know your thoughts about Apache Kafka in the automotive industry. Try out the demo and share your feedback. Let’s build some exciting new and innovative use cases for the automotive industry.

The post Apache Kafka in the Automotive Industry appeared first on Kai Waehner.