Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases across all industries.

What is a Virtual Power Plant?

A Virtual Power Plant (VPP) is a network of decentralized energy resources—such as home batteries, solar panels, and smart grid systems—that function as a single unit. Unlike a traditional power plant that generates electricity from a centralized location, a VPP aggregates power from many small, distributed sources. This allows energy to be dynamically stored and shared, helping to balance supply and demand in real time.

VPPs are crucial in the shift to renewable energy. The traditional power grid was designed around fossil fuel plants that could easily adjust output. Renewable energy sources like solar and wind are intermittent—they don’t generate power on demand. By connecting thousands of batteries and solar panels in homes and businesses, a VPP can smooth out fluctuations in power generation and consumption. This prevents blackouts, reduces energy costs, and enables homes and businesses to participate in energy markets.

How Tesla’s Virtual Power Plant Fits Its Business Model

Tesla is not just an automaker. It is a sustainable energy company. Tesla’s product ecosystem includes electric vehicles, solar panels, home batteries (Powerwall), grid-scale energy storage (Megapack), and energy management software (Autobidder).

The Tesla Virtual Power Plant (VPP) ties all these elements together. Homeowners with Powerwalls store excess solar power during the day and feed it back to the grid when needed. Tesla’s Autobidder software automatically optimizes energy use and market participation, turning home batteries into revenue-generating assets.

For Tesla, the VPP strengthens its energy business, creating a scalable model that maximizes battery efficiency, stabilizes grids, and expands the role of software in energy markets. Tesla is not just selling batteries; it is selling energy intelligence.

Virtual Energy Platform and ESG (Environmental, Social, and Governance) Goals

Tesla’s energy platform is a perfect example of how data streaming and real-time decision-making align with ESG principles:

• Environmental Impact: VPPs reduce reliance on fossil fuels by making renewable energy more reliable.
• Social Benefit: By enabling energy independence, VPPs provide power during outages and extreme weather conditions.
• Governance & Regulation: VPPs allow consumers to participate in energy markets, fostering decentralized energy ownership.

Tesla’s approach is smart grid innovation at scale—real-time data makes the grid more dynamic, efficient, and resilient.

My article “Green Data, Clean Insights: How Apache Kafka and Flink Power ESG Transformations” covers other real-world data streaming deployments in the energy sector like EON.

Tesla’s Energy Platform: A Network of Connected Home Energy Systems

Tesla’s VPP connects thousands of homes with Powerwalls, solar panels, and grid services. These systems work together to provide electricity on demand, reacting to supply fluctuations in real-time.

Key Functions of Tesla’s VPP:

1. Energy Storage & Redistribution: Batteries store solar energy during the day and discharge at night or during peak demand.
2. Grid Stabilization: The VPP balances energy supply and demand to prevent outages and fluctuations.
3. Market Participation: Homeowners can sell excess power back to the grid, monetizing their batteries.
4. Disaster Resilience: The VPP provides backup power during blackouts, storms, and grid failures.

This requires real-time data processing at massive scale—something traditional batch-based data architectures cannot handle.

Apache Kafka and Real-Time Data Streaming at Tesla

Tesla operates in many domains—automotive, energy, and AI. Across all these areas, Apache Kafka plays a critical role in enabling real-time data movement and stream processing.

In 2018, Tesla already processed trillions of IoT messages with Apache Kafka:

Tesla leverages stream processing to handle trillions of IoT events daily, using Apache Kafka to ingest, process, and analyze data from its vehicle fleet in real time. By implementing efficient data partitioning, fast and slow data lanes, and scalable infrastructure, Tesla optimizes vehicle performance, predicts failures, and enhances manufacturing efficiency.

These strategies demonstrate how real-time data streaming is essential for managing large-scale IoT ecosystems, ensuring low-latency insights while maintaining operational stability. To learn more about these use cases read Tesla’s blog post “Stream Processing with IoT Data: Challenges, Best Practices, and Techniques“.

The following sections explore Tesla’s innovation for its virtual power plant, as discussed in an excellent presentation at QCon.

Tesla Energy Platform: Architecture of the Virtual Power Plant Powered by Apache Kafka

Tesla’s VPP uses Apache Kafka for:

1. Telemetry Ingestion: Streaming data from millions of Powerwalls, solar panels, and Megapacks into the cloud.
2. Command & Control: Sending real-time control commands to batteries and grid services.
3. Market Participation: Autobidder analyzes real-time data and adjusts energy prices dynamically.

The event-driven architecture allows Tesla to react to energy demand in milliseconds—critical for balancing the grid.

Tesla’s Energy Platform is the software foundation of the VPP. It integrates OT (Operational Technology), IoT (Internet of Things), and IT (Information Technology) to control distributed energy assets.

Tesla Applications Built on the Energy Platform

Tesla’s Energy Platform powers a suite of applications that optimize energy management, market participation, and grid stability through real-time data streaming and automation.

Autobidder

• Optimizes energy trading in real time.
• Automatically bids into energy markets.

I wrote about about other data streaming success stories for energy trading with Apache Kafka and Flink, including Uniper, re.alto and Powerledger.

Distributed Virtual Power Plant

• Aggregates thousands of Powerwalls into a single energy asset.
• Provides grid stabilization and peak load balancing.

If you are interested in other smart grid infrastructures, check out “Apache Kafka for Smart Grid, Utilities and Energy Production“. The articles covers how data streaming realizes IT/OT integration. And some hybrid cloud IoT deployments.

Battery Control (Command & Control)

• Ensures optimal charging and discharging of batteries.
• Minimizes costs while maximizing energy efficiency.

Market Participation

• Allows homeowners and businesses to profit from energy markets.
• Ensures seamless grid integration of Tesla’s energy products.

Key Components of Tesla’s Energy Platform: Apache Kafka, WebSockets, Akka Streams

The combination of data streaming with Apache Kafka and the last-mile IoT integration via WebSockets builds the central nervous system of Tesla’s Energy Platform:

1. Apache Kafka (Event Streaming):
   • Streams telemetry data from Powerwalls every second.
   • Ensures durability and reliability of data streams.
   • Allows real-time energy aggregation across thousands of homes.
2. WebSockets (Last-Mile IoT Integration):
   • Provides low-latency bidirectional communication with Powerwalls.
   • Used to send real-time commands to home batteries.
3. Pub/Sub (Command & Control):
   • Enables distributed energy resource coordination.
   • Ensures resilient messaging between systems.
4. Business Logic (Applications & Microservices):
   • Tesla’s services are built with Scala and Python.
   • Uses gRPC & HTTP for inter-service communication.
5. Digital Twins (Real-Time State Management):
   • Digital models of physical assets ensure real-time decision-making.
   • Tesla uses Akka Streams for stateful event processing.
6. Kubernetes (Cloud Infrastructure):
   • Ensures scalability and resilience of Tesla’s energy microservices.

Interesting side note: While most energy companies I have seen rely on Kafka Streams or Apache Flink for stateful event processing, Tesla takes an interesting approach by leveraging Akka Streams (based on Akka’s Actor Model) to manage real-time digital twins of its energy assets. This choice provides fine-grained control over streaming workflows, but unlike Kafka Streams or Flink, Akka lacks widespread community adoption, making it a less common choice for many large-scale energy platforms. Kafka and Flink are a match made in heaven for most data streaming use cases.

Best Practice: Shift Left Architecture with Data Products for High-Volume IoT Data

Tesla leverages several data processing best practices to improve efficiency and consistency:

• Canonical Kafka Topics: Data is filtered and structured at the source.
• Consistent Downstream Services: Every consumer gets clean, structured data.
• Real-Time Aggregation of Thousands of Batteries: A unique challenge that forms the foundation of the virtual power plant.

This data-first approach ensures Tesla’s energy platform can scale to millions of distributed assets.

Today, many people refer to the Shift Left Architecture when applying these best practices for processing data efficiently and continuously to provide data product in real-time and good quality:

In Tesla’s Energy Platform, the data comes from IoT interfaces. WebSockets provide the last-mile integration and feed the events into the data streaming platform for continuous processing before the ingestion into the operational and analytical applications.

Tesla’s Energy Vision: How Streaming Data Will Shape Tomorrow’s Power Grids

Tesla’s Virtual Power Plant is not just about batteries—it’s about software, real-time data, and automation.

Why Data Streaming Matters for Tesla’s Energy Platform:

1. Scalability: Can handle millions of energy devices.
2. Resilience: Works even when devices go offline.
3. Real-Time Decision Making: Adjusts energy distribution within milliseconds.
4. Market Optimization: Autobidder ensures maximum revenue for battery owners.

Tesla’s VPP is a blueprint for the future of energy—one where real-time data streaming and intelligent software optimize renewable energy. By leveraging Apache Kafka, WebSockets, and stream processing, Tesla is redefining how energy is generated, distributed, and consumed.

This is not just an innovation in power generation—it’s an AI-driven energy revolution.

How do you leverage data streaming in the energy and automotive sector? Follow me on LinkedIn or X (former Twitter) to stay in touch and discuss. Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter. And make sure to download my free book about data streaming use cases across all industries.

The post Tesla Energy Platform – The Power of Data Streaming with Apache Kafka appeared first on Kai Waehner.

Industrial Automation – The OT/IT Bridge

An OT/IT Bridge in industrial automation refers to the integration between Operational Technology (OT) systems, which manage real-time industrial processes, and Information Technology (IT) systems, which handle data, business operations, and analytics. This bridge is crucial for modern Industrial IoT (IIoT) environments, as it enables seamless data flow between machines, sensors, and industrial control systems (PLC, SCADA) on the OT side, and business management applications (ERP, MES) on the IT side.

The OT/IT Bridge facilitates real-time data synchronization. It allows industries to monitor and control their operations more efficiently, implement condition monitoring/predictive maintenance, and perform advanced analytics. The OT/IT bridge helps overcome the traditional siloing of OT and IT systems by integrating real-time data from production environments with business decision-making tools. Data Streaming frameworks like Kafka and Flink, often combined with specialized platforms for the last-mile IoT integration, act as intermediaries to ensure data consistency, interoperability, and secure communication across both domains.

This bridge enhances overall productivity and improves the OEE by providing actionable insights that help optimize performance and reduce downtime across industrial processes.

OT/IT Hierarchy – Different Layers based on ISA-95 and the Purdue Model

The OT/IT Levels 0-5 framework is commonly used to describe the different layers in industrial automation and control systems, often following the ISA-95 or Purdue model:

• Level 0: Physical Process: This is the most basic level, consisting of the physical machinery, equipment, sensors, actuators, and production processes. It represents the actual processes being monitored or controlled in a factory or industrial environment.
• Level 1: Sensing and Actuation: At this level, sensors, actuators, and field devices gather data from the physical processes. This includes things like temperature sensors, pressure gauges, motors, and valves that interact directly with the equipment at Level 0.
• Level 2: Control Systems: Level 2 includes real-time control systems such as Programmable Logic Controllers (PLCs) and Distributed Control Systems (DCS). These systems interpret the data from Level 1 and make real-time decisions to control the physical processes.
• Level 3: Manufacturing Operations Management (MOM): This level manages and monitors production workflows. It includes systems like Manufacturing Execution Systems (MES), which ensure that production runs smoothly and aligns with the business’s operational goals. It bridges the gap between the physical operations and higher-level business planning.
• Level 4: Business Planning and Logistics: This is the IT layer that includes systems for business management, enterprise resource planning (ERP), and supply chain management (SCM). These systems handle business logistics such as order processing, materials procurement, and long-term planning.
• Level 5: Enterprise Integration: This level encompasses corporate-wide IT functions such as financial systems, HR, sales, and overall business strategy. It ensures the alignment of all operations with the broader business goals.

In summary, Levels 0-2 focus on the OT (Operational Technology) side—real-time control and monitoring of industrial processes, while Levels 3-5 focus on the IT (Information Technology) side—managing data, logistics, and business operations.

While the modern, cloud-native IIoT world is not strictly hierarchical anymore (e.g. there is also lots of edge computing like sensor analytics), these layers are still often used to separate functions and responsibilities. Industrial IoT data platforms, including the data streaming platform, often connect to several of these layers in a decoupled hub and spoke architecture.

Industrial IoT Middleware

Industrial IoT (IIoT) Middleware is a specialized software infrastructure designed to manage and facilitate the flow of data between connected industrial devices and enterprise systems. It acts as a mediator that connects various industrial assets, such as machines, sensors, and controllers, with IT applications and services such as MES or ERP, often in a cloud or on-premises environment.

This middleware provides a unified interface for managing the complexities of data integration, protocol translation, and device communication to enable seamless interoperability among heterogeneous systems. It often includes features like real-time data processing, event management, scalability to handle large volumes of data, and robust security mechanisms to protect sensitive industrial operations.

In essence, IIoT Middleware is critical for enabling the smart factory concept, where connected devices and systems can communicate effectively, allowing for automated decision-making, predictive maintenance, and optimized production processes in industrial settings.

By providing these services, IIoT Middleware enables industrial organizations to optimize operations, enhance Overall Equipment Effectiveness (OEE), and improve system efficiency through seamless integration and real-time data analytics.

Relevant Industries for IIoT Middleware

Industrial IoT Middleware is essential across various industries that rely on connected equipment, sensors or vehicles and data-driven processes to optimize operations. Some key industries where IIoT Middleware is particularly needed include:

• Manufacturing: For smart factories, IIoT Middleware enables real-time monitoring of production lines, predictive maintenance, and automation of manufacturing processes. It supports Industry 4.0 initiatives by integrating machines, robotics, and enterprise systems.
• Energy and Utilities: IIoT Middleware is used to manage data from smart grids, power plants, and renewable energy sources. It helps in optimizing energy distribution, monitoring infrastructure health, and improving operational efficiency.
• Oil and Gas: In this industry, IIoT Middleware facilitates the remote monitoring of pipelines, drilling rigs, and refineries. It enables predictive maintenance, safety monitoring, and optimization of extraction and refining processes.
• Transportation and Logistics: IIoT Middleware is critical for managing fleet operations, tracking shipments, and monitoring transportation infrastructure. It supports real-time data analysis for route optimization, fuel efficiency, and supply chain management.
• Healthcare: In healthcare, IIoT Middleware connects medical devices, patient monitoring systems, and healthcare IT systems. It enables real-time monitoring of patient vitals, predictive diagnostics, and efficient management of medical equipment.
• Agriculture: IIoT Middleware is used in precision agriculture to connect sensors, drones, and farm equipment. It helps in monitoring soil conditions, weather patterns, and crop health, leading to optimized farming practices and resource management.
• Aerospace and Defense: IIoT Middleware supports the monitoring and maintenance of aircraft, drones, and defense systems. It ensures the reliability and safety of critical operations by integrating real-time data from various sources.
• Automotive: In the automotive industry, IIoT Middleware connects smart vehicles, assembly lines, and supply chains. It enables connected car services, autonomous driving, and the optimization of manufacturing processes.
• Building Management: For smart buildings and infrastructure, IIoT Middleware integrates systems like HVAC, lighting, and security. It enables real-time monitoring and control, energy efficiency, and enhanced occupant comfort.
• Pharmaceuticals: In pharmaceuticals, IIoT Middleware helps monitor production processes, maintain regulatory compliance, and ensure the integrity of the supply chain.

These industries benefit from IIoT Middleware by gaining better visibility into their operations. The digitalization of shop floor and business processes improves decision-making and drives efficiency through automation and real-time data analysis.

Industrial IoT Middleware Layers in OT/IT

While modern, cloud-native IoT architectures don’t always use an hierarchical model anymore, Industrial IoT (IIoT) middleware typically operates at Level 3 (Manufacturing Operations Management) and Level 2 (Control Systems) in the OT/IT hierarchy.

At Level 3, IIoT middleware integrates data from control systems, sensors, and other devices, coordinating operations, and connecting these systems to higher-level IT layers such as MES and ERP systems. At Level 2, the middleware handles real-time data exchange between industrial control systems (like PLCs) and IT infrastructure, ensuring data flow and interoperability between the OT and IT layers.

This middleware acts as a bridge between the operational technology (OT) at Levels 0-2 and the business-oriented IT systems at Levels 4-5.

Edge and Cloud Vendors for Industrial IoT

The industrial IoT space provides many solutions from various software vendors. Let’s explore the different options and their trade-offs.

Traditional “Legacy” Solutions

Traditional Industrial IoT (IIoT) solutions are often characterized by proprietary, monolithic architectures that can be inflexible and expensive to implement and maintain. These traditional platforms, offered by established industrial vendors like PTC ThingWorx, Siemens MindSphere, GE Predix, and Osisoft PI, are typically designed to meet specific industry needs but may lack the scalability, flexibility, and cost-efficiency required for modern industrial applications. However, while these solutions are often called “legacy” do a solid job integrating with proprietary PLCs, SCADA systems and data historians. They still operate the shop floor in most factories worldwide.

Emerging Cloud Solutions

In contrast to legacy systems, emerging cloud-based IIoT solutions offer elastic, scalable, and (hopefully) cost-efficient alternatives that are fully managed by cloud service providers. These platforms, such as AWS IoT Core, enable industrial organizations to quickly deploy and scale IoT applications while benefiting from the cloud’s inherent flexibility, reduced operational overhead, and integration with other cloud services.

However, emerging cloud solutions for IIoT can face challenges:

• Latency and real-time processing limitations, making them less suitable for time-sensitive industrial applications.
• High network transfer cost from the edge to the cloud.
• Security and compliance concerns arise when transferring sensitive operational data to the cloud, particularly in regulated industries.
• Depending on reliable internet connectivity, which can be a significant drawback in remote or unstable environments.
• Very limited connectivity to proprietary (legacy) protocols such as Siemens S7 or Modbus.

The IIoT Enterprise Architecture is a Mix of Vendors and Platforms

Threre is no black and white comparing different solutions. The current IIoT landscape in real world deployments features a mix of traditional industrial vendors and new cloud-native solutions. Companies like Schneider Electric’s EcoStruxure still provide robust industrial platforms, while newer entrants like AWS IoT Core are gaining traction due to their modern, cloud-centric approaches. The shift towards cloud solutions reflects the growing demand for more agile and scalable IIoT infrastructures.

The reality in the industrial space is that:

• OT/IT is usually hybrid edge to cloud, not just cloud
• Most cloud-only solutions do not provide the right security, SLAs, latency, cost
• IoT is a complex space. “Just” a OPC-UA or MQTT connector is not sufficient in most scenarios.

Data Streaming for Industrial IoT in the OT/IT World with Kafka and Flink

Data streaming with Apache Kafka and Flink is a powerful approach that enables the continuous flow and processing of real-time data across various systems. However, to be clear: Data streaming is NOT a silver bullet. It is complementary to other IoT middleware. And some modern, cloud-native industrial software is built on top of data streaming technologies like Kafka and Flink under the hood.

In the context of Industrial IoT, data streaming plays a crucial role by seamlessly integrating and processing data from numerous IoT devices, equipment, PLCs, MES and ERP in real-time. This capability enhances decision-making processes and operational efficiency by providing continuous insights, allowing industries to optimize their operations and respond proactively to changing conditions. The last-mile integration is usually done by complementary IIoT technologies providing sophisticated connectivity to OPC-UA, MQTT and proprietary legacy protocols like S7 or Modbus.

Apache Kafka and Flink in IT Environments

In data center and cloud settings, Kafka and Flink are used to provide continuous processing and data consistency across IT applications including sales and marketing, B2B communication with partners, and eCommerce. Data streaming facilitates data integration, processing and analytics to enhance the efficiency and responsiveness of IT operations and business; no matter if data sources or sinks are real-time, batch or request-response APIs.

Apache Kafka as an OT/IT Bridge

Kafka serves as a critical bridge between Operational Technology (OT) and Information Technology (IT) by enabling real-time data synchronization at scale. This integration ensures data consistency across different systems, supporting seamless communication and coordination between industrial operations and business systems.

Apache Kafka and Flink in OT Edge Applications

At the edge of operational technology, Kafka and Flink provide a robust backbone for use cases such as condition monitoring and predictive maintenance. By processing data locally and in real-time, these technologies improve the Overall Equipment Effectiveness (OEE), and support advanced analytics and decision-making directly within industrial environments.

IoT Success Story: Industrial Edge Intelligence with Helin and Confluent

Helin is a company specializes in providing advanced data solutions focusing on real-time data integration and analytics, particularly in the context of industrial and operational environments. Its industry focus on maritime and energy sector, but this is relevant across all IIoT industries.

Helin presented about its Industrial Edge Intelligence Platform at Confluent’s Data in Motion Tour in Utrecht, Netherlands in. 2024. The IIoT platform includes capabilities for data streaming, processing, and visualization to help organizations leverage their data more effectively for decision-making and operational improvements.

Helin’s platform bridges the OT and IT worlds by seamlessly integrating industrial edge analytics with multi-tenant cloud solutions:

The above architecture diagram shows how Helin maps to the OT/IT hierarchy:

• OT – 0,1,2,3
  • 1: Sensors, Actuators, Field Devices
  • 2: Remote I/O
  • 3: Controller
• DMZ / Gateway – 3.5
• BIZ (= IT) – 4,5
  • 4 OT Applications (MES, SCADA, etc)
  • 5 – outside of Helin – IT Applications (ERP, CRM, DWH, etc)

The strategy and value of Helin’s IoT platform is relevant for most industrial organizations: Making dumb assets smart by extracting data in real-time and utilize AI to transform it into significant business value and actionable insights for the maritime & energy sectors.

Business Value: Fuel Reduction, Increased Revenue, Saving Human Lives

Helin presented three success stories with huge business value:

• 8% Fuel reduction: Helin’s platform reduced the fuel consumption for Boskalis 8% by delivering real-time insights to vessel operators offshore.
• 20% Revenue: An increase of revenue for the solar parks of Sunrock with 20% by optimizing their assets by the platform.
• Saving human lives: Optimization of drilling operations while increasing the safety of the crew on oil rigs by reducing human errors.

Data Streaming with Kafka and Flink in the Cloud

Why does the Helin IoT Platform use Kafka? Helin brought up a few powerful arguments:

Helin started with self-managed Kafka and cumbersome Python scripts…

… and transitioned to fully managed Kafka in Confluent Cloud:

As a next step, Helin is migrating from cumbersome and unreliable Python mappings to Apache Flink for scalable and reliable data processing.

Please note that the last-mile IoT connectivity at the edge (SCADA, PLC, etc.) is implemented with technologies like OPC-UA, MQTT or custom integrations. You can see a common best practice: Choose and combine the right tools for the job.

Data Streaming with Kafka and Flink Empowers Edge-to-Cloud Industrial IoT Middleware

Data streaming plays a crucial role in bridging OT and IT in industrial automation. By enabling continuous data flow between the edge and the cloud, Kafka and Flink ensure that both operational data from sensors and machinery, and IT applications like ERP and MES, remain synchronized in real-time. Additionally, data consistency with non-real-time systems like a legacy batch system or a cloud-native data lakehouse are guaranteed out-of-the-box.

The real-time integration powered by Kafka and Flink improves the overall operational efficiency (OEE) and enables specific use cases such as enhanced predictive maintenance, condition monitoring. As industries increasingly adopt edge computing alongside cloud solutions, these data streaming tools provide the scalability, flexibility, and low-latency performance needed to drive Industrial IoT initiatives forward.

Helin’s Industrial Edge Intelligence platform is an excellent example for an IIoT middleware. It leverages Apache Kafka and Flink to integrate real-time data from industrial assets and enabling predictive analytics and operational optimization. By using this platform, companies like Boskalis achieved 8% fuel savings, and Sunrock increased revenue by 20%. These real world scenarios demonstrate the platform’s ability to drive significant business value through real-time insights and decision-making in industrial projects.

How does your OT/IT integration look like today? Do you plan to optimize the infrastructure with data streaming? How does the hybrid architecture look like? What are the use cases? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Industrial IoT Middleware for Edge and Cloud OT/IT Bridge powered by Apache Kafka and Flink appeared first on Kai Waehner.

The Digitalization of Airports and the Aviation Industry

Digitalization transforms airport operations and improves the experience of employees and passengers. It affects various aspects of airport operations, passenger experiences, and overall efficiency.

Schiphol Group is a Dutch company that owns and operates airports in the Netherlands. The company is primarily known for operating Amsterdam Airport Schiphol, which is one of the busiest and most important airports in Europe. The Schiphol Group is involved in a range of activities related to airport management, including aviation and non-aviation services.

Schiphol Group describes its journey of becoming a leading autonomous airport until 2050:

Data Streaming with Kafka and Flink for Airport Operations and Passenger Experience

Data streaming with Apache Kafka and Apache Flink enables airport and aviation systems to process and analyze real-time data from various sources, such as flight information, passenger movements, and baggage tracking, enhancing operational efficiency and passenger experience.

These technologies facilitate predictive maintenance, personalized services, and improved security measures through the continuous flow and immediate processing of critical data at any scale reliably.

Continuous Stream Processing in an Event-driven Architecture with Kafka Streams, KSQL, or Flink

Continuous processing of incoming events in real-time enables transparency and context-specific decision making. OpenCore, an IT consultancy in Germany, presented already in 2018 at Kafka Summit San Francisco how stream processing with technologies like Kafka Streams, KSQL or Apache Flink serves the real-time needs of an airport.

Think about the technical IoT events ingested from aircraft, gates, retail stores, passenger mobile apps, and many other interfaces…

… and how continuous correlation of data in real-time enables use cases such as predictive forecasting, planning, maintenance, plus scenarios like cross-organization loyalty platforms, advertisement, and recommendation engines for improving the customer experience and increasing revenue:

Airport Digitalization with Data Streaming using Apache Kafka and Flink

Real-time data beats slow data. That’s true for almost any use in the aviation industry, including airports, airlines, and other involved organizations. Additionally, data consistency matters across organizations.

Here are key areas where digitalization affects airports. While compiling this list, I realized I wrote about many of these scenarios in the past because other industry already deployed these use cases. Hence, each section includes a reference to another article where data streaming with Kafka and Flink is already applied in this context.

1. Passenger Experience

As frequent traveller myself, I put this at the beginning of the list. Examples:

• Self-service Kiosks: Check-in, baggage drop, and boarding processes have become faster and more efficient.
• Mobile Applications: Passengers can book tickets, receive real-time flight updates, and access boarding passes.
• Biometric Systems: Facial recognition and fingerprint scanning expedite security checks and boarding.

The past decade already significantly improved the passenger experience. But it still needs to get better. And data consistency matters. Today, a flight delay or cancellation is not shared consistently across the customer mobile app, airport screens, and customer service of the airline and airport.

Reference to data streaming in financial services: Operational and transactional systems leverage Kafka for data consistency, not because of its real-time capabilities. Apache Kafka ensures data consistency with its durable commit log, timestamps, and guaranteed ordering. Kafka connects to real-time and non-real-time systems (files, batch, HTTP/REST APIs).

2. Operational Efficiency

Automation with IoT sensors, paperless processes, and software innovation enables more cost-efficient and reliable airport operations. Examples:

• Automated Baggage Handling: RFID tags and automated systems track and manage luggage, reducing errors and lost baggage).
• Predictive Maintenance: IoT sensors and data analytics predict equipment failures before they occur, ensuring smoother operations.
• Air Traffic Management: Advanced software systems enhance the coordination and efficiency of air traffic control.

Reference to data streaming in manufacturing: Condition monitoring and predictive maintenance leverage stream processing with Apache Kafka and Flink for many years already, either in the cloud or at the edge and shop floor level for Industrial IoT (IIoT) use cases.

3. Security, Safety and Health Enhancements

Safety and health are one of the most important aspects at any airport. Airports continuously improved security, monitoring, and surveillance because of terrorist attacks, the Covid pandemic, and many other dangerous scenarios.

• Advanced Screening Technologies: AI-powered systems and improved scanning technologies detect threats more effectively.
• Cybersecurity: Protecting sensitive data and systems from cyber threats is crucial, requiring robust digital security measures.
• Health Monitoring: Temperature measurements and people tracking were introduced during the Covid pandemic in many airports.

Reference to data streaming in Real Estate Management: Apache Kafka and Flink improve real estate maintenance and operations, optimize space usage, provide better employee experience, and better defense against cyber attacks. Check out “IoT Analytics with Kafka and Flink for Real Estate and Smart Building” and “Apache Kafka as Backbone for Cybersecurity” for more details.

4. Sustainability and Energy Management

Sustainability and energy management in airports involve optimizing energy use and reducing environmental impact through efficient resource management and implementing eco-friendly technologies. Examples:

• Smart Lighting and HVAC Systems: Automated systems reduce energy consumption and enhance sustainability.
• Data Analytics: Monitoring and optimizing resource usage helps reduce the carbon footprint of airports.

Sustainability and energy management in an airport can be significantly enhanced by using Apache Kafka and Apache Flink to stream and analyze real-time data from smart meters and HVAC systems, optimizing energy consumption and reducing environmental impact.

Reference to data streaming in Environmental, Social, and Governance (ESG) across industries: Kafka and Flink’s real-time data processing capabilities build a powerful alliance with ESG principles. Beyond just buzzwords, I wrote about real-world deployments with Kafka and Flink and architectures across industries to show the value of data streaming for better ESG ratings.

5. Customer Service and Communication

Customer service is crucial for each airport. While lots of information comes from airlines (like delays, cancellations, seat changes, etc.), the airport provides the critical communication backend with display, lounges, service personal, and so on.  Examples to improve the customer experience:

• AI Chatbots: Provide 24/7 customer support for inquiries and assistance with Generative AI (GenAI) embedded into the existing business processes.
• Digital Signage: Real-time updates on flight information, gate changes, and other announcements improve communication.
• Loyalty Integration: Airports do not provide a loyalty platform, but they integrate more and more with airlines (e.g., to reward miles for shopping).

Reference to data streaming in retail: The retail industry is years ahead with providing a hyper-personalized customer experience. “Omnichannel Retail and Customer 360 in Real Time with Apache Kafka” and “Customer Loyalty and Rewards Platform with Data Streaming” tell you more. GenAI is a fundamental change for customer services. Kafka and Flink play a critical role for GenAI to provide contextual, up-to-date information from transactional systems into the large language model (LLM).

6. Revenue Management

Airport revenue management involves optimizing income from aviation and non-aviation sources through demand forecasting and strategic resource allocation. Examples:

• Dynamic Pricing: Algorithms adjust prices for parking, retail spaces, and other services based on demand and other factors.
• Personalized Marketing: Data analytics help target passengers with tailored offers and promotions.

Reference to data streaming in retail: While the inventory looks different for an airport, the principles from retail can be adopted one-to-one. Instead of TVs or clothes, the inventory is the parking lot, lounge seat, and similar. Advertising is another great example. Airports can learn from many digital natives how they built a real-time digital ads platform with Kafka and Flink. This can be adopted to retail media in the airport, but also to any physical inventory management.

7. Emergency Response and Safety

Emergency response and safety at the airport involve coordinating real-time monitoring, quick decision-making, and efficient resource deployment to ensure the safety and security of passengers, staff, and infrastructure during emergencies. Examples:

• Real-time Monitoring: IoT devices and sensors provide live data on airport conditions, aiding in faster response times.
• Digital Simulation and Training: Virtual reality and simulation technologies enhance training for emergency scenarios.
• Seamless Connectivity: Stable Wi-Fi and 5G Networks with good latency and network slicing for safety-critical use cases.

Reference to data streaming in Industrial IoT: Safety-critical applications require hard real-time. This is NOT Kafka, Flink, or any similar IT technology. Instead, this is embedded systems, robotics, and programming languages like C or Rust. However, data streaming integrates the OT/IT world for near real-time data correlation and analytics in edge or hybrid cloud architectures. Every relevant data set from aircraft, gates, and other equipment is continuously monitored to ensure a safe airport environment.

Data Sharing with Kafka between Airport, Airlines and other B2B Partners like Retail Stores

Cross-organization data sharing is crucial for any airport and airline. Today, most integrations are implemented with APIs (usually HTTP/REST) or still even file-based systems. This works well for some use cases. But data streaming – by nature – is perfect for sharing streaming data like transactions, sensor data, location-based services, etc. in real-time between organizations:

As Apache Kafka is the de facto standard for data streaming, many companies directly replicate data to partners using the Kafka protocol. AsyncAPI as an open standard (beyond Kafka) and integration via HTTP on top of Kafka (via Kafka Connect API connectors) are other common patterns.

Real-World Success Stories for Data Streaming in the Aviation Industry

Several real world success stories exist for deployments of data streaming with Apache Kafka and Flink in airports and airlines. Let’s explore a few case studies and refer to further material.

Schiphol Group (Amsterdam Airport)

Roel Donker and Christiaan Hoogendoorn from Schiphol Group presented at the Data in Motion Tour 2024 in Utrecht, Netherlands. This was an excellent presentation with various data streaming use cases across fields like application integration, data analytics, internet of things, and artificial intelligence.

On its journey to an autonomous airport until 2025, the digitalization involves many technologies and software/cloud services. Schiphol Group transitioned from open source Apache Kafka to Confluent Cloud for cost-efficiency, elasticity, and multi-tenancy.

The company runs operational and analytical data streaming workloads with different SLAs. The integration team uses the data streaming platform to integrate with both the legacy and the new world, also 3rd party like airlines, GDS, police, etc (all point-to-point and with different interfaces).

Here are a few examples of the scenarios Schiphol Group explored:

Schiphol Group: Data Platform with Apache Kafka

Schiphol uses Apache Kafka as a core integration platform. The various use cases require different Kafka clusters depending on the uptime SLA, scalability, security, and latency requirements. Confluent Cloud fully manages the data streaming platform, including connectors to various data sources and sinks:

Kafka connects critical PostgreSQL databases, Databricks analytics platform, applications running in containers on Red Hat OpenShift, and others.

3Scale is used as complementary API gateway for request-response communication. The latter is not a surprise, but very common. HTTP/REST APIs and Apache Kafka complement each other. API Management solutions such as 3Scale, MuleSoft, Apigee or Kong connect to Kafka via HTTP or other interfaces.

Schiphol Group: IoT with Apache Kafka

Some use cases at Schiphol Group require connectivity and processing of IoT data. That’s not really a big surprise in the aviation industry, where airports and airlines rely on data-driven business processes:

Kafka Connect and stream processing connect and combine IoT data and feed relevant context into other IT applications.

Connectivity covers various infrastructures and networks, including:

• Private LoRa networks
• Passenger flow management system(FMS)
• BLIP (the supplier delivering IoT devices in the terminal measuring real-time how crowded areas are so people can be redirected when needed)
• Wi-Fi location services (like heatmaps for crowd management)

Schiphol Group: AI and Machine Learning with Apache Kafka

Artificial Intelligence (AI) requires various technologies and concepts to add business value. Predictive analytics, active learning, batch model training, debugging and testing the entire pipeline, and many other challenges need to be solved. Apache Kafka is the data fabric of many AI/ML infrastructures.

Here is how Kafka provides the foundation of an event-driven AI architecture at Schiphol Group:

The combination of Apache Kafka and AI/ML technologies enables various valuable use cases at Schiphol Group, including:

• Analysis of historical data (root cause analysis, critical path & process analysis, reporting)
• Insights on real-time data (insight on turnaround process with one shared truth, real time insight on ramp capacity and turnaround progress per ramp, real-time insight on ramp safety, input for E2E insight Airside
• Predictions (input for dynamic gate management, input for autonomous vehicles, input for predicting delays)

Lufthansa, Southwest, Cathay Pacific, and many other Airlines…

I met plenty of airlines that already use data streaming in production for different scenarios. Fortunately, a few of these airlines were happy to share their stories in the public:

• Southwest Airlines (Data in Motion Tour 2024 in Dallas): Single pane of glass with the ability to view all flight operations and sync their three key schedules: aircraft, passengers, workforce.
• Cathay Pacific (Data in Motion Tour 2024 in Singapore): Rebranded to Cathay because of transitioning from focus on passenger transport to adding cargo and lifestyle / shopping experiences.
• Lufthansa (Webinar 2023): Operations steering, IT modernization (from MQ and ESB to Confluent), and real-time analytics with AI/ML.

The Lufthansa success story is available in its own blog post (including video recording). For even more examples, including Singapore Airlines, Air France, and Amadeus, check out the overview article “Apache Kafka in the Airline, Aviation and Travel Industry“.

Apache Kafka and Flink as Data Fabric for Operational and Analytical Airport Use Cases

Schiphol Group’s vision of an autonomous Amsterdam Airport in 2050 shows where the aviation industry is going: Automated business processes, continuous monitoring and processing of IoT infrastructure, and data-driven decision making and passenger experiences.

Airports like Amsterdam, similarly like airlines such as Lufthansa, Southwest or Cathay, modernize existing IT infrastructure, transition to hybrid cloud architectures, and innovate with new use cases (often learning from other industries like financial services, retail or manufacturing).

Data Streaming with Apache Kafka and Flink plays a crucial role in this journey. Data processing at any scale to provide consistent and good quality data in real-time enables any downstream application (including batch and API) to build reliable operational and analytical systems.

How do you leverage data streaming with Kafka and Flink in the aviation industry? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post The Digitalization of Airport and Airlines with IoT and Data Streaming using Kafka and Flink appeared first on Kai Waehner.

Apache Kafka at the Edge and Hybrid Cloud

Apache Kafka is a distributed event streaming platform that enables building real-time streaming data pipelines and applications by providing capabilities for publishing, subscribing to, storing, and processing streams of records in a scalable and fault-tolerant way.

Various examples exist for Kafka deployments on the edge. These use cases are related to several of the above categories and requirements, such as low hardware footprint, disconnected offline processing, hundred of locations, and hybrid architectures.

Use Cases for Apache Kafka at the Edge

I have worked with enterprises across industries and the globe on the following scenarios:

• Public Sector: Local administration in each city, smart city projects including public transportation, traffic management, integration of various connected car platforms from different carmakers, cybersecurity (including IoT use cases such as capturing and processing camera images)
• Transportation / Logistics / Railway / Aviation: Track and trace, Kafka in the trains for offline and local processing / storage, traveller information (delayed or canceled flight / train / bus), real-time loyalty platforms (class upgrade, lounge access)
• Manufacturing (Automotive, Aerospace, Semiconductors, Chemical, Food, and others): IoT aftermarket customer services, OEM in machines and vehicles, embedding into standard software such as ERP or MES systems, cybersecurity, a digital twin of devices/machines/production lines/processes, production line monitoring in factories for predictive maintenance/quality control/production efficiency, operations dashboards and line wellness (on-site for the plant manager, and aggregated global KPIs for executive management), track&trace and geofencing on the shop floor
• Energy / Utility / Oil & Gas: Smart home, smart buildings, smart meters, monitoring of remote machines (e.g., for drilling, windmills, mining), pipeline and refinery operations (e.g., predictive failure or anomaly detection)
• Telecommunications / Media: OSS real-time monitoring/problem analysis/metrics reporting/root cause analysis/action response of the network devices and infrastructure (routers, switches, other network devices), BSS customer experience and OTT services (mobile app integration for millions of users), 5G edge (e.g., street sensors)
• Healthcare: Track & trace in the hospital, remote monitoring, machine sensor analytics
• Retailing / Food / Restaurants / Banking: Customer communication, cross-/up-selling, loyalty system, payments in retail stores, perpetual inventory, Point-of-Sale (PoS) integration for (local) payments and (remote) CRM integration, EFTPOS (Electronic funds transfer at point of sale)

Benefits for Kafka at the Edge AND in the Cloud

Deploying the same technology in hybrid environments is not a new idea. Project teams see tremendous benefits when using Kafka at the edge and in the data center or cloud:

• Same APIs, concepts, development tools and testing
• Same architecture for streaming, storing, processing and connecting systems, even if at very different scale
• Real-time synchronization between multiple environments included out-of-the-box via the Kafka protocol

Let’s explore how ARM CPUs fit into this discussion.

What is ARM CPU?

An ARM CPU refers to a family of CPUs based on the Advanced RISC Machine (ARM) architecture, which is a type of Reduced Instruction Set Computing (RISC) architecture. ARM CPUs ave a reputation for their high performance, power efficiency, and low cost. These characteristics make them particularly popular in mobile devices such as smartphones, tablets, and an increasingly wide range of other devices like IoT (Internet of Things) gadgets, servers, and even desktop computers.

The ARM architecture performs operations with a smaller number of computer instructions, allowing it to achieve high performance with lower power consumption compared to more complex instruction set computing (CISC) architectures like x86 used by Intel and AMD CPUs. This efficiency is a key advantage for battery-powered devices, where energy conservation is critical.

ARM Holdings, the company behind the ARM architecture, does not manufacture CPUs but licenses the architecture to other companies. These companies can then implement their own ARM-based processors, potentially customizing them for specific needs. This licensing model has led to a wide adoption of ARM processors across various segments of the technology industry.

ARM32 vs. ARM64

ARM architectures come in different versions, primarily distinguished by their instruction set architectures and addressing capabilities. The most commonly referenced are ARMv7 and ARMv8 (also called AArch64) correspond to 32-bit and 64-bit processing capabilities, respectively.

Newer hardware for industrial PCs and home computers incorporates ARMv8 (64-bit). It is the foundation for smartphones, tablets, servers, and processors like Apple’s A-series chips in iPhones and iPads. Even the cloud providers use the ARM architecture to build new processors for cloud computing, like Amazon’s Graviton. ARMv8 processors can run both 32-bit and 64-bit applications, offering greater versatility and performance.

Key Features and Benefits of ARM CPUs

The key features and benefits of ARM CPUs include:

• Power Efficiency: Their design allows for significant power savings, extending battery life in portable devices.
• Performance: While historically seen as less powerful than their x86 counterparts, modern ARM processors offer competitive performance, especially in multi-core configurations.
• Customization: Companies can license the ARM architecture and customize their own chips, allowing for optimized processors that meet specific product requirements.
• Ecosystem: A broad adoption across mobile, embedded, and increasingly in server and desktop markets ensures a robust ecosystem of software and development tools.

ARM CPUs are central to the development of mobile computing and are becoming more important in other areas, including edge computing, data centers, and as part of the shift towards more energy-efficient computing solutions.

Why ARM CPUs at the Edge (e.g., for Industrial IoT)?

ARM architecture is favored for edge computing, including Industrial IoT. It provides high power efficiency and performance within compact form factors. These characteristics ensure devices can handle compute-intensive tasks locally. Only relevant data is transmitted to the cloud, which saves bandwidth and decreases latency.

The efficiency of ARM CPUs is crucial for industrial applications where real-time processing and long battery life are essential. ARM’s versatility and low power consumption make it ideal for the diverse needs of edge computing in various environments.

For instance, in manufacturing, ARM-powered sensors on machines enable predictive maintenance by monitoring conditions like vibration and temperature. These sensors process data locally, offering real-time alerts on potential failures, reducing downtime, and saving costs. ARM’s efficiency supports widespread deployment, making it ideal for continuous, autonomous monitoring in industrial environments.

Why ARM in the Cloud?

ARM’s efficiency and performance advantages are driving its adoption in cloud computing. ARM-based processors, like Amazon’s AWS Graviton, offer an attractive mix of high performance and lower power consumption compared to traditional x86 CPUs. This efficiency translates into cost savings and reduced environmental impact for cloud service providers and their customers.

AWS Graviton, specifically designed for cloud workloads, exemplifies how ARM architecture can optimize operations in data centers, enhancing the performance of web servers, containerized applications, and microservices at a lower cost. This shift towards ARM in the cloud represents a significant move towards more energy-efficient and cost-effective data center operations.

Apache Kafka on ARM – A Match Made in Heaven for Edge and Cloud Workloads

Using ARM architecture together with Apache Kafka, a distributed streaming platform, offers several advantages, especially in scenarios that demand high throughput, scalability, and energy efficiency.

1. Energy Efficiency and Cost-Effectiveness: ARM processors are known for their low power consumption, which makes them cost-effective for running distributed systems like Kafka. Deploying Kafka on ARM-based servers can reduce operational costs, particularly in large-scale environments where energy consumption can significantly affect the budget.
2. Scalability: Kafka handles large volumes of data and high throughput, characteristics that align well with the scalability of ARM processors in cloud environments. ARM’s efficiency enables scaling out Kafka clusters more economically, allowing for the processing of streaming data in real-time without incurring high energy or hardware costs.
3. Edge Computing: Kafka is a common choice for real-time data processing and aggregation in edge computing scenarios. ARM’s dominance in IoT and edge devices makes it a natural fit for these use cases. Running Kafka on ARM enables efficient data processing closer to the source, reducing latency and bandwidth usage by minimizing the need to send large volumes of data to central data centers.
4. Eco-Friendly Solutions: With growing environmental concerns, ARM’s energy efficiency contributes to more sustainable computing solutions. Deploying Kafka on ARM can be part of an eco-friendly strategy for organizations looking to minimize their carbon footprint.
5. Innovative Use Cases: Combining Kafka with ARM opens up new possibilities for innovative applications in IoT, real-time analytics, and mobile applications. The efficiency of ARM allows for cost-effective experimentation and deployment of new services that require real-time data processing and streaming capabilities.

Examples and Case Studies for Kafka at the Edge

Overall, the combination of ARM and Apache Kafka supports the development of efficient, scalable, and sustainable data processing architectures, particularly suited for modern applications that require real-time performance with minimal energy consumption.

For several use cases, architectures and case studies about data streaming at the edge and hybrid cloud, check out my related articles:

• Use Cases and Architectures for Kafka at the Edge
• Apache Kafka is the New Black at the Edge in Industrial IoT, Logistics and Retailing
• Infrastructure Checklist for Apache Kafka at the Edge
• Kafka for Real-Time Replication between Edge and Hybrid Cloud
• Apache Kafka in Air-Gapped Zero-Trust Environments with Data Diode / Unidirectional Gateway

Most of these blog posts are a few years old. But they are as relevant today as at the time of writing them. Actually, the official support of ARM CPU at the edge completely changes the conversations about challenges and solutions of deploying Kafka on edge infrastructure. The deployment of Kafka at the edge was never easier. If you buy a new Industrial PC (IPC) today, it will have enough hardware power to run Kafka and its ecosystem for data integration and stream processing easily.

Confluent Platform on ARM Infrastructure for Edge Deployments

Confluent Platform is Confluent’s data streaming platform for self-managed deployments of Apache Kafka. Most deployments operate in a traditional data center. However, this is more and more shifting to deploy at the edge, i.e., outside of a data center, too.

Since version 7.6., Confluent Platform officially supports ARM64 Linux architectures. Confluent Platform’s architecture allows you to run it wherever your IT systems are, across a global footprint. This includes running it as a mission-critical cluster in data centers, but also on edge sites like retail stores, ships or factories, or as a single broker on edge devices.

Confluent itself recognized the benefits of ARM64 CPUs: They moved the entire AWS fleet for the fully managed Confluent Cloud to ARM-based images in the past months.

The Confluent Server Broker powered by Apache Kafka enables end-to-end data pipelines:

• Collecting data from any source
• Persists the data in the event storage with separate compute and storage
• Processes the events with stream processing
• Share data with downstream applications
• Replicate selected events across the WAN through the native Kafka protocol via Cluster Linking.

Now, you can deploy this in production on low-cost, small-footprint ARM64 architecture infrastructure at the edge and also synchronize with a data center or cloud Kafka cluster.

Kafka + ARM = Cost-Effective and Sustainable

The article outlined the synergistic relationship between Apache Kafka and ARM CPUs. It enables efficient, scalable, and sustainable data processing architectures for edge and hybrid cloud environments.

The adoption of ARM in cloud computing marks a significant shift towards more sustainable and performance-optimized computing solutions. The combination of Kafka and ARM CPUs is poised to drive innovation in real-time analytics, IoT, and mobile applications. A few great examples:

• AWS Graviton to operate Kafka cost-efficient in the public cloud.
• Confluent Platform’s compatibility and support for ARM64 architectures at the edge.

The sustainability of energy-efficient ARM CPUs is a perfect segue to the data streaming article “Green Data, Clean Insights: How Kafka and Flink Power ESG Transformations“.

Do you already use ARM processors in your edge or cloud Kafka environment? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post ARM CPU for Cost-Effective Apache Kafka at the Edge and Cloud appeared first on Kai Waehner.

A Green Future with Improved ESG

ESG stands for Environmental, Social, and Governance. It’s a set of criteria that investors use to evaluate a company’s impact on the world beyond just financial performance. Environmental factors assess how a company manages its impact on nature, social criteria examine how it treats people (both internally and externally), and governance evaluates a company’s leadership, ethics, and internal controls. Essentially, it’s a way for investors to consider the broader impact and sustainability of their investments.

ESG Buzzword Bingo with Scores and Ratings

ESG seems to be an excellent marketing hit. In the ESG world, phrases like “climate risk“, “social responsibility“, and “impact investing” are all the rage. Everyone’s throwing around terms like “carbon footprint”, “diversity and inclusion,” and “ethical governance” like confetti at a party. It’s all about showing off your commitment to a better, more sustainable future.

To be more practical, ESG scores or ratings are quantitative assessments that measure a company’s performance in terms of environmental, social, and governance factors.

These scores reflect a company’s performance on environmental, social, and governance factors. High scores mean you’re the A-lister of the sustainability world, while low scores might land you in the “needs improvement” category.

Various rating agencies or organizations that evaluate how well a company aligns with sustainable and responsible business practices assign scores. Investors and stakeholders use these scores to make informed decisions, considering not only financial performance but also a company’s impact on the environment, its treatment of employees, and the effectiveness of its governance structure. It’s a way to encourage businesses to prioritize sustainability and social responsibility.

No matter if you use these scores or your own (maybe more reasonable) KPIs, let’s find out how data streaming helps to improve the ESG in your company.

Data Streaming to Improve ESG and Sustainability

Data streaming with Kafka and Flink is a dynamic synergy. Apache Kafka efficiently handles the high-throughput, fault-tolerant ingestion of real-time data. Apache Flink processes and analyzes this streaming data with low-latency,

The combination empowers complex event processing and enabling timely insights for applications ranging from ESG monitoring to financial analytics. Together, Kafka and Flink form a powerful duo, orchestrating the seamless flow of information, transforming how organizations harness the value of real-time data in diverse use cases.

Apache Kafka: The De Facto Standard for Data Streaming

The Kafka API became the de facto standard for event-driven architectures and event streaming. Similar to Amazon S3 for object storage. Two proof points:

• Use cases across all industries and infrastructure. Including various kinds of transactional and analytic workloads. Edge, hybrid, multi-cloud. I collected a few examples across verticals that use Apache Kafka to show the prevalence across markets.
• Adoption by various open-source frameworks and many software/cloud vendors. Check out my blog post if you are interested in a comparison of Kafka vendors such as Confluent, Cloudera, Red Hat or Amazon MSK and related technologies like Azure Event Hubs, AWS Kinesis, Redpanda, or Apache Pulsar.

Apache Flink is gaining a similar adoption right now. The early adopters in Silicon valley use it in combination with Kafka for years already. Software and cloud vendors adopt it similarly to Kafka. Both together will have a grand future across all industries.

A New Software Category with Apache Kafka and Flink

In December 2013, the research company published “The Forrester Wave: Streaming Data Platforms, Q4 2023“. Get free access to the report here. The leaders are Microsoft, Google and Confluent, followed by Oracle, Amazon, Cloudera, and a few others.

You might agree or disagree with the positions of a specific vendor regarding its offering or strategy strength. But the emergence of this new wave is a proof that data streaming is a new software category; not just yet another hype or next-generation ETL / ESB / iPaaS tool.

ESG and Sustainability Use Cases for Kafka and Flink

Using Kafka and Flink for data streaming in the context of ESG brings serious positive impact. Imagine a continuous flow of real-time data on environmental metrics, social impact indicators, and governance practices. Here’s how data streaming with Kafka and Flink helps:

1. Real-time Monitoring: With Kafka and Flink, you can monitor and analyze ESG-related data in real time. This means instant insights into a company’s environmental practices, social initiatives, and governance structures. This immediate feedback loop allows for quicker response to any deviations from sustainability goals.
2. Proactive Risk Management: Streaming data enables the identification of potential ESG risks as they emerge. Whether it’s an environmental issue, a social controversy, or a governance concern, early detection allows companies to address these issues promptly, reducing the overall impact on their ESG performance.
3. Transparent Reporting: ESG reporting requires transparency. By streaming data, companies can provide stakeholders with real-time updates on their sustainability efforts across the entire data pipeline with data governance and lineage. This transparency builds trust among investors, customers, and the public.
4. Automated Compliance Monitoring: ESG regulations are evolving, and compliance is crucial. With Kafka and Flink, you can automate the monitoring of compliance with ESG standards. This reduces the risk of overlooking any regulatory changes and ensures that a company stays on the right side of environmental, social, and governance requirements.
5. Data Integration: Data streaming facilitates the integration of diverse ESG-related data sources. Whether it’s data from IoT devices, social media sentiment analysis, or governance-related documents, these tools enable seamless integration, providing a comprehensive view of a company’s ESG performance. Good data quality ensures correct reporting, no matter if real-time with stream processing or batch in a data lake or data warehouse.

Leveraging Kafka and Flink for data streaming in the ESG space enables companies to move from retrospective analysis to proactive management, fostering a culture of continuous improvement in environmental impact, social responsibility, and governance practices.

ESG Data Sources for the Data Streaming Platform

In the ESG context, data sources for Kafka and Flink are diverse, reflecting the multidimensional nature of environmental, social, and governance factors. Here are some key data sources:

1. Environmental Data:
   • IoT Devices: Sensors measuring air and water quality, energy consumption, and emissions.
   • Satellite Imagery: Monitoring changes in land use, deforestation, and other environmental indicators.
   • Weather Stations: Real-time weather data influencing environmental conditions.
2. Social Data:
   • Social Media: Analyzing sentiments and public perceptions related to a company’s social initiatives.
   • Employee Feedback: Gathering insights from internal surveys or employee feedback platforms.
   • Community Engagement Platforms: Monitoring community interactions and feedback.
3. Governance Data:
   • Financial Reports: Analyzing financial transparency, disclosure practices, and governance structures.
   • Legal and Regulatory Documents: Tracking compliance with governance regulations and changes.
   • Board Meeting Minutes: Extracting insights into decision-making and governance discussions.
4. Supply Chain Data:
   • Supplier Information: Assessing the ESG practices of suppliers and partners.
   • Logistics Data: Monitoring the environmental impact of transportation and logistics.
5. ESG Ratings and Reports:
   • Third-Party ESG Ratings: Integrating data from ESG rating agencies to gauge a company’s overall performance.
   • Sustainability Reports: Extracting information from company-issued sustainability reports.
6. Internal Operational Data:
   • Employee Diversity Metrics: Tracking diversity and inclusion metrics within the organization.
   • Internal Governance Documents: Analyzing internal policies and governance structures.

By leveraging Kafka to ingest and process data from these sources in real-time, organizations can create a comprehensive and up-to-date view of their ESG performance, enabling them to make informed decisions and drive positive change.

ESG Data Sinks from Kafka and Flink

In the ESG context with Kafka and Flink, data sinks play a crucial role in storing and further using the processed information. Here are some common data sinks:

1. Databases:
   • Time-Series Databases: Storing time-stamped ESG metrics for historical analysis.
   • Data Warehouses: Aggregating and storing ESG data for business intelligence and reporting.
2. Analytics Platforms (aka Data Lake):
   • Big Data Platforms (e.g., Hadoop, Spark): Performing complex analytics on ESG data for insights and trend analysis.
   • Machine Learning Models: Using ESG data to train models for predictive analytics and pattern recognition.
3. Visualization Tools:
   • Dashboarding Platforms (e.g., Tableau, Power BI): Creating visual representations of ESG metrics for stakeholders and decision-makers.
   • Custom Visualization Apps: Building specialized applications for interactive exploration of ESG data.
4. Alerting Systems:
   • Monitoring and Alerting Tools: Generating alerts based on predefined thresholds for immediate action on critical ESG events.
   • Notification Systems: Sending notifications to relevant stakeholders based on specific ESG triggers.
5. Compliance and Reporting Systems:
   • ESG Reporting Platforms: Integrating with systems that automate ESG reporting to regulatory bodies.
   • Internal Compliance Systems: Ensuring adherence to internal ESG policies and regulatory requirements.
6. Integration with ESG Management Platforms:
   • ESG Management Software: Feeding data into platforms designed for comprehensive ESG performance management.
   • ESG Rating Agencies: Providing necessary data for third-party ESG assessments and ratings.

By directing Kafka-streamed ESG data to these sinks, organizations can effectively manage, analyze, and report on their environmental, social, and governance performance, fostering continuous improvement and transparency.

Real World Deployments with Kafka and Flink for a Better ESG Rating

The following case studies mainly focus on green energy, decarbonization and other relevant techniques to achieve a sustainable world. Social and governance aspects are as relevant and can leverage in similar ways from data streaming to provide accurate, curated, and consistent data in real-time.

Fun fact: Almost all the case studies come out of Europe. Would be great to see similar success stories about improved ESG in other regions.

EON: Smart Grid for Energy Production and Distribution with Cloud-Native Apache Kafka

EON explained to us already years ago in 2018 at a Confluent event in Munich how the mankind’s energy world is transforming:

From a linear pipe model…

• System-centric
• Fossil-fuel and nuclear
• Large-scale and central

… to a connected energy world…

• Green and clean technologies
• Smaller-scale, distributed
• Micro Grids, user centric

… with the EON Streaming Platform Smart Grid Infrastructure

The EON Streaming Platform is built on the following paradigms to provide a cloud-native smart grid infrastructure:

• IoT scale capabilities of public cloud providers
• EON microservices that are independent of cloud providers
• Real-time data integration and processing powered by Apache Kafka

aedifion: Efficient Management of Real Estate with MQTT and Kafka

The aedifion cloud platform is a solution for optimizing building portfolios. A simple digital software upgrade enables you to get an optimally managed portfolio and sustainably reduce your operating costs and CO₂ emissions. Connectivity includes Industrial IoT, smart sensors, MQTT, AMQP, HTTP, and others.

A few notes about the architecture of aedifion’s cloud platform:

• Digital, data-driven monitoring, analytics and controls products to operate buildings better and meet Environmental, Social, and Corporate Governance (ESG) goals
• Secure connectivity and reliable data collection with fully managed Confluent Cloud
• Deprecated MQTT-based pipeline – Kafka serves as a reliable buffer between producers (Edge Devices) and consumers (backend microservices) smoothing over bursts and temporary application downtimes

The latter point is very interesting. While Kafka and MQTT are a match made in heaven for many use cases, MQTT is not needed if you have a stable internet connection in a building and only hundreds of connections to devices. For bad networks or thousands of connections, MQTT is still the best choice for the last mile integration.

Ampeers Energy: Decarbonization for the Real Estate with OPC-UA and Confluent Cloud

The building sector is one of the major sources of climate-damaging CO₂ emissions. AMPEERS ENERGY achieves a significant CO₂ reduction of up to 90% – through intelligent software and established partners for the necessary hardware and services. The technology determines the optimal energy concept and automates processes to market locally generated energy, for example, to tenants.

The service provides district management with IoT-based forecasts and optimization, and local energy usage accounting. The real-time analytics of time-series data is implemented with OPC-UA, Confluent Cloud and TimescaleDB.

The solution is either deployed as fully managed services or with hybrid edge nodes if security or compliance requires this.

PAUL Tech AG: Energy Savings and Optimized ESG Ratings without PLC / SCADA

The PAUL Tech AG digitalizes central building systems by installing intelligent hardware in water-bearing systems. PAUL optimizes and automates volume flows by using artificial intelligence, resulting in significant energy and CO₂  savings.

The platform helps to optimize ESG ratings and enables energy savings up to 40 percent.

PAUL’s legacy architecture shows the traditional deployment patterns of the past using complex, expensive, proprietary PLCs and SCADA systems. The architecture is expensive, does not scale, and is hard to maintain:

PAUL’s cloud-native architecture is open and elastic. It does not require PLCs and SCADA systems. Instead, the open standards like MQTT and Sparkplug connect the Industrial IoT with the IT environments. Apache Kafka in the AWS cloud connects, streams and processes the IoT data and shares information with dashboards and other data management systems:

SBB: Trains use Adaptive Steering with Sustainability Effect

SBB (Swiss Federal Railways / Schweizerische Bundesbahnen in German) is the national railway company of Switzerland. It’s known for its extensive network of trains connecting various cities and regions within Switzerland, as well as international routes to neighboring countries.

The company uses adaptive steering for more punctuality and energy efficiency. This saves electricity and ensures greater punctuality and comfort.

The so called RCS module Adaptive Direction Control (ADL) aims to reduce power consumption by preventing unnecessary train stops and thus energy-intensive restarts. RCS-ADL calculates the energy-optimized speed and sends this to the train driver’s tablet.

Based on these calculations, RCS-ADL provides the corresponding speed recommendations. These are purely driving recommendations and not driving regulations. The external signals still have priority.

SBB creates the following sustainable benefits:

• Energy is produced by early electric braking.
• Energy is saved by avoiding stopping.
• No loss of time because of the acceleration phase.
• The train leaves the point of conflict at maximum speed.

The event-driven architecture is powered by Apache Kafka and Confluent. It consumes and correlates the following events:

• Speed recommendation to train driver
• Calculations of train speeds
• Recognition on unplanned stops
• Prognosis / statistical analysis on likelihood of stop
• Disposition system (Fleet management + train driver availability)
• Operating situation (Healthy, conflict situation, ..)
• Train steering
• Rolling stock material
• Train timetable
• Infrastructure data

SBB provides Open APIs for relevant train and customer information. Most endpoints are still API-based, i.e. HTTP/REST. Some streaming endpoints provide a direct public Kafka API:

Streaming data sharing is a common trend and complementary to existing API gateways and management tools like Mulesoft, Apigee or Kong. Read my article “Streaming Data Exchange with Kafka and a Data Mesh in Motion” for more details.

EverySens: Decarbonize Freight Transport with a Cloud-Native Transport Management System (TMS)

Everysens offers a Transport Visibility & Management System (TVMS): A new cloud-native generation of Transport Management System (TMS) integrating real-time visibility deep into the logistics processes of its customers.

Here are a few of the statistics of Everysens’ website:

The last bullet point impressively shows how real-time data streaming enables reduction of unnecessary carbonization. The solution avoided hundreds of thousands of truck rides. After building a cloud-native transport management system (TMS), merging visibility and TMS adds tremendous business value:

Everysens presented its data streaming journey at the Data in Motion Tour Paris 2023. Their evolution looks very similar to other companies. Data streaming with Kafka and Flink is a journey, not a race!

The technical architecture is not all that surprising. Kafka is the data hub and connects, streams, and processes events across the data pipeline:

Noteworthy that many sources and sinks are traditional relational databases like MySQL or PostgreSQL. One of the most underestimated features of Kafka is its event store and the capability to truly decouple systems and guarantee data consistency across real-time and non-real-time systems.

Powerledger: Green Energy Trading with Blockchain, Kafka and MongoDB

Powerledger develops software solutions for the tracking, tracing and trading of renewable energy. The solution is powered by Blockchain, Kafka and MongoDB for green energy trading.

The combination of data streaming and blockchain is very common in FinTech and cryptocurrency projects. Here are a few interesting facts about the platform:

• Blockchain-based energy trading platform
• Facilitating peer-to-peer (P2P) trading of excess electricity from rooftop solar power installations and virtual power plants
• Traceability with non-fungible tokens (NFTs) representing renewable energy certificates (RECs)
• Decentralised rather than the conventional unidirectional market
• Benefits: Reduced customer acquisition costs, increased customer satisfaction, better prices for buyers and sellers (compared with feed-in and supply tariffs), and provision for cross-retailer trading
• Apache Kafka via Confluent Cloud as a core piece of the integration infrastructure, specifically to ingest data from smart electricity meters and feed it into the trading system

Here is an example architecture for the integration between data streaming and trading via NFT tokens:

• Apache Kafka and Blockchain – Comparison and a Kafka-native Implementation
• Apache Kafka as Data Hub for Crypto, DeFi, NFT, Metaverse – Beyond the Buzz
• Apache Kafka in Crypto and FinServ for Cybersecurity and Fraud Detection

Streaming for a Better World: Unleashing ESG Potential with Kafka and Flink

This article explored how organizations can achieve Environmental, Social, and Governance (ESG) excellence through the utilization of innovative tools like Apache Kafka and Flink. By leveraging the real-time data processing capabilities of these platforms, companies can translate their sustainability aspirations into concrete actions, driving meaningful change and enhancing their ESG ratings across diverse industries.

The case studies showed the value of data streaming to improve ESG scores across use cases, such as managing real estate, monitoring IoT data streams, or trading renewable energy. In a similar way, project teams can evaluate how used software solutions and cloud services are committed to developing a strong ESG program and strategy to conduct business with integrity and build a sustainable future. As an example for data streaming, look at Confluent’s ESG Report to understand how ESG matters for many software vendors, too.

How do you build sustainable software projects to improve your ESG ratings? Is data streaming and real-time data correlation part of the story already? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Green Data, Clean Insights: How Kafka and Flink Power ESG Transformations appeared first on Kai Waehner.

MQTT Summit in Munich

In December 2023, I attended the MQTT Summit Connack. HiveMQ sponsored the event. The agenda included various industry experts. The talks covered industrial IoT deployments, unified namespace, Sparkplug B, security and fleet management, and use cases for Kafka combined with MQTT like connected vehicles or smart city (my talk).

It was a pleasure to meet many industry peers of the MQTT community, independent consultants, and software vendors. I learned a lot about the adoption of MQTT in the real world, best practices, and a few trade-offs of Sparkplug B. The following sections summarize my trends for MQTT of this event combined with experiences I had this year in customer meetings around the world.

Special thanks to Kudzai Manditereza of HiveMQ to organize this great event with many international attendees across industries:

What is MQTT?

MQTT stands for Message Queuing Telemetry Transport. MQTT is a lightweight and open-source messaging protocol designed for small sensors and mobile devices with high-latency or unreliable networks. IBM originally developed MQTT in the late 1990s and later became an open standard.

MQTT follows a publish/subscribe model, where devices (or clients) communicate through a central message broker. The key components in MQTT are:

1. Client: The device or application that connects to the MQTT broker to send or receive messages.
2. Broker: The central hub that manages the communication between clients. It receives messages from publishing clients and routes them to subscribing clients based on topics.
3. Topic: A hierarchical string that acts as a label for a message. Clients subscribe to topics to receive messages and publish messages to specific topics.

When to use MQTT?

The publish/subscribe model allows for efficient communication between devices. When a client publishes a message to a specific topic, all other clients subscribed to that topic receive the message. This decouples the sender and receiver, enabling a scalable and flexible communication system.

The MQTT standard is known for its simplicity, low bandwidth usage, and support for unreliable networks. These characteristics make it well-suited for Internet of Things (IoT) applications, where devices often have limited resources and may operate under challenging network conditions. Good MQTT implementations provide a scalable and reliable platform for IoT projects.

MQTT has gained widespread adoption in various industries for IoT deployments, home automation, and other scenarios requiring lightweight and efficient communication.

I discuss the following four market trends for MQTT in the following sections. These have huge impact on the adoption and making a decision to choose MQTT:

1. MQTT in the Public Cloud
2. Data Governance for MQTT
3. MQTT vs. OPC-UA Debates
4. MQTT and Apache Kafka for OT/IT Data Processing

Trend 1: MQTT in the Public Cloud

Most companies have a cloud first strategy. Go serverless if you can! Focus on business problems, faster time-to-market, and an elastic infrastructure are the consequence.

Mature MQTT cloud services exist. At Confluent, we work a lot with HiveMQ. The combination even provides a fully managed integration between both cloud offerings.

Having said that, not everything can or should go to the (public) cloud. Security, latency and cost often make a deployment in the data center or at the edge (e.g., in a smart factory) the preferred or mandatory option. Hybrid architectures allow the combination of both options for building the most cost-efficient but also reliable and secure IoT infrastructure. I talked about zero-trust and air-gapped environments leveraging unidirectional hardware for the most critical use cases in another blog..

Automation and Security are the Typical Blockers for Public Cloud

Key for success, especially in hybrid architectures, is automation and fleet management with CI/CD and GitOps for multi-cluster management. Many projects leverage Kubernetes as a cloud-native infrastructure for the edge and private cloud. However, in the public cloud, the first option should always be a fully managed service (if security and other requirements allow it).

Be careful when adopting fully-managed MQTT cloud services: Support for MQTT is not always equal across the cloud vendors. Many vendors do not implement the entire protocol, miss features, and require usage limitations. HiveMQ wrote a great article showing this. The article is a bit outdated (and opinionated, of course, as a competing MQTT vendor). But it shows very well how some vendors provide offerings that are far away from a good MQTT cloud solution.

The hardest problem for public cloud adoption of MQTT is security! Double check the requirements early. Latency, availability or specific features are usually not the problem. The deployment and integration need to be compliant and follow the cloud strategy. As Industrial IoT projects always have to include some kind of edge story, it is a tougher discussion than sales or marketing projects.

Trend 2: Data Governance for MQTT

Data governance is crucial across the enterprise. From an IoT and MQTT perspective, the two main topics are unified namespace as the concept and Sparkplug B as the technology.

Unified Namespace for Industrial IoT

In the context of Industrial Internet of Things (IIoT), a unified namespace (UNS) typically refers to a standardized and cohesive way of naming and organizing devices, data, and resources within an industrial network or ecosystem. The goal is to provide a consistent naming structure that facilitates interoperability, data sharing, and management of IIoT devices and systems.

The term Unified Namespace (in Industrial IoT) was coined and popularized by Walker Reynolds, an expert and content creator for Industrial IoT.

Concepts of Unified Namespace

Here are some key aspects of a unified namespace in Industrial IoT:

1. Device Naming: Devices in an IIoT environment may come from various manufacturers and have different functionalities. A unified namespace ensures that devices are named consistently, making it easier for administrators, applications, and other devices to identify and interact with them.
2. Data Naming and Tagging: IIoT involves the generation and exchange of vast amounts of data. A unified namespace includes standardized naming conventions and tagging mechanisms for data points, variables, or attributes associated with devices. This consistency is crucial for applications that need to access and interpret data across different devices.
3. Interoperability: A unified namespace promotes interoperability by providing a common framework for devices and systems to communicate. When devices and applications follow the same naming conventions, it becomes easier to integrate new devices into existing systems or replace components without causing disruptions.
4. Security and Access Control: A well-defined namespace contributes to security by enabling effective access control mechanisms. Security policies can be implemented based on the standardized names and hierarchies, ensuring that only authorized entities can access specific devices or data.
5. Management and Scalability: In large-scale industrial environments, having a unified namespace simplifies device and resource management. It allows for scalable solutions where new devices can be added or replaced without requiring extensive reconfiguration.
6. Semantic Interoperability: Beyond just naming, a unified namespace may include semantic definitions and standards. This helps in achieving semantic interoperability, ensuring that devices and systems understand the meaning and context of the data they exchange.

Overall, a unified namespace in Industrial IoT is about establishing a common and standardized structure for naming devices, data, and resources, providing a foundation for efficient, secure, and scalable IIoT deployments. Standards organizations and industry consortia often play a role in developing and promoting these standards to ensure widespread adoption and compatibility across diverse industrial ecosystems.

Sparkplug B: Interoperability and Standardized Communication for MQTT Topics and Payloads

Unified Namespace is the theoretical concept for interoperability. The standardized implementation for payload structure enforcement is Sparkplug B. This is a specification created at the Eclipse foundation and turned into an ISO standard later.

Sparkplug B provides a set of conventions for organizing data and defining a common language for devices to exchange information. Here is an example of HiveMQ depicting how a unified namespace makes communication between devices, systems, and sites easier:

Key features of Sparkplug B include:

1. Payload Structure: Sparkplug B defines a specific format for the payload of MQTT messages. This format includes fields for information such as timestamp, data type, and value. This standardized payload structure ensures that devices can consistently understand and interpret the data being exchanged.
2. Topic Namespace: The specification defines a standardized topic namespace for MQTT messages. This helps in organizing and categorizing messages, making it easier for devices to discover and subscribe to relevant information.
3. Birth and Death Certificates: Sparkplug B introduces the concept of “Birth” and “Death” certificates for devices. When a device comes online, it sends a Birth certificate with information about itself. Conversely, when a device goes offline, it sends a Death certificate. This mechanism aids in monitoring the status of devices within the IIoT network.
4. State Management: The specification includes features for managing the state of devices. Devices can publish their current state, and other devices can subscribe to receive updates. This helps in maintaining a synchronized view of device states across the network.

Sparkplug B is intended to enhance the interoperability, scalability, and efficiency of IIoT deployments by providing a standardized framework for MQTT communication in industrial environments. Its adoption can simplify the integration of diverse devices and systems within an industrial ecosystem, promoting seamless communication and data exchange.

Limitations of Sparkplug B

Sparkplug B has a few limitations, such as:

• Only supports Quality of Service (QoS) 0 providing at most once message delivery guarantees.
• Limits in the structure of topic namespaces.
• Very device centric (but MQTT is for many “things”)

Understand the pros and cons of Sparkplug B. It is perfect for some use cases. But the above limitations are blockers for some others. Especially, only supporting QoS 0 is a huge limitation for mission-critical use cases.

Trend 3: MQTT vs. OPC-UA Debates

MQTT has many benefits compared to other industrial protocols. However, OPC-UA is another standard in the IoT space that gets at least as much traction in the market as MQTT. The debate about choosing the right IoT standard is controversial, often led by emotions and opinions, and still absolutely valid to discuss.

OPC-UA (Open Platform Communications Unified Architecture) is a machine-to-machine communication protocol for industrial automation. It enables seamless and secure communication and data exchange between devices and systems in various industrial settings.

OPC UA has become a widely adopted standard in the industrial automation and control domain, providing a foundation for secure and interoperable communication between devices, machines, and systems. Its open nature and support from industry organizations contribute to its widespread use in applications ranging from manufacturing and process control to energy management and more.

If you look at the promises of MQTT and OPC-UA, a lot of overlapping exists:

• Scalable
• Reliable
• Real-time
• Open
• Standardized

All of them are true for both standards. Still, trade-offs exist. I won’t start a flame war here. Just search for “MQTT vs. OPC-UA”. You will find many blog posts, articles and videos. Most are very opinionated (and often driven by a vendor). Reality is that the industry adopted both MQTT and OPC-UA widely.

And while the above characteristics might all be true for both standards in general, the details make the difference for specific implementations. For instance, if you try to connect plenty of Siemens S3 PLCs via OPC-UA, then you quickly realize that the number of parallel connections is not as scalable as the OPC-UA standard specification tells you.

When to Choose MQTT vs. OPC-UA?

The clear recommendation is starting with the business problem, not the technology. Evaluate both standards and their implementations, supported interfaces, vendors cloud services, etc. Then choose the right technology.

Here is what I use as a simplified rule of thumb if you have to start a technical discussion:

• MQTT: Use cases for connected IoT devices, vehicles, and other interfaces with support for lightweight infrastructure, large number of connections, and/or bad networks.
• OPC-UA: Use cases for industrial automation to connect heavy equipment, PLCs, SCADA systems, data historians, etc.

This is just a rule of thumb. And the situation changes. Modern PLCs and other equipment add support for multiple protocols to be more flexible. But, nowadays, you rarely have an option anyway because specific equipment, devices, or vehicles only support one or the other. And you can still be happy: Otherwise, you need to use another IIoT platform to connect to proprietary legacy protocols like S3, Modbus, et al.

MQTT and OPC-UA Gotchas

A few additional gotchas I realized from various customer conversations around the world in the past quarters:

• In theory, MQTT and OPC-UA work well together, i.e., MQTT is the underlying transportation protocol for OPC-UA. I did not see this yet in the real world (no statistical evidence, just my personal experience). But what I see is the combination of OPC-UA for the last mile integration to the PLC and then forwarding the data to other consumers via MQTT. All in a single gateway, usually a proprietary IoT platform.
• OPC-UA defines many sub-standards for different industries or use cases. In theory, this is great. In practice, I see this more like the WS-* hell in the SOAP/WSDL web service world where most projects moved to a much simpler HTTP/REST architectures. Similarly, most integrations I see to OPC-UA use simple, custom-coded clients in Java or other programming languages – because the tools don’t support the complex standards.
• IoT vendors pitch any possible integration scenario in marketing. I am amazed that MQTT and OPC-UA platforms directly integrate with MES and ERP system like SAP, and any data warehouse and data lake, like Google Big Query, Snowflake, or Databricks. But that’s only the theory. Should you really do this? And did you ever try to connect SAP ECC to MQTT or OPC-UA? Good luck from a technical, and even harder, from an organizational perspective. And do you want tight coupling and point-to-point communication in between the OT world and the ERP? In most cases, it is a good thing to have a clear separation of concerns between different business units, domains, and use cases. Choose the right tool and enterprise architecture; not just for the POC and first pipeline, but for the entire long-term strategy and vision.

The last point brings me to another growing trend: The combination of MQTT for IoT / OT workloads and data streaming with Apache Kafka for the integration with the IT world.

Trend 4: MQTT and Apache Kafka for OT/IT Data Processing

Contrary to MQTT, Apache Kafka is NOT an IoT platform. Instead, Kafka is an event streaming platform and used the underpinning of an event-driven architecture for various use cases across industries. It provides a scalable, reliable, and elastic real-time platform for messaging, storage, data integration, and stream processing. Apache Kafka and MQTT are a perfect combination for many IoT use cases.

Let’s explore the pros and cons of both technologies from the IoT perspective.

Trade-offs of MQTT

MQTT’s pros:

• Lightweight
• Built for thousands of connections
• All programming languages supported
• Built for poor connectivity / high latency scenarios
• High scalability and availability (depending on broker implementation)•ISO Standard
• Most popular IoT protocol (competing with OPC UA)

MQTT’s cons:

• Adoption mainly in IoT use cases
• Only pub/sub, not stream processing
• No reprocessing of events

I wrote a blog series about MQTT in conjunction with Apache Kafka with many more technical details and real-world case studies across industries.

The first blog post explores the relation between MQTT and Apache Kafka. Afterward, the other four blog posts discuss various use cases, architectures, and reference deployments.

• Part 1 – Overview: Relation between Kafka and MQTT, pros and cons, architectures
• Part 2 – Connected Vehicles: MQTT and Kafka in a private cloud on Kubernetes; use case: remote control and command of a car
• Part 3 – Manufacturing: MQTT and Kafka at the edge in a smart factory; use case: Bidirectional OT-IT integration with Sparkplug B between PLCs, IoT Gateways, Data Historian, MES, ERP, Data Lake, etc.
• Part 4 – Mobility Services: MQTT and Kafka leveraging serverless cloud infrastructure; use case: Traffic jam prediction service using machine learning
• Part 5 – Smart City: MQTT at the edge connected to fully-managed Kafka in the public cloud; use case: Intelligent traffic routing by combining and correlating different 1st and 3rd party services

The following presentation is from my talk at the MQTT Summit. It explores various use cases and reference architectures for MQTT and Apache Kafka:

If you have a bad network, tens of thousands of clients, or the need for a lightweight push-based messaging solution, then MQTT is the right choice. Elsewhere, Kafka, a powerful event streaming platform, is probably the right choice for real-time messaging, data integration, and data processing. In many IoT use cases, the architecture combines both technologies. And even in the industrial space, various projects use Kafka for use cases like building a cloud-native data historian or real-time condition monitoring and predictive maintenance.

Data Governance for MQTT with Sparkplug and Kafka (and Beyond)

Unified Namespace and the concrete implementation with Sparkplug B is excellent for data governance in IoT workloads with MQTT. In a similar way, the Schema Registry defines the data contracts for Apache Kafka data pipelines.

Schema Registry should be the foundation of any Kafka project! Data contracts (aka Schemas, similar to Swagger in REST/HTTP APIs) enforce good data quality and interoperability between independent microservices in the Kafka ecosystem. Each business unit and its data products can choose any technology or API. But data sharing with others works only with good (enforced) data quality.

You can see the issue: Each technology uses its own data governance technology. If you add your favorite data lake, you will add another concept, like Apache Iceberg, to define the data tables for analytics storage systems. And that’s okay! Each data governance suite is optimized for its workloads and requirements. A company-wide master data management failed in the last two decades because each software category has different requirements.

Hence, one clear trend I see is an enterprise-wide data governance strategy across the different systems (with technologies like Collibra or Azure Purview). It has open interfaces and integrates with specific data contracts like Sparkplug B for MQTT, Schema Registry for Kafka, Swagger for HTTP/REST applications,  or Iceberg for data lakes. Don’t try to solve the entire enterprise-wide data governance strategy with a single technology. It will fail! We have seen this before…

Legacy PLC (S7, Modbus, BACnet, etc.) with MQTT or Kafka?

MQTT and Kafka enable reliable and scalable end-to-end data pipelines between IoT and IT systems. At least, if you can use modern APIs and standards. Most IoT projects today are still brownfield. A lot of legacy PLCs, SCADA systems, and data historians only support proprietary protocols like Siemens S7, Modbus, BACnet, and so on.

MQTT or Kafka don’t support these legacy protocols out-of-the-box. Another middleware is required. Usually, enterprises choose a dedicated IoT platform for this. That means more cost and complexity, and slower projects.

In the Kafka world, Apache PLC4X is a great open source option if you want to build a modern, cloud-native data historian with Kafka. The framework provides integration with many legacy protocols. And it offers a Kafka Connect connector. The main issue is no official vendor support behind. Companies cannot buy support with a 24/7 business model for mission-critical applications. And that’s typically a blocker for any industrial deployment.

As MQTT is only a pub/sub message broker, it cannot help with legacy protocol integration. HiveMQ tries to solve this challenge with a new framework called HiveMQ Edge: A software-based industrial edge protocol converter. It is a young project and just kicking off. The core is open source. The first supported legacy protocol is Modbus. I think this is an excellent product strategy. I hope the project gets traction and evolves to support many other legacy IIoT technologies to modernize the brownfield shop floor. The project actually also supports OPC-UA. We will see how much demand that feature creates, too.

MQTT and Sparkplug Adoption Grows Year-By-Year for IoT Use Cases

In the IoT world, MQTT and OPC UA have established themselves as open and platform-independent standards for data exchange in Industrial IoT and Industry 4.0 use cases. Data Streaming with Apache Kafka is the data hub for integrating and processing massive volumes of data at any scale in real-time. The “Trinity of Data Streaming in IoT explores the combination of MQTT, OPC-UA and Apache Kafka” in more detail.

MQTT adoption grows year by year with the need for more scalable, reliable and open IoT communication between devices, equipment, vehicles, and the IT backend. The sweet spots of MQTT are unreliable networks, lightweight (but reliable and scalable) communication and infrastructure, and connectivity to thousands of things.

Maturing trends like the Unified Namespace with Sparkplug B, fully managed cloud services, and combined usage with Apache Kafka make MQTT one of the most relevant IoT standards across verticals like manufacturing, automotive, aviation, logistics, and smart city.

But don’t get fooled by architecture pictures and theory. For example, most diagrams for MQTT and Sparkplug show integrations with the ERP (e.g., SAP) and Data Lake (e.g., Snowflake). Should you really integrate directly from the OT world into the analytics platform? Most times, the answer is no because of cost, decoupling of business units, legal issues, and other reasons. This is where the combination of MQTT and Kafka (or another integration platform) shines.

How do you use MQTT and Sparkplug today? What are the use cases? Do you combine it with other technologies, like Apache Kafka, for end-to-end integration across the OT/IT pipeline? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post MQTT Market Trends: Cloud, Unified Namespace, Sparkplug, Kafka Integration appeared first on Kai Waehner.

I look at trends in the insurance sector to explore how data streaming helps as a business enabler, including customer stories from Allianz, Generali, Policygenius, and more. A complete slide deck and on-demand video recording are included.

General trends in the insurance industry

The insurance industry is fundamental for various aspects of modern life and society, including financial protection, healthcare, life insurance, property protection, business continuity, transportation and travel, etc.

McKinsey & Company expects that “major shifts in insurance operating models are under way” in the following years through tech-driven innovation:

Data streaming in the insurance industry

Adopting trends like claim automation, expediting payloads, virtual assistants, or usage-based insurance is only possible if enterprises in the insurance sector can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means):

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

“Apache Kafka in the Insurance Industry” is a great starting point to learn more about data streaming in the industry, including a few Kafka-powered case studies not covered in this blog post – such as

• Centene: Integration and data processing at scale in real-time
• Swiss Mobiliar: Decoupling and orchestration
• Humana: Real-time data integration and analytics
• freeyou: Stateful streaming analytics
• Tesla: Carmaker and utility company, now also a car insurer

Architecture trends for data streaming

The insurance industry applies various trends for enterprise architectures for cost, flexibility, security, and latency reasons. The three major topics I see these days at customers are:

• Focus on business logic and faster time-to-market by leveraging fully managed SaaS
• Event-driven architectures (in combination with request-response communication) to enable domain-driven design and independent innovation
• Omnichannel customer services across the customer lifecycle

Let’s look deeper into some enterprise architectures that leverage data streaming for insurance use cases

Event-driven patterns for insurance

Domain-driven design with decoupled applications and microservices is vital to enabling change and innovation in the enterprise architecture of insurers. Apache Kafka’s unique capabilities to provide real-time messaging at scale plus storing events as long as needed for true decoupling creates independence for business units.

Business processes are not orchestrated by a monolithic application or middleware. Instead, teams build decentralized data products with their technologies, APIs, SaaS, and communication paradigms like streaming, batch, or request-response:

As the heart of the enterprise architecture, data streaming is real-time, scalable, and reliable. The streaming platform collects, stores, processes, and shares data. But the underlying event store makes integrating with non-real-time systems like a legacy file-based application on the mainframe or the cloud-native data lake, data warehouse, or lakehouse possible.

Omnichannel customer journey

Customers expect consistent information across all channels, including mobile apps, web browsers, brick-and-mortar agencies, and 3rd party partner services. Kafka’s event log of the streaming platform ensures that historical and current events can be correlated to make the right automated decision, alert, recommendation, etc.

The following diagram shows how historical marketing data from a SaaS platform is combined with other historical data from the product configurator to feed the real-time decisions of the salesperson in the brick-and-mortar agency when the customer is entering the store:

Data consistency is one of the most challenging problems in the IT and software world. Apache Kafka ensures data consistency across all applications and databases, whether these systems operate in real-time, near-real-time, or batch.

New customer stories for data streaming in the insurance sector

So much innovation is happening in the insurance sector. Automation and digitalization change how insurers process claims, build customer relationships, and create new business models with enterprises of other verticals.

Most insurance providers use a cloud-first approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure. And elastic scalability gets even more critical with all the growing real-time expectations and mobile app capabilities.

Here are a few customer stories from worldwide insurance organizations:

• Generali built a critical integration platform between legacy databases on-premises and cloud-native workloads running on Google Cloud.
• Allianz has separate projects for brownfield integration and IT modernization and greenfield innovation for real-time analytics to implement use cases like pricing for insurance products, analytics for customer 360 views, and fraud management.
• Ladder leverages machine learning in real-time for an online, direct-to-consumer, full-stack approach to life insurance.
• Roosevelt built an agnostic claim processing and B2B platform to automatically process the vast majority of claims for over 20 million members, saving hundreds of millions of dollars in treatment costs…
• Policygenius is an online insurance marketplace with 30+ million customers for life, disability, home, and auto insurance; the data streaming platform checks with real agents to deliver quotes from leading insurance companies in real-time.

Resources to learn more

This blog post is just the starting point. Learn more about data streaming in the insurance industry in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-demand video recording

The video recording explores the telecom industry’s trends and architectures for data streaming. The primary focus is the data streaming case studies.

I am excited to have presented this webinar in my interactive light board studio:

This creates a much better experience, especially in a time after the pandemic where many people are “Zoom fatigue”.

Check out our on-demand recording:

Slides

If you prefer learning from slides, check out the deck used for the above recording:

Case studies and lightboard videos for data streaming in the insurance industry

The state of data streaming for insurance is fascinating. New use cases and case studies come up every month. This includes better data governance across the entire organization, real-time data collection and processing data across legacy applications in the data center and modern cloud infrastructures, data sharing and B2B partnerships for new business models, and many more scenarios.

We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories. Stay tuned; I will update the links in the next few weeks.

And this is just the beginning. Every month, we will talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, telcos, gaming, and so on… Check out my other blog posts.

Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post The State of Data Streaming for Insurance appeared first on Kai Waehner.

The State of Data Streaming for Manufacturing in 2023

The evolution of industrial IoT, manufacturing 4.0, and digitalized B2B and customer relations require modern, open, and scalable information sharing. Data streaming allows integrating and correlating data in real-time at any scale. Trends like software-defined manufacturing and data streaming help modernize and innovate the entire engineering and sales lifecycle.

I have recently presented an overview of trending enterprise architectures in the manufacturing industry and data streaming customer stories from BMW, Mercedes, Michelin, and Siemens. A complete slide deck and on-demand video recording are included:

This blog post explores one of the enterprise architectures and case studies in more detail: Modernization of legacy and proprietary monoliths and SCADA systems to a scalable, open platform with real-time data integration capabilities.

What is a SCADA System? And how does Data Streaming help?

Supervisory control and data acquisition (SCADA) is a control system architecture comprising computers, networked data communications, and graphical user interfaces for high-level supervision of machines and processes. It also covers sensors and other devices, such as programmable logic controllers, which interface with process plants or machinery.

Data streaming helps connect high-volume sensor data from machines, PLCs, robots, and other IoT devices. This is possible in real-time at scale with stream processing. The de facto standard for data streaming is Apache Kafka and its ecosystems, like Kafka Stream and Kafka Connect.

Enterprises leverage Apache Kafka as the next generation of Data Historians. Integrating and pre-processing the events with data streaming is a prerequisite for data correlation with information systems like the MES or ERP (that might run at the edge or more often in the cloud).

50hertz: A cloud-native SCADA system built with Apache Kafka

50hertz is a transmission system operator for electricity in Germany. The company secures electricity supply to 18 million people in northern and eastern Germany.

The infrastructure must operate 24 hours, seven days a week. Various shift teams and a mission-critical SCADA infrastructure supervise and control the OT systems.

50hertz next-generation Modular Control Center System (MCCS) leverages a central, scalable, event-based integration platform based on Confluent:

The first four containers include the Supervisory & Control (SCADA), Load Frequency Control (LFC), and Time Series Management & Forecasting applications. Each container can have multiple services/functions that follow the event-based microservices pattern.

50hertz provides central governance for security, protocols, and data schemas (CIM compliant) between platform containers/ modules. The cloud-native 24/7 SCADA system is developed in the cloud and deployed in safety-critical edge environments.

50hertz presented their OT/IT and SCADA modernization leveraging data streaming with Apache Kafka at the Confluent Data in Motion tour 2021. Unfortunately, the on-demand video recording is available only in German. Therefore, in another blog post, I wrote more about the case study: “A cloud-native SCADA System for Industrial IoT built with Apache Kafka“.

Lightboard Video: How Data Streaming Modernizes SCADA and OT/IT

Here is a five-minute lightboard video that describes how data streaming helps with modernizing monolith and proprietary SCADA infrastructure and OT/IT environments:

If you liked this video, make sure to follow the YouTube channel for many more lightboard videos across all industries.

Apache Kafka glues together the old and new OT/IT World

The 50Hertz case study showed how to modernize an existing legacy infrastructure with cloud-native technologies, whether you deploy at the edge or in the public cloud. For more case studies, check out the free “The State of Data Streaming in Manufacturing” on-demand recording or read the related blog post.

A common question in these scenarios is the proper communication and integration protocol when you move away from proprietary legacy PLCs and OT interfaces. MQTT and OPC-UA established themselves as excellent standards with different sweet spots. Data Streaming with Apache Kafka is complementary, not competitive. Learn more by reading “OPC UA, MQTT, and Apache Kafka – The Trinity of Data Streaming in IoT“.

How do you leverage data streaming in your manufacturing use cases? Do you deploy at the edge, in the cloud, or both? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post Modernizing SCADA Systems and OT/IT Integration with Data Streaming appeared first on Kai Waehner.

I look at trends in the utilities sector to explore how data streaming helps as a business enabler, including customer stories from SunPower, 50hertz, Powerledger, and more. A complete slide deck and on-demand video recording are included.

General trends in the energy & utilities industry

The energy & utilities industry is fundamental for a sustainable future. Garter explores the Top 10 Trends Shaping the Utility Sector in 2023: “In 2023, power and water utilities will continue to face a variety of forces that will challenge their business and operating models and shape their technology investments.

Utility technology leaders must confidently compose the future for their organizations in the midst of uncertainty during this energy transition volatile period — the future that requires your organizations to be both agile and resilient.”

From system-centric and large to smaller-scale and distributed

The increased use of digital tools makes the expected structural changes in the energy system possible:

Energy AI use cases

Artificial Intelligence (AI) with technologies like Machine Learning (ML) and Generative AI (GenAI) is a hot topic across all industries. Innovation around AI disrupts many business models, tasks, business processes, and labor.

NVIDIA created an excellent diagram showing the various opportunities for AI in the energy & utilities sector. It separates the scenarios by segment: upstream, midstream, downstream, power generation, and power distribution:

Cybersecurity: The threat is real!

McKinsey & Company explains that “the cyberthreats facing electric-power and gas companies include the typical threats that plague other industries: data theft, billing fraud, and ransomware. However, several characteristics of the energy sector heighten the risk and impact of cyberthreats against utilities:”

Data streaming in the energy & utilities industry

Adopting trends like predictive maintenance, track&trace, proactive sales and marketing, or threat intelligence is only possible if enterprises in the energy sector can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means):

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

“Apache Kafka for Smart Grid, Utilities and Energy Production” is a great starting point to learn more about data streaming in the industry, including a few case studies not covered in this blog post – such as

• EON: Smart grid for energy production and distribution with Apache Kafka
• Devon Energy: Kafka at the edge for hybrid integration and analytics in the cloud
• Tesla: Kafka-based data platform for trillions of data points per day

5 Ways Utilities Accomplish More with Real-Time Data

“After creating a collaborative team that merged customer experience and digital capabilities, one North American utility went after a 30 percent reduction in its cost-to-serve customers in some of its core journeys.”

As the Utilities Analytics Institute explains: “Utilities need to ensure that the data they are collecting is high quality, specific to their needs, preemptive in nature, and, most importantly, real-time.” The following five characteristics are crucial to add value with real-time data:

1. High-Quality Data
2. Data Specific to Your Needs
3. Make Your Data Proactive
4. Data Redundancy
5. Data is Constantly Changing

Real-Time Data for Smart Meters and Common Praxis

Smart meters are a perfect example of increasing business value with real-time data streaming. As Clou Global confirms: “The use of real-time data in smart grids and smart meters is a key enabler of the smart grid“.

Possible use cases include:

1. Load Forecasting
2. Fault Detection
3. Demand Response
4. Distribution Automation
5. Smart Pricing

Processing and correlating events from smart meters with stream processing is just one IoT use case. You can leverage “Apache Kafka and Apache Flink for many Industrial IoT and Manufacturing 4.0 use cases“.

And there is so much more if you expand your thinking from upstream through midstream to downstream applications to “transform the global supply chain with data streaming and IoT“.

Cloud adoption in utilities & energy sector

Accenture points out that 84% use Cloud SaaS solutions and 79% use Cloud PaaS Solutions in the energy & utilities market for various reasons:

• New approach to IT
• Incremental adoption
• Improved scalability, efficiency, agility and security
• Unlock most business value

This is a general statistic, but this applies to all components in the data-driven enterprise, including data streaming. A company does not just move a specific application to the cloud; this would be counter-intuitive from a cost and security perspective. Hence, most companies start with a hybrid architecture and bring more and more workloads to the public cloud.

Architecture trends for data streaming

The energy & utilities industry applies various trends for enterprise architectures for cost, flexibility, security, and latency reasons. The three major topics I see these days at customers are:

• Global data streaming
• Edge computing and hybrid cloud integration
• OT/IT modernization

Let’s look deeper into some enterprise architectures that leverage data streaming for energy & utilities use cases.

Global data streaming across data centers, clouds and the edge

Energy and utilities require data infrastructure everywhere. While most organizations have a cloud-first strategy, there is no way around running some workloads at the edge outside a data center for cost, security, or latency reasons.

Data streaming is available everywhere:

Data synchronization across environments, regions and clouds is possible with open-source Kafka tools like MirrorMaker. However, this requires additional infrastructure and development/operations efforts. Innovative solutions like Confluent’s Cluster Linking leverage the Kafka protocol for real-time replication. This enables much easier deployments and significantly reduced network traffic.

Edge computing and hybrid cloud integration

Kafka deployments look different depending on where it needs to be deployed.

Fully managed serverless offerings like Confluent Cloud are highly recommended in the public cloud to focus on business logic with reduced time-to-market and TCO.

In a private cloud, data center or edge environment, most companies deploy on Kubernetes today to provide a similar cloud-native experience.

Kafka can also be deployed on industrial PCs (IPC) and other industrial hardware. Many use cases exist for data streaming at the edge. Sometimes, a single broker (without high availability) is good enough.

No matter how you deploy data streaming workloads, a key value is the unidirectional or bidirectional synchronization between clusters. Often, only curated and relevant data is sent to the cloud for cost reasons. Also, command & control patterns can start a business process in the cloud and send events to the edge.

OT/IT modernization with data streaming

The energy sector operates many monoliths, inflexible and closed software and hardware products. This is changing in this decade. OT/IT modernization and the digital transformation require open APIs, flexible scale, and decoupled applications (from different vendors).

Many companies leverage Apache Kafka to build a postmodern data historian to complement or replace existing expensive OT middleware:

Just to be clear: Kafka and any other IT software like Spark, Flink, Amazon Kinesis, and so on are NOT hard real-time. It cannot be used for safety-critical use cases with deterministic systems like autonomous driving or robotics. That is C, Rust, or other embedded software.

However, data streaming connects the OT and IT worlds. As part of that, connectivity with robotic systems, intelligent vehicles, and other IoT devices is the norm for improving logistics, integration with ERP and MES, aftersales, etc.

Learn more about this discussion in two articles:

• Apache Kafka for the integration with OT and IoT devices in Automotive and Industrial IoT
• Apache Kafka for real-time trading use cases (like energy trading)

New customer stories for data streaming in the energy & utilities sector

So much innovation is happening in the energy & utilities sector. Automation and digitalization change how utilities monitor infrastructure, build customer relationships, and create completely new business models.

Most energy service providers use a cloud-first approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure. And elastic scalability gets even more critical with all the growing networks, 5G workloads, autonomous vehicles, drones, and other innovations.

Here are a few customer stories from worldwide energy & utilities organizations:

• 50hertz: A grid operator modernization of the legacy, monolithic and proprietary SCADA infrastructure to cloud-native microservices and a real-time data fabric powered by data streaming. More details: A cloud-native SCADA System for Industrial IoT built with Apache Kafka.
• SunPower: Solar solutions across the globe where 6+ million devices in the field send data to the streaming platform. However, sensor data alone is not valuable! Fundamentals for delivering customer value include measurement ingestion, metadata association, storage, and analytics.
• aedifion: Efficient management of real estate to operate buildings better and meet environmental, social, and corporate governance (ESG) goals. Secure connectivity and reliable data collection are implemented with Confluent Cloud (and deprecated the existing MQTT-based pipeline).
• Ampeers Energy: Decarbonization for the real estate. The service provides district management with IoT-based forecasts and optimization, and local energy usage accounting. The real-time analytics of time-series data is implemented with OPC-UA, Confluent Cloud and TimescaleDB.
• Powerledger: Green energy trading with blockchain-based tracking, tracing and trading of renewable energy from rooftop solar power installations and virtual power plants. Non-fungible tokens (NFTs) representing renewable energy certificates (RECs) in. a decentralised rather than the conventional unidirectional market. Confluent Cloud ingests data from smart electricity meters. Learn more: data streaming and blockchain.

Resources to learn more

This blog post is just the starting point. Learn more about data streaming in the energy & utilities industry in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-demand video recording

The video recording explores the telecom industry’s trends and architectures for data streaming. The primary focus is the data streaming case studies. Check out our on-demand recording:

Slides

If you prefer learning from slides, check out the deck used for the above recording:

Case studies and lightboard videos for data streaming in the energy & utilities industry

The state of data streaming for energy & utilities is fascinating. New use cases and case studies come up every month. This includes better data governance across the entire organization, real-time data collection and processing data across hybrid edge and cloud infrastructures, data sharing and B2B partnerships for new business models, and many more scenarios.

We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories. Stay tuned; I will update the links in the next few weeks and publish a separate blog post for each story and lightboard video.

And this is just the beginning. Every month, we will talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, telcos, gaming, and so on… Check out my other blog posts.

Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post The State of Data Streaming for Energy & Utilities appeared first on Kai Waehner.

The State of Data Streaming for Manufacturing in 2023

The evolution of industrial IoT, manufacturing 4.0, and digitalized B2B and customer relations require modern, open, and scalable information sharing. Data streaming allows integrating and correlating data in real-time at any scale. Trends like software-defined manufacturing and data streaming help modernize and innovate the entire engineering and sales lifecycle.

I have recently presented an overview of trending enterprise architectures in the manufacturing industry and data streaming customer stories from BMW, Mercedes, Michelin, and Siemens. A complete slide deck and on-demand video recording are included:

This blog post explores one of the enterprise architectures and case studies in more detail: Data streaming between edge infrastructure (like a smart factory) and applications in the data center or public cloud.

What is a Smart Factory? And how does Data Streaming help?

Smart Factory is a term from research in manufacturing technology. It refers to the vision of a production environment in which manufacturing plants and logistics systems primarily organize themselves without human intervention to produce the desired products.

The technical basis is cyber-physical systems, i.e., physical manufacturing objects and virtual images in a centralized system. Digital Twins often play a crucial role in smart factories for simulation, engineering, condition monitoring, predictive maintenance, and other tasks.

In the broader context, the Internet of Things (IoT) is the foundation of a smart factory. Communication between the product (e.g., workpiece) and the manufacturing system continues to be part of this future scenario: The product brings its manufacturing information in machine-readable form, e.g., on an RFID chip. This data controls the product’s path through the production system and the individual production steps. Other transmission technologies, such as WLAN, Bluetooth, color coding, or QR codes, are also being experimented with.

Data streaming helps connect high-volume sensor data from machines, PLCs, robots, and other IoT devices. Integrating and pre-processing the events with data streaming is a prerequisite for data correlation with information systems like the MES or ERP (that might run at the edge or more often in the cloud). The latter is possible in real-time at scale with stream processing. The de facto standard for data streaming is Apache Kafka and its ecosystems, like Kafka Stream and Kafka Connect.

BMW Group: Data from 30 Smart Factories Streamed to the Cloud

BMW Group needed to make all data generated by its 30+ production facilities and worldwide sales network available in real-time to anyone across the global business.

The data ingested by BMW from its smart factories into the cloud with data streaming enables simple access to the data for visibility and new automation applications by any business unit.

The Apache Kafka ecosystem facilitates the decoupling between logistics and production systems. Transparent data flows and the flexibility of building innovative new services are possible with this access to events from everywhere in the company.

Stability is vital in manufacturing across the supply chain. This begins with Tier 1 and Tier 2 suppliers up to the aftersales and service management. Direct integration from the shop floor to serverless Confluent Cloud on Azure ensures a mission-critical data streaming environment for data pipelines between edge and cloud.

The use case enables reliable data sharing across the logistics and supply chain processes for BMW’s global plants.

Data streaming enables:

• Access to information about the right stock in place (physically and in ERP systems like SAP)
• Just in time, just in sequence process optimization
• Improved overall equipment effectiveness (OEE) for a lot of critical applications
• Last-mile integration between Apache Kafka and IoT standards like OPC-UA

Read more about BMW’s success story for IoT and cloud-native data streaming.

Lightboard Video: How Data Streaming Connects Smart Factory and Cloud

Here is a five-minute lightboard video that describes how data streaming helps with the integration between production facilities (or any other edge environments) and the cloud:

If you liked this video, make sure to follow the YouTube channel for many more lightboard videos across all industries.

IoT and Edge are not contradictory to Cloud and Data Streaming

The BMW case study shows how you can build reliable real-time synchronization between smart factories and cloud applications. However, there are more options. For more case studies, check out the free “The State of Data Streaming in Manufacturing” on-demand recording or read the related blog post.

MQTT is combined with Kafka regularly if the use case requires supporting bad networks or millions of IoT clients. Another alternative is data streaming at the edge with highly available Kafka clusters on industrial PCs, e.g., for air-gapped environments, or embedded single Kafka brokers, e.g., deployment in a machine.

Humans are still crucial for the success of a smart factory. Improving the OEE requires a smart combination of software, robots, and people. Augmented Reality leveraging Data Streaming is an excellent example. VR/AR platforms like Unity enable remote services, training, or simulation. Apache Kafka is the foundation for real-time data sharing across these different technologies and interfaces.

How do you leverage data streaming in your manufacturing use cases? Do you deploy at the edge, in the cloud, or both? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post Data Streaming from Smart Factory to Cloud appeared first on Kai Waehner.