I spoke with Jussi Ahtikari (Chief AI Officer at Virta) at a HotTopics C-Suite Exchange about Virta business model around EV charging networks and how they leverage data streaming. The following is a summary of this excellent success story about an innovative EV charging platform.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases, including several success stories around Kafka and Flink to improve ESG.

The Evolution and Challenges of Electric Vehicle (EV) Charging

The global shift towards electric vehicles (EVs) is accelerating, driven by the surge in variable renewable energy (wind, solar) production, need for sustainable and more cost-efficient transportation solutions, government incentives, and rapid advancements in battery technology. EV charging infrastructure plays a critical role in making this transition successful. It ensures that drivers have access to reliable and efficient charging options while keeping the costs of energy and charging operations in check and energy system in balance.

The innovation in EV charging goes beyond simply providing power to vehicles. Intelligent charging networks, dynamic pricing models, and energy management solutions are transforming the industry. Sustainability is also a key factor, as efficient energy consumption and integration with renewable energy system contribute to environmental, social, and governance (ESG) goals.

While the user and charged energy volumes grow, the real time interplay with the energy system, demand fluctuations, complex billing systems, and real-time station availability updates require a scalable and resilient data infrastructure. Delays in processing real-time data can lead to inefficient energy distribution, poor user experience, and lost revenue.

Virta: Innovating the Future of EV Charging

Virta is a digital cloud platform for electric vehicle (EV) charging businesses and a global leader in connecting of smart charging infrastructure and EV battery capacity with the renewable energy system via bi-directional charging (V2G) and demand response (V1G).

The digital Virta EV Energy platform provides a comprehensive suite of solutions for charging businesses to launch and manage their own EV charging networks. Virta full-service charging platform enables Charging Network and Business Management, Transactions, Pricing, Payments and Invoicing, EV Driver and Fleet Services, Roaming, Energy Management, and Virtual Power Plant services.

Its Charge Point Management System (CPMS) supports over 450 charger models, allowing seamless integration with third-party infrastructure. Virta is the only provider combining CPMS with energy flexibility platform.

Virta Platform Connecting 100,000+ Charging Stations Serving Millions of EV Drivers

The Virta platform is utilised by professional charge point operators (CPOs) and e-mobility service providers (EMPs) across energy, petrol, retail, automotive and real estate industries in 36 countries in Europe and South-East Asia. Virta is headquartered in Helsinki, Finland.

Virta manages real-time data from well over 100,000 EV charging stations, serving millions of EV drivers, and processes approximately 40 GB of real-time data every hour. Including roaming partnerships, the platform offers EV drivers access to in total over 620,000 public charging stations in over 60 countries.

With this scale, real-time responsiveness is critical. Each time a charging station sends a signal—for example, when a driver starts charging—the platform must immediately trigger a series of actions:

• Start billing
• Update real-time status in mobile apps
• Notify roaming networks
• Update metrics and statistics
• Conduct fraud checks

At the early days of electric mobility all of these operations could be handled in a monolithic system using tightly coupled and synchronized code. According to Jussi Ahtikari, Chief AI Officer at Virta, this would have made the system “complex, difficult to maintain, and hard to scale” as data volumes grew. Therefore the team identified early a need for a more modular, scalable, and real-time architecture to support its rapid growth and evolving service portfolio.

Innovative Industry Partnerships: Virta and Valeo

Virta is also exploring new opportunities in the EV ecosystem through its partnership with Valeo, a leader in automotive and energy solutions. The companies are working on integrating Valeo’s Ineez charging technology with Virta’s CPMS platform to enhance fleet charging, leasing services, and vehicle-to-grid (V2G) capabilities.

Vehicle-to-grid technology enables EVs to act as distributed energy storage, feeding excess power back into the grid during peak demand. This innovation is expected to play a critical role in balancing electricity supply and demand, contributing to cheaper electricity and more stable renewables based energy system.

The Role of Data Streaming in ESG and EV Charging

Sustainability and environmental responsibility are key drivers of ESG initiatives in industries such as energy, transportation, and manufacturing. Data streaming plays a crucial role in achieving ESG goals by enabling real-time monitoring, predictive maintenance, and energy efficiency improvements.

In the EV charging industry, real-time data streaming supports:

• Grid load balancing: Preventing energy spikes and ensuring optimal distribution of power. Example: Tesla’s Energy Platform.
• Dynamic pricing: Adjusting charging costs based on demand and electricity availability. Example: Dynamic pricing for road tolling.
• Fraud prevention: Detecting unauthorized access or energy theft. Example: Fraud prevention in under 60 seconds.
• Predictive maintenance: Identifying potential failures before they occur, reducing downtime. Example: Condition monitoring in manufacturing.
• User experience improvements: Providing real-time station availability updates and billing transparency. Example: Improved user experience in mobile apps.

Foreseeing the growing need for these real-time insights led Virta to adopt a data streaming approach with Confluent.

Virta’s Data Streaming Transformation

To maintain its rapid growth and provide an exceptional customer experience, Virta needed a scalable, real-time data streaming solution. The company turned to Confluent’s data streaming platform (DSP), powered by Apache Kafka, to process millions of messages per hour and ensure seamless operations.

Scaling Challenges and the Need for Real-Time Processing

Virta’s rapid growth to scale of millions of charging events and tens of gigawatt hours of charged energy on a monthly basis in Europe and South-East Asia resulted in massive volumes of data that needed to be processed instantly. Something legacy systems, based on sequential authorization, would have struggled with.

Without real-time updates, large scale charging operations would face issues such as:

• Unclear station availability
• Slow transaction processing
• Inaccurate billing information

Initially, Virta worked with open-source Apache Kafka but found managing high-volume data streams at scale to be increasingly resource-intensive. Therefore the team sought an enterprise-grade solution that would remove operational complexities while providing robust real-time capabilities.

Deploying A Data Streaming Platform for Scalable EV Charging

Confluent has become the backbone of Virta’s real-time data architecture. With Confluent’s event streaming platform, Virta is able to maintain a modern event-driven microservices architecture. Instead of tightly coupling all business logic into one system, each charging event—such as a driver starting a session—is published as a single, centralized event. Independent microservices subscribe to that event to trigger specific actions like billing, mobile app updates, roaming notifications, fraud detection, and more.

Here is a diagram of Virta’s cloud-Native microservices architecture powered by AWS, Confluent Cloud, Snowflake, Redis, OpenSearch, and other technologies:

This architectural shift with an event-driven architecture and the data streaming platform as central nervous system has significantly improved scalability, maintainability, and fault isolation. It has also accelerated innovation with fast roll-out times of new services, including audit trails, improved data governance through schemas, and the foundation for AI-powered capabilities—all built on clean, real-time data streams.

Key Benefits of a SaaS Data Streaming Platform for Virta

As a fully managed data streaming platform, Confluent Cloud has eliminated the need for Virta to maintain Kafka clusters manually, allowing its engineering teams to focus on innovation rather than infrastructure management:

• Elastic scalability: Automatically scales up to handle peak loads, ensuring uninterrupted service.
• Real-time processing: Supports 45 million messages per hour, enabling immediate updates on charging status and availability.
• Simplified development: Tools such as Schema Registry and pre-built APIs provide a standardized approach for developers, speeding up feature deployment.

Data Streaming Landscape: Spoilt for Choice – Open Source Kafka, Confluent, and many other Vendors

To navigate the evolving data streaming landscape, Virta chose a cloud-native, enterprise-grade platform that balances reliability, scalability, cost-efficiency, and ease of use. While many streaming technologies exist, Confluent offered the right trade-offs between operational simplicity and real-time performance at scale.

Read more about the different data streaming frameworks, platforms and cloud services in the data streaming landscape overview:

Business Impact of a Data Streaming Platform

By leveraging Confluent Cloud as its cloud-native and serverless data streaming platform, Virta has realized significant business benefits:

1. Faster Time to Market

Virta’s teams can now deploy new app features, charge points, and business services more quickly. The company has regained the agility of a startup, rolling out improvements without infrastructure bottlenecks.

2. Instant Updates for Customers and Operators

With real-time data streaming, Virta can update station availability and configuration changes in less than a second. This ensures that customers always have the latest information at their fingertips.

3. Cost Savings through Usage-Based Pricing

Virta’s shift to a usage-based pricing model has optimized its operational expenses. Instead of maintaining excess capacity, the company only pays for the resources it consumes.

4. Future-Ready Infrastructure for Advanced Analytics

Virta is building the future of real-time analytics, predictive maintenance, and smart billing by integrating Confluent with Snowflake’s AI-powered data cloud.

By decoupling data streams with Kafka, Virta ensures data consistency, scalability, and agility—enabling advanced analytics without operational bottlenecks.

Beyond EV Charging: Broader Energy and ESG Use Cases

Virta’s success with real-time data streaming highlights broader applications across the energy and ESG sectors. Similar data-driven solutions are being deployed for:

• Smart grids: Real-time monitoring of electricity distribution to optimize supply and demand.
• Renewable energy integration: Managing wind and solar power fluctuations with predictive analytics.
• Industrial sustainability: Tracking carbon emissions and optimizing resource utilization.

The Future of EV Charging with Real-Time Data Streaming using Kafka and Flink

The transition to electric mobility requires more than just an increase in charging stations. The ability to process and act on data in real time is critical to optimizing the use and costs of energy and infrastructure, enhancing user experience, and driving sustainability.

Virta’s usage of a serverless data streaming platform demonstrates the power of real-time data streaming in enabling scalable, efficient, and future-ready EV charging solutions. By eliminating infrastructure constraints, improving responsiveness, and reducing operational costs, Virta is setting new industry standards for innovation in mobility and energy management.

The EV charging landscape will tenfold within the next ten years, and especially with the mass adoption of bi-directional charging (V2G), integrate seamlessly with the energy system. Real-time data streaming will serve as the cornerstone for this evolution, helping businesses navigate challenges while unlocking new opportunities for sustainability and profitability.

For more data streaming success stories and use cases, make sure to download my free ebook. Please let me know your thoughts, feedback and use cases on LinkedIn and stay in touch via my newsletter.

The post Virta’s Electric Vehicle (EV) Charging Platform with Real-Time Data Streaming: Scalability for Large Charging Businesses appeared first on Kai Waehner.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases across all industries.

What is a Virtual Power Plant?

A Virtual Power Plant (VPP) is a network of decentralized energy resources—such as home batteries, solar panels, and smart grid systems—that function as a single unit. Unlike a traditional power plant that generates electricity from a centralized location, a VPP aggregates power from many small, distributed sources. This allows energy to be dynamically stored and shared, helping to balance supply and demand in real time.

VPPs are crucial in the shift to renewable energy. The traditional power grid was designed around fossil fuel plants that could easily adjust output. Renewable energy sources like solar and wind are intermittent—they don’t generate power on demand. By connecting thousands of batteries and solar panels in homes and businesses, a VPP can smooth out fluctuations in power generation and consumption. This prevents blackouts, reduces energy costs, and enables homes and businesses to participate in energy markets.

How Tesla’s Virtual Power Plant Fits Its Business Model

Tesla is not just an automaker. It is a sustainable energy company. Tesla’s product ecosystem includes electric vehicles, solar panels, home batteries (Powerwall), grid-scale energy storage (Megapack), and energy management software (Autobidder).

The Tesla Virtual Power Plant (VPP) ties all these elements together. Homeowners with Powerwalls store excess solar power during the day and feed it back to the grid when needed. Tesla’s Autobidder software automatically optimizes energy use and market participation, turning home batteries into revenue-generating assets.

For Tesla, the VPP strengthens its energy business, creating a scalable model that maximizes battery efficiency, stabilizes grids, and expands the role of software in energy markets. Tesla is not just selling batteries; it is selling energy intelligence.

Virtual Energy Platform and ESG (Environmental, Social, and Governance) Goals

Tesla’s energy platform is a perfect example of how data streaming and real-time decision-making align with ESG principles:

• Environmental Impact: VPPs reduce reliance on fossil fuels by making renewable energy more reliable.
• Social Benefit: By enabling energy independence, VPPs provide power during outages and extreme weather conditions.
• Governance & Regulation: VPPs allow consumers to participate in energy markets, fostering decentralized energy ownership.

Tesla’s approach is smart grid innovation at scale—real-time data makes the grid more dynamic, efficient, and resilient.

My article “Green Data, Clean Insights: How Apache Kafka and Flink Power ESG Transformations” covers other real-world data streaming deployments in the energy sector like EON.

Tesla’s Energy Platform: A Network of Connected Home Energy Systems

Tesla’s VPP connects thousands of homes with Powerwalls, solar panels, and grid services. These systems work together to provide electricity on demand, reacting to supply fluctuations in real-time.

Key Functions of Tesla’s VPP:

1. Energy Storage & Redistribution: Batteries store solar energy during the day and discharge at night or during peak demand.
2. Grid Stabilization: The VPP balances energy supply and demand to prevent outages and fluctuations.
3. Market Participation: Homeowners can sell excess power back to the grid, monetizing their batteries.
4. Disaster Resilience: The VPP provides backup power during blackouts, storms, and grid failures.

This requires real-time data processing at massive scale—something traditional batch-based data architectures cannot handle.

Apache Kafka and Real-Time Data Streaming at Tesla

Tesla operates in many domains—automotive, energy, and AI. Across all these areas, Apache Kafka plays a critical role in enabling real-time data movement and stream processing.

In 2018, Tesla already processed trillions of IoT messages with Apache Kafka:

Tesla leverages stream processing to handle trillions of IoT events daily, using Apache Kafka to ingest, process, and analyze data from its vehicle fleet in real time. By implementing efficient data partitioning, fast and slow data lanes, and scalable infrastructure, Tesla optimizes vehicle performance, predicts failures, and enhances manufacturing efficiency.

These strategies demonstrate how real-time data streaming is essential for managing large-scale IoT ecosystems, ensuring low-latency insights while maintaining operational stability. To learn more about these use cases read Tesla’s blog post “Stream Processing with IoT Data: Challenges, Best Practices, and Techniques“.

The following sections explore Tesla’s innovation for its virtual power plant, as discussed in an excellent presentation at QCon.

Tesla Energy Platform: Architecture of the Virtual Power Plant Powered by Apache Kafka

Tesla’s VPP uses Apache Kafka for:

1. Telemetry Ingestion: Streaming data from millions of Powerwalls, solar panels, and Megapacks into the cloud.
2. Command & Control: Sending real-time control commands to batteries and grid services.
3. Market Participation: Autobidder analyzes real-time data and adjusts energy prices dynamically.

The event-driven architecture allows Tesla to react to energy demand in milliseconds—critical for balancing the grid.

Tesla’s Energy Platform is the software foundation of the VPP. It integrates OT (Operational Technology), IoT (Internet of Things), and IT (Information Technology) to control distributed energy assets.

Tesla Applications Built on the Energy Platform

Tesla’s Energy Platform powers a suite of applications that optimize energy management, market participation, and grid stability through real-time data streaming and automation.

Autobidder

• Optimizes energy trading in real time.
• Automatically bids into energy markets.

I wrote about about other data streaming success stories for energy trading with Apache Kafka and Flink, including Uniper, re.alto and Powerledger.

Distributed Virtual Power Plant

• Aggregates thousands of Powerwalls into a single energy asset.
• Provides grid stabilization and peak load balancing.

If you are interested in other smart grid infrastructures, check out “Apache Kafka for Smart Grid, Utilities and Energy Production“. The articles covers how data streaming realizes IT/OT integration. And some hybrid cloud IoT deployments.

Battery Control (Command & Control)

• Ensures optimal charging and discharging of batteries.
• Minimizes costs while maximizing energy efficiency.

Market Participation

• Allows homeowners and businesses to profit from energy markets.
• Ensures seamless grid integration of Tesla’s energy products.

Key Components of Tesla’s Energy Platform: Apache Kafka, WebSockets, Akka Streams

The combination of data streaming with Apache Kafka and the last-mile IoT integration via WebSockets builds the central nervous system of Tesla’s Energy Platform:

1. Apache Kafka (Event Streaming):
   • Streams telemetry data from Powerwalls every second.
   • Ensures durability and reliability of data streams.
   • Allows real-time energy aggregation across thousands of homes.
2. WebSockets (Last-Mile IoT Integration):
   • Provides low-latency bidirectional communication with Powerwalls.
   • Used to send real-time commands to home batteries.
3. Pub/Sub (Command & Control):
   • Enables distributed energy resource coordination.
   • Ensures resilient messaging between systems.
4. Business Logic (Applications & Microservices):
   • Tesla’s services are built with Scala and Python.
   • Uses gRPC & HTTP for inter-service communication.
5. Digital Twins (Real-Time State Management):
   • Digital models of physical assets ensure real-time decision-making.
   • Tesla uses Akka Streams for stateful event processing.
6. Kubernetes (Cloud Infrastructure):
   • Ensures scalability and resilience of Tesla’s energy microservices.

Interesting side note: While most energy companies I have seen rely on Kafka Streams or Apache Flink for stateful event processing, Tesla takes an interesting approach by leveraging Akka Streams (based on Akka’s Actor Model) to manage real-time digital twins of its energy assets. This choice provides fine-grained control over streaming workflows, but unlike Kafka Streams or Flink, Akka lacks widespread community adoption, making it a less common choice for many large-scale energy platforms. Kafka and Flink are a match made in heaven for most data streaming use cases.

Best Practice: Shift Left Architecture with Data Products for High-Volume IoT Data

Tesla leverages several data processing best practices to improve efficiency and consistency:

• Canonical Kafka Topics: Data is filtered and structured at the source.
• Consistent Downstream Services: Every consumer gets clean, structured data.
• Real-Time Aggregation of Thousands of Batteries: A unique challenge that forms the foundation of the virtual power plant.

This data-first approach ensures Tesla’s energy platform can scale to millions of distributed assets.

Today, many people refer to the Shift Left Architecture when applying these best practices for processing data efficiently and continuously to provide data product in real-time and good quality:

In Tesla’s Energy Platform, the data comes from IoT interfaces. WebSockets provide the last-mile integration and feed the events into the data streaming platform for continuous processing before the ingestion into the operational and analytical applications.

Tesla’s Energy Vision: How Streaming Data Will Shape Tomorrow’s Power Grids

Tesla’s Virtual Power Plant is not just about batteries—it’s about software, real-time data, and automation.

Why Data Streaming Matters for Tesla’s Energy Platform:

1. Scalability: Can handle millions of energy devices.
2. Resilience: Works even when devices go offline.
3. Real-Time Decision Making: Adjusts energy distribution within milliseconds.
4. Market Optimization: Autobidder ensures maximum revenue for battery owners.

Tesla’s VPP is a blueprint for the future of energy—one where real-time data streaming and intelligent software optimize renewable energy. By leveraging Apache Kafka, WebSockets, and stream processing, Tesla is redefining how energy is generated, distributed, and consumed.

This is not just an innovation in power generation—it’s an AI-driven energy revolution.

How do you leverage data streaming in the energy and automotive sector? Follow me on LinkedIn or X (former Twitter) to stay in touch and discuss. Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter. And make sure to download my free book about data streaming use cases across all industries.

The post Tesla Energy Platform – The Power of Data Streaming with Apache Kafka appeared first on Kai Waehner.

Blog series: Apache Kafka in the Public Sector and Government

This blog series explores why many governments and public infrastructure sectors leverage event streaming for various use cases. Learn about real-world deployments and different architectures for Kafka in the public sector:

1. Life is a Stream of Events
2. Smart City
3. Citizen Services
4. Energy and Utilities (THIS POST)
5. National Security

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts once published.

As a side note: If you wonder why healthcare is not on the above list. Healthcare is another blog series on its own. While the government can provide public health care through national healthcare systems, it is part of the private sector in many other cases.

Energy, Utilities, Smart Grid, and the Public Sector

The energy sector is different in countries and even states. Public utilities are subject to public control and regulation, ranging from local community-based groups to statewide government monopolies. Hence, some markets are private businesses, or entirely controlled by the government, or a mix of both. For instance, here is the complex US Regulated vs. Deregulated Electricity Market:

Nevertheless, one thing is clear: The energy sector is changing; no matter if the government entirely regulates the market or not:

Let’s look at a few real-world examples for Apache Kafka in the Energy Sector, its relation to the public sector, and a few possible enterprise architectures.

Kafka Examples for Public Utilities

First of all, I already wrote about data in motion powered by Kafka in the energy sector. I also had a great panel discussion about edge and hybrid architecture in a panel discussion about Kafka and 5G networks in the oil and gas and mining industry.

Let’s now take a look at two more examples:

• Stadtwerke Leipzig: A government-owned electricity provider
• Tesla: A private company heavily influenced by the public administration

Stadtwerke Leipzig – Digital Customer Interface for Public Utilities

Stadtwerke Leipzig is a municipal energy utility in central Germany that provides electricity, natural gas, and district heating. They are wholly owned by LVV Leipziger Versorgungs- und Verkehrsgesellschaft, in which the City of Leipzig holds a 100% stake.

Leipziger Stadtwerke built a digital customer interface to connect public utilities, grid operators, the housing industry, end-consumer, industrial customers:

The picture is of bad quality, unfortunately, and not available in a better version. Though, the essential point is that the long green rectangle in the middle is Apache Kafka. Kafka is the central nervous system to connect edge devices, proprietary protocols, and open standards such as MQTT, OPC-UA, XML, JSON, etc. This way, the OT and IT world are connected with a single, scalable real-time pipeline.

Instead of having various data silos, the data is now accessible by any interested consumer in real-time at scale. Hence, this architecture solves one of the biggest challenges in energy infrastructures: Getting value out of the massive volumes of OT data. Nevertheless, the enterprise architecture allows different technologies and brownfield integration. Kafka provides automatic backpressure handling and preprocessing.

Leipziger Stadtwerke combines Kafka with other great technologies to build innovative digital services. For instance, Kunbus edge devices (a PI with custom Linux) and over-the-air updates (OTA) with Mender.

Tesla – Streaming IoT Data for Innovative Services

Tesla is a private enterprise, not within the public sector. However, living in Germany, I see how much related the company is with the public administration, government, law, etc. The Gigafactory in Berlin is in the press every week. The innovation around electric cars is a widespread public discussion; even German competitors like Volkswagen admit Tesla’s innovative business. As the public sector often does not talk to the public about its projects, I thought Tesla’s Kafka success story is still worth mentioning in this post.

Why?

Well, because Tesla has a considerable energy business (they don’t just sell cars), innovates like not many other car and energy companies need to collaborate with governments across the globe regarding law compliance, charging infrastructure, and other crucial topics.

Tesla processes trillions of messages per day for IoT use cases with Kafka. The data comes from connected cars, power packs, factories,  charging stations, etc. Tesla’s Kafka Summit talk showed exciting information about their Kafka journey:

Hybrid IoT Architecture for the Energy Sector

IT architectures in the public sector look very similar to the private sector. The main difference is the more limited usage of public cloud providers. Nevertheless, most energy infrastructures require a hybrid approach with edge computing outside a data center or cloud.

Let’s take a look at a few example architecture for energy production from upstream and midstream to downstream:

Event Streaming enables data integration and data processing in motion, whether it has to happen at the edge or in the data center/cloud.

Edge Computing with Kafka in Disconnected Offline Mode

From the perspective of the edge, data is often filtered, preprocessed, and aggregated at the edge for latency, security, or cost reasons:

Disconnected data processing at the edge is crucial in many energy, and utilities use cases. It has to work even without an internet connection in “offline mode”:

The same is valid on the consumer side. The point-of-sale (POS) has to run 24/7 for transactional workloads, no matter if there is an internet connection:

I covered edge use cases for Kafka and security implications with Kafka in a zero-trust air-gapped environment in separate posts.

Cybersecurity – The Threat is Real for Public Sector and the Energy Infrastructure

Cybersecurity is crucial everywhere in the public sector, including citizen services, smart city, and mobility services. But in these “convenience use cases”, we “only” talk about data privacy. Yes, this is very important. But in the energy sector, we are talking about safety and human lives at risk. The Colonial Pipeline ransomware attack in May 2021 in the US is just one of many successful attacks in the past few quarters.

National security is a huge topic for the energy sector. Electric utilities can be affected by cyberattacks across the whole value chain. McKinsey has an exciting diagram explaining this:

The discussion around cybersecurity is a primer to the last post of this blog series.

Of course, my general blog series about Apache Kafka for Cybersecurity (Situational Awareness, Threat Intelligence, Forensics, Zero Trust, SIEM/SOAR Modernization) is helpful, too.

Data in Motion for Reliable and Scalable Smart Grid Infrastructure

This post showed a few real-world examples and architectures for data in motion in hybrid architectures in the energy industry. The private sector has fewer examples than the public sector. But the architectures look the same, no matter who is responsible.

The private energy sector needs to collaborate with the government and public administration like the public energy sector. The integration and processing of data in motion with Apache Kafka is a game-changer for improving existing processes and building new innovative solutions.

For instance, Tesla is a very innovative private company with cutting business models that are only possible if you collect, aggregate, and leverage data streams from various data sources. Tesla’s new car insurance service is an excellent example of this. The insurance business is backed by data from many IoT sensors and applied in real-time to provide context-specific information. That’s the way to go for the public sector, too.

How do you leverage event streaming in the public sector? Are you working on energy/utility projects or building a smart grid? What technologies and architectures do you use? What projects did you already work on or are in the planning? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka in the Public Sector – Part 4: Energy and Utilities appeared first on Kai Waehner.

Apache Kafka + MQTT Blog Series

The first blog post explores the relation between MQTT and Apache Kafka. Afterward, the other four blog posts discuss various use cases, architectures, and reference deployments.

• Part 1 – Overview: Relation between Kafka and MQTT, pros and cons, architectures
• Part 2 – Connected Vehicles: MQTT and Kafka in a private cloud on Kubernetes; use case: remote control and command of a car
• Part 3 – Manufacturing: MQTT and Kafka at the edge in a smart factory; use case: Bidirectional OT-IT integration with Sparkplug between PLCs, IoT Gateways, Data Historian, MES, ERP, Data Lake, etc.
• Part 4 – Mobility Services: MQTT and Kafka leveraging serverless cloud infrastructure; use case: Traffic jam prediction service using machine learning
• Part 5 – Smart City (THIS POST): MQTT at the edge connected to fully-managed Kafka in the public cloud; use case: Intelligent traffic routing by combining and correlating 3rd party services

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Use Case: Smart City and 5G

A smart city is an urban area that uses different types of electronic Internet of Things (IoT) sensors to collect data and then use insights gained from that data to manage assets, resources, and services efficiently.

A smart city provides many benefits for civilization and city management. Some of the goals are:

• Improved Pedestrian Safety
• Improved Vehicle Safety
• Proactively Engaged First Responders
• Reduced Traffic Congestion
• Connected / Autonomous Vehicles
• Improved Customer Experience
• Automated Business Processes

I covered the use cases in more detail in the post “Event Streaming with Kafka as Foundation for a Smart City“. For a specific 5G example, check out “Building a Smart Factory with Apache Kafka and 5G Campus Networks“.

Let’s now explore the relation of Kafka and MQTT for smart city use cases.

Architecture: MQTT and Kafka for a Smart City

The following architecture shows an infrastructure deployed at a stadium:

In this example, both MQTT and Kafka are deployed close to the stadium. For instance, AWS Wavelength is an innovative infrastructure option to build low latency 5G use cases. The connected “regular AWS cloud region” is still used for use cases that do not require low latency.

The combination of Kafka and MQTT enables connectivity and real-time data processing for various use cases:

• Parking information and smart navigation.
• Location-based shopping and restaurant experiences, including innovative scenarios such as monitoring of queues and geofencing.
• Integration of loyalty platforms to earn rewards and points.
• Live information about the game or concert
• Lottery drawing experiences while watching a sports game.

The possibilities are endless. Integration with 1st and 3rd party applications will create completely new opportunities to improve the customer experience, increase safety, and improve operational efficiency.

The stadium example is a particular scenario to explore the added value of processing data in motion. Let’s take a look at other real-world examples that leverage MQTT and Kafka.

Example: Cloud-based Traffic Control Systems @Berlex

The Swedish company Berlex designs and manufactures new ways to improve traffic safety.

Berlex provides cloud-based portable traffic signals. Their innovative R6 traffic signal is one of the first mobile traffic signals controlled by a cloud-based service. Berlex’s connected solution allows customers to monitor the new traffic signals on a smartphone, computer, or tablet anytime and from anywhere. MQTT enables real-time information delivery and constant monitoring.

The cloud-based service reduces the time that their customers need to spend in dangerous traffic work zones. The system enables customers to carry out numerous tasks such as checking the battery status of a traffic signal or performing an inspection remotely, with no need for risky and time-consuming on-site intervention.

Each portable R6 traffic signal is equipped with a radar that allows the signal to see traffic. Sensors within the signals publish detailed information on the current status of the signal as MQTT data. The Berlex Connect cloud service captures the continuous stream of MQTT data from each signal and shares the information with the appropriate subscribers.

To prevent interruption of the traffic signal operation, high availability is essential for the system. Berlex customers monitor the real-time information on individual portals with customized user roles that fit their specific use case.

Read the complete case study from HiveMQ for more details about this successful smart city project.

Example: The Life of Citizens as a Stream of Events @ NAV

NAV (Norwegian Work and Welfare Department) currently distributes more than one-third of the national budget to Norway or abroad citizens. NAV assists people through all phases of life within work, family, health, retirement, and social security. Events happening throughout a person’s life determines which services we provide to them, how we provide them and when we provide them.

In most countries, each person has to apply for these services resulting in many tasks handled manually by various caseworkers in the organization. Their access to insight and useful information is limited and often hard to find, causing frustration to both our caseworkers and our users. By streaming a person’s life events through our Kafka pipelines, NAV revolutionized the way users experience government services and the way the employees work:

NAV and the government as a whole have access to vast amounts of data about the citizens, reported by health institutions, employers, various government agencies, or the users themselves. Some data is distributed by large batches, while others are available on-demand through APIs. The data is ingested into streams using Kafka, Streams API, and Java microservices. NAV distributes and acts on events about birth, death, relationships, employment, income, and business processes to vastly improve the user experience, provide real-time insight and reduce the need to apply for services the government already knows are needed.

NAV chose Confluent Platform to implement to get valuable insight from life and business events. Security is a key concern. Compliance with GDPR is essential for the success of this project.

More details about NAV’s Kafka usage in their Kafka Summit presentation.

Kafka + MQTT = Smart City

In conclusion, Apache Kafka and MQTT are a perfect combination for smart city and 5G use cases. Follow the blog series to learn about use cases such as connected vehicles, manufacturing, mobility services, and smart city. Every blog post also includes real-world deployments from companies across industries. It is key to understand the different architectural options to make the right choice for your project.

What are your experiences and plans in IoT projects? What use case and architecture did you implement? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka and MQTT (Part 5 of 5) – Smart City and 5G appeared first on Kai Waehner.

Smart Grid – The Energy Production and Distribution of the Future

The energy sector includes corporations that primarily are in the business of producing or supplying energy such as fossil fuels or renewables.

What is a Smart Grid?

A smart grid is an electrical grid that includes a variety of operation and energy measures,, including smart meters, smart appliances, renewable energy resources, and energy-efficient resources. Electronic power conditioning and control of the production and distribution of electricity are important aspects of the smart grid.

The European Union Commission Task Force for Smart Grids provides smart grid definition as:

“A Smart Grid is an electricity network that can cost-efficiently integrate the behavior and actions of all users connected to it – generators, consumers and those that do both – to ensure economically efficient, sustainable power system with low losses and high levels of quality and security of supply and safety. A smart grid employs innovative products and services, together with intelligent monitoring, control, communication, and self-healing technologies to:

1. Better facilitate the connection and operation of generators of all sizes and technologies.
2. Allow consumers to play a part in optimizing the operation of the system.
3. Provide consumers with greater information and options for how they use their supply.
4. Significantly reduce the environmental impact of the whole electricity supply system.
5. Maintain or even improve the existing high levels of system reliability, quality, and security of supply.
6. Maintain and improve existing services efficiently.”

Technologies and Evolution of a Smart Grid

The Roll-out of smart grid technology also implies a fundamental re-engineering of the electricity services industry, although typical usage of the term is focused on the technical infrastructure. Key smart grid technologies include solar power, smart meters, microgrids, and self-optimizing systems:

Requirements for a Smart Grid and Modern Energy Infrastructure

• Reliability: The smart grid uses technologies such as state estimation which improve fault detection and allow self-healing of the network without the intervention of technicians. This will ensure a more reliable supply of electricity and reduced vulnerability to natural disasters or attack.
• Flexibility in network topology: Next-generation transmission and distribution infrastructure will be better able to handle possible bidirectional energy flows, allowing for distributed generation such as from photovoltaic panels on building roofs, but also charging to/from the batteries of electric cars, wind turbines, pumped hydroelectric power, the use of fuel cells, and other sources.
• Efficiency: Numerous contributions to the overall improvement of the efficiency of energy infrastructure are anticipated from the deployment of smart grid technology, in particular including demand-side management, for example, turning off air conditioners during short-term spikes in electricity price, reducing the voltage when possible on distribution lines through Voltage/VAR Optimization (VVO), eliminating truck-rolls for meter reading, and reducing truck-rolls by improved outage management using data from Advanced Metering Infrastructure systems. The overall effect is less redundancy in transmission and distribution lines and greater utilization of generators, leading to lower power prices.
• Sustainability: The smart grid’s improved flexibility permits greater penetration of highly variable renewable energy sources such as solar power and wind power, even without the addition of energy storage.
• Market-enabling: The smart grid allows for systematic communication between suppliers (their energy price) and consumers (their willingness-to-pay. It permits both the suppliers and the consumers more flexible and sophisticated operational strategies.
• Cybersecurity: Provide a secure infrastructure with encrypted and authenticated communication and real-time anomaly detection at scale across the supply chain.

Architectures with Kafka for a Smart Grid

From a technical perspective, use cases such as load adjustment/load balancing or peak curtailment/leveling and time of use pricing cannot be implemented with traditional, monolith software like they were used in the past in the energy industry.

A smart grid requires a cloud-native infrastructure that is flexible, scalable, elastic, and reliable. All of that in combination with real-time data integration and data processing. These requirements explain why more and more energy companies rely heavily on event streaming with Apache Kafka and its ecosystem.

Energy Production and Distribution with a Hybrid Architecture

Many energy companies have a cloud-first strategy. They build new innovative applications in the cloud. Especially in the energy industry, this makes a lot of sense. The need for elastic and scalable data processing is key to success. However, not everything can run in the cloud. Most energy-related use cases required data processing at the edge, too. This is true for energy production and energy distribution.

Here is an example architecture leveraging Apache Kafka in the cloud and at the edge:

The integration in the cloud requires connecting to modern technologies such as Snowflake data warehouse, Google’s Machine Learning services based on TensorFlow, or 3rd party SaaS services like Salesforce. The edge is different. Connectivity is required for machines, equipment, sensors, PLCs, SCADA systems, and many other systems. Kafka Connect is a perfect, Kafka-native tool to implement these integrations in real-time at scale.

Replication in real-time between the edge sites and the cloud is another important case where tools like MirrorMaker 2 or Confluent’s Cluster Linking fit perfectly.

The continuous processing of data (aka stream processing) is possible with Kafka-native components like Kafka Streams or ksqlDB. Using an external tool such as Apache Flink is also a good fit.

Event Streaming for Energy Production at the Edge with a 5G Campus Network

Kafka at the edge is the new black. Energy production a great example:

More details about deploying Kafka at edge sites is described in the post “Building a Smart Factory with Apache Kafka and 5G Campus Networks“.

The edge is often disconnected from the cloud or remote data centers. Mission-critical applications have to run 24/7 in a decoupled way even if the internet connection is not available or not stable:

Example: Real-Time Outage Management with Kafka in the Utilities Sector

Let’s take a look at an example implemented in the utilities sector: Continous processing of smart meter sensor data with Kafka and ksqlDB:

The preprocessing and filtering happens at the edge, as shown in the above picture. However, the aggregation and monitoring of all the assets of the smart grid (including smart home, smart buildings, powerlines, switches, etc.) happen in the cloud:

Real-time data processing is not just important for operations. Huge added value comes from the improved customer experience. For instance, the outage management operations tool can alert a customer in real-time:

Let’s now take a look at a few real-world examples of energy-related use cases.

Kafka Real-World Deployments in the Energy Sector

This section explores three very different deployments and architectures of Kafka-based infrastructure to build smart grids and energy production-related use cases: EON, WPX Energy, and Tesla.

EON – Smart Grid for Energy Production and Distribution with Kafka

The EON Streaming Platform is built on the following paradigms to provide a cloud-native smart grid infrastructure:

• IoT scale capabilities of public cloud providers
• EON microservices that are independent of cloud providers
• Real-time data integration and processing powered by Apache Kafka

WPX Energy – Kafka at the Edge for Integration and Analytics

WPX Energy (now merged into Devon Energy) is a company in the oil&gas industry. The digital transformation creates many opportunities to improve processes and reduce costs in this vertical. WPX leverages Confluent Platform on Hivecell edge hardware to realize edge processing and replication to the cloud in real-time at scale.

The solution is designed for true real-time decision making and future potential closed-loop control optimization.  WPX conducts edge stream processing to enable true real-time decision-making at well site. They also replicate business-relevant data streams produced by machine learning models and analytical preprocessed data at the well site to the cloud, enabling WPX to harness the full power of its real-time events.

Kafka deployments at the edge (i.e., outside a data center) come up more and more in the energy industry, but also in factories, restaurants, retail stores, banks, and hospitals.

Tesla – Kafka-based Data Platform for Trillions of Data Points per Day

Tesla is not just a car maker. Telsa is a tech company writing a lot of innovative and cutting-edge software. They provide an energy infrastructure for cars with their Telsa Superchargers, solar energy production at their Gigafactories, and much more. Processing and analyzing the data from their smart grids and the integration with the rest of the IT backend services in real-time is a key piece of their success:

Tesla has built a Kafka-based data platform infrastructure “to support millions of devices and trillions of data points per day”. Tesla showed an interesting history and evolution of their Kafka usage at a Kafka Summit in 2019:

Kafka + OT Middleware (OSIsoft PI, Siemens MindSphere, et al)

A common and very relevant question is the relation between Apache Kafka and traditional OT middleware such as OSIsoft PI or Siemens MindSphere.

TL;DR: Apache Kafka and OT middleware complement each other. Kafka is NOT an IoT platform. OT middleware is not built for the integration and correlation of OT data with the rest of the enterprise IT. Kafka and OT middleware connect to each other very well. Both sides provide integration options, including REST APIs, native Kafka Connect connectors, and more. Hence, in many cases, enterprises leverage and integrate both technologies instead of choosing just one of them.

Please check out the following blogs/slides/videos to understand how Apache Kafka and OT middleware complement each other very well:

• Apache Kafka vs. Middleware (MQ, ETL, ESB): Blog or Slides/Video.

• Domain-driven Design (DDD) with Apache Kafka for true decoupling and flexible integration infrastructures

Slides: Kafka-based Smart Grid and Energy Use Cases and Architectures

The following slide deck goes into more details about this topic:

The Future of Kafka for the Energy Sector and Smart Grid

Kafka is relevant in many use cases for building an elastic and scalable smart grid infrastructure. Even beyond, many projects utilize Kafka heavily for customer 360, payment processing, and many other use cases. Check out the various “Use Cases and Architectures for Apache Kafka across Industries“. Energy companies can apply many of these use cases, too.

If you have read this far and actually wonder what “real-time” actually means in the context of Kafka and the OT/IT convergence, please check out the post “Kafka is NOT hard real-time“.

What are your experiences and plans for event streaming in the energy industry? Did you already build a smart grid with Apache Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka for Smart Grid, Utilities and Energy Production appeared first on Kai Waehner.