Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases.

The Business of Mobility Services (Ride-Hailing, Food Delivery, Taxi Aggregators, etc.)

Mobility services have become an essential part of modern urban life. They offer convenience and efficiency through ride-hailing, food delivery, car-sharing, e-scooters, taxi aggregators, and micro-mobility options. Companies such as Uber, Lyft, FREE NOW (former MyTaxi; acquired by Lyft recently), Grab, Careem, and DoorDash connect millions of passengers, drivers, restaurants, retailers, and logistics partners to enable seamless transactions through digital platforms.

These platforms operate in highly dynamic environments where real-time data is crucial for pricing, route optimization, customer experience, and fraud detection. However, this very nature of mobility services also makes them prime targets for fraudulent activities. Fraud in this sector can lead to financial losses, reputational damage, and deteriorating customer trust.

To effectively combat fraud, mobility services must rely on real-time data streaming with technologies such as Apache Kafka and Apache Flink. These technologies enable continuous event processing and allow platforms to detect and prevent fraud before transactions are finalized.

Why Fraud is a Major Challenge in Mobility Services

Fraudsters continually exploit weaknesses in digital mobility platforms. Some of the most common fraud types include:

1. Fake Rides and GPS Spoofing: Drivers manipulate GPS data to simulate trips that never occurred. Passengers use location spoofing to receive cheaper fares or exploit promotions.

2. Payment Fraud and Stolen Credit Cards: Fraudsters use stolen payment methods to book rides or order food.

3. Fake Drivers and Passengers: Fraudsters create multiple accounts and pretend to be both the driver and passenger to collect incentives. Some drivers manipulate fares by manually adjusting distances in their favor.

4. Promo Abuse: Users create multiple fake accounts to exploit referral bonuses and promo discounts.

5. Account Takeovers and Identity Fraud: Hackers gain access to legitimate accounts, misusing stored payment information. Fraudsters use fake identities to bypass security measures.

Fraud not only impacts revenue but also creates risks for legitimate users and drivers. Without proper fraud prevention measures, ride-hailing and delivery companies could face serious losses, both financially and operationally.

The Unseen Enemy: Core Challenges in Mobility Fraud
Detection

Traditional fraud detection relies on batch processing and manual rule-based systems. However, these approaches are no longer effective due to the speed and complexity of modern mobile apps with real-time experiences combined with modern fraud schemes.

Key challenges in mobility fraud detection include:

• Fraud occurs in real-time, requiring instant detection and prevention before transactions are completed.
• Millions of events per second must be processed, requiring scalable and efficient systems.
• Fraud patterns constantly evolve, making static rule-based approaches ineffective.
• Platforms operate across hybrid and multi-cloud environments, requiring seamless integration of fraud detection systems.

How Data Streaming with Apache Kafka and Flink Enables Real-Time Fraud Detection

To overcome these challenges, real-time streaming analytics powered by Apache Kafka and Apache Flink provide an effective solution.

Apache Kafka: The Backbone of Event-Driven Fraud Detection

Kafka serves as the core event streaming platform. It captures and processes real-time data from multiple sources such as:

• GPS location data
• Payment transactions
• User and driver behavior analytics
• Device fingerprints and network metadata

Kafka provides:

• High-throughput data streaming, capable of processing millions of events per second to support real-time decision-making.
• An event-driven architecture that enables decoupled, flexible systems—ideal for scalable and maintainable mobility platforms.
• Seamless scalability across hybrid and multi-cloud environments to meet growing demand and regional expansion.
• Always-on reliability, ensuring 24/7 data availability and consistency for mission-critical services such as fraud detection, pricing, and trip orchestration.

An excellent success story about the transition to data streaming comes from DoorDash: Why DoorDash migrated from Cloud-native Amazon SQS and Kinesis to Apache Kafka and Flink.

Apache Flink: Continuous Stream Processing for Fraud Detection in Real-Time

Apache Flink enables real-time fraud detection through advanced event correlation and applied AI:

• Detects anomalies in GPS data, such as sudden jumps, route manipulation, or unrealistic movement patterns.
• Analyzes historical user behavior to surface signs of account takeovers or other forms of identity misuse.
• Joins multiple real-time streams—including payment events, location updates, and account interactions—to generate accurate, low-latency fraud scores.
• Applies machine learning models in-stream, enabling the system to flag and stop suspicious transactions before they are processed.
• Continuously adapts to new fraud patterns, updating models with fresh data in near real-time to reflect evolving user behavior and emerging threats.

With Kafka and Flink, fraud detection can shift from reactive to proactive to stop fraudulent transactions before they are completed.

I already covered various data streaming success stories from financial services companies such as Paypal, Capital One and ING Bank in a dedicated blog post. And a separate case study from about “Fraud Prevention in Under 60 Seconds with Apache Kafka: How A Bank in Thailand is Leading the Charge“.

Real-World Fraud Prevention Stories from Mobility Leaders

Fraud is not just a technical issue—it’s a business-critical challenge that impacts trust, revenue, and operational stability in mobility services. The following real-world examples from industry leaders like FREE NOW (Lyft), Grab, and Uber show how data streaming with advanced stream processing and AI are used around the world to detect and stop fraud in real time, at massive scale.

FREE NOW (Lyft): Detecting Fraudulent Trips in Real Time by Analyzing GPS Data of Cars

FREE NOW operates in more than 150 cities across Europe with 48 million users. It integrates multiple mobility services, including taxis, private vehicles, car-sharing, e-scooters, and bikes.

The company was recently acquired by Lyft, the U.S.-based ride-hailing giant known for its focus on multimodal urban transport and strong presence in North America. This acquisition marks Lyft’s strategic entry into the European mobility ecosystem, expanding its footprint beyond the U.S. and Canada.

Fraud Prevention Approach leveraging Data Streaming (presented at Kafka Summit)

• Uses Kafka Streams and Kafka Connect to analyze GPS trip data in real-time.
• Deploys fraud detection models that identify anomalies in trip routes and fare calculations.
• Operates data streaming on fully managed Confluent Cloud and applications on Kubernetes for scalable fraud detection.

Example: Detecting Fake Rides

1. A driver inputs trip details into the app.
2. Kafka Streams predicts expected trip fare based on distance and duration.
3. GPS anomalies and unexpected route changes are flagged.
4. Fraud alerts are triggered for suspicious transactions.

By implementing real-time fraud detection with Kafka and Flink, FREE NOW (Lyft) has significantly reduced fraudulent trips and improved platform security.

Grab: AI-Powered Fraud Detection for Ride-Hailing and Delivery with Data Streaming and AI/ML

Grab is a leading mobility platform in Southeast Asia, handling millions of transactions daily. Fraud accounts for 1.6 percent of total revenue loss in the region.

To address these significant fraud numbers, Grab developed GrabDefence—an AI-powered fraud detection engine that leverages real-time data and machine learning to detect and block suspicious activity across its platform.

Fraud Detection Approach

• Uses Kafka Streams and machine learning for fraud risk scoring.
• Leverages Flink for feature aggregation and anomaly detection.
• Detects fraudulent transactions before they are completed.

Example: Fake Driver and Passenger Fraud

1. Fraudsters create accounts as both driver and passenger to claim rewards.
2. Kafka ingests device fingerprints, payment transactions, and ride data.
3. Flink aggregates historical fraud behavior and assigns risk scores.
4. High-risk transactions are blocked instantly.

With GrabDefence built with data streaming, Grab reduced fraud rates to 0.2 percent, well below the industry average. Learn more about GrabDefence in the Kafka Summit talk.

Uber: Project RADAR – AI-Powered Fraud Detection with Human Oversight

Uber processes millions of payments per second globally. Fraud detection is complex due to chargebacks and uncollected payments.

To combat this, Uber launched Project RADAR—a hybrid system that combines machine learning with human reviewers to continuously detect, investigate, and adapt to evolving fraud patterns in near real time. Low latency is not required in this scenario. And humans are in the loop of the business process. Hence, Apache Spark is sufficient for Uber.

Fraud Prevention Approach

• Uses Kafka and Spark for multi-layered fraud detection.
• Implements machine learning models to detect chargeback fraud.
• Incorporates human analysts for rule validation.

Example: Chargeback Fraud Detection

1. Kafka collects all ride transactions in real time.
2. Stream processing detects anomalies in payment patterns and disputes.
3. AI-based fraud scoring identifies high-risk transactions.
4. Uber’s RADAR system allows human analysts to validate fraud alerts.

Uber’s combination of AI-driven detection and human oversight has significantly reduced chargeback-related fraud.

Data Streaming with Kafka and Flink Provides Real-Time Fraud Detection in Mobility Services

Fraud in mobility services is a real-time challenge that requires real-time solutions that work 24/7, even at extreme scale for millions of events. Traditional batch processing systems are too slow, and static rule-based approaches cannot keep up with evolving fraud tactics.

By leveraging data streaming with Apache Kafka in conjunction with Kafka Streams or Apache Flink, mobility platforms can:

• Process millions of events per second to detect fraud in real time.
• Prevent fraudulent transactions before they occur.
• Use AI-driven real-time fraud scoring for accurate risk assessment.
• Adapt dynamically through continuous learning to evolving fraud patterns.

Mobility platforms such as Uber, Grab, and FREE NOW (Lyft) are leading the way in using real-time streaming analytics to protect their platforms from fraud. By implementing similar approaches, other mobility businesses can enhance security, reduce financial losses, and maintain customer trust.

Real-time fraud prevention in mobility services is not an option; it is a necessity. The ability to detect and stop fraud in real time will define the future success of ride-hailing, food delivery, and urban mobility platforms.

Stay ahead of the curve! Subscribe to my newsletter for insights into data streaming and connect with me on LinkedIn to continue the conversation. And download my free book about data streaming use cases.

The post Fraud Detection in Mobility Services (Ride-Hailing, Food Delivery) with Data Streaming using Apache Kafka and Flink appeared first on Kai Waehner.

Today, in 2022, Apache Kafka is the central nervous system of many applications in various areas related to the automotive and manufacturing industry for processing analytical and transactional data in motion across edge, hybrid, and multi-cloud deployments. This article explores the automotive event streaming landscape, including connected vehicles, smart manufacturing, supply chain optimization, aftersales, mobility services, and innovative new business models.

The Event Streaming Landscape for Automotive and Manufacturing

Every business domain leverages Event Streaming with Apache Kafka in the automotive and manufacturing industry. Data in motion helps everywhere. The infrastructure and deployment differ depending on the use case and requirements. I have seen everything at carmakers and manufacturers across the globe:

• Cloud-first strategy with all new business applications in the public cloud deployed and connected across regions and even continents
• Hybrid integration scenarios between legacy applications in the data center and modern cloud-native services the public cloud
• Edge computing in a smart factory for low latency, cost-efficient data processing, and cybersecurity
• Embedded Kafka brokers in machines and vehicles at the disconnected edge

This spread of use cases is impressive. The following diagram depicts a high-level overview:

The following sections describe the automotive and manufacturing landscape for event streaming in more detail:

• Manufacturing 4.0
• Supply Chain Optimization
• Mobility Services
• New Business Models

If you are mainly interested in real-world Kafka deployments with examples from BMW, Porsche, Audi, Tesla, and other OEMs, check out the article “Real-World Deployments of Kafka in the Automotive Industry“.

If you want to understand why Kafka makes such a difference in automotive and manufacturing, check out the article “Apache Kafka in the Automotive Industry“. This article explores the business motivation for these game-changing concepts of data in motion for the digitalization of the automotive industry.

Before you start reading the below section, I want to clearly emphasize that Kafka is not the silver bullet for every problem. “When NOT to use Apache Kafka?” digs deep into this discussion.

I keep the following sections relatively short to give a high-level overview. Each section contains links to more deep-dive articles about the topics.

Manufacturing 4.0

Industrial IoT (IIoT) respectively Industry 4.0 changes how the shop floor and production lines produce goods. Automation, process efficiency, and a much better Overall Equipment Effectiveness (OEE) enable cost reduction and flexibility in the production process:

Smart Factory

A smart factory is not necessarily a newly built building like a Tesla Gigafactory. Many enterprises install smart technology like networked sensors for temperature or vibrations measurements into old factories. Improving the Overall Equipment Effectiveness (OEE) is the primary goal of most use cases. Many scenarios leverage Kafka for continuously processing sensor and telemetry data in motion:

• Connectivity to machines with modern, open technologies such as MQTT
• Visualization and monitoring of equipment and assets (often called Digital Twin or Digital Threat)
• Quality assurance such as condition monitoring with stateless stream processing
• Predictive maintenance of machines, robots, productions lines with stateful streaming analytics and machine learning
• Surveillance and safety-critical video monitoring by processing images and videos
• Cybersecurity for situational awareness and threat intelligence
• Smart Buildings for maintenance and operations, smarter energy consumptions, optimized space usage, better employee experience

Legacy Modernization with Open APIs and Hybrid Cloud

Factories exist for decades after they are built. Digitalization and the modernization of legacy technologies are some of the biggest challenges in IIoT projects. Such an initiative usually includes several tasks:

• Complex integration with proprietary legacy protocols such as Siemens S7, Allan Bradley, Modbus, et al., for instance, with a dedicated open-source framework such as Apache PLC4X running on Kafka Connect
• Simple integration with open standards such as HTTP and REST-based web services and a REST Proxy for Kafka
• Deployment of a modern, open, scalable data historian replacing or complementing monolithic, proprietary data historians
• Postmodern MES and ERP architectures upgrading or replacing legacy proprietary non-scalable MES and ERP systems, for instance, the integration between legacy SAP systems and Kafka
• Lift and shift from on-premise data centers to the public cloud with hybrid synchronization and replication using the Kafka protocol and cluster linking

Continuous Data-driven Engineering and Product Development

Last but not least, an opportunity many people underestimate: Continuous data streaming with Kafka enables new possibilities in software engineering and product development for IoT and automotive projects.

For instance, developing and deploying the “big loop” for machine learning of advanced driver-assistance systems (ADAS) or self-driving functions based on sensor data from the fleet is a new way of software engineering. Tesla’s Kafka-based data platform is a fantastic example. A related use case in engineering is the ingest of sensor data during and after test drives.

Supply Chain Optimization

Supply chain processes and solutions are very complex. The Covid pandemic showed how only flexible enterprises could survive, stay profitable, and provide a great customer experience, even in disastrous external events.

Here are the top 5 critical challenges of supply chains:

• Time Frames are Shorter
• Rapid Change
• Zoo of Technologies and Products
• Historical Models are No Longer Viable
• Lack of visibility

Only real-time data streaming and correlation solve these supply chain challenges end-to-end across regions and companies:

In its detailed blog post, I covered Supply Chain Optimization (SCM) with Apache Kafka. Check it out to learn about real-world supply chain use cases from Bosch, BMW, Walmart, and other companies.

Intra-logistics and Global Distribution Networks

Logistics and supply chains within a factory, distribution center, or store require real-time data integration and processing to provide efficient processing of goods and a great customer experience. Batch processes or manual interaction by human workers cannot implement these use cases. Examples include:

• Real-Time Locating System (RTLS) for transportation and logistics within a building to monitor robots, driverless transport systems (DTS), and manual processes for improved safety, controlled security, and optimized operations and productivity
• Inventory management for optimized and customized production processes with a better B2B integration in real-time – the same story and benefits as in a modern omnichannel retail architecture
• Globally distributed networks powered by a global Kafka infrastructure
• Augmented reality for the digitalization and automation of worker tasks

Track & Trace and Fleet Management

Real-time logistics is a game-changer for fleet management and track & trace use cases.

• Commercial motor vehicles such as cars, vans, trucks, specialist vehicles (such as mobile construction machinery), forklifts, and trailers
• Private vehicles used for work (the ‘grey fleet’)
• Aviation machinery such as aircraft (planes and helicopters)
• Ships
• Rail cars
• Non-powered assets such as generators, tanks, gearboxes

All the following aspects are not new. The difference is that event streaming allows to continuously execute these tasks in real-time to act on new information in motion:

• Visualization
• Location-based services
• Routing and navigation
• Estimated time of arrival
• Alerting
• Proactive recalculation
• Monitoring of the assets and mechanical components of a vehicle

Most companies have a cloud-first strategy for building such a platform. However, some cases require edge computing either via local 5G location for low latency use cases or embedded Kafka brokers for disconnected data collection and analytics within the vehicles.

Streaming Data Exchange for B2B Collaboration with Partners

Real-time data is not just relevant without a company. OEMs and Tier 1 and Tier 2 suppliers benefit in the same way from data streams. The same is true for car dealerships, end customers, and any other consumer of the data. Hence, a clear trend in the market is the emergence of a Kafka-based streaming data exchange across companies to build a data mesh.

I have often seen this situation in the past: The OEM leverages event streaming. The Tier 1 supplier leverages event streaming. The used ERP solution is built on Kafka, too. All leverage the capabilities of scalable real-time data streaming. It makes little sense to integrate with partners and software vendors via web service APIs, such as SOAP or HTTP/REST. Instead, a streaming interface is a natural choice to hand streaming data to partners.

The following example from the automotive industry shows how independent stakeholders (= domains in different enterprises) use a cross-company streaming data exchange:

Mobility Services

Every OEM, supplier, or innovative startup in the automotive space thinks about providing a mobility service either on top of the goods they sell or as an independent service.

Most mobility services on your mobile apps used today for business or privately are only possible because of a scalable real-time backbone powered by event streaming:

The possibilities for mobility services are endless. A few examples that are mainstream today already:

• Omnichannel retail and aftersales to buy additional car features online, for instance, more power, seat heater, up-to-date navigation, self-driving software (okay, the latter one is not mainstream yet, but Tesla shows where it goes)
• Connected Cars for ride-hailing, scooter rental, taxi services, food delivery
• 3rd party integration for adding services that a company does not want to build by themselves

Today’s most successful and widely adopted mobility services are independent of a specific carmaker or supplier.

Examples of prominent Kafka-powered consumer mobility services are Uber and Lyft in the US, Grab in Asia, and FREENOW in Europe. Here Technologies is an excellent example for a B2B mobility service providing mapping information so that companies can build new or improve existing applications on top of it.

A good starting point to learn more is my blog post about Apache Kafka and MQTT for mobility services and transportation.

New Business Models

The access to real-time data enables companies to build entirely new business models on top of their existing products:

A few examples:

• Next-generation car rental with excellent customer experience, context-specific coupons, loyalty platform, and car rental fleets with other services from the carmaker.
• Reinventing car insurance based on real-time driving information about each driver to build driver-specific pricing based on real-time analysis of the driver behavior instead of legacy approaches using statistical models with attributes like driver age, number of accidents in the past, etc.
• Data provider for monetization enables other companies to build new business models with your car data – for instance, working with a government to make a smart city traffic system or a mobility service startup to analyze and correlate car data across OEMs.

This evolution is just the beginning of the usage of streaming data. I have seen many customers build a first streaming pipeline for one use case. However, new business divisions will leverage the data for innovations when the platform is there.

The Data is in Motion in Automotive and Manufacturing

The landscape for Apache Kafka in the automotive and manufacturing industry showed that Apache Kafka is the central nervous system of many applications in various areas for processing analytical and transactional data in motion.

This article explored use cases such as connected vehicles, smart manufacturing, supply chain optimization, aftersales, mobility services, and innovative new business models. The possibilities for data in motion are almost endless. The automotive and manufacturing industry is still in the very early stages of leveraging data in motion.

Where do you use Apache Kafka and its ecosystem in the automotive and manufacturing industry? Do you deploy in the public cloud, in your data center, or at the edge outside a data center? What other technologies do you combine with Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka Landscape for Automotive and Manufacturing appeared first on Kai Waehner.

Apache Kafka + MQTT Blog Series

The first blog post explores the relation between MQTT and Apache Kafka. Afterward, the other four blog posts discuss various use cases, architectures, and reference deployments.

• Part 1 – Overview: Relation between Kafka and MQTT, pros and cons, architectures
• Part 2 – Connected Vehicles: MQTT and Kafka in a private cloud on Kubernetes; use case: remote control and command of a car
• Part 3 – Manufacturing: MQTT and Kafka at the edge in a smart factory; use case: Bidirectional OT-IT integration with Sparkplug between PLCs, IoT Gateways, Data Historian, MES, ERP, Data Lake, etc.
• Part 4 – Mobility Services: MQTT and Kafka leveraging serverless cloud infrastructure; use case: Traffic jam prediction service using machine learning
• Part 5 – Smart City (THIS POST): MQTT at the edge connected to fully-managed Kafka in the public cloud; use case: Intelligent traffic routing by combining and correlating 3rd party services

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Use Case: Smart City and 5G

A smart city is an urban area that uses different types of electronic Internet of Things (IoT) sensors to collect data and then use insights gained from that data to manage assets, resources, and services efficiently.

A smart city provides many benefits for civilization and city management. Some of the goals are:

• Improved Pedestrian Safety
• Improved Vehicle Safety
• Proactively Engaged First Responders
• Reduced Traffic Congestion
• Connected / Autonomous Vehicles
• Improved Customer Experience
• Automated Business Processes

I covered the use cases in more detail in the post “Event Streaming with Kafka as Foundation for a Smart City“. For a specific 5G example, check out “Building a Smart Factory with Apache Kafka and 5G Campus Networks“.

Let’s now explore the relation of Kafka and MQTT for smart city use cases.

Architecture: MQTT and Kafka for a Smart City

The following architecture shows an infrastructure deployed at a stadium:

In this example, both MQTT and Kafka are deployed close to the stadium. For instance, AWS Wavelength is an innovative infrastructure option to build low latency 5G use cases. The connected “regular AWS cloud region” is still used for use cases that do not require low latency.

The combination of Kafka and MQTT enables connectivity and real-time data processing for various use cases:

• Parking information and smart navigation.
• Location-based shopping and restaurant experiences, including innovative scenarios such as monitoring of queues and geofencing.
• Integration of loyalty platforms to earn rewards and points.
• Live information about the game or concert
• Lottery drawing experiences while watching a sports game.

The possibilities are endless. Integration with 1st and 3rd party applications will create completely new opportunities to improve the customer experience, increase safety, and improve operational efficiency.

The stadium example is a particular scenario to explore the added value of processing data in motion. Let’s take a look at other real-world examples that leverage MQTT and Kafka.

Example: Cloud-based Traffic Control Systems @Berlex

The Swedish company Berlex designs and manufactures new ways to improve traffic safety.

Berlex provides cloud-based portable traffic signals. Their innovative R6 traffic signal is one of the first mobile traffic signals controlled by a cloud-based service. Berlex’s connected solution allows customers to monitor the new traffic signals on a smartphone, computer, or tablet anytime and from anywhere. MQTT enables real-time information delivery and constant monitoring.

The cloud-based service reduces the time that their customers need to spend in dangerous traffic work zones. The system enables customers to carry out numerous tasks such as checking the battery status of a traffic signal or performing an inspection remotely, with no need for risky and time-consuming on-site intervention.

Each portable R6 traffic signal is equipped with a radar that allows the signal to see traffic. Sensors within the signals publish detailed information on the current status of the signal as MQTT data. The Berlex Connect cloud service captures the continuous stream of MQTT data from each signal and shares the information with the appropriate subscribers.

To prevent interruption of the traffic signal operation, high availability is essential for the system. Berlex customers monitor the real-time information on individual portals with customized user roles that fit their specific use case.

Read the complete case study from HiveMQ for more details about this successful smart city project.

Example: The Life of Citizens as a Stream of Events @ NAV

NAV (Norwegian Work and Welfare Department) currently distributes more than one-third of the national budget to Norway or abroad citizens. NAV assists people through all phases of life within work, family, health, retirement, and social security. Events happening throughout a person’s life determines which services we provide to them, how we provide them and when we provide them.

In most countries, each person has to apply for these services resulting in many tasks handled manually by various caseworkers in the organization. Their access to insight and useful information is limited and often hard to find, causing frustration to both our caseworkers and our users. By streaming a person’s life events through our Kafka pipelines, NAV revolutionized the way users experience government services and the way the employees work:

NAV and the government as a whole have access to vast amounts of data about the citizens, reported by health institutions, employers, various government agencies, or the users themselves. Some data is distributed by large batches, while others are available on-demand through APIs. The data is ingested into streams using Kafka, Streams API, and Java microservices. NAV distributes and acts on events about birth, death, relationships, employment, income, and business processes to vastly improve the user experience, provide real-time insight and reduce the need to apply for services the government already knows are needed.

NAV chose Confluent Platform to implement to get valuable insight from life and business events. Security is a key concern. Compliance with GDPR is essential for the success of this project.

More details about NAV’s Kafka usage in their Kafka Summit presentation.

Kafka + MQTT = Smart City

In conclusion, Apache Kafka and MQTT are a perfect combination for smart city and 5G use cases. Follow the blog series to learn about use cases such as connected vehicles, manufacturing, mobility services, and smart city. Every blog post also includes real-world deployments from companies across industries. It is key to understand the different architectural options to make the right choice for your project.

What are your experiences and plans in IoT projects? What use case and architecture did you implement? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka and MQTT (Part 5 of 5) – Smart City and 5G appeared first on Kai Waehner.

Apache Kafka + MQTT Blog Series

The first blog post explores the relation between MQTT and Apache Kafka. Afterward, the other four blog posts discuss various use cases, architectures, and reference deployments.

• Part 1 – Overview: Relation between Kafka and MQTT, pros and cons, architectures
• Part 2 – Connected Vehicles: MQTT and Kafka in a private cloud on Kubernetes; use case: remote control and command of a car
• Part 3 – Manufacturing: MQTT and Kafka at the edge in a smart factory; use case: Bidirectional OT-IT integration with Sparkplug between PLCs, IoT Gateways, Data Historian, MES, ERP, Data Lake, etc.
• Part 4 – Mobility Services (THIS POST): MQTT and Kafka leveraging serverless cloud infrastructure; use case: Traffic jam prediction service using machine learning
• Part 5 – Smart City: MQTT at the edge connected to fully-managed Kafka in the public cloud; use case: Intelligent traffic routing by combining and correlating 3rd party services

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Use Case: Mobility as a Service (MaaS) and Transportation

Transportation is changing significantly these days. Mobility Services – often called Mobility-as-a-Service (MaaS) – is a type of service that through a joint digital channel enables users to plan, book, and pay for multiple types of mobility services.

The concept describes a shift away from personally-owned modes of transportation and towards mobility provided as a service. This is enabled by combining transportation services from public and private transportation providers through a unified gateway that creates and manages the trip, which users can pay for with a single account. Users can pay per trip or a monthly fee for a limited distance. The key concept behind MaaS is to offer travelers mobility solutions based on their travel needs. Specialist urban mobility applications are also expanding their offerings to enable MaaS, such as Transit, Uber, and Lyft.

Travel planning typically begins in a journey planner. For example, a trip planner can show that the user can get from one destination to another by using a train/bus combination. The user can then choose their preferred trip based on cost, time, and convenience. At that point, any necessary bookings (e.g. calling a taxi, reserving a seat on a long-distance train) would be performed as a unit. It is expected that this service should allow roaming, that is, the same end-user app should work in different cities, without the user needing to become familiar with a new app or to sign up for new services.

As you can hopefully already imagine, plenty of innovative new use cases are possible by combining Apache Kafka and MQTT for MaaS. And most of these scenarios require data integration and data processing at scale in real-time.

Architecture: MQTT and Kafka for Mobility Services

Mobility services are often separated from other core IT infrastructure. MaaS – as the term says – is just consumed as a service. Hence, most mobility services I have seen run in the cloud. The following diagram shows an intelligent navigation service built with MQTT, Kafka, and Machine Learning:

A few notes on the architecture:

• As mobility services connect to moving vehicles, smartphones, or other things, the cloud is perfect. No need to operate the infrastructure. Just focus on building applications.
• Many mobility services integrate other 1st or 3rd party services. For instance, there is no need to build yet another mapping service. If you need one for building your innovative new application, just embed HERE Technologies (that actually provides a public Kafka interface as the preferred integration option instead of HTTP!) or any other available mapping service.
• Regional services with low latency are often very relevant for mobility services. Hence, multiple MQTT and Kafka clusters are the norm, not an exception.

Let’s take a look at some real-world examples for cutting-edge mobility services in the transportation industry.

Example: Cloud Ecosystem for Next-Generation Mobility @ ZF

ZF Friedrichshafen AG is a global automotive supplier that enables vehicles to see, think, and act. With a broad range of systems for passenger cars, commercial vehicles, and industrial technology, ZF offers comprehensive solutions for established vehicle manufacturers as well as newly emerging transport and mobility service providers.

ZFs Connectivity Suite enables new business models for mobility as a service (MaaS) and transportation as a service (TaaS). The ProCV gateway device allows each vehicle to communicate using MQTT. The gateway provides a secure and reliable channel for transferring telemetry data from the car to the cloud and remote commands from the cloud to each vehicle.

Applications can exchange data such as real-time positioning information, remote commands to the vehicle, and vehicle-generated alerts. Some possible use cases:

• Remote diagnostics for technical insight & management of vehicle performance
• Fleet monitoring
• Secure & reliable middleware between connected vehicles & cloud services

Read the case study from HiveMQ for more details about ZF’s IoT gateway.

Example: Real-Time Traveler Information @ Deutsche Bahn

Deutsche Bahn (the German railway) has a very complex network of short-distance and long-distance trains. Hence, delays and cancellations are common, not an exception. Hence, at least the traveler information should work well to send real-time notifications to customers.

For that reason, Deutsche Bahn has built a single source of truth traveler information platform with Confluent:

The mobility service integrates via Kafka with many legacy and modern applications. The mobile app shows real-time status updates about each train. While train delays and cancellations cannot be avoided completely, the app at least allows you to get to a lounge or grab a coffee if the delay is more than just a few minutes. I use the app every week myself and can confirm that the customer experience improved significantly.

Not every interface is or will be real-time. Kafka helps!

Fun fact: The first proof of concept to build this traveler information app used a traditional messaging queue. In theory, this is is sufficient as you “just” need to send status updates in real-time to the mobile app. Unfortunately, a few issues came up quickly:

• Not every interface is real-time! In addition to messaging sources such as JMS or MQTT, the platform needed to integrate with databases, files, web services, and other legacy systems. Hence, data integration and is a key piece of the puzzle.
• Data storage is important to handle backpressure and decouple applications. Slow consumers fall behind. Analytics workloads take data in batches, not in real-time. Web applications consume specific events via request-response queries.
• Sending events from A to B is just part of the problem! The real added value comes by correlating the data from streaming and non-streaming applications and databases in real-time. Kafka-native Stream processing frameworks such as Kafka Streams or ksqlDB help to process data in motion.

For the above reasons, Deutsche Bahn re-started their proof of concept. Their existing project used different frameworks for messaging, caching, integration, and processing. This setup was replaced with Kafka. Kafka Connect integrates applications and databases. Kafka Streams processes the data in motion. The Kafka storage handles backpressure and slow consumers.  All of this is built into Kafka out-of-the-box. And it scales much better. Today, the traveler information system is live and creates a much better customer experience.

Find more details about the traveler information system from Deutsche Bahn in their Confluent blog post.

Kafka + MQTT = Mobility Services and Transportation

In conclusion, Apache Kafka and MQTT are a perfect combination for mobility services and transportation. Follow the blog series to learn about use cases such as connected vehicles, manufacturing, mobility services, and smart city. Every blog post also includes real-world deployments from companies across industries. It is key to understand the different architectural options to make the right choice for your project.

What are your experiences and plans in IoT projects? What use case and architecture did you implement? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka and MQTT (Part 4 of 5) – Mobility Services and Transportation appeared first on Kai Waehner.