Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases.

The Business of Mobility Services (Ride-Hailing, Food Delivery, Taxi Aggregators, etc.)

Mobility services have become an essential part of modern urban life. They offer convenience and efficiency through ride-hailing, food delivery, car-sharing, e-scooters, taxi aggregators, and micro-mobility options. Companies such as Uber, Lyft, FREE NOW (former MyTaxi; acquired by Lyft recently), Grab, Careem, and DoorDash connect millions of passengers, drivers, restaurants, retailers, and logistics partners to enable seamless transactions through digital platforms.

These platforms operate in highly dynamic environments where real-time data is crucial for pricing, route optimization, customer experience, and fraud detection. However, this very nature of mobility services also makes them prime targets for fraudulent activities. Fraud in this sector can lead to financial losses, reputational damage, and deteriorating customer trust.

To effectively combat fraud, mobility services must rely on real-time data streaming with technologies such as Apache Kafka and Apache Flink. These technologies enable continuous event processing and allow platforms to detect and prevent fraud before transactions are finalized.

Why Fraud is a Major Challenge in Mobility Services

Fraudsters continually exploit weaknesses in digital mobility platforms. Some of the most common fraud types include:

1. Fake Rides and GPS Spoofing: Drivers manipulate GPS data to simulate trips that never occurred. Passengers use location spoofing to receive cheaper fares or exploit promotions.

2. Payment Fraud and Stolen Credit Cards: Fraudsters use stolen payment methods to book rides or order food.

3. Fake Drivers and Passengers: Fraudsters create multiple accounts and pretend to be both the driver and passenger to collect incentives. Some drivers manipulate fares by manually adjusting distances in their favor.

4. Promo Abuse: Users create multiple fake accounts to exploit referral bonuses and promo discounts.

5. Account Takeovers and Identity Fraud: Hackers gain access to legitimate accounts, misusing stored payment information. Fraudsters use fake identities to bypass security measures.

Fraud not only impacts revenue but also creates risks for legitimate users and drivers. Without proper fraud prevention measures, ride-hailing and delivery companies could face serious losses, both financially and operationally.

The Unseen Enemy: Core Challenges in Mobility Fraud
Detection

Traditional fraud detection relies on batch processing and manual rule-based systems. However, these approaches are no longer effective due to the speed and complexity of modern mobile apps with real-time experiences combined with modern fraud schemes.

Key challenges in mobility fraud detection include:

• Fraud occurs in real-time, requiring instant detection and prevention before transactions are completed.
• Millions of events per second must be processed, requiring scalable and efficient systems.
• Fraud patterns constantly evolve, making static rule-based approaches ineffective.
• Platforms operate across hybrid and multi-cloud environments, requiring seamless integration of fraud detection systems.

How Data Streaming with Apache Kafka and Flink Enables Real-Time Fraud Detection

To overcome these challenges, real-time streaming analytics powered by Apache Kafka and Apache Flink provide an effective solution.

Apache Kafka: The Backbone of Event-Driven Fraud Detection

Kafka serves as the core event streaming platform. It captures and processes real-time data from multiple sources such as:

• GPS location data
• Payment transactions
• User and driver behavior analytics
• Device fingerprints and network metadata

Kafka provides:

• High-throughput data streaming, capable of processing millions of events per second to support real-time decision-making.
• An event-driven architecture that enables decoupled, flexible systems—ideal for scalable and maintainable mobility platforms.
• Seamless scalability across hybrid and multi-cloud environments to meet growing demand and regional expansion.
• Always-on reliability, ensuring 24/7 data availability and consistency for mission-critical services such as fraud detection, pricing, and trip orchestration.

An excellent success story about the transition to data streaming comes from DoorDash: Why DoorDash migrated from Cloud-native Amazon SQS and Kinesis to Apache Kafka and Flink.

Apache Flink: Continuous Stream Processing for Fraud Detection in Real-Time

Apache Flink enables real-time fraud detection through advanced event correlation and applied AI:

• Detects anomalies in GPS data, such as sudden jumps, route manipulation, or unrealistic movement patterns.
• Analyzes historical user behavior to surface signs of account takeovers or other forms of identity misuse.
• Joins multiple real-time streams—including payment events, location updates, and account interactions—to generate accurate, low-latency fraud scores.
• Applies machine learning models in-stream, enabling the system to flag and stop suspicious transactions before they are processed.
• Continuously adapts to new fraud patterns, updating models with fresh data in near real-time to reflect evolving user behavior and emerging threats.

With Kafka and Flink, fraud detection can shift from reactive to proactive to stop fraudulent transactions before they are completed.

I already covered various data streaming success stories from financial services companies such as Paypal, Capital One and ING Bank in a dedicated blog post. And a separate case study from about “Fraud Prevention in Under 60 Seconds with Apache Kafka: How A Bank in Thailand is Leading the Charge“.

Real-World Fraud Prevention Stories from Mobility Leaders

Fraud is not just a technical issue—it’s a business-critical challenge that impacts trust, revenue, and operational stability in mobility services. The following real-world examples from industry leaders like FREE NOW (Lyft), Grab, and Uber show how data streaming with advanced stream processing and AI are used around the world to detect and stop fraud in real time, at massive scale.

FREE NOW (Lyft): Detecting Fraudulent Trips in Real Time by Analyzing GPS Data of Cars

FREE NOW operates in more than 150 cities across Europe with 48 million users. It integrates multiple mobility services, including taxis, private vehicles, car-sharing, e-scooters, and bikes.

The company was recently acquired by Lyft, the U.S.-based ride-hailing giant known for its focus on multimodal urban transport and strong presence in North America. This acquisition marks Lyft’s strategic entry into the European mobility ecosystem, expanding its footprint beyond the U.S. and Canada.

Fraud Prevention Approach leveraging Data Streaming (presented at Kafka Summit)

• Uses Kafka Streams and Kafka Connect to analyze GPS trip data in real-time.
• Deploys fraud detection models that identify anomalies in trip routes and fare calculations.
• Operates data streaming on fully managed Confluent Cloud and applications on Kubernetes for scalable fraud detection.

Example: Detecting Fake Rides

1. A driver inputs trip details into the app.
2. Kafka Streams predicts expected trip fare based on distance and duration.
3. GPS anomalies and unexpected route changes are flagged.
4. Fraud alerts are triggered for suspicious transactions.

By implementing real-time fraud detection with Kafka and Flink, FREE NOW (Lyft) has significantly reduced fraudulent trips and improved platform security.

Grab: AI-Powered Fraud Detection for Ride-Hailing and Delivery with Data Streaming and AI/ML

Grab is a leading mobility platform in Southeast Asia, handling millions of transactions daily. Fraud accounts for 1.6 percent of total revenue loss in the region.

To address these significant fraud numbers, Grab developed GrabDefence—an AI-powered fraud detection engine that leverages real-time data and machine learning to detect and block suspicious activity across its platform.

Fraud Detection Approach

• Uses Kafka Streams and machine learning for fraud risk scoring.
• Leverages Flink for feature aggregation and anomaly detection.
• Detects fraudulent transactions before they are completed.

Example: Fake Driver and Passenger Fraud

1. Fraudsters create accounts as both driver and passenger to claim rewards.
2. Kafka ingests device fingerprints, payment transactions, and ride data.
3. Flink aggregates historical fraud behavior and assigns risk scores.
4. High-risk transactions are blocked instantly.

With GrabDefence built with data streaming, Grab reduced fraud rates to 0.2 percent, well below the industry average. Learn more about GrabDefence in the Kafka Summit talk.

Uber: Project RADAR – AI-Powered Fraud Detection with Human Oversight

Uber processes millions of payments per second globally. Fraud detection is complex due to chargebacks and uncollected payments.

To combat this, Uber launched Project RADAR—a hybrid system that combines machine learning with human reviewers to continuously detect, investigate, and adapt to evolving fraud patterns in near real time. Low latency is not required in this scenario. And humans are in the loop of the business process. Hence, Apache Spark is sufficient for Uber.

Fraud Prevention Approach

• Uses Kafka and Spark for multi-layered fraud detection.
• Implements machine learning models to detect chargeback fraud.
• Incorporates human analysts for rule validation.

Example: Chargeback Fraud Detection

1. Kafka collects all ride transactions in real time.
2. Stream processing detects anomalies in payment patterns and disputes.
3. AI-based fraud scoring identifies high-risk transactions.
4. Uber’s RADAR system allows human analysts to validate fraud alerts.

Uber’s combination of AI-driven detection and human oversight has significantly reduced chargeback-related fraud.

Data Streaming with Kafka and Flink Provides Real-Time Fraud Detection in Mobility Services

Fraud in mobility services is a real-time challenge that requires real-time solutions that work 24/7, even at extreme scale for millions of events. Traditional batch processing systems are too slow, and static rule-based approaches cannot keep up with evolving fraud tactics.

By leveraging data streaming with Apache Kafka in conjunction with Kafka Streams or Apache Flink, mobility platforms can:

• Process millions of events per second to detect fraud in real time.
• Prevent fraudulent transactions before they occur.
• Use AI-driven real-time fraud scoring for accurate risk assessment.
• Adapt dynamically through continuous learning to evolving fraud patterns.

Mobility platforms such as Uber, Grab, and FREE NOW (Lyft) are leading the way in using real-time streaming analytics to protect their platforms from fraud. By implementing similar approaches, other mobility businesses can enhance security, reduce financial losses, and maintain customer trust.

Real-time fraud prevention in mobility services is not an option; it is a necessity. The ability to detect and stop fraud in real time will define the future success of ride-hailing, food delivery, and urban mobility platforms.

Stay ahead of the curve! Subscribe to my newsletter for insights into data streaming and connect with me on LinkedIn to continue the conversation. And download my free book about data streaming use cases.

The post Fraud Detection in Mobility Services (Ride-Hailing, Food Delivery) with Data Streaming using Apache Kafka and Flink appeared first on Kai Waehner.

General trends in the financial services industry

Researchers, analysts, startups, and last but not least, labs and the first real-world rollouts of traditional players show a few upcoming trends in the financial services industry:

• Digital Transformation in Financial Services (see Gartner’s Hype Cycle for Digital Banking Transformation)
• Mainframe: A Critical Part Of Modern IT Strategies (see Deloitte’s Forrester Report about the value and support of mainframes in the modern IT world)
• Public Cloud and Open Banking APIs (see Forrester’s article about the history of banks moving to the cloud)

Let’s explore the goals and impact of these trends.

Innovation: Digital banking transformation

Gartner says that four technologies have the potential for high levels of transformation in the banking sector and are likely to mature within the next couple of years:

• Banking as a Service (BaaS) can be a discrete or broad set of financial service functions exposed by chartered banks or regulated entities to power new business models deployed by other banking market participants – fintechs, neobanks, traditional banks, and other third parties.
• Chatbots in banks will affect all areas of communication between machines and humans.
• Public Cloud for Banking is becoming highly transformational to the banking industry since banks can achieve greater efficiency and agility by moving workloads to the cloud.
• Social Messaging Payment Apps rely on instant messaging platforms to originate payment transactions. The messaging app interface is used to register payment accounts and to initiate and monitor related transactional activity.

Gartner’s Hype Cycle for Digital Banking Transformation shows the state of these and other trends in the financial services sector:

Mainframe: A critical part of modern IT strategies

Deloitte published an interesting analysis by Forrester. It confirms my experience from customer meetings: “Unparalleled processing power and high security make the mainframe a strategic component of hybrid environments”. Or: The Mainframe is here to stay:

The mainframe has many disadvantages: Monolithic, legacy protocols and programming languages, inflexible, expensive, etc. However, the mainframe is battle-tested and works well for many mission-critical applications worldwide.

And look at the specifications of a modern mainframe: The IBM z15 was announced in 2019 with up to 40TB RAM and 190 Cores. Wow. Impressive! But it typically costs millions $$$ (variable software costs not included).

Public cloud and Open API: Data sharing in real-time at elastic scale 

Many banks worldwide – usually more conservative enterprises – have a cloud-first strategy in 2023. Not just for analytics or reporting (that’s a great way to get started, though), but also for mission-critical workloads like core banking. And we are talking about the public cloud here (e.g., AWS, Azure, GCP, Alibaba). Not just private cloud deployments in their own data center built with Kubernetes or similar container-based, cloud-native technologies.

The Forrester article “A Short History Of Financial Services In The Cloud” is an excellent reminder: “But in 2015, Capital One shocked the world with its pronouncement of going all in on Amazon Web Services (AWS) and even migrating existing applications — an unheard-of notion for any company across any industry. Capital One claimed AWS could better secure its workloads than Capital One’s highly qualified security staff. Following this announcement, an onslaught of digital-native banks followed suit.”

The cloud is not cheaper. But it provides a flexible and elastic infrastructure to focus on business and innovation, not operations of IT. Open APIs are normal in cloud-native infrastructure. Open Banking trends and standards like PSD2 (European regulation for electronic payment services) make payments more secure, ease innovation, and enable easier integration between B2B partners.

Data streaming in financial services

Adopting trends like mobile social payment apps or open banking APIs is only possible if enterprises in the financial services world can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means):

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

“Apache Kafka in the Financial Services Industry” is a good article for starting with an industry-specific point of view on data streaming.

This is just one example. Data streaming with the Apache Kafka ecosystem and cloud services are used throughout the supply chain of the finance industry. Search my blog for various articles related to this topic: Search Kai’s blog.

Data streaming as a business enabler

Forbes published a great article about “high-frequency data and event streaming in banking“. Here are a few of Forbes’ pivotal use cases for data streaming in financial services:

Retail Banking

• From a limited view of the customer to a 360-degree view of the customer.
• Hyper-personalized customer experiences.
• App-first customer interactions.

Payments Processing

• Typical T+1 or T+2 batch of settlement cycle to real-time transfers
• Stringent event-driven infrastructure with decoupled producers and consumers.

Capital Markets

• From overnight pricing models to real-time sensitivities & market risk.
• Managing the complexities of timing and volume related to co-located execution
• T+1 trading Settlements to Automated clearing and settlement (T+0).

Open API for flexibility and faster time to market

Real-time data beats slow data in almost all use cases. But as essential is data consistency and an Open API approach across all systems, including non-real-time legacy systems and modern request-response APIs.

Apache Kafka’s most underestimated feature is the storage component based on the append-only commit log. It enables loose coupling for domain-driven design with microservices and independent data products in a data mesh.

Here is an example of an open banking architecture for all payment information systems and external interfaces:

Architecture trends for data streaming

The financial services industry applies various trends for enterprise architectures for cost, flexibility, security, and latency reasons. The three major topics I see these days at customers are:

• Legacy modernization by offloading and migrating from monoliths to cloud-native
• Hyper-personalized customer experience with omnichannel banking and context-specific decisions
• Real-time analytics with stateless and stateful data correlation for transactional and analytical workloads
• Mission-critical data streaming across data centers and clouds for high availability and compliance

Legacy modernization and data offloading

I covered the steps for a successful legacy modernization in various blog posts. Mainframe offloading is an excellent example. While existing transactional workloads still run on the mainframe using IBM DB2, VSAM, CICS, etc., event changes are pushed to the data streaming platform and stored in the Kafka log. New applications can be built with any technology and communication paradigm to access the data:

Hyper-personalized customer experience

Customers expect a great customer experience across devices (like a web browser or mobile app) and human interactions (e.g., in a bank branch). Data streaming enables a context-specific omnichannel banking experience by correlating real-time and historical data at the right time in the right context:

“Omnichannel Retail and Customer 360 in Real Time with Apache Kafka” goes into more detail. This is a great example where the finance sector can learn from other industries that had to innovate a few years earlier to solve a specific challenge or business problem.

Real-time analytics with stream processing

Real-time data beats slow data. That’s true for most analytics scenarios. If you detect fraud after the fact in your data warehouse, it is nice… But too late! Instead, you need to detect and prevent fraud before it happens.

This requires intelligent decision-making in real-time (< 1 second end-to-end). Data Streaming’s stream processing capabilities with technologies like Kafka Streams, KSQL, or Apache Flink are built for these use cases. Stream processing is scalable for millions of transactions per second and is reliable to guarantee zero data loss.

Here is an example of stateless transaction monitoring of payment spikes (i.e., looking at one event at a time):

More powerful stateful stream processing aggregates and correlates events from one or more data sources together continuously in real time:

The article “Fraud Detection with Apache Kafka, KSQL and Apache Flink” explores stream processing for real-time analytics in more detail, shows an example with embedded machine learning, and covers several real-world case studies.

Mission-critical data streaming with a stretched Kafka cluster across regions

The most critical use cases require business continuity, even if disaster strikes and a data center or complete cloud region goes down.

Data streaming powered by Apache Kafka enables various architectures and deployments for different SLAs. Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. A single Kafka cluster stretched across regions (like the US East, Central, and West) is the most resilient way to deploy data streaming:

Please note that this capability of Multi-Region Clusters (MRC) for Kafka is only available in the Confluent Platform as a commercial product.

In my other blog post, learn about architecture patterns for Apache Kafka that may require multi-cluster solutions and see real-world examples with their specific requirements and trade-offs. That blog explores scenarios such as disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments, and global Kafka.

New customer stories for data streaming in the financial services industry

So much innovation is happening in the financial services industry. Automation and digitalization change how we process payment, prevent fraud, communicate with partners and customers, and so much more.

Most FinServ enterprises use a cloud-first (but still hybrid) approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure.

Here are a few customer stories from worldwide FinServ enterprises across industries:

• Erste Bank: Hyper-personalized mobile banking
• Singapore Stock Exchange (SGX): Modernized trading platform with a hybrid architecture
• Citigroup: Global payment applications with redundancy and scalability to support 99.9999% uptime
• Raiffeisen Bank International: Hybrid data mesh across countries with end-to-end data governance
• Capital One: Context-specific fraud detection and prevention in real-time
• 10X Banking: Cloud-native core banking platform using a modern Kappa architecture

Find more details about these case studies in the below slide deck and video recording.

Resources to learn more

This blog post is just the starting point. Learn more in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-demand video recording

The video recording explores the FinServ industry’s trends and architectures for data streaming. The primary focus is the data streaming case studies. Check out our on-demand recording:

Slides

If you prefer learning from slides, check out the deck used for the above recording:

Case studies and lightboard videos for data streaming in financial services

The state of data streaming for financial services in 2023 is fascinating. New use cases and case studies come up every month. This includes better data governance across the entire organization, collecting and processing data from payment interfaces in real-time, data sharing and B2B partnerships with Open Banking APIs for new business models, and many more scenarios.

We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories. Here are the videos for the FinServ sector, each one includes a case study:

• Episode 1: Data Streaming in Real Life: Banking
• Episode 2: Core Banking
• Episode 3: Fraud Detection
• Episode 4: Cost Reduction Through Mainframe Offloading
• Episode 5: Data Mesh

And this is just the beginning. Every month, we will talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, and so on…

Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post The State of Data Streaming for Financial Services appeared first on Kai Waehner.

Disruptive Trends in Retail

A few general trends completely change the retail industry:

• Highly competitive market, work to thin margins
• Moving from High Street (brick & mortar) to Online (OnmiChannel)
• Personalized Customer Experience – optimal buyer journey

These trends require retail companies to create new business models, provide a great customer experience, and improve operational efficiencies:

Event Streaming with Apache Kafka in Retail

Many use cases for event streaming are not new. Instead, Apache Kafka enables faster processing at a larger scale with a lower cost and reduced risk:

Hence, Kafka is not just used for greenfield projects in the retail industry. It very often complements existing applications in a brownfield architecture. Plenty of material explores this topic in more detail. For instance, check out the following:

• Kafka as modern middleware as a replacement or together with other MQ / ETL / ESB infrastructure
• Kafka in conjunction with API Management tools
• Integration with legacy mainframe applications
• Event Streaming as the foundation of postmodern ERP, logistics, transportation systems

Let’s now take a look at a few public examples that leverage all the above capabilities.

Real World Use Cases for Kafka in Retail

Various deployments across the globe leverage event streaming with Apache Kafka for very different use cases. Consequently, Kafka is the right choice, no matter if you need to optimize the supply chain, disrupt the market with innovative business models, or build a context-specific customer experience. Here are a few examples:

• Walmart – Real-Time Inventory System: 8,500 nodes processing 11 billion events per day deliver an omnichannel experience so every customer can shop the way they want to
• Target – Omnichannel Distribution and Logistics: Data correlation of events from distribution centers, stores, digital channels, and customer interactions enable real-time omnichannel scale
• Nuuly – Clothing Rental Subscription Service: Very different from a typical e-commerce model with the need for a real-time event-driven architecture
• AO.com – Context-specific Customer 360: Hyper-personalized online retail experience, turning each customer visit into a one-on-one marketing opportunity via a correlation of historical customer data with real-time digital signals
• Mojix – Retail and Supply Chain IoT Platform: Real-time operational intelligence across the edge and the cloud improves inventory accuracy, enables operational intelligence, supports omnichannel sales

The architectures of retail deployments often leverage a fully-managed serverless infrastructure with Confluent Cloud or deploy in hybrid architectures across data centers, clouds, and edge sites. Let’s now take a look at one example.

Omnichannel and Customer 360 across the Supply Chain with Kafka

Omnichannel retail requires the combination of various different tasks and applications across the whole supply chain. Some tasks are real-time while others are batch or historical data:

• Customer interactions, including website, mobile app, on-site in store
• Reporting and analytics, including business intelligence and machine learning
• R&D and manufacturing
• Marketing, loyalty system, and aftersales

The real business value is generated by correlating all the data from these systems in real-time. That’s where Kafka is a perfect fit due to its combination of different capabilities: Real-time message at scale, storage for decoupling and caching, data integration, continuous data processing.

Hybrid Architecture from Edge to Cloud

The following picture shows a possible retail architecture leveraging event streaming. It runs many mission-critical workloads and integrations in the cloud. However, the context-specific recommendations, point of sale payment and loyalty processing, and other relevant use cases are executed at the disconnected edge in each retail store:

I will dig deeper into this architecture and talk about specific requirements and challenges solved with Kafka at the edge and in the cloud for retailers. For now, check out the following posts to learn about global Kafka deployments and Kafka at the edge in the retail stores:

• Kafka is the new black at the edge in retail stores
• Use cases and architectures for Kafka outside the data center
• Architectures for hybrid and global Kafka deployments

Slides and Video – Disruption in Retail with Kafka

The following slide deck explores the usage of Kafka in retail in more detail:

Also, here is a link to the on-demand video recording:

Software (including Kafka) is Eating Retail

In conclusion, Event Streaming with Apache Kafka plays a key role in this evolution of re-inventing the retail business. Walmart, Target, and many other retail companies rely on Apache Kafka and its ecosystem to provide a real-time infrastructure to make the customer happy, increase revenue, and stay competitive in this tough industry.

What are your experiences and plans for event streaming in the retail industry? Did you already build applications with Apache Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Use Cases for Apache Kafka in Retail appeared first on Kai Waehner.