Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And download my free book about data streaming use cases, including various use cases from the retail industry.

What is Retail Media?

Retail media is transforming how brands advertise by leveraging first-party data from retailers to create highly targeted ads within their ecosystems. Instead of relying solely on third-party data from traditional digital advertising platforms, retail media allows companies to reach consumers at the point of purchase—whether online, in-store, or via mobile apps.

Retail media is one of the fastest-growing and most strategic revenue streams for retailers today. It has transformed from a niche digital advertising concept into a multi-billion-dollar industry, changing how retailers monetize their data and engage with brands. Below are the key reasons retail media is crucial for retailers.

Retailers like Amazon, Walmart, and Albertsons are leading the way in monetizing their digital real estate, offering brands access to sponsored product placements, banner ads, video ads, and personalized promotions based on shopping behavior. This shift has made retail media one of the fastest-growing sectors in digital advertising, expected to exceed $100 billion globally in the coming years.

The Digitalization of Retail Media

Retail media has grown from traditional in-store promotions to a fully digitized, data-driven advertising ecosystem. The rise of e-commerce, mobile apps, and connected devices has enabled retailers to:

• Collect granular consumer behavior data in real time
• Offer personalized promotions to drive higher conversion rates
• Provide advertisers with measurable ROI and closed-loop attribution
• Leverage AI and machine learning for dynamic ad targeting

By integrating digital advertising with real-time customer data and real-time inventory, retailers can provide contextually relevant promotions across multiple touchpoints. The key to success lies in seamlessly connecting online and offline shopping experiences—a challenge that data streaming with Apache Kafka and Flink helps solve.

Online, Brick-and-Mortar, and Hybrid Retail Media

Retail media strategies vary depending on whether a retailer operates online, in-store, or in a hybrid model:

• Online-Only Retail Media: Retail giants like Amazon and eBay leverage vast amounts of digital consumer data to offer programmatic ads, sponsored products, and personalized recommendations directly on their websites and apps.
• Brick-and-Mortar Retail Media: Traditional retailers like Target and Albertsons are integrating digital signage, in-store Wi-Fi promotions, and AI-powered shelf displays to engage customers while shopping in physical stores.
• Hybrid Retail Media: Retailers like Walmart and Kroger are bridging the gap between digital and physical shopping experiences with omnichannel marketing strategies, personalized mobile app promotions, and AI-powered customer insights that drive both online and in-store purchases.

Omnichannel vs. Unified Commerce in Retail Media

Retailers are moving beyond omnichannel marketing, where customer interactions happen across multiple channels, to unified commerce, where all customer data, inventory, and marketing campaigns are synchronized in real time.

• Omnichannel: Offers a seamless shopping experience across different platforms but often lacks real-time data integration.
• Unified Commerce: Uses real-time data streaming to unify customer behavior, inventory management, and personalized advertising for a more cohesive experience.

For example, a unified commerce strategy allows a retailer to:

• Track customer interactions in real time across e-commerce, mobile, and in-store purchases
• Deliver personalized product recommendations based on live shopping behavior
• Optimize ad placements and promotions based on real-time demand and inventory availability

This level of integration is only possible with real-time data streaming using technologies such as Apache Kafka and Apache Flink.

How Data Streaming with Apache Kafka and Flink Powers Retail Media

Retail media networks require real-time data processing at scale to manage millions of customer interactions across online and offline touchpoints. Kafka and Flink provide the foundation for a scalable, event-driven infrastructure that enables retailers to:

• Process customer behavior in real time: Tracking clicks, searches, and in-store activity instantly
• Deliver hyper-personalized ads and promotions: AI-driven dynamic ad targeting
• Optimize inventory and pricing: Aligning promotions with real-time stock levels
• Measure campaign performance instantly: Providing brands with real-time attribution and insights

With Apache Kafka as the backbone for data streaming and Apache Flink for real-time analytics, retailers can ingest, analyze, and act on consumer data within milliseconds.

Here are a few examples of input data sources, stream processing applications, and outputs for other systems:

Input Data Sources for Retail Media

1. Customer transaction data (e.g., point-of-sale purchases, online orders)
2. Website and app interactions (e.g., product views, searches, cart additions)
3. Loyalty program data (e.g., customer preferences, purchase frequency)
4. Third-party ad networks (e.g., campaign performance data, audience segments)
5. In-store sensor and IoT data (e.g., foot traffic, digital shelf interactions)

Stream Processing Applications for Retail Media

1. Real-time advertisement personalization engine (customizes promotions based on live behavior)
2. Dynamic pricing optimization (adjusts ad bids and discounts in real-time)
3. Customer segmentation & targeting (creates audience groups based on behavioral signals)
4. Fraud detection & clickstream analysis (identifies bot traffic and fraudulent ad clicks)
5. Omnichannel attribution modeling (correlates ads with online and offline purchases)

Output Systems for Retail Media

1. Retail media network platforms (e.g., sponsored product listings, display ads)
2. Programmatic ad exchanges (e.g., Google Ads, The Trade Desk, Amazon DSP)
3. CRM & marketing automation tools (e.g., Salesforce, Adobe Experience Cloud)
4. Business intelligence dashboards (e.g., Looker, Power BI, Tableau)
5. In-store digital signage & kiosks (personalized promotions for physical shoppers)

Key Retail Media Use Cases for Data Streaming with Kafka & Flink

Real-time data streaming with Kafka and Flink enables critical retail media use cases by processing vast amounts of data from customer interactions, inventory updates and advertising platforms. The ability to analyze and act on data instantly allows retailers to optimize ad placements, enhance personalization, and measure the effectiveness of marketing campaigns with unprecedented accuracy. Below are some of the most impactful retail media applications powered by event-driven architectures.

Personalized In-Store Promotions

Retailers can use real-time customer location data, combined with purchase history and preferences, to deliver highly personalized promotions through mobile apps or digital signage. By incorporating location-based services (LBS), the system detects when a shopper enters a specific section of a store and triggers a targeted discount or special offer. For example, a customer browsing the beverage aisle might receive a notification offering 10% off their favorite soda, increasing the likelihood of an impulse purchase.

Dynamic Ad Placement & Bidding

Kafka and Flink power real-time programmatic advertising, enabling retailers to dynamically adjust ad placements and bids based on customer activity and shopping trends. This allows advertisers to serve the most relevant ads at the optimal time, maximizing engagement and conversions. For instance, Walmart Connect continuously analyzes in-store and online behavior to adjust which ads appear on product pages or search results, ensuring brands reach the right shoppers at the right moment.

Inventory-Aware Ad Targeting

Real-time inventory tracking ensures that advertisers only bid on ads for products that are in stock and ready for fulfillment, reducing wasted ad spend and improving customer satisfaction. This integration between retail media networks and inventory systems prevents scenarios where customers click on an ad only to find the item unavailable. For example, if a popular TV model is running low in a specific store, the system can prioritize ads for a similar in-stock product, ensuring a seamless shopping experience.

Fraud Detection & Brand Safety

Retailers must protect their media platforms from click fraud, fake engagement, and suspicious transactions, which can distort performance metrics and drain marketing budgets.

Kafka and Flink enable real-time fraud detection by analyzing patterns in ad clicks, user behavior, and IP addresses to identify bots or fraudulent activity. For example, if an unusual spike in ad impressions originates from a single source, the system can immediately block the traffic, safeguarding advertisers’ investments.

Real-Time Attribution & Measurement

Retail media networks must provide advertisers with instant insights into ad performance by linking online interactions to in-store purchases.

Kafka enables event-driven attribution models, allowing brands to measure how digital ads drive real-world sales. For example, if a customer clicks on an ad for running shoes, visits a store, and buys them later, the platform tracks the conversion in real time, ensuring brands understand the full impact of their campaigns. Solutions like Segment (built on Kafka) provide robust customer data platforms (CDPs) that help retailers unify and analyze customer journeys.

Retail Media as an Advertising Channel for Third-Party Brands

Retailers are increasingly leveraging third-party data sources to bridge the gap between retail media networks and adjacent industries, such as quick-service restaurants (QSRs).

Kafka enables seamless data exchange between grocery stores, delivery apps, and restaurant chains, optimizing cross-industry advertising. For example, a burger chain could dynamically adjust digital menu promotions based on real-time data from a retail partner—if a grocery store’s sales data shows a surge in plant-based meat purchases, the restaurant could prioritize ads for its new vegan burger, ensuring more relevant and effective marketing.

Albertsons’ New Retail Media Strategy Leveraging Data Streaming

One of the most innovative retail media success stories comes from Albertsons. Albertsons is one of the largest supermarket chains in the United States, operating over 2,200 stores under various banners, including Safeway, Vons, and Jewel-Osco, and providing groceries, pharmacy services, and household essentials.

I explored Albertsons in another article about its revamped loyalty platform to retain customers for life. Data streaming is essential and a key strategic part of Albertsons’ enterprise architecture:

When I hosted a webinar with Albertsons around two years ago on their data streaming strategy, retail media was one of the bullet points. But I didn’t yet realize until now how crucial it would become for retailers:

• Retail Media Network Expansion: Albertsons has launched its own retail media network, leveraging first-party data to create highly targeted advertising campaigns.
• Real-Time Personalization: With real-time data streaming, Albertsons can provide personalized promotions based on customer purchase history, in-store behavior, and digital engagement.
• AI-Powered Insights: Albertsons uses AI and machine learning on top of streaming data pipelines to optimize ad placements, campaign effectiveness, and dynamic pricing strategies.
• Data Monetization: By offering data-driven advertising solutions, Albertsons is monetizing its shopper data while enhancing the customer experience with relevant, timely promotions.

Business Value of Real-Time Retail Media

Retailers that adopt data streaming with Kafka and Flink for retail media strategies to unlock massive business value:

• New Revenue Streams: Retail media monetization drives ad sales growth
• Higher Conversion Rates: Real-time targeting improves customer engagement
• Better Customer Insights: Streaming analytics enables deep behavioral insights
• Competitive Advantage: Retailers with real-time personalization outperform rivals
• Better Customer Experience: Retail media reduces friction and enhances the shopping journey through personalized promotions

The Future of Retail Media is Real-Time and Context-Specific Data Streaming

Retail media is no longer just about placing ads on retailer websites—it’s about delivering real-time, data-driven advertising experiences across every consumer touchpoint.

With Kafka and Flink powering real-time data streaming, retailers can:

• Unify online and offline shopping experiences
• Enhance personalization with AI-driven insights
• Maximize ad revenue with real-time campaign optimization

As retailers like Albertsons, Walmart, and Amazon continue to innovate, the future of retail media will be hyper-personalized, data-driven, and real-time.

How is your organization using real-time data for retail media? Stay ahead of the curve in retail innovation! Subscribe to my newsletter for insights into data streaming and connect with me on LinkedIn to continue the conversation. And download my free book about data streaming use cases and success stories in the retail industry.

The post Retail Media with Data Streaming: The Future of Personalized Advertising in Commerce appeared first on Kai Waehner.

What is a Loyalty Platform?

Data Streaming with Apache Kafka for the Next-Generation Loyalty Platform

Apache Kafka is a distributed data streaming platform for building real-time data pipelines and streaming applications.

While it may not be the alone technology for building a loyalty platform, there are several reasons companies incorporate Apache Kafka into the enterprise architecture of a loyalty platform:

1. Real-Time Data Processing: Apache Kafka excels at handling real-time data streams. In a loyalty platform, real-time processing is crucial for activities such as updating customer points, sending instant notifications, and providing timely rewards.
2. Message Durability: Kafka persists messages on disk, ensuring durability even in the event of a system failure. This feature is important for loyalty platforms, as it helps prevent data loss and ensures the data consistency and integrity of customer transaction records across real-time and non-real-time applications.
3. Scalability: Loyalty platforms can experience varying levels of user engagement and data volume. Apache Kafka scales horizontally, allowing the platform to handle increased loads by adding more Kafka brokers to the cluster. This scalability is valuable for accommodating growth in the number of users and transactions.
4. Data Integration: Loyalty platforms often need to integrate with various data sources and systems, such as customer databases, e-commerce platforms, and CRM systems. Kafka’s ability to act as a data integration hub facilitates the seamless flow of data between different components of the loyalty platform and external systems.
5. Fault Tolerance and Reliability: Apache Kafka provides fault tolerance by replicating data across multiple brokers in a cluster. This ensures that, even if a broker fails, data is not lost, contributing to the reliability of the loyalty platform.
6. Event-Driven Architecture: Loyalty platforms can benefit from an event-driven architecture, where events trigger actions and updates across the system. Kafka’s publish-subscribe model enables a decoupled, event-driven approach, making it easier to introduce new features or integrations without tightly coupling components.
7. Event Sourcing: Apache Kafka supports the event sourcing pattern, which is relevant for loyalty platforms where events like customer transactions, point accruals, and redemptions need to be captured and stored as a sequence of immutable events. This can simplify data modeling and auditing. Tiered Storage for Kafka makes replayability even easier, more scalable, and cost-efficient – with no another data lake.
8. Analytics and Monitoring: Kafka provides tools for real-time analytics and monitoring, which can be valuable for gaining insights into user behavior, loyalty program effectiveness, and system performance. Integrating Kafka with other analytics platforms allows for a comprehensive view of the loyalty platform’s operations.

Use Cases for a Kafka-powered Loyalty Platform

While specific implementations may vary, here are some examples of how Apache Kafka can be used in loyalty platforms:

1. Real-Time Points Accumulation: Kafka can process and handle real-time events related to customer transactions. As users make purchases or engage with the loyalty platform, these events can be captured in Kafka topics. The platform can then process these events in real-time with stream processing using Kafka Streams or Apache Flink to update customer points balances and trigger relevant notifications.
2. Event-Driven Rewards and Offers: Kafka’s publish-subscribe model allows for an event-driven approach to managing rewards and offers. When a customer becomes eligible for a reward or offer, the loyalty platform can publish an event to the appropriate Kafka topic. Subscribers, such as notification services or backend processors, can then react to these events in real-time, ensuring timely communication and fulfillment of rewards via push notifications to a mobile app or location-based service.
3. Cross-Channel Integration: Loyalty platforms often operate across multiple channels, including online stores, mobile apps, and physical stores. Kafka can facilitate the integration of these channels by serving as a central hub for loyalty-related events. For example, customer interactions in different channels can generate events in Kafka, allowing the loyalty platform to maintain a consistent view of customer activity across all touchpoints – no matter if the interface is real-time, batch, or an API.
4. Customer Engagement Tracking: Kafka can capture and process events related to customer engagement. Events such as logins, clicks, or interactions with loyalty program features can be streamed to Kafka topics. This data can then be used for real-time analytics, allowing the platform to understand customer behavior and adjust loyalty strategies accordingly.
5. Fault-Tolerant Transaction Processing: Loyalty platforms deal with critical customer transactions, including point redemptions and reward fulfillment. Kafka’s fault-tolerant architecture and Transaction API ensures these transactions are reliably processed even in the face of hardware failures or other issues. This helps maintain the integrity of customer balances and transaction history.
6. Scalable Data Processing: As the user base and transaction volume of a loyalty platform grows, Kafka’s scalability becomes crucial. Loyalty platforms can leverage Kafka to scale horizontally, distributing the processing load across multiple Kafka brokers to accommodate increased data throughput.
7. Audit Trail and Compliance: Kafka’s log-based architecture makes it well-suited for maintaining an audit trail of loyalty-related events. This is valuable for compliance purposes, ensuring that all customer interactions and transactions are recorded in a secure and tamper-evident manner. Replayability of historical data in guaranteed order is a common Kafka sweet spot.
8. Integration with External Systems: Loyalty platforms often need to integrate with external systems, such as payment gateways, CRM systems, or analytics platforms. Kafka can act as a central integration point, enabling seamless communication between the loyalty platform and these external systems. Kafka’s data integration capabilities have significant benefits compared to ETL, ESB and iPaaS tools.

These examples highlight how Apache Kafka plays a pivotal role in building a robust and scalable architecture for loyalty platforms, providing the infrastructure for real-time processing, fault tolerance, and integration with various components.

Example: Real-Time Rewards Platform for Video Streaming built with Apache Kafka

While Apache Kafka offers these advantages, it’s important to note that building a loyalty and reward platform involves various technologies and platforms. Kafka is a scalable real-time data fabric for the architecture of a loyalty platform and complementary to other open-source, commercial or SaaS transactional and analytics applications.

Here is an example of a loyalty and rewards platform built around video streaming platforms like Twitch. Kafka connects to different APIs and interfaces, correlates the information, and ingested the data into downstream applications:

Many of these interfaces provide bi-directional communication. For instance, the Salesforce CRM is updated with new information from an influencer, i.e., video streamer. On the other side, the Twitch subscribers receive notifications like drops or rewards based on information consumed from Salesforce CRM.

Real World Case Studies Across Industries for Kafka-based Loyalty and Rewards Platforms

Plenty of success stories exist for customer loyalty and reward systems built around Apache Kafka as data fabric and integration hub. Let’s explore some case studies across industries around the world:

• Retail: Albertsons (United States)
• Airlines: Virgin Australia
• Telco: Globe Telecom (Asia)
• Automotive / Manufacturing: Porsche (Germany)
• Sports and Gaming: Disney+ Hotstar (Asia)
• Public Sector: Service NSW (Australia)

Retail: Albertsons – Revamped Loyalty Platform to Retain Customers for Life

Albertsons is the second largest American grocery company with 2200+ stores and 290,000+ employees. “Customers for Life” is the primary goal of its CEO: “We want our customers to interact with us daily […] doubling down on our omnichannel engagement with customers beyond just transactions.”

For this reason, Albertsons choose Apache Kafka as the strategic data integration platform. The cloud-native architecture handles extreme retail scenarios like Christmas or Black Friday:

Albertsons use cases include:

• Scalable workforce management and supply chain modernization
• Inventory updates from 2200 stores to the cloud services in near real time
• Distributing offers and customer clips to each store in near real time
• Feed the data in real time to forecast engines for supply chain order forecast, demand planning, and other tasks
• Ingesting transactions in near real time to data lake for reporting and analytics
• New retail media network

Here is an example business process of Albertsons revamped loyalty platform. The data streaming platform connects to various applications and databases, then processes and shares the events:

Plenty of other success stories for loyalty platforms build with Kafka exist in the retail world. For instance:

• Kmart (Australia): The Loyalty platform “OnePass” provides customers a more seamless shopping experience, better customer personalization, and offers. The instant customer feedback improves the retention rate. Turning paper receipts into valuable digital data with streaming also saves millions of dollars. For instance, better analytics forecasts enable accurate product and ranging options and better stock and inventory planning.
• Intersport (Austria): The central nervous system for real-time data is powered by Confluent Cloud. Its loyalty platform offers a real-time bonus point system. The data hub shares personalized marketing and omnichannel customer experience across online and stores. Data integration ensures data consistency across the ERP, financial accounting (SAP FI) and 3rd Party B2B, 100s of Point-of-Sale (POS), legacy batch FTP applications.

Aviation: Virgin Australia – Workflow Orchestration between Airline, GDS, CRM and Travel Booking

Telco: Globe Telecom – Personalized Rewards Points

Manufacturing / Automotive: Porsche – Digital Service Platform for Customers, Fans, and Enthusiasts

Porsche is a German automotive manufacturer (one of Volkswagen’s subsidiaries) specializing in high-performance sports cars, SUVs, and sedans, known for its iconic designs and a strong emphasis on driving dynamics and engineering excellence.

Providing points or rewards for buying cars is a difficult strategic for car makers. Most people buy or lease a car every few years. Hence, car makers often focus on providing a great customer experience during the buying process. ‘My Porsche’ is Porsche’s digital service platform for customers, fans, and enthusiasts across multiple channels like website, mobile app, and the dealership on site:

A customer 360 view across multiple channels is difficult to implement. Porsche has a central data stream strategy across data centers, clouds, and regions to ensure data consistency across real-time, batch, and request response APIs. Porsche’s Streamzilla is the automakers’s one-stop-shop for all data streaming needs powered by Apache Kafka.

Streamzilla enables the data-driven company. It is one central platform, providing a unifying data architecture for setting clean standards and enabling cross-boundary data usage with a single source of truth.

Streamzilla is simplified by design: A single source of truth. The pipeline provides transparency about the cluster state, while increasing productivity and enhancing fault-tolerance & repeatability through automated execution.

Check out Porsche’s Kafka Summit talk or more details about omnichannel customer 360 architectures leveraging Apache Kafka to learn more.

Sports and Gaming: Disney+ Hotstar – Gamification of Live TV Events and Integration of Affiliates

Hotstar (acquired by Disney) is a popular Indian over-the-top (OTT) streaming service that offered a wide range of content, including movies, TV shows, sports, and original programming. “Hotstar Watch N Play” combines gamification and loyalty, which creates a win-win-win for Hotstar, users and affiliates. The feature was introduced to engage users further during live sports events.

Here’s a general overview of how gamification of a live sports event works with “Hotstar Watch N Play”:

1. Live Streaming of Sports: Hotstar provides live streaming of various sports events, focusing particularly on cricket matches, which are immensely popular in India.
2. Interactive Experience: The “Watch N Play” feature makes the viewing experience more interactive. Users could predict outcomes, answer trivia questions, and take part in polls related to the live sports event they were watching.
3. Points and Rewards: Users earn points based on the accuracy of their predictions and their participation in the interactive elements. These points can redeem rewards or showcased on leaderboards.
4. Leaderboards and Social Interaction: Hotstar often incorporates leaderboards to highlight top scorers among users. This adds a competitive element to the experience, encouraging users to compete with friends and other viewers. Users can also share their achievements on social media.
5. Engagement and Gamification: “Watch N Play” enhances user engagement by adding a gamified layer to the streaming experience. By blending entertainment with interactivity, Hotstar keeps viewers actively involved during live events.

This infrastructure has to run at extreme scale. Millions of actions have to be processed each second. No surprise that Disney+ Hotstar chose Kafka as the heart of this infrastructure. Kafka Connect integrates with various data sources and sinks.

IoT Integration is often also part of such a customer 360 implementation. Use cases include:

• Live e-sports events, TV, video streaming and news stations
• Fan engagement
• Audience communication
• Entertaining features for Alexa, Google Home or sports-specific hardware

Public Sector: NSW Australia – Single View of the Citizen and Partner Onboarding

The public sector includes various government organizations, agencies, and institutions at the national, regional, and local levels that provide public goods and services to the citizens

Obviously, a loyalty and rewards platform for the public sector must look differently (and has another name). The enterprise architecture is similar to all the private sector examples I covered above, though the goals and benefits are different.

Service New South Wales (NSW) is an Australian NSW government agency that delivers the best possible customer experience for people who want to apply for a bushfire support grant, get an energy rebate, manage their driver license, or access any of the many other government services and transactions available within the state of New South Wales.

The agency is part of the NSW government’s greater push to become “the world’s most customer-centric government by 2030.”

Service NSW partners with 70-plus teams and /agencies and offers 200 products, delivering about 1,300 different services and transactions. It’s an enormous effort and creates a complex integration problem from a technology point of view.

There are three components to the Apache Kafka data streaming architecture that make the single view of a citizen possible. The first is the onboarding of partners. In order to build that single view of the customer, Service NSW has to first collect information from 70+ product teams. These include all kinds of data sources, some with public networks, some with private.

Those integration partners include end-user-facing platforms like Salesforce and the interfaces of the apps customers see, such as the Service NSW Mobile App or MyServiceNSW web app.

Apache Kafka = Data Hub for Customer Loyalty

Loyalty points systems and customer rewards are crucial across most industries for long-term customer retention, increased revenue and visibility of a brand. Contextual information at the right time and data consistency across different applications require a reliable and scalable data hub.

Apache Kafka is the de facto standard for data streaming. Modern enterprise architectures leverage Kafka and its ecosystem to provide a better customer experience with accurate and innovative loyalty services.

How does your loyalty platform and rewards system look like? Do you already leverage data streaming in the enterprise architecture? Or even build context-specific recommendations with stream processing technologies like Kafka Streams or Apache Flink? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Customer Loyalty and Rewards Platform with Apache Kafka appeared first on Kai Waehner.