Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And download my free book about data streaming use cases, including various use cases from the retail industry.

What is Retail Media?

Retail media is transforming how brands advertise by leveraging first-party data from retailers to create highly targeted ads within their ecosystems. Instead of relying solely on third-party data from traditional digital advertising platforms, retail media allows companies to reach consumers at the point of purchase—whether online, in-store, or via mobile apps.

Retail media is one of the fastest-growing and most strategic revenue streams for retailers today. It has transformed from a niche digital advertising concept into a multi-billion-dollar industry, changing how retailers monetize their data and engage with brands. Below are the key reasons retail media is crucial for retailers.

Retailers like Amazon, Walmart, and Albertsons are leading the way in monetizing their digital real estate, offering brands access to sponsored product placements, banner ads, video ads, and personalized promotions based on shopping behavior. This shift has made retail media one of the fastest-growing sectors in digital advertising, expected to exceed $100 billion globally in the coming years.

The Digitalization of Retail Media

Retail media has grown from traditional in-store promotions to a fully digitized, data-driven advertising ecosystem. The rise of e-commerce, mobile apps, and connected devices has enabled retailers to:

• Collect granular consumer behavior data in real time
• Offer personalized promotions to drive higher conversion rates
• Provide advertisers with measurable ROI and closed-loop attribution
• Leverage AI and machine learning for dynamic ad targeting

By integrating digital advertising with real-time customer data and real-time inventory, retailers can provide contextually relevant promotions across multiple touchpoints. The key to success lies in seamlessly connecting online and offline shopping experiences—a challenge that data streaming with Apache Kafka and Flink helps solve.

Online, Brick-and-Mortar, and Hybrid Retail Media

Retail media strategies vary depending on whether a retailer operates online, in-store, or in a hybrid model:

• Online-Only Retail Media: Retail giants like Amazon and eBay leverage vast amounts of digital consumer data to offer programmatic ads, sponsored products, and personalized recommendations directly on their websites and apps.
• Brick-and-Mortar Retail Media: Traditional retailers like Target and Albertsons are integrating digital signage, in-store Wi-Fi promotions, and AI-powered shelf displays to engage customers while shopping in physical stores.
• Hybrid Retail Media: Retailers like Walmart and Kroger are bridging the gap between digital and physical shopping experiences with omnichannel marketing strategies, personalized mobile app promotions, and AI-powered customer insights that drive both online and in-store purchases.

Omnichannel vs. Unified Commerce in Retail Media

Retailers are moving beyond omnichannel marketing, where customer interactions happen across multiple channels, to unified commerce, where all customer data, inventory, and marketing campaigns are synchronized in real time.

• Omnichannel: Offers a seamless shopping experience across different platforms but often lacks real-time data integration.
• Unified Commerce: Uses real-time data streaming to unify customer behavior, inventory management, and personalized advertising for a more cohesive experience.

For example, a unified commerce strategy allows a retailer to:

• Track customer interactions in real time across e-commerce, mobile, and in-store purchases
• Deliver personalized product recommendations based on live shopping behavior
• Optimize ad placements and promotions based on real-time demand and inventory availability

This level of integration is only possible with real-time data streaming using technologies such as Apache Kafka and Apache Flink.

How Data Streaming with Apache Kafka and Flink Powers Retail Media

Retail media networks require real-time data processing at scale to manage millions of customer interactions across online and offline touchpoints. Kafka and Flink provide the foundation for a scalable, event-driven infrastructure that enables retailers to:

• Process customer behavior in real time: Tracking clicks, searches, and in-store activity instantly
• Deliver hyper-personalized ads and promotions: AI-driven dynamic ad targeting
• Optimize inventory and pricing: Aligning promotions with real-time stock levels
• Measure campaign performance instantly: Providing brands with real-time attribution and insights

With Apache Kafka as the backbone for data streaming and Apache Flink for real-time analytics, retailers can ingest, analyze, and act on consumer data within milliseconds.

Here are a few examples of input data sources, stream processing applications, and outputs for other systems:

Input Data Sources for Retail Media

1. Customer transaction data (e.g., point-of-sale purchases, online orders)
2. Website and app interactions (e.g., product views, searches, cart additions)
3. Loyalty program data (e.g., customer preferences, purchase frequency)
4. Third-party ad networks (e.g., campaign performance data, audience segments)
5. In-store sensor and IoT data (e.g., foot traffic, digital shelf interactions)

Stream Processing Applications for Retail Media

1. Real-time advertisement personalization engine (customizes promotions based on live behavior)
2. Dynamic pricing optimization (adjusts ad bids and discounts in real-time)
3. Customer segmentation & targeting (creates audience groups based on behavioral signals)
4. Fraud detection & clickstream analysis (identifies bot traffic and fraudulent ad clicks)
5. Omnichannel attribution modeling (correlates ads with online and offline purchases)

Output Systems for Retail Media

1. Retail media network platforms (e.g., sponsored product listings, display ads)
2. Programmatic ad exchanges (e.g., Google Ads, The Trade Desk, Amazon DSP)
3. CRM & marketing automation tools (e.g., Salesforce, Adobe Experience Cloud)
4. Business intelligence dashboards (e.g., Looker, Power BI, Tableau)
5. In-store digital signage & kiosks (personalized promotions for physical shoppers)

Key Retail Media Use Cases for Data Streaming with Kafka & Flink

Real-time data streaming with Kafka and Flink enables critical retail media use cases by processing vast amounts of data from customer interactions, inventory updates and advertising platforms. The ability to analyze and act on data instantly allows retailers to optimize ad placements, enhance personalization, and measure the effectiveness of marketing campaigns with unprecedented accuracy. Below are some of the most impactful retail media applications powered by event-driven architectures.

Personalized In-Store Promotions

Retailers can use real-time customer location data, combined with purchase history and preferences, to deliver highly personalized promotions through mobile apps or digital signage. By incorporating location-based services (LBS), the system detects when a shopper enters a specific section of a store and triggers a targeted discount or special offer. For example, a customer browsing the beverage aisle might receive a notification offering 10% off their favorite soda, increasing the likelihood of an impulse purchase.

Dynamic Ad Placement & Bidding

Kafka and Flink power real-time programmatic advertising, enabling retailers to dynamically adjust ad placements and bids based on customer activity and shopping trends. This allows advertisers to serve the most relevant ads at the optimal time, maximizing engagement and conversions. For instance, Walmart Connect continuously analyzes in-store and online behavior to adjust which ads appear on product pages or search results, ensuring brands reach the right shoppers at the right moment.

Inventory-Aware Ad Targeting

Real-time inventory tracking ensures that advertisers only bid on ads for products that are in stock and ready for fulfillment, reducing wasted ad spend and improving customer satisfaction. This integration between retail media networks and inventory systems prevents scenarios where customers click on an ad only to find the item unavailable. For example, if a popular TV model is running low in a specific store, the system can prioritize ads for a similar in-stock product, ensuring a seamless shopping experience.

Fraud Detection & Brand Safety

Retailers must protect their media platforms from click fraud, fake engagement, and suspicious transactions, which can distort performance metrics and drain marketing budgets.

Kafka and Flink enable real-time fraud detection by analyzing patterns in ad clicks, user behavior, and IP addresses to identify bots or fraudulent activity. For example, if an unusual spike in ad impressions originates from a single source, the system can immediately block the traffic, safeguarding advertisers’ investments.

Real-Time Attribution & Measurement

Retail media networks must provide advertisers with instant insights into ad performance by linking online interactions to in-store purchases.

Kafka enables event-driven attribution models, allowing brands to measure how digital ads drive real-world sales. For example, if a customer clicks on an ad for running shoes, visits a store, and buys them later, the platform tracks the conversion in real time, ensuring brands understand the full impact of their campaigns. Solutions like Segment (built on Kafka) provide robust customer data platforms (CDPs) that help retailers unify and analyze customer journeys.

Retail Media as an Advertising Channel for Third-Party Brands

Retailers are increasingly leveraging third-party data sources to bridge the gap between retail media networks and adjacent industries, such as quick-service restaurants (QSRs).

Kafka enables seamless data exchange between grocery stores, delivery apps, and restaurant chains, optimizing cross-industry advertising. For example, a burger chain could dynamically adjust digital menu promotions based on real-time data from a retail partner—if a grocery store’s sales data shows a surge in plant-based meat purchases, the restaurant could prioritize ads for its new vegan burger, ensuring more relevant and effective marketing.

Albertsons’ New Retail Media Strategy Leveraging Data Streaming

One of the most innovative retail media success stories comes from Albertsons. Albertsons is one of the largest supermarket chains in the United States, operating over 2,200 stores under various banners, including Safeway, Vons, and Jewel-Osco, and providing groceries, pharmacy services, and household essentials.

I explored Albertsons in another article about its revamped loyalty platform to retain customers for life. Data streaming is essential and a key strategic part of Albertsons’ enterprise architecture:

When I hosted a webinar with Albertsons around two years ago on their data streaming strategy, retail media was one of the bullet points. But I didn’t yet realize until now how crucial it would become for retailers:

• Retail Media Network Expansion: Albertsons has launched its own retail media network, leveraging first-party data to create highly targeted advertising campaigns.
• Real-Time Personalization: With real-time data streaming, Albertsons can provide personalized promotions based on customer purchase history, in-store behavior, and digital engagement.
• AI-Powered Insights: Albertsons uses AI and machine learning on top of streaming data pipelines to optimize ad placements, campaign effectiveness, and dynamic pricing strategies.
• Data Monetization: By offering data-driven advertising solutions, Albertsons is monetizing its shopper data while enhancing the customer experience with relevant, timely promotions.

Business Value of Real-Time Retail Media

Retailers that adopt data streaming with Kafka and Flink for retail media strategies to unlock massive business value:

• New Revenue Streams: Retail media monetization drives ad sales growth
• Higher Conversion Rates: Real-time targeting improves customer engagement
• Better Customer Insights: Streaming analytics enables deep behavioral insights
• Competitive Advantage: Retailers with real-time personalization outperform rivals
• Better Customer Experience: Retail media reduces friction and enhances the shopping journey through personalized promotions

The Future of Retail Media is Real-Time and Context-Specific Data Streaming

Retail media is no longer just about placing ads on retailer websites—it’s about delivering real-time, data-driven advertising experiences across every consumer touchpoint.

With Kafka and Flink powering real-time data streaming, retailers can:

• Unify online and offline shopping experiences
• Enhance personalization with AI-driven insights
• Maximize ad revenue with real-time campaign optimization

As retailers like Albertsons, Walmart, and Amazon continue to innovate, the future of retail media will be hyper-personalized, data-driven, and real-time.

How is your organization using real-time data for retail media? Stay ahead of the curve in retail innovation! Subscribe to my newsletter for insights into data streaming and connect with me on LinkedIn to continue the conversation. And download my free book about data streaming use cases and success stories in the retail industry.

The post Retail Media with Data Streaming: The Future of Personalized Advertising in Commerce appeared first on Kai Waehner.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases.

Data Streaming with Apache Kafka and Flink in the Media Industry

Data streaming technologies like Apache Kafka and Flink are revolutionizing the media industry by enabling real-time data processing at an unprecedented scale. Media platforms, including Over-The-Top (OTT) services operated by telcos and media companies, leverage these technologies to deliver video, audio, and other content directly to viewers over the internet. The OTT services bypass traditional cable or satellite channels.

As these platforms cater to growing audiences with diverse needs, data streaming serves as the backbone for seamless content delivery, real-time user engagement, and operational efficiency. Data streaming ensures a superior viewing experience at scale.

Netflix is a leading global media company renowned for its extensive use of Apache Kafka and Flink. The media company powers critical use cases such as real-time personalization, anomaly detection, and monitoring at extreme scale. Its data streaming architecture processes billions of events daily, ensuring seamless content delivery and exceptional viewer experiences for a global audience.

Use Cases for Data Streaming in the Media Industry

Data streaming with technologies like Apache Kafka and Flink is transforming the media industry by enabling real-time data processing for seamless content delivery, personalized experiences, and operational efficiency.

1. Live Video Streaming: Data streaming with Apache Kafka serves as a central event hub for managing log data, metadata, and control signals associated with live video streaming. It processes real-time data related to user interactions, stream health, and session analytics to ensure ultra-low latency and a seamless experience for live events like concerts and sports. The actual video streams are handled by Content Delivery Networks (CDNs).
2. On-Demand Content Delivery: Media platforms use Kafka to reliably manage data pipelines, delivering movies, TV shows, and other content to millions of users.
3. Personalized Recommendations: By integrating Kafka with analytics tools, platforms provide tailored suggestions based on user behavior, increasing viewer engagement and satisfaction.
4. Real-Time Ad Targeting: Kafka enables real-time ad insertion by processing user events and contextual data, ensuring ads are relevant and timely.
5. Monitoring and Anomaly Detection: Media companies use Kafka to monitor backend systems in real time, detecting and resolving issues proactively to ensure a smooth user experience.
6. Churn Prediction: By analyzing behavioral patterns in real time, platforms can predict user churn and take corrective actions, such as offering discounts or new content recommendations.

Learn more about data streaming use cases in the telco and media industry from real world customer stories like Dish Network, British Telecom, Globe Telecom, Swisscom, and more:

Business Value of Data Streaming in Media

Data streaming technologies like Apache Kafka and Flink drive transformative business value in the media industry by enabling real-time insights, efficiency, and innovation:

• Enhanced User Experience: Real-time at any scale capabilities enable faster content delivery, personalized recommendations, and reduced buffering.
• Cost Optimization: Streamlined pipelines improve infrastructure utilization and reduce operational costs. The Shift Left Architecture is adopted across business units.
• Revenue Growth: Precision in ad targeting and churn reduction leads to higher revenues.
• Competitive Edge: Real-time analytics and content delivery position companies as leaders in their field.

Disney+ Hotstar (Disney) and JioCinema (Viacom18): Streaming Giants Shaping India’s Media Landscape

Disney+ Hotstar revolutionized OTT streaming in India with a robust freemium model. Catering to a diverse audience, it provided an extensive library of movies, TV shows, and sports, including exclusive streaming rights for the Indian Premier League (IPL), the world’s most popular cricket league. By blending free content with premium subscriptions, it attracted millions of users, leveraging IPL viewership as a major growth driver.

JioCinema, part of Reliance Jio, employs a mass-market approach, offering free streaming supported by Reliance’s vast 5G network. It gained significant traction by taking over the IPL digital streaming rights in 2023 in 4K resolution to over 32 million concurrent viewers, breaking records for live streaming.

Each platform used respectively uses IPL strategically—Hotstar with a premium model and JioCinema for mass-market penetration. Post-merger, the unified platform could combine these approaches, delivering enhanced IPL experiences powered by a consolidated Kafka-based streaming infrastructure.

Both platforms share a commitment to innovation, scalability, and user engagement, making them ideal candidates for heavy Apache Kafka usage.

Data Streaming Architectures with Kafka and Flink

Both Disney+ Hotstar and JioCinema (Viacom18) are renowned for their openness in discussing their technical data streaming architectures, similar to Netflix. They frequently presented at conferences like Kafka Summit and industry events, sharing insights about their data streaming strategies and implementations.

This transparency achieves several goals:

• Showcasing Innovation: Highlighting their advanced use of Kafka and Flink establishes their leadership in tech innovation.
• Talent Acquisition: Open discussions attract engineers who want to work on cutting-edge systems.
• Industry Collaboration: Sharing experiences fosters collaboration within the streaming and open-source communities.

By examining their presentations and publications, we gain a deeper understanding of their use of Kafka to achieve extreme scalability and efficiency.

Data Streaming Solves the Challenges and Extreme Scale of OTT Services in the Media Industry

Running platforms of this scale comes with its share of challenges:

• Massive Throughput: Kafka handles billions of messages daily, requiring extensive partitioning and scaling strategies.
• Fault Tolerance: Platforms implement advanced disaster recovery and replication strategies to ensure zero downtime, even during critical events like IPL.
• Cost vs. Performance Trade-Offs: Streaming 4K video for millions of users demands balancing high infrastructure costs with user expectations.

Data streaming with Apache Kafka and Flink is a key piece of the data strategy to solve these challenges.

Disney+ Hotstar: Gamification at Extreme Scale

Disney+ Hotstar’s “Watch N Play” feature transformed live sports streaming, particularly cricket, into an interactive experience. Viewers predict outcomes, answer trivia, and participate in polls, earning points for rewards or leaderboard rankings, adding a competitive and social element to the platform.

Hotstar’s presentation from Kafka Summit 2019 is still very impressive and worth watching. Here is a summary about the OTT services serving millions of cricket fans:

Powered by Apache Kafka, Disney+ Hotstar’s infrastructure processed millions of real-time interactions per second. The integration of data sources via Kafka Connect enables seamless analytics and rewards. This gamified approach enhances user engagement and extends to broader applications like e-sports, interactive TV, and IoT-driven fan experiences, making Hotstar a leader in innovative streaming.

Disney+ Hotstar runs ~15 different Kafka Connect clusters with over 2000+ connectors and auto-scaling based on traffic, as they presented in another Kafka Summit talk in 2021.

Single Message Transforms (SMT) are used within the Kafka Connect integration for stateless streaming ETL. Integration use cases include masking/filtering of PlI, sampling of data, and schema validation and enforcement.

JioCinema: Multiple Kafka Clusters and Deployment Strategies

JioCinema leverages a robust enterprise architecture built on Apache Kafka, Flink, and Spark. As showcased at Kafka Summit India 2024, data streaming is central to its platform, enabling real-time analytics, personalized recommendations, and seamless content delivery.

Initially, JioCinema operated a single Kafka cluster handling 1,000+ topics and 100,000+ partitions for diverse use cases.

Over time, the platform transitioned to multiple Kafka clusters with different SLAs and architectures, optimizing uptime, performance, and costs for specific workloads, as explained by Kushal Khandelwal, Head of Data Platform.

This shift from a monolithic to a segmented architecture highlights the scalability and flexibility of Kafka. This approach ensures JioCinema meets the demands of high traffic and complex SLAs. Their success reflects the common journey of organizations scaling data streaming infrastructures to achieve operational excellence.

Use Cases for Kafka in Disney+ Hotstar and JioCinema

Disney+ Hotstar and JioCinema rely on Apache Kafka to power diverse use cases, from IPL cricket streaming to real-time personalization and ad targeting.

IPL Cricket Streaming at Massive Scale

The Indian Premier League (IPL) is the crown jewel of streaming in India, drawing millions of concurrent viewers. Here’s how Kafka and Flink support IPL’s massive scale:

• Concurrent Viewers: During IPL 2023, JioCinema hit a record of over 32 million concurrent viewers, streaming matches in 4K resolution. Disney+ Hotstar has also scaled to tens of millions of viewers in past IPL seasons.
• Data Throughput: JioCinema and Hotstar handle millions of messages per second with Kafka, ensuring uninterrupted video delivery.
• Kafka Infrastructure: Reports reveal that JioCinema operates over 100 Kafka clusters, managing tens of thousands of partitions. These clusters handle not only video streaming but also ancillary tasks, like ad placement and user analytics.
• Connectors: Both platforms rely on hundreds of Kafka Connect connectors to integrate with databases, storage systems, and real-time analytics platforms.

On-Demand Streaming and Catalog Management

Both platforms use Kafka to deliver on-demand content to millions of users, ensuring quick access to movies and TV shows. Kafka’s reliable event streaming guarantees smooth playback and dynamic scaling during peak usage.

Real-Time Personalization and Recommendations

Personalization is central to user retention. Kafka streams user behavior data to machine learning systems in real time, enabling both platforms to recommend content tailored to individual preferences. Customer loyalty and Rewards platform often leverage Kafka and Flink under the hood.

Ad Targeting and Revenue Optimization

By processing user data in real time, Kafka enables precise ad targeting with context-specific advertisements. This not only improves ad effectiveness but also enhances viewer experience by ensuring ads are contextually relevant. Many real-time advertising platforms are powered by a data streaming platform using Apache Kafka and Flink.

Content Quality Monitoring

Both platforms use Kafka for continuous real-time monitoring of video stream quality, automatically adjusting bitrate or rerouting streams during disruptions to maintain a consistent viewing experience.

Data Streaming for M&A, Merger and Migrations

The merger of Disney+ Hotstar and JioCinema presents a significant opportunity to integrate their Kafka-based infrastructures, paving the way for a unified, more efficient system. Such transitions are a natural fit for Apache Kafka and its ecosystem. Migrations are a core capability. Tools like MirrorMaker and Cluster Linking allow seamless data movement between clusters for continuous replication and a later lift and shift. The usage of data streaming for migration projects enables zero-downtime and business continuity.

Here are some opportunities and benefits of data streaming for integrations and migrations:

1. Integrated Pipelines: A combined Kafka architecture could streamline content delivery, reduce costs, and support advanced analytics, providing an optimized infrastructure for their vast user base.
2. Expanded Use Cases: The merger might drive innovations such as multi-angle viewing, live interactive features, and more personalized experiences powered by real-time data.
3. Hybrid and Multi-Cloud Flexibility: Transitions like these often span hybrid and multi-cloud environments, making Kafka’s flexibility essential for connecting and scaling across platforms.
4. Multi-Organization Integration: Merging Kafka clusters across distinct organizations, as in this case, is a common use case where Kafka’s tools excel.
5. Technical Leadership: Both platforms are transparent about their Kafka implementations, and we can anticipate new insights from their efforts to integrate and scale, highlighting lessons for the broader streaming industry.

In conclusion, Kafka and Flink are not just enablers but drivers of success for Disney+ Hotstar and JioCinema. Data streaming at scale creates new benchmarks for innovation and user experience in the media industry.

Do you see similar opportunities in your organization? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter. And make sure to download my free book about data streaming use cases.

The post Data Streaming with Apache Kafka and Flink in the Media Industry: Disney+ Hotstar and JioCinema appeared first on Kai Waehner.

I look at trends in the games industry to explore how data streaming helps as a business enabler, including customer stories from Kakao Games, Mobile Premier League (MLP), Demonware / Blizzard, and more. A complete slide deck and on-demand video recording are included.

General trends in the gaming industry

The global gaming market is bigger than the music and film industries combined! Digitalization plays a huge factor for the growth in the past years. The gaming industry has various business models connecting players, fans, vendors, and other stakeholders:

• Hardware sales: Game consoles, VR sets, glasses
• Game sales: Physical and digital
• Free-to-play + in-game purchases: One-time in-game purchases (skins, champions, miscellaneous), gambling (loot boxes)
• Game-as-a-service (subscription): Seasonal in-game purchases like passes for theme events, mid-season invitational & world championship, passes for competitive play
• Game-Infrastructure-as-a-Service: High-performance state synchronization, multiplayer, matchmaking, gaming statistics
• Merchandise sales: T-shirts, souvenirs, fan equipment
• Community: Esports broadcast, ticket sales, franchising fees
• Live betting
• Video streaming: Subscriptions, advertisements, rewards,

Growth and innovation require cloud-native infrastructure

Most industries require a few specific characteristics. Instant payments must be executed in real time without data loss. Telcom infrastructure monitors huge volumes of logs in near-real-time. The retail industry needs to scale up for events like Christmas or Black Friday and scale down afterward.The gaming industry combines all the characteristics of other industries:

• Real-time data processing
• Scalability for millions of players
• High availability, at least for transactional data
• Decoupling for innovation and faster roll-out of new features
• Cost efficiency because cloud networking for huge volumes is expensive
• The flexibility of adopting various innovative technologies and APIs
• Elasticity for critical events a few times a year
• Standards-based integration for integration with SaaS, B2B, and mobile apps
• Security for trusted customer data
• Global and vendor-independent cloud infrastructure to deploy across countries

The good news is that data streaming powered by Apache Kafka and Apache Flink provides all these characteristics on a single platform, especially if you choose a fully managed SaaS offering.

Data streaming in the gaming industry

Adopting gaming trends like in-game purchases, customer-specific discounts, and massively multiplayer online games (MMOG) is only possible if enterprises in the games sector can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means):

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

“Apache Kafka in the Gaming Industry” is a great starting point to learn more about data streaming in the games sector, including a few Kafka-powered case studies not covered in this blog post – such as

• Big Fish Games: Live operations by monitoring real-time analytics of game telemetry and context-specific recommendations for in-game purchases
• Unity: Monetization network for player rewards, banner ads, playable advertisements, and cross-promotions.
• William Hill: Trading platform for gambling and betting
• Disney+ Hotstar: Gamification of live sport video streaming

Architecture trends for data streaming

The gaming industry applies various trends for enterprise architectures for cost, elasticity, security, and latency reasons. The three major topics I see these days at customers are:

• Fully managed SaaS to focus on business logic and faster time-to-market
• Event-driven architectures (in combination with request-response communication) to enable domain-driven design and flexible technology choices
• Data mesh for building new data products and real-time data sharing with internal platforms and partner APIs

Let’s look deeper into some enterprise architectures that leverage data streaming for gaming use cases.

Cloud-native elasticity for seasonal spikes

The games sector has extreme spikes in workloads. For instance, specific game events increase the traffic 10x and more. Only cloud-native infrastructure enables a cost-efficient architecture.

Epic Games presented at an AWS Summit in 2018 already how elasticity is crucial for data-driven architecture.

Make sure to use a truly cloud-native Apache Kafka service for gaming infrastructure. Adding brokers is relatively easy. Removing brokers is much harder. Hence, a fully-managed SaaS should take over the complex operations challenges of distributed systems like Kafka and Flink for you. The separation of compute and storage is another crticial piece of a cloud-native Kafka architecture to ensure cost-efficient scale.

Data mesh for real-time data sharing

Data sharing across business units is important for any organization. The gaming industry has to combine very interesting (different) data sets, like big data game telemetry, monetization and advertisement transactions, and 3rd party interfaces.

Data consistency is one of the most challenging problems in the games sector. Apache Kafka ensures data consistency across all applications and databases, whether these systems operate in real-time, near-real-time, or batch.

One sweet spot of data streaming is that you can easily connect new applications to the existing infrastructure or modernize existing interfaces, like migrating from an on-premise data warehouse to a cloud SaaS offering.

New customer stories for data streaming in the gaming sector

So much innovation is happening in the gaming sector. Automation and digitalization change how gaming companies process game telemetry data, build communities and customer relationships with VIPs, and create new business models with enterprises of other verticals.

Most gaming companies use a cloud-first approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure. And elastic scalability gets even more critical with all the growing real-time expectations and mobile app capabilities.

Here are a few customer stories from worldwide gaming organizations:

• Kakao Games: Log analytics and fraud prevention
• Mobile Premier League (MLP): Mobile eSports and digital gaming
• Demonware / Blizzard: Network and gaming infrastructure
• WhatNot: Retail gamification and social commerce
• Vimeo: Video streaming observability

Resources to learn more

This blog post is just the starting point. Learn more about data streaming in the gaming industry in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-demand video recording

The video recording explores the gaming industry’s trends and architectures for data streaming. The primary focus is the data streaming case studies.

I am excited to have presented this webinar in my interactive light board studio:

This creates a much better experience, especially in a time after the pandemic, where many people are “Zoom fatigue”.

Check out our on-demand recording:

Slides

If you prefer learning from slides, check out the deck used for the above recording:

Case studies and lightboard videos for data streaming in the gaming industry

The state of data streaming for gaming in 2023 is fascinating. New use cases and case studies come up every month. This includes better end-to-end observability in real-time across the entire organization, telemetry data collection from gamers, data sharing and B2B partnerships with engines like Unity or video platforms like Twitch, new business models for ads and in-game purchases, and many more scenarios.

We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories. Here is an example for real-time fraud detection with data streaming.

Gaming is just one of many industries that leverages data streaming with Apache Kafka and Apache Flink.. Every month, we talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, telcos, gaming, and so on… Check out my other blog posts.

Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post The State of Data Streaming for Gaming appeared first on Kai Waehner.

What is a digital advertising platform?

How does Apache Kafka help build an advertising platform?

Apache Kafka combines real-time messaging at any scale with true decoupling through its event store. The data streaming platform collects data, correlates real-time and historical events with stream processing, and shares created information with downstream consumers.

One of the most underestimated capabilities is the out-of-the-box capability of Apache Kafka to ensure data consistency across real-time and non-real-time systems. The heart of the enterprise architecture is real-time, scalable, and reliable. But any near real-time, batch or request-response communication can produce or consume at its own pace with its own API or programming language.

Apache Flink to the rescue for real-time bidding and fraud detection

Apache Flink is ideal for data correlation. No matter if the task is data integration (aka streaming ETL) or advanced stateful business and application logic. Apache Kafka and Apache Flink are a match made in heaven for data streaming.

Real-world advertising platforms built with Kafka and Flink

Real-world success stories show how data streaming with Kafka and Flink helps build a next-generation advertising platform. These technologies solve the abovementioned challenges to provide real-time and consistent information across all applications.

Advertising platforms are either directly embedded into customer-facing applications or built as software or SaaS products that other companies buy and leverage.

The following success stories explore ad platforms built with data streaming:

• Pinterest: Image-sharing and natural engagement with ads (Kafka Streams).
• Buzzvil: Lock screen advertisement for smartphones (Kafka and Confluent Cloud).
• TV-Insight: Live decisions of regular TV ad blocks (Kafka, Flink, and Confluent Cloud).
• Unity: Monetization network for gaming (Kafka and Confluent).
• Uber Eats: Ads in the mobile food delivery app (Kafka, Flink, Pinot).
• Reddit: Ads placing including real-time budget planning without over-delivery or under-delivery (Kafka, Flink, Druid).

Pinterest – Social media natural engagement with ads

Pinterest is an American image-sharing and social media service designed to enable the saving and discovery of information (specifically “ideas”) like recipes, home, style, motivation, and inspiration on the internet.

The content of ads is very close to the actual content. Naturally, users engage with the content and ads:

Pinterest talked about its Kafka-powered advertising platform for the first time in 2018 at a Kafka Summit. The Ad platform leverages Kafka for the data ingestion pipeline and stream processing with Kafka Streams to enable a real-time feedback loop. Recommendation engine (via machine learning), budgeting, and new ads exploration are some of the critical use cases.

The continuous feedback loop enables real-time updates in seconds. Stateful stream processing with Kafka Streams correlates events from users, ads, budget, and other interfaces to decide on ads serving.

Real-time (even at an extreme scale) is critical for Pinterest. When a new ad is created, the ads platform does not know about the user engagement with this ad on different surfaces. The faster the ads platform knows about the performance of the newly created ad, the better value can be provided to the user.

There is a balance between exploiting good ads and exploring new ads. The solution was adding a boosting factor to new ads to increase the probability of winning an auction.

Listen to the talk from Pinterest for more details, best practices, and lessons learned in developing and operating a scalable, real-time advertising platform with stateful stream processing using Kafka Streams.

Buzzvil – Lock screen advertising platform

Buzzvil provides a lock screen advertising platform that connects partners and advertisers:

Buzvill’s advertising platform is data-driven and built with Apache Kafka in the cloud. It optimizes ad spending through automation, behavioral analytics, audience targeting, rewards programs, and more. Data streaming enables a single source of truth for real-time ad transaction data.

They built the ad platform with Apache Kafka in a fully managed Confluent Cloud to focus on business logic and faster time-to-market.

Data streaming with Apache Kafka enables 18x faster data updates for ad bidding. Confluent Cloud saves 20-30% infrastructure cost.

TV-Insight – Live decisions of regular TV ad blocks

TV-Insight developed a solution to help Joint Industry Committees (JIC), Broadcasters, and Advertisers to improve and evolve the data quality of existing TV measurement panels using return path data of connected devices.

The essential difference between TV-Insight and all other “panel boosting” initiatives and products is that TV-Insight uses real-time data. Therefore, it can provide a live TV reach for live decisions of regular TV ad blocks.

The TV-Insight application collects data from the Smart TV or Set-Top Box via GDPR compliance device tracking. The live extrapolation enables advertising optimization:

The technical architecture and data pipeline look like the following. Apache Kafka is the real-time messaging platform and event store. Apache Kafka’s stateful stream processing correlates events to calculate real-time ad serving in the advertising platform.

Unity Ads – Monetization network for gaming

Unity is a cross-platform game engine developed by Unity Technologies. The engine has since been gradually extended to support a variety of desktop, mobile, console, and virtual reality platforms. The engine can create three-dimensional (3D) and two-dimensional (2D) games, interactive simulations, and other experiences. Industries outside video gaming have adopted the engine, such as film, automotive, architecture, engineering, and construction.

In 2019, Unity apps and content were installed 33 billion times, reaching 3 billion devices worldwide.

The 3D development platform and game engine is not the only product of Unity Technologies. Unity Ads is one of the largest monetization networks in the world:

• Reward players for watching ads
• Incorporate banner ads
• Incorporate Augmented Reality (AR) ads
• Playable ads
• Cross-Promotions
• IAPs (in-app purchases)

Unity is a data-driven company:

• Average about half a million events per second
• Handles millions of dollars in monetary transactions
• Data infrastructure based on Apache Kafka

A single data pipeline provides the foundational infrastructure for analytics, R&D, monetization, cloud services, etc., for real-time and batch processing leveraging Apache Kafka:

• Real-time monetization network
• Feed machine learning models in real-time
• Data lake went from two-day latency down to 15 minutes

If you want to learn about Unity’s success story of migrating this platform from self-managed Kafka to the cloud, read the post on the Confluent Blog: “How Unity uses Confluent for real-time event streaming at scale“.

Uber Eats – Ads embedded into food delivery app

Uber provides an exciting food delivery app capability: Uber Eats allows ads embedding. With this ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more.

Uber wrote an excellent article that focuses on how they leveraged open source technology to build Uber’s first near real-time exactly-once events processing system. Uber leverages Kafka, Flink, and Pinot for its advertising platform. This perfectly combines the right technologies.

As Uber writes: “With every ad served, there are corresponding events per user (impressions, clicks). The responsibility of the ad events processing system is to manage the flow of events, cleanse them, aggregate clicks and impressions, attribute them to orders, and provide this data in an accessible format for reporting and analytics as well as dependent clients (e.g., other ads systems).”

While speed, scale, and reliability are always crucial for such a system, I want to emphasize the part about accuracy and why exactly-once processing with Kafka and Flink was a critical piece of the architecture.

The Aggregation Job implemented with Apache Flink does a lot of the heavy lifting: Data cleansing, persistence for order attribution, aggregation, and record UUID generation.

Exactly-once with Kafka and Flink is very important, as their blog post explains: “Uber can’t afford to overcount events. Double counting clicks results in overcharging advertisers and overreporting the success of ads. Both being poor customer experiences, this requires processing events exactly-once. Uber is the marketplace in which ads are being served, therefore our ad attribution must be 100% accurate.”

Reddit – Ads Placing with Budget Planning avoiding Over- or Under-Delivery

Reddit is an American social news aggregation, content rating, and discussion website. Registered users submit content to the site such as links, text posts, images, and videos, which other members then vote up or down.

Reddit’s ads platform allows advertisers to create ad campaigns and set both daily and lifetime budgets for a campaign. Here is Reddit’s decision tree to place advertisements:

The data pipeline leverages Kafka, Flink, and Druid to analyze campaign budgets in real-time. The platform leverages real-time plus historical user activity data to decide which ad to place. All within 30 milliseconds to avoid over-delivery and under-delivery (budget spent too quickly / slowly).

Watch Reddit’s talk from Druid Summit “Low Latency Real-Time Ads Pacing Queries” to learn more about their ads platform and use cases.

Real-Time Advertisment with Kafka and Flink

Real-world success stories from Pinterest, Uber, Unity, buzzwil, and TV-Insight showed how to embed real-time advertising into your applications or build a dedicated marketing product.

Data streaming with Apache Kafka and Apache Flink enables context-specific advertising at scale in real time. The cloud makes it possible to focus on business logic and faster time-to-market with a fully managed data streaming platform.

How do you leverage data streaming in marketing and advertising use cases? Do you deploy at the edge, in the cloud, or both? Or do you integrate 3rd party marketing platforms into your advertising platforms? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post How to Build a Real-Time Advertising Platform with Apache Kafka and Flink appeared first on Kai Waehner.