Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases, including several success stories around gaming, loyalty platforms, and personalized advertising.

Fantasy Sports: Real-Time Gaming Meets Real-World Sports

Fantasy sports allows users to create virtual teams based on real-life athletes. As matches unfold, players earn points based on the performance of their selected athletes. The better the team performs, the higher the user’s score—and the bigger the prize.

Key characteristics of fantasy gaming:

• Multi-sport experience: Users can play across cricket, football, basketball, and more.
• Live interaction: Scoring is updated in real time as matches progress.
• Contests and leagues: Players join public or private contests, often with cash prizes.
• Peak traffic patterns: Most activity spikes in the minutes before a match begins.

This user behavior creates a unique business and technology challenge. Millions of users make critical decisions at the same time, just before the start of each game. The result: extreme concurrency, massive request volumes, and a hard dependency on data accuracy and low latency.

Real-time infrastructure isn’t optional in this model. It’s fundamental to user trust and business success.

Dream11: A Fantasy Sports Giant with Massive Scale

Founded in India, Dream11 is the largest fantasy sports platform in the country—and one of the biggest globally. With over 230 million users, it dominates fantasy gaming across cricket and 11 other sports. The platform sees traffic that rivals the world’s largest digital services.

Bipul Karnanit from Dream11 presented very interesting overview at Current 2025 in Bangalore India. Here are a few statistics about Dream11’s scale:

• 230M users
• 12 sports
• 12,000 matches/year
• 44TB data per day
• 15M+ peak concurrent users
• 43M+ peak transactions/day

During major events like the IPL, Dream11 experiences hockey-stick traffic curves, where tens of millions of users log in just minutes before a match begins—making lineup changes, joining contests, and waiting for live updates.

This creates a business-critical need for:

• Low latency
• Guaranteed data consistency
• Fault tolerance
• Real-time analytics and scoring
• High developer productivity to iterate fast

Apache Kafka at the Heart of Dream11’s Platform

To meet these demands, Dream11 uses Apache Kafka as the foundation of its real-time data infrastructure. Kafka powers the messaging between services that manage user actions, match scores, payouts, leaderboards, and more.

Apache Kafka enables:

• Event-driven microservices
• Scalable ingestion and processing of user and game data
• Loose coupling between systems with data products for operational and analytical consumers
• High throughput with guaranteed ordering and durability

Solving Kafka Consumer Challenges at Scale

As the business grew, Dream11’s engineering team encountered challenges with Kafka’s standard consumer APIs, particularly around rebalancing, offset management, and processing guarantees under peak load.

To address these issues, Dream11 built a custom Java-based Kafka consumer library—a foundational component of its internal platform that simplifies Kafka integration across services and boosts developer productivity.

Dream11 Kafka Consumer Library:

• Purpose: A custom-built Java library designed to handle high-volume Kafka message consumption at Dream11 scale.
• Key Benefit: Abstracts away low-level Kafka consumer details, simplifying tasks like offset management, error handling, and multi-threading, allowing developers to focus on business logic.
• Simple Interfaces: Provides easy-to-use interfaces for processing records.
• Increased Developer Productivity: Standardized library lead to faster development and fewer errors.

This library plays a crucial role in enabling real-time updates and ensuring seamless gameplay—even under the most demanding user scenarios.

For deeper technical insights, including how Dream11 decoupled polling and processing, implemented at-least-once delivery, and improved throughput with custom worker pools, watch the Dream11 engineering session from Current India 2025 presented by Bipul Karnanit.

Fantasy Sports, Real-Time Expectations, and Business Value

Dream11’s business success is built on user trust, real-time responsiveness, and high-quality gameplay. With millions of users relying on accurate, timely updates, the platform can’t afford downtime, data loss, or delays.

Data Streaming with Apache Kafka enables Dream11 to:

• React to user interactions instantly
• Deliver consistent data across microservices and devices
• Scale dynamically during live events
• Streamline the development and deployment of new features

This is not just a backend innovation—it’s a competitive advantage in a space where milliseconds matter and trust is everything.

Dream11’s Kafka Journey: The Backbone of Fantasy Sports at Scale

Fantasy sports is one of the most demanding environments for real-time data platforms. Dream11’s approach—scaling Apache Kafka to serve hundreds of millions of events with precision—is a powerful example of aligning architecture with business needs.

As more industries adopt event-driven systems, Dream11’s journey offers a clear message: Apache Kafka is not just a messaging layer—it’s a strategic platform for building reliable, low-latency digital experiences at scale.

Whether you’re in gaming, finance, telecom, or logistics, there’s much to learn from the way fantasy sports leaders like Dream11 harness data streaming to deliver world-class services.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases, including several success stories around gaming, loyalty platforms, and personalized advertising.

The post Powering Fantasy Sports at Scale: How Dream11 Uses Apache Kafka for Real-Time Gaming appeared first on Kai Waehner.

I look at trends in the games industry to explore how data streaming helps as a business enabler, including customer stories from Kakao Games, Mobile Premier League (MLP), Demonware / Blizzard, and more. A complete slide deck and on-demand video recording are included.

General trends in the gaming industry

The global gaming market is bigger than the music and film industries combined! Digitalization plays a huge factor for the growth in the past years. The gaming industry has various business models connecting players, fans, vendors, and other stakeholders:

• Hardware sales: Game consoles, VR sets, glasses
• Game sales: Physical and digital
• Free-to-play + in-game purchases: One-time in-game purchases (skins, champions, miscellaneous), gambling (loot boxes)
• Game-as-a-service (subscription): Seasonal in-game purchases like passes for theme events, mid-season invitational & world championship, passes for competitive play
• Game-Infrastructure-as-a-Service: High-performance state synchronization, multiplayer, matchmaking, gaming statistics
• Merchandise sales: T-shirts, souvenirs, fan equipment
• Community: Esports broadcast, ticket sales, franchising fees
• Live betting
• Video streaming: Subscriptions, advertisements, rewards,

Growth and innovation require cloud-native infrastructure

Most industries require a few specific characteristics. Instant payments must be executed in real time without data loss. Telcom infrastructure monitors huge volumes of logs in near-real-time. The retail industry needs to scale up for events like Christmas or Black Friday and scale down afterward.The gaming industry combines all the characteristics of other industries:

• Real-time data processing
• Scalability for millions of players
• High availability, at least for transactional data
• Decoupling for innovation and faster roll-out of new features
• Cost efficiency because cloud networking for huge volumes is expensive
• The flexibility of adopting various innovative technologies and APIs
• Elasticity for critical events a few times a year
• Standards-based integration for integration with SaaS, B2B, and mobile apps
• Security for trusted customer data
• Global and vendor-independent cloud infrastructure to deploy across countries

The good news is that data streaming powered by Apache Kafka and Apache Flink provides all these characteristics on a single platform, especially if you choose a fully managed SaaS offering.

Data streaming in the gaming industry

Adopting gaming trends like in-game purchases, customer-specific discounts, and massively multiplayer online games (MMOG) is only possible if enterprises in the games sector can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means):

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

“Apache Kafka in the Gaming Industry” is a great starting point to learn more about data streaming in the games sector, including a few Kafka-powered case studies not covered in this blog post – such as

• Big Fish Games: Live operations by monitoring real-time analytics of game telemetry and context-specific recommendations for in-game purchases
• Unity: Monetization network for player rewards, banner ads, playable advertisements, and cross-promotions.
• William Hill: Trading platform for gambling and betting
• Disney+ Hotstar: Gamification of live sport video streaming

Architecture trends for data streaming

The gaming industry applies various trends for enterprise architectures for cost, elasticity, security, and latency reasons. The three major topics I see these days at customers are:

• Fully managed SaaS to focus on business logic and faster time-to-market
• Event-driven architectures (in combination with request-response communication) to enable domain-driven design and flexible technology choices
• Data mesh for building new data products and real-time data sharing with internal platforms and partner APIs

Let’s look deeper into some enterprise architectures that leverage data streaming for gaming use cases.

Cloud-native elasticity for seasonal spikes

The games sector has extreme spikes in workloads. For instance, specific game events increase the traffic 10x and more. Only cloud-native infrastructure enables a cost-efficient architecture.

Epic Games presented at an AWS Summit in 2018 already how elasticity is crucial for data-driven architecture.

Make sure to use a truly cloud-native Apache Kafka service for gaming infrastructure. Adding brokers is relatively easy. Removing brokers is much harder. Hence, a fully-managed SaaS should take over the complex operations challenges of distributed systems like Kafka and Flink for you. The separation of compute and storage is another crticial piece of a cloud-native Kafka architecture to ensure cost-efficient scale.

Data mesh for real-time data sharing

Data sharing across business units is important for any organization. The gaming industry has to combine very interesting (different) data sets, like big data game telemetry, monetization and advertisement transactions, and 3rd party interfaces.

Data consistency is one of the most challenging problems in the games sector. Apache Kafka ensures data consistency across all applications and databases, whether these systems operate in real-time, near-real-time, or batch.

One sweet spot of data streaming is that you can easily connect new applications to the existing infrastructure or modernize existing interfaces, like migrating from an on-premise data warehouse to a cloud SaaS offering.

New customer stories for data streaming in the gaming sector

So much innovation is happening in the gaming sector. Automation and digitalization change how gaming companies process game telemetry data, build communities and customer relationships with VIPs, and create new business models with enterprises of other verticals.

Most gaming companies use a cloud-first approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure. And elastic scalability gets even more critical with all the growing real-time expectations and mobile app capabilities.

Here are a few customer stories from worldwide gaming organizations:

• Kakao Games: Log analytics and fraud prevention
• Mobile Premier League (MLP): Mobile eSports and digital gaming
• Demonware / Blizzard: Network and gaming infrastructure
• WhatNot: Retail gamification and social commerce
• Vimeo: Video streaming observability

Resources to learn more

This blog post is just the starting point. Learn more about data streaming in the gaming industry in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-demand video recording

The video recording explores the gaming industry’s trends and architectures for data streaming. The primary focus is the data streaming case studies.

I am excited to have presented this webinar in my interactive light board studio:

This creates a much better experience, especially in a time after the pandemic, where many people are “Zoom fatigue”.

Check out our on-demand recording:

Slides

If you prefer learning from slides, check out the deck used for the above recording:

Case studies and lightboard videos for data streaming in the gaming industry

The state of data streaming for gaming in 2023 is fascinating. New use cases and case studies come up every month. This includes better end-to-end observability in real-time across the entire organization, telemetry data collection from gamers, data sharing and B2B partnerships with engines like Unity or video platforms like Twitch, new business models for ads and in-game purchases, and many more scenarios.

We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories. Here is an example for real-time fraud detection with data streaming.

Gaming is just one of many industries that leverages data streaming with Apache Kafka and Apache Flink.. Every month, we talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, telcos, gaming, and so on… Check out my other blog posts.

Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post The State of Data Streaming for Gaming appeared first on Kai Waehner.

Think about wonderful customer experiences with gamification when watching a match, live betting, location-based services in the stadium, automated payments, coupons, integration with connected fan shops and shopping malls, and so on.

Improving the sport and the related matches itself is another excellent enhancement, including analyzing and monitoring gameplay, the health of players, security, and other use cases.

The current state of table tennis

World Table Tennis (WTT) is a business created by the International Table Tennis Federation (ITTF) to manage the official professional Table Tennis series of events and its commercial rights. Table tennis is more significant than you might think: There are over 200 member associations across the globe within the ITTF.

World Table Tennis also leads the digital sports transformation and commercializes its software application for real-time event scoring worldwide with Apache Kafka.

Previously, ITTF scoring was processed manually with a desktop-based, on-venue results system (OVR) – an on-premises solution to process match data that calculated rankings and records, then sent event information to other systems, such as scoreboards.

Real-time data is essential in the sporting world. The ITTF team re-engineered their data system in 18 months, moving from solely on-premises infrastructure to a cloud-native data system that uses fully managed Confluent Cloud with Apache Kafka as its central nervous system.

Real-time analytics with Kafka to provide stats, engage with fans, and integrate with gaming and betting

Vatsan Rama (Director of IT, ITTF Group) talked in the Confluent podcast about Streaming Real-Time Sports Analytics with Apache Kafka for World Table Tennis. Here are several exciting use cases for real-time analytics around table tennis:

• Real-time stats of scores and any other interesting facts in a match are sent to scoreboards, media broadcasters, betting providers, and other 3rd party consumers
• The empire kicks off the recording of a live feed (stream of events)
• Analysis of player acting in real-time and comparing it to historical data (including advanced use cases like ball spin)
• Smart referees using video analytics in real-time (like fault, net, offsite, foul, etc.)
• Stateful statistics during the broadcast, e.g., the longest ball play (rally) in the last 24 months
• Batch analytics of historical data for coaching and player preparation against the next opponent
• Worldwide consolidation of data from events and leagues across the globe and across different (sub) organizations
• Customer 360 with mobile apps and real-time clickstream analytics to know the fans better and increase revenue (aka fan engagement)
• Data exchange with business partners, e.g., low latency with SLAs for critical use cases like a live betting API integration
• Innovative new business models of integration with cutting-edge technologies like blockchain, NFTs, and crypto

Why data streaming with Kafka makes the difference in processing and sharing sports data

Data connectivity across various interfaces, APIs, and systems plus correlation of the data in real-time is key for modernizing the data infrastructure for any sports use case:

Apache Kafka is the de facto standard for data streaming. Let’s look at why data streaming is a perfect solution for modernizing sports use cases:

• Real-time data integration and processing: Most innovative sports use cases only work well if the information across systems is correlated in real-time. Kafka Connect, Kafka Streams, KSQL, and other components allow using a single infrastructure for data processing.
• Storage: True decoupling and backpressure handling are crucial as slow consumers have different data processing capabilities than real-time consumers. The replayability of historical information does not require yet another database or data lake. Tiered Storage enables cost-efficient long-term storage in Kafka.
• Hybrid edge infrastructure: Some use cases require low-latency or offline compute power. Kafka is perfect, being a single platform for real-time integration, data processing, and storage. Real-time replication between separate Kafka environments provides out-of-the-box support for edge analytics and sometimes disconnected environments.
• Data governance across stakeholders and environments: Data privacy, access control, compliance, and zero trust are critical characteristics of any modern IT infrastructure. The Kafka ecosystem monitors and enforces the end-to-end data flow using a Schema Registry for defining contracts between independent data producers and downstream consumers and additional tools on top of data lineage and distributed tracing.
• Fully managed cloud-first approach: The cloud enables focusing on business problems and innovation. Only manage your own Kafka clusters if serverless SaaS is impossible for security, cost, or latency reasons! Don’t trust marketing and make sure your Kafka service is indeed fully managed, not just partially managed, where you take over the risk and operation burden.
• Omnichannel customer 360: Most businesses and fans require access to information across different interfaces (including web browsers, mobile apps, devices, smart point of sale, location-based services, and so on). Kafka’s unique combination of real-time messaging and storage provides out-of-the-box support for building decoupling customer 360 applications.
• Data sharing and open API for B2B exchange: Most sports use cases hold various data sets that enable new internal and external use cases. Data sharing across business units and 3rd party business partners or public Open APIs in real-time allows innovation to improve the customer experience or establish brand new business models. Kafka and related cloud services enable real-time data exchange.
• Proactive cybersecurity: Digitalization comes with its risks. Stars use social networks. Stadiums and shops get connected. Cameras monitor players and environments. And so on. Real-time situational awareness and threat intelligence are crucial to protect the data, and people are essential in a world where everything is digital and integrated.
• Integration with blockchain, crypto, and NFT: Beyond the current crypto winter, many innovative use cases will come for the metaverse and decentralized identity management. one example is selling instant moments in sports via NFTs. Kafka is the middleware between regular applications and the NFT and crypto trading platforms.

Reimagine sports and turn customers into fans with data streaming using Apache Kafka

The post Reimagine sports and gaming with data streaming: A table tennis success story built with Apache Kafka appeared first on Kai Waehner.

Learn about:

• Real-time analytics and data correlation of game telemetry
• Monetization network for real-time advertising and in-app purchases
• Payment engine for betting
• Detection of financial fraud and cheating
• Chat function in games and cross-games
• Monitor the results of live operations like weekend events or limited time offers
• Real-time analytics on metadata and chat data for marketing campaigns

The Evolution of the Gaming Industry

The gaming industry must process billions of events per day in real-time and ensure consistent and reliable data processing and correlation across gameplay interactions and backend analytics. Deployments must run globally and work for millions of users 24/7 on 365 days a year.

These requirements are valid for hardcore games and blockbusters, including massively multiplayer online role-playing games (MMORPG), first-person shooters, and multiplayer online battle arenas (MOBA), but also mid-core and casual games. Reliable and scalable real-time integration with consumer devices like smartphones and game consoles is as essential as cooperating with online streaming services like Twitch and betting providers.

Business Models in the Gaming Industry

Gaming is not just about games anymore. Though, even in the games industry, the option of playing games diverse from consoles and PCs to mobile games, casino games, online games, and various other options. In addition to the games, people also engage via professional eSports, $$$ tournaments, live video streaming, and real-time betting.

This is a crazy evolution, isn’t it? Here are some of the business models relevant today in the gaming industry:

• Hardware sales
• Game sales
• Free-to-play + in-game purchases, such as skins or champions
• Gambling (Loot boxes)
• Game-as-a-service (Subscription)
• Seasonal in-game purchases like passes for theme events, mid-season invitational & world championship, passes for competitive play
• Game-Infrastructure-as-a-Service
• Merchandise sales
• Communities including eSports broadcast, ticket sales, franchising fees
• Live betting
• Video streaming, including ads, rewards, etc.
• …

Evolution of “AI” (Artificial Intelligence) in Gaming

Artificial Intelligence (business rules, statistical models, machine learning, deep learning) is vital for many use cases in Gaming. These use cases include:

• In-game AI: Non-playable characters (NPC), environments, features
• Fraud detection: Cheating, financial fraud, child abuse
• Game analytics: Retention, game changes (real-time delivery or via next patch/update)
• Research: Find new algorithms, improve AI, adopt to business problems

Many of the use cases I explore use AI in conjunction with event streaming and Kafka in the following.

Hybrid Gaming Architectures for Event Streaming with Apache Kafka

A vast demand for building an open, flexible, scalable platform and real-time processing are the reasons why so many gaming-related projects use Apache Kafka. I will not discuss Kafka here and assume you know why Kafka became the de facto standard for event streaming.

What’s more interesting is the different deployments and architectures I have seen in the wild. Infrastructures in the gaming industry are often global. Sometimes cloud-only, sometimes hybrid with local on-premises installations. Betting is usually regional (mainly because of laws and compliance reasons). Games typically are global. If a game is excellent, it gets deployed and rolled out across the world.

Let’s now take a look at several different use cases and architectures in the gaming industry. Most of these examples are relevant in all gaming-related use cases, including games, mobile, betting, gambling, and video streaming.

Infrastructure Operations – Live Monitoring and Troubleshooting

Monitoring the results of live operations is essential for every mission-critical infrastructure. Use cases include:

• Game clients, game servers, game services
• Service health 24/7
• Special events such as weekend tournaments, limited time offers and user acquisition campaigns

Immediate and correct troubleshooting require real-time monitoring. You need to be able to answer questions like “Who creates the problem? Client? ISP? The game itself?”

Let’s take a look at a typical example in the gaming industry: A new marketing campaign:

• “Play for free over the weekend”
• Scalability – Huge extra traffic
• Monitoring – Was the marketing campaign successful? How profitable is the game/business?
• Real-time (e.g., alerting)
• Batch (e.g., analytics and reporting of success with Snowflake)

A lot of diverse data has to be integrated, correlated, and monitored to keep the infrastructure running and to troubleshoot issues.

Elasticity Is the Key for Success in the Games Industry

A key challenge in infrastructure monitoring is the required elasticity. You cannot just provision some hardware, deploy the software, and operate it 24 hours 365 days a year. Gaming infrastructures require elasticity. No matter if you care about online games, betting, or video streaming.

Chris Dyl, Director of Platform at Epic Games, pointed this out well at AWS Summit 2018: “We have an almost ten times difference in workloads between peak and low-peak. Elasticity is really, really important for us in any particular region at the cloud providers”.

Confluent provides elasticity for any Kafka deployment, no matter if the event streaming platform runs self-managed at the edge or fully managed in the cloud. Check out “Scaling Apache Kafka to 10+ GB Per Second in Confluent Cloud” to see how Kafka can be scaled automatically in the cloud. Self-managed Kafka gets elastic by using tools such as Self-Balancing Kafka, Tiered Storage, and Confluent Operator for Kubernetes.

Game Telemetry – Real-time Analytics and Data Correlation with Kafka

Game Telemetry describes how the player plays the game. Player information includes business logic such as user actions (button clicks, shooting, use item) or game environment metrics (quests, level up), and technical information like login from a specific server, IP address, location.

Global Gaming requires proxies all over the world to guarantee regional latency for millions of clients. Besides, a central analytics cluster (with anonymized data) correlates data from across the globe. Here are some use cases for using game telemetry:

• Game monitoring
• How well do players progress through the game and what problems occurred
• Live operations – Adjust the gameplay
• Server-side changes while the player is playing the game (e.g., time-limited event, give reward)
• Real-time updates to improve the game or align to audience needs (or in other words: Recommend an item / upgrade / skin / additional in-game purchase

Most use cases require processing big data streams in real-time:

Big Fish Games

Big Fish Games is an excellent example of live operations leveraging Apache Kafka and its ecosystem. They develop casual and mid-core games. 2.5 billion games were installed on smartphones and computers in 150 countries, representing over 450 unique mobile games and over 3,500 unique PC games.

Live operations use real-time analytics of game telemetry data. For instance, Big Fish Games increases revenue while the player plays the game by making context-specific recommendations for in-game purchases in real-time. Kafka Streams is used for continuous data correlation in real-time at scale.

Check out the details in the Kafka Summit Talk “How Big Fish Games developed real-time analytics“.

Monetization Network

Monetization networks are a fundamental component in most gaming companies. Use cases include:

• In-game advertising
• Micro-transactions and in-game purchases: Sell Skins, Upgrade to the next level…
• Game-Infrastructure-as-a-Service: Multi-platform-and-store-integration, matchmaking, advertising,   player identity and friends, cross-play, lobbies, leader boards, achievements, game analytics, …
• Partner network: Cross-sell game data, game SDK, game analytics, …

A monetization network looks like the following:

Unity Ads – Monetization network

Unity is a fantastic example. In 2019, content installed 33 billion times, reaching 3 billion devices worldwide. The company provides a real-time 3D development platform.

Unity operates one of the largest monetization networks in the world:

• Reward players for watching ads
• Incorporate banner ads
• Incorporate Augmented Reality (AR) ads
• Playable ads
• Cross-Promotions

Unity is a data-driven company:

• Average about half a million events per second
• Handles millions of dollars of monetary transactions
• Data infrastructure based on Confluent Platform, Confluent Cloud and Apache Kafka

A single data pipeline provides the foundational infrastructure for analytics, R&D, monetization, cloud services, etc. for real-time and batch processing leveraging Apache Kafka:

• Real-time monetization network
• Feed machine learning models in real-time
• Data lake went from two-day latency down to 15 minutes

If you want to learn about their success story migrating this platform from self-managed Kafka to fully-managed Confluent Cloud, read Unity’s post on the Confluent Blog: “How Unity uses Confluent for real-time event streaming at scale“.

Chat Function within Games and Cross-Platform

Building a chat platform is not a trivial task in today’s world. Chatting means send text, in-game screenshots, in-game items, and other things. Millions of events have to be processed in real-time. Cross-platform chat platforms need to support various technologies, programming languages, and communication paradigms such as real-time, batch, request-response:

The characteristics of Kafka make it the perfect infrastructure for chat platforms due to high scalability, real-time processing, and real decoupling, including backpressure handling.

Payment Engine

Payment infrastructure needs to be real-time, scalable, reliable, and technology-independent. No matter if your solution is built for games, betting, casino, 3D game engines, video streaming, or any other 3rd services.

Most payment engines in the gaming industry are built on top of Apache Kafka. Many of these companies provide public information about their real-time betting infrastructure. Here is one example of an architecture:

One example use case is the implementation of a betting delay and approval system in live bets. Stateful streaming analytics is required to improve the margin:

Kafka-native technologies like Kafka Streams or ksqlDB enable a straightforward implementation of these scenarios.

William Hill – A Secure and Reliable Real-time Microservice Architecture

William Hill went from a monolith to a flexible, scalable microservice architecture:

• Kafka as central, reliable streaming infrastructure
• Kafka for messaging, storage, cache and processing of data
• Independent decoupled microservices
• Decoupling and replayability
• Technology independence
• High throughput + low latency + real-time

William Hill’s trading platform leverages Kafka as the heart of all events and transactions:

• “process-to-process” execution in real-time
• Integration with analytic models for real-time machine learning
• Various data sources and data sinks (real-time, batch, request-response)

Bookmaker business == Banking Business (including Legacy Middleware and Mainframes)

Not everyone can start from greenfield. Legacy middleware and mainframe integration, offloading, and replacement is a common scenario.

Betting usually is a regulated market. PII data is often processing on-premise in a regional data center. Non-PII data can be offloaded to the cloud for Analytics.

Legacy technologies like mainframe are a crucial cost factor, monolithic and inflexible. I covered the relation between Kafka and Mainframe in detail in the following post:

And here is the story about Kafka vs. Legacy Middleware (MQ, ETL, EBS).

Streaming Analytics for Retention, Compliance, and Customer Experience

Data quality is critical for legal compliance. Responsible gaming compliance. Client retention is vital to keep engagement and revenue growth.

Plenty of real-time streaming analytics use cases exist in this environment. Some examples where Kafka-native frameworks like Kafka Streams or ksqlDB can provide the foundation for a reliable and scalable solution:

• Player winning / losing streak
• Player conversion – from registration to wage (within x min)
• Game achievement of the player
• Fraud detection – e.g., payment windows
• Long-running windows per player over days/months
• Tournaments
• Incentive unhappy players with an additional free credit
• Reports to regulator – replay old events in a guaranteed order
• Geolocation to enable features, limitations or commissions

Stream processing is also relevant for many other use cases, including fraud detection, as you will see in the next section.

Fraud Detection in Gaming with Kafka

Real-time analytics for detecting anomalies is a widespread scenario in any payment infrastructure. In Gaming, two different kinds of fraud exist:

• Cheating: Fake accounts, bots, …
• Financial fraud: match-fixing, stolen credit cards, …

Here is an example of doing streaming analytics for fraud detection with Kafka, its ecosystem, and machine learning:

Here is an example of detecting financial fraud and cheating with Jupyter notebooks and Python to analyze data pre-processed with ksqlDB:

Customer 360 – Recommendations, Loyalty System, Social Integration

Customer 360 is critical for real-time and context-specific acquisition, engagement, and retention. Use cases include:

• Real-Time Event Streaming
  • Game event triggers
  • Personalized statistics and odds
  • Player segmentation
  • Campaign orchestration (“player journey”)
• Loyalty system
  • Rewards e.g., upgrade, exclusive in-game content, beta keys for the announcement event
  • Avoid customer churn
  • Cross-selling
• Social Network integration
  • Twitter, Facebook, …
  • Example: Candy Crush (I guess every Facebook user has seen ads for this game)
• Partner integration
  • API Management

The following architecture depicts the relation between various internal and external components of a customer 360 solution:

Customer 360 at Sky Betting & Gaming

Sky Betting & Gaming has built a real-time streaming architecture for customer 360 use cases with Kafka’s ecosystem.

Here is a quote of why they choose Kafka-native frameworks like Kafka Streams instead of a zoo of technologies like Hadoop, Spark, Storm, and others:

“Most of our streaming data is in the form of topics on a Kafka cluster. This means we can use tooling designed around Kafka instead of general streaming solutions with Kafka plugins/connectors.

Kafka itself is a fast-moving target, with client libraries constantly being updated; waiting for these new libraries to be included in an enterprise distribution of Hadoop or any off the shelf tooling is not really an option. Finally, the data in our first use-case is user-generated and needs to be presented back to the user as quickly as possible.”

Disney+ Hotstar – Telco-OTT for millions of cricket fans in India

In India, people love cricket. Millions of users watch live streams on their smartphones. But they are not just watching it. Instead, gambling is also part of the story. For instance, you can bet on the result of the next play. People compete with each other and can win rewards.

This infrastructure has to run at extreme scale. Millions of actions have to be processed each second. No surprise that Disney+ Hotstar chose Kafka as the heart of this infrastructure:

IoT Integration is often also part of such a customer 360 implementation. Use cases include:

• Live eSports events, TV, video streaming and news stations
• Fan engagement
• Audience communication
• Entertaining features for Alexa, Google Home or sports-specific hardware

Cross-Company Kafka Integration

Last but not least, let’s talk about a trend I see in many industries: Streaming replication across departments and companies.

Most companies in the gaming industry use event streaming with Kafka at the heart of their business. However, connecting to the outside world (i.e., other departments, partners, 3rd party services) is typically done via HTTP / REST APIs. A total anti-pattern! Not scalable! Why not directly stream the data?

I see more and more companies moving to this approach.

API Management is an elaborate discussion on its own. Therefore, I have a dedicated blog post about the relation between Kafka and API Management:

Slides and Video – Kafka in the Gaming Industry

Here are the slides and on-demand video recording discussing Apache Kafka in the gaming industry in more detail:

As you learned in this post, Kafka is used everywhere in the gaming industry. No matter if you focus on games, betting, or video streaming.

What are your experiences with modernizing the infrastructure and applications in the gaming industry? Did you or do you plan to use Apache Kafka and its ecosystem? What is your strategy? Let’s connect on LinkedIn and discuss it!

The post Apache Kafka in Gaming (Games Industry, Bookmaker, Betting, Gambling, Video Streaming) appeared first on Kai Waehner.