Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And download my free book about data streaming use cases, including a dedicated chapter about the telco industry.

Data Streaming in the Telco Industry

Telecommunications networks generate vast amounts of data every second. Every call, message, internet session, device interaction, and network event produces valuable information. Historically, much of this data was processed in batches — often hours or even days after it was collected. This delayed model no longer meets the needs of modern Telcos, partners, and customers.

Data streaming transforms how Telcos handle information. Instead of storing and processing data later, it is ingested, processed, and acted upon in real time as it is generated. This enables continuous intelligence across all parts of the network and business.

Learn more about “The Top 20 Problems with Batch Processing (and How to Fix Them with Data Streaming)“.

Business Value of Data Streaming in the Telecom Sector

Key benefits of data streaming for Telcos include:

• Real-Time Visibility: Immediate insight into network health, customer behavior, fraud attempts, and service performance.
• Operational Efficiency: Faster detection and resolution of issues reduces downtime, improves customer satisfaction, and lowers operating costs.
• New Revenue Opportunities: Real-time data enables new services such as dynamic pricing, personalized offers, and proactive customer support.
• Enhanced Security and Compliance: Immediate anomaly detection and instant auditability support regulatory requirements and protect against cyber threats.

Technologies like Apache Kafka and Apache Flink are now core components of Telco IT architectures. They allow Telcos to integrate massive, distributed data flows from radio access networks (RAN), 5G core systems, IoT ecosystems, billing and support platforms, and customer devices.

Modern Telcos use data streaming to not only improve internal operations but also to deliver trusted, secure, and differentiated services to external partners such as MVNOs, IoT platforms, and enterprise customers.

Learn More about Data Streaming in Telco

Learn more about data streaming in the telecommunications sector:

• How Data Streaming and AI Help Telcos to Innovate: Top 5 Trends from MWC 2025
• How Apache Kafka helps Dish Wireless building cloud-native 5G Telco Infrastructure
• The State of Data Streaming for Telco
• Apache Kafka for Telco-OTT (Telecom Sector) and Media Applications
• Data Streaming with Apache Kafka and Flink in the Media Industry: Disney+ Hotstar and JioCinema

Data streaming is not an allrounder to solve every problem. Hence, a modern enterprise architecture combines data streaming with purpose-built telco-specific platforms and SaaS solutions, and data lakes/warehouses/lakehouses like Snowflake or Databricks for the analytical workloads.

I already wrote about the combination of data streaming platforms like Confluent together with Snowflake and Microsoft Fabric. A blog series about data streaming with Confluent combined with AI and analytics using Databricks is coming right after this blog post here.

Building a Real-Time Data Sharing Platform in the Telco Industry with Data Streaming

By mastering real-time data streaming, Telcos unlock the ability to share valuable insights securely and efficiently with internal divisions, IoT platforms, and enterprise customers.

Mobile Virtual Network Operators (MVNOs) — companies that offer mobile services without owning their own network infrastructure — are an equally important group of consumers. As an MVNO delivers niche services, competitive pricing, and tailored customer experiences, real-time data sharing becomes essential to support their growth and enable differentiation in a highly competitive market.

A strong real-time data sharing platform in the telco industry integrates multiple types of components and stakeholders, organized into four critical areas:

Data Sources

A real-time data platform aggregates information from a wide range of technical systems across the Telco infrastructure.

• Radio Access Network (RAN) Metrics: Capture real-time information about signal quality, handovers, and user session performance.
• 5G Core Network Functions: Manage traffic flows, session lifecycles, and device mobility through UPF, SMF, and AMF components.
• Operational Support Systems (OSS) and Business Support Systems (BSS): Provide data for service assurance, provisioning, customer management, and billing processes.
• IoT Devices: Send continuous telemetry data from connected vehicles, industrial assets, healthcare monitors, and consumer electronics.
• Customer Premises Equipment (CPE): Supply performance and operational data from routers, gateways, modems, and set-top boxes.
• Billing Events: Stream usage records, real-time charging information, and transaction logs to support accurate billing.
• Customer Profiles: Update subscription plans, user preferences, device types, and behavioral attributes dynamically.
• Security Logs: Capture authentication events, threat detections, network access attempts, and audit trail information.

Stream Processing

Stream processing technologies ensure raw events are turned into enriched, actionable data products as they move through the system.

• Real-Time Data Ingestion: Continuously collect and process events from all sources with low latency and high reliability.
• Data Aggregation and Enrichment: Transform raw network, billing, and device data into structured, valuable datasets.
• Actionable Data Products: Create enriched, ready-to-consume information for operational and business use cases across the ecosystem.

Data Governance

Effective governance frameworks guarantee that data sharing is secure, compliant, and aligned with commercial agreements.

• Policy-Based Access Control: Enforce business, regulatory, and contractual rules on how data is shared internally and externally.
• Data Protection Techniques: Apply masking, anonymization, and encryption to secure sensitive information at every stage.
• Compliance Assurance: Meet regulatory requirements like GDPR, CCPA, and telecom-specific standards through real-time monitoring and enforcement.

Data Consumers

Multiple internal and external stakeholders rely on tailored, policy-controlled access to real-time data streams to achieve business outcomes.

• MVNO Partners: Consume real-time network metrics, subscriber insights, and fraud alerts to offer better customer experiences and safeguard operations.
• Internal Telco Divisions: Use operational data to improve network uptime, optimize marketing initiatives, and detect revenue leakage early.
• IoT Platform Services: Rely on device telemetry and mobility data to improve fleet management, predictive maintenance, and automated operations.
• Enterprise Customers: Integrate real-time network insights and SLA compliance monitoring into private network and corporate IT systems.
• Regulatory and Compliance Bodies: Access live audit streams, security incident data, and privacy-preserving compliance reports as required by law.

Key Data Products Driving Value for Data Sharing in the Telco Industry

In modern Telco architectures, data products act as the building blocks for a data mesh approach, enabling decentralized ownership, scalable integration with microservices, and direct access for consumers across the business and partner ecosystem.

The right data products accelerate time-to-insight and enable additional revenue streams. Leading Telcos typically offer:

• Network Quality Metrics: Monitoring service degradation, latency spikes, and coverage gaps continuously.
• Customer Behavior Analytics: Tracking app usage, mobility patterns, device types, and engagement trends.
• Fraud and Anomaly Detection Feeds: Capturing unusual usage, SIM swaps, or suspicious roaming activities in real time.
• Billing and Charging Data Streams: Delivering session records and consumption details instantly to billing systems or MVNO partners.
• Device Telemetry and Health Data: Providing operational status and error signals from smartphones, CPE, and IoT devices.
• Subscriber Profile Updates: Streaming changes in service plans, device upgrades, or user preferences.
• Location-Aware Services Data: Powering geofencing, smart city applications, and targeted marketing efforts.
• Churn Prediction Models: Scoring customer retention risks based on usage behavior and network experience.
• Network Capacity and Traffic Forecasts: Helping optimize resource allocation and investment planning.
• Policy Compliance Monitoring: Ensuring real-time validation of internal and external SLAs, privacy agreements, and regulatory requirements.

These data products can be offered via APIs, secure topics, or integrated into partner platforms for direct consumption.

How Each Data Consumer Gains Strategic Value

Real-time data streaming empowers each data consumer within the Telco ecosystem to achieve specific business outcomes, drive operational excellence, and unlock new growth opportunities based on continuous, trusted insights.

Internal Telco Divisions

Real-time insights into network behavior allow proactive incident management and customer support. Marketing teams optimize campaigns based on live subscriber data, while finance teams minimize revenue leakage by tracking billing and usage patterns instantly.

MVNO Partners

Access to live network quality indicators helps MVNOs improve customer satisfaction and loyalty. Real-time fraud monitoring protects against financial losses. Tailored subscriber insights enable MVNOs to offer personalized plans and upsells based on actual usage.

IoT Platform Services

Large-scale telemetry streaming enables better device management, predictive maintenance, and operational automation. Real-time geolocation data improves logistics, fleet management, and smart infrastructure performance. Event-driven alerts help detect and resolve device malfunctions rapidly.

Enterprise Customers

Private 5G networks and managed services depend on live analytics to meet SLA obligations. Enterprises integrate real-time network telemetry into their own systems for smarter decision-making. Data-driven optimizations ensure higher uptime, better resource utilization, and enhanced customer experiences.

Building a Trusted Data Ecosystem for Telcos with Real-Time Streaming and Hybrid Cloud

Real-time data sharing is no longer a luxury for Telcos — it is a competitive necessity. A successful platform must balance openness with control, ensuring that every data exchange respects privacy, governance, and commercial boundaries.

Hybrid cloud architectures play a critical role in this evolution. They enable Telcos to process, govern, and share real-time data across on-premises infrastructure, edge environments, and public clouds seamlessly. By combining the flexibility of cloud-native services with the security and performance of on-premises systems, hybrid cloud ensures that data remains accessible, scalable, cost-efficient and compliant wherever it is needed.

By deploying scalable data streaming solutions across a hybrid cloud environment, Telcos enable secure, real-time data sharing with MVNOs, IoT platforms, enterprise customers, and internal business units. This empowers critical use cases such as dynamic quality of service monitoring, real-time fraud detection, customer behavior analytics, predictive maintenance for connected devices, and SLA compliance reporting — all without compromising performance or regulatory requirements.

The future of telecommunications belongs to those who implement real-time data streaming and controlled data sharing — turning raw events into strategic advantage faster, more securely, and more effectively than ever before.

How do you share data in your organization? Do you already leverage data streaming or still operate in batch mode? Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases.

The post Real-Time Data Sharing in the Telco Industry for MVNO Growth and Beyond with Data Streaming appeared first on Kai Waehner.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases.

Data Streaming with Apache Kafka and Flink in the Media Industry

Data streaming technologies like Apache Kafka and Flink are revolutionizing the media industry by enabling real-time data processing at an unprecedented scale. Media platforms, including Over-The-Top (OTT) services operated by telcos and media companies, leverage these technologies to deliver video, audio, and other content directly to viewers over the internet. The OTT services bypass traditional cable or satellite channels.

As these platforms cater to growing audiences with diverse needs, data streaming serves as the backbone for seamless content delivery, real-time user engagement, and operational efficiency. Data streaming ensures a superior viewing experience at scale.

Netflix is a leading global media company renowned for its extensive use of Apache Kafka and Flink. The media company powers critical use cases such as real-time personalization, anomaly detection, and monitoring at extreme scale. Its data streaming architecture processes billions of events daily, ensuring seamless content delivery and exceptional viewer experiences for a global audience.

Use Cases for Data Streaming in the Media Industry

Data streaming with technologies like Apache Kafka and Flink is transforming the media industry by enabling real-time data processing for seamless content delivery, personalized experiences, and operational efficiency.

1. Live Video Streaming: Data streaming with Apache Kafka serves as a central event hub for managing log data, metadata, and control signals associated with live video streaming. It processes real-time data related to user interactions, stream health, and session analytics to ensure ultra-low latency and a seamless experience for live events like concerts and sports. The actual video streams are handled by Content Delivery Networks (CDNs).
2. On-Demand Content Delivery: Media platforms use Kafka to reliably manage data pipelines, delivering movies, TV shows, and other content to millions of users.
3. Personalized Recommendations: By integrating Kafka with analytics tools, platforms provide tailored suggestions based on user behavior, increasing viewer engagement and satisfaction.
4. Real-Time Ad Targeting: Kafka enables real-time ad insertion by processing user events and contextual data, ensuring ads are relevant and timely.
5. Monitoring and Anomaly Detection: Media companies use Kafka to monitor backend systems in real time, detecting and resolving issues proactively to ensure a smooth user experience.
6. Churn Prediction: By analyzing behavioral patterns in real time, platforms can predict user churn and take corrective actions, such as offering discounts or new content recommendations.

Learn more about data streaming use cases in the telco and media industry from real world customer stories like Dish Network, British Telecom, Globe Telecom, Swisscom, and more:

Business Value of Data Streaming in Media

Data streaming technologies like Apache Kafka and Flink drive transformative business value in the media industry by enabling real-time insights, efficiency, and innovation:

• Enhanced User Experience: Real-time at any scale capabilities enable faster content delivery, personalized recommendations, and reduced buffering.
• Cost Optimization: Streamlined pipelines improve infrastructure utilization and reduce operational costs. The Shift Left Architecture is adopted across business units.
• Revenue Growth: Precision in ad targeting and churn reduction leads to higher revenues.
• Competitive Edge: Real-time analytics and content delivery position companies as leaders in their field.

Disney+ Hotstar (Disney) and JioCinema (Viacom18): Streaming Giants Shaping India’s Media Landscape

Disney+ Hotstar revolutionized OTT streaming in India with a robust freemium model. Catering to a diverse audience, it provided an extensive library of movies, TV shows, and sports, including exclusive streaming rights for the Indian Premier League (IPL), the world’s most popular cricket league. By blending free content with premium subscriptions, it attracted millions of users, leveraging IPL viewership as a major growth driver.

JioCinema, part of Reliance Jio, employs a mass-market approach, offering free streaming supported by Reliance’s vast 5G network. It gained significant traction by taking over the IPL digital streaming rights in 2023 in 4K resolution to over 32 million concurrent viewers, breaking records for live streaming.

Each platform used respectively uses IPL strategically—Hotstar with a premium model and JioCinema for mass-market penetration. Post-merger, the unified platform could combine these approaches, delivering enhanced IPL experiences powered by a consolidated Kafka-based streaming infrastructure.

Both platforms share a commitment to innovation, scalability, and user engagement, making them ideal candidates for heavy Apache Kafka usage.

Data Streaming Architectures with Kafka and Flink

Both Disney+ Hotstar and JioCinema (Viacom18) are renowned for their openness in discussing their technical data streaming architectures, similar to Netflix. They frequently presented at conferences like Kafka Summit and industry events, sharing insights about their data streaming strategies and implementations.

This transparency achieves several goals:

• Showcasing Innovation: Highlighting their advanced use of Kafka and Flink establishes their leadership in tech innovation.
• Talent Acquisition: Open discussions attract engineers who want to work on cutting-edge systems.
• Industry Collaboration: Sharing experiences fosters collaboration within the streaming and open-source communities.

By examining their presentations and publications, we gain a deeper understanding of their use of Kafka to achieve extreme scalability and efficiency.

Data Streaming Solves the Challenges and Extreme Scale of OTT Services in the Media Industry

Running platforms of this scale comes with its share of challenges:

• Massive Throughput: Kafka handles billions of messages daily, requiring extensive partitioning and scaling strategies.
• Fault Tolerance: Platforms implement advanced disaster recovery and replication strategies to ensure zero downtime, even during critical events like IPL.
• Cost vs. Performance Trade-Offs: Streaming 4K video for millions of users demands balancing high infrastructure costs with user expectations.

Data streaming with Apache Kafka and Flink is a key piece of the data strategy to solve these challenges.

Disney+ Hotstar: Gamification at Extreme Scale

Disney+ Hotstar’s “Watch N Play” feature transformed live sports streaming, particularly cricket, into an interactive experience. Viewers predict outcomes, answer trivia, and participate in polls, earning points for rewards or leaderboard rankings, adding a competitive and social element to the platform.

Hotstar’s presentation from Kafka Summit 2019 is still very impressive and worth watching. Here is a summary about the OTT services serving millions of cricket fans:

Powered by Apache Kafka, Disney+ Hotstar’s infrastructure processed millions of real-time interactions per second. The integration of data sources via Kafka Connect enables seamless analytics and rewards. This gamified approach enhances user engagement and extends to broader applications like e-sports, interactive TV, and IoT-driven fan experiences, making Hotstar a leader in innovative streaming.

Disney+ Hotstar runs ~15 different Kafka Connect clusters with over 2000+ connectors and auto-scaling based on traffic, as they presented in another Kafka Summit talk in 2021.

Single Message Transforms (SMT) are used within the Kafka Connect integration for stateless streaming ETL. Integration use cases include masking/filtering of PlI, sampling of data, and schema validation and enforcement.

JioCinema: Multiple Kafka Clusters and Deployment Strategies

JioCinema leverages a robust enterprise architecture built on Apache Kafka, Flink, and Spark. As showcased at Kafka Summit India 2024, data streaming is central to its platform, enabling real-time analytics, personalized recommendations, and seamless content delivery.

Initially, JioCinema operated a single Kafka cluster handling 1,000+ topics and 100,000+ partitions for diverse use cases.

Over time, the platform transitioned to multiple Kafka clusters with different SLAs and architectures, optimizing uptime, performance, and costs for specific workloads, as explained by Kushal Khandelwal, Head of Data Platform.

This shift from a monolithic to a segmented architecture highlights the scalability and flexibility of Kafka. This approach ensures JioCinema meets the demands of high traffic and complex SLAs. Their success reflects the common journey of organizations scaling data streaming infrastructures to achieve operational excellence.

Use Cases for Kafka in Disney+ Hotstar and JioCinema

Disney+ Hotstar and JioCinema rely on Apache Kafka to power diverse use cases, from IPL cricket streaming to real-time personalization and ad targeting.

IPL Cricket Streaming at Massive Scale

The Indian Premier League (IPL) is the crown jewel of streaming in India, drawing millions of concurrent viewers. Here’s how Kafka and Flink support IPL’s massive scale:

• Concurrent Viewers: During IPL 2023, JioCinema hit a record of over 32 million concurrent viewers, streaming matches in 4K resolution. Disney+ Hotstar has also scaled to tens of millions of viewers in past IPL seasons.
• Data Throughput: JioCinema and Hotstar handle millions of messages per second with Kafka, ensuring uninterrupted video delivery.
• Kafka Infrastructure: Reports reveal that JioCinema operates over 100 Kafka clusters, managing tens of thousands of partitions. These clusters handle not only video streaming but also ancillary tasks, like ad placement and user analytics.
• Connectors: Both platforms rely on hundreds of Kafka Connect connectors to integrate with databases, storage systems, and real-time analytics platforms.

On-Demand Streaming and Catalog Management

Both platforms use Kafka to deliver on-demand content to millions of users, ensuring quick access to movies and TV shows. Kafka’s reliable event streaming guarantees smooth playback and dynamic scaling during peak usage.

Real-Time Personalization and Recommendations

Personalization is central to user retention. Kafka streams user behavior data to machine learning systems in real time, enabling both platforms to recommend content tailored to individual preferences. Customer loyalty and Rewards platform often leverage Kafka and Flink under the hood.

Ad Targeting and Revenue Optimization

By processing user data in real time, Kafka enables precise ad targeting with context-specific advertisements. This not only improves ad effectiveness but also enhances viewer experience by ensuring ads are contextually relevant. Many real-time advertising platforms are powered by a data streaming platform using Apache Kafka and Flink.

Content Quality Monitoring

Both platforms use Kafka for continuous real-time monitoring of video stream quality, automatically adjusting bitrate or rerouting streams during disruptions to maintain a consistent viewing experience.

Data Streaming for M&A, Merger and Migrations

The merger of Disney+ Hotstar and JioCinema presents a significant opportunity to integrate their Kafka-based infrastructures, paving the way for a unified, more efficient system. Such transitions are a natural fit for Apache Kafka and its ecosystem. Migrations are a core capability. Tools like MirrorMaker and Cluster Linking allow seamless data movement between clusters for continuous replication and a later lift and shift. The usage of data streaming for migration projects enables zero-downtime and business continuity.

Here are some opportunities and benefits of data streaming for integrations and migrations:

1. Integrated Pipelines: A combined Kafka architecture could streamline content delivery, reduce costs, and support advanced analytics, providing an optimized infrastructure for their vast user base.
2. Expanded Use Cases: The merger might drive innovations such as multi-angle viewing, live interactive features, and more personalized experiences powered by real-time data.
3. Hybrid and Multi-Cloud Flexibility: Transitions like these often span hybrid and multi-cloud environments, making Kafka’s flexibility essential for connecting and scaling across platforms.
4. Multi-Organization Integration: Merging Kafka clusters across distinct organizations, as in this case, is a common use case where Kafka’s tools excel.
5. Technical Leadership: Both platforms are transparent about their Kafka implementations, and we can anticipate new insights from their efforts to integrate and scale, highlighting lessons for the broader streaming industry.

In conclusion, Kafka and Flink are not just enablers but drivers of success for Disney+ Hotstar and JioCinema. Data streaming at scale creates new benchmarks for innovation and user experience in the media industry.

Do you see similar opportunities in your organization? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter. And make sure to download my free book about data streaming use cases.

The post Data Streaming with Apache Kafka and Flink in the Media Industry: Disney+ Hotstar and JioCinema appeared first on Kai Waehner.

What is a 5G telco network?

5G, short for “fifth generation,” is the latest generation of wireless communication technology for mobile devices and networks. It represents a significant advancement over its predecessor, 4G LTE (fourth generation).

Technical benefits of 5G

Here are some key features and characteristics of 5G:

1. Faster speeds: 5G offers significantly faster data transfer speeds than 4G. It delivers download speeds of up to several gigabits per second, allowing for nearly instantaneous downloads of large files, high-definition video streaming, and low-latency online gaming.
2. Low latency: One of the most notable improvements in 5G is its low latency, which refers to the delay between sending and receiving data. This low latency is crucial for applications like autonomous vehicles, remote surgery, and real-time augmented and virtual reality experiences.
3. Increased capacity: 5G networks can support a much larger number of devices simultaneously within the same geographic area. This is essential for the growing number of connected devices in the Internet of Things (IoT) ecosystem.
4. Enhanced connectivity: 5G networks use a variety of frequency bands, including higher-frequency millimeter-wave (mmWave) bands and mid-band frequencies. These higher frequencies enable faster speeds but require more infrastructure, including small cells and antennas placed closer together.
5. Improved network slicing: 5G allows for network slicing, which means network operators can allocate specific portions of the network to meet the needs of different applications and services. This enhances the flexibility and efficiency of the network.
6. Security: Cellular has advantages in authentication, ruggedization, and built-in systematic threat detection/mitigation versus other commonly used networks (Wi-Fi, LPWAN, etc.). 5G was completely redesigned vs. 4G to adopt a zero-trust paradigm, including within the network and additional features that protect user identity. In contrast to a monolithic “trusted telco” network, 5G more closely aligns with the best practices of the most sophisticated enterprise networks.

Business opportunities of 5G

5G enables a massive potential for New Applications across all industries. 5G technology opens the door to various innovative applications and services. It can revolutionize entire industries, like healthcare, transportation, and entertainment.

Here are a few concrete examples:

• Intelligent infrastructure: Deploying 5G infrastructure to enable smart city applications, such as traffic management, environmental monitoring, and public safety.
• Connected and autonomous vehicles (CAVs): Building and providing connectivity solutions for autonomous cars, drones, and other autonomous vehicles that rely on low-latency 5G networks.
• Factory automation: Implementing 5G for real-time monitoring and control industrial processes in smart factories.
• Supply chain optimization: Using 5G for visibility, tracking, and management.
• Enhanced retail customer experience: Leveraging 5G for augmented reality shopping experiences, cashless stores, and personalized marketing.
• Inventory management: Implementing real-time inventory tracking and management using 5G-connected sensors.
• Cloud gaming: Launching cloud gaming platforms that leverage 5G’s low latency and high bandwidth for streaming games.
• Augmented and Virtual Reality (AR/VR): Developing immersive AR and VR experiences enabled for a new entertainment.
• And so much more…

Overall, 5G represents a significant leap forward in wireless technology. 5G offers faster speeds, lower latency, and improved connectivity to support the increasing demands of our increasingly connected world.

Dish Wireless Standalone 5G infrastructure

“Dish Wireless” refers to the wireless division of Dish Network Corporation, a company primarily known for providing satellite television services in the United States. It is Dish Network’s venture into the wireless telecommunications industry.

The marketing slogan is “The DISH 5G Open RAN network is flexible, scalable and transparent. With DISH Wireless, the only limit is your imagination.” I love it because it is true! The infrastructure is cloud-native, and data streaming is a core piece of it.

Here are some key points about Dish Wireless:

1. Entry into wireless: Dish Network acquired a substantial amount of wireless spectrum through various acquisitions, including assets from bankrupt companies like Clearwire and Sprint, to enter the wireless market.
2. 5G network deployment: Dish Wireless has been working on building its own 5G wireless network, aiming to compete with other major wireless carriers in the United States. This network is being developed using 5G technology and provides broadband internet and mobile services.
3. Building infrastructure: To build its 5G network, Dish Wireless has invested in infrastructure development, including cell towers, small cell sites (coming soon), and other network components. They are working to deploy a nationwide network gradually. When writing this article in September 2023, Dish Wireless already covers >73% of the US population!
4. Wireless services: Dish Wireless offers a range of wireless services, including mobile phone plans, home internet services, and potentially other IoT (Internet of Things) services.

Dish Wireless 5G from a technical perspective

The Dish infrastructure innovates in many ways compared to most other telco networks operating today:

• OpenRAN: Built on open standard interfaces between all telco components. Allows Dish Wireless to mix and match vendor software and radios.
• Virtualization: The entire infrastructure is built on commodity x86 hardware. This provides flexibility and compatibility across many vendors, such as AWS or VMware.
• Cloud-native: Containerized software, CI/CD style deployment, open observability (key for data generation), chaos testing, etc. These are “firsts” for 5G at scale deployment and revolutionary and might be a turning point in the entire global telco industry.

Data streaming with Apache Kafka in the telco industry

The evolution of telco infrastructure, customer services, and new business models requires real-time end-to-end visibility, fancy mobile apps, and integration with pioneering technologies like 5G for low latency or augmented reality for innovation.

Many enterprises in the telecom sector leverage data streaming with Apache Kafka in various use cases across OSS, BSS, and OTT services.

Learn about trends, architectures, and customer stories from Dish Network, British Telecom, Globe Telecom, Swisscom, and more in the blog post “The State of Data Streaming for Telco in 2023“.

And here is a blog and video recording about Cloud-Native 5G, MEC and OSS/BSS/OTT Telco with Apache Kafka and Kubernetes.

This post specifically looks at the usage of data streaming at Dish Wireless.

How Dish leverages data streaming powered by Apache Kafka for 5G infrastructure

Brian Mengwasser (Vice President, Head of Marketplace and App Design at DISH Wireless) explored the general strategy, technical architecture, and opportunities for customers of Dish Wireless 5G products. You can find the on-demand recording below.

Dish Wireless cloud-native 5G network

Dish Wireless had the benefit (and challenge) of starting from scratch. The entire 5G infrastructure is cloud-native and provides elastic scalability.

Dish Wireless connectivity platform

The Dish Wireless connectivity platform enables real-time communication from any cloud to any device:

As you can see, the 5G infrastructure requires the combination of many critical software and hardware vendors. AWS, IBM, Dell, VMware, Oracle, and Confluent.

Dish Wireless data platform powered by 5G and Apache Kafka

The enterprise architecture of the Dish Wireless data platform enables the building of decoupled data products. The network stack comprises various components like RAN Core applications, traffic inspection, and observability platforms. The data-driven applications include data engines, service orchestrators, correlation tools, and integration with 3rd party interfaces.

The central nervous system of the enterprise architecture is the data streaming platform Confluent. It collects from various data sources, processes and correlates real-time and historical data, and shares information with downstream applications like mobile apps, data lakes, and data warehouses.

How to innovate with real-time use cases combining 5G networking and Apache Kafka

5G is an innovative technology that allows building new real-time applications not possible before. However, infrastructure alone does not solve the problem. Software is needed on top of the infrastructure. Therefore, 5G and data streaming are a match made in heaven. Both support real-time data processing at an elastic scale.

Let’s look at a few examples of combining 5G and data streaming with Apache Kafka.

Connected data streaming in retail

The retail industry has many use cases that can leverage 5G infrastructure and Apache Kafka for logistics, point of sale, and location-based customer services.

The state of data streaming for retail in 2023 explores the art of the possible. Use cases include omnichannel customer experiences, hybrid shopping models, and hyper-personalized recommendations. Data streaming allows integrating and correlating data in real-time at any scale. I explore customer stories from Walmart, Albertsons, Otto, AO.com, and more,

Connected data streaming in manufacturing

Manufacturing requires real-time information and data consistency across the entire supply chain, including shop floor, supplier integration, intralogistics, interlogistics, and customer-facing aftersales and service. Apache Kafka and 5G help to build hybrid edge deployments:

The state of data streaming for manufacturing in 2023 explores the evolution of industrial IoT, manufacturing 4.0, and digitalized B2B and customer relations. Most use cases require modern, open, and scalable information sharing. The foci are trending enterprise architectures in the manufacturing industry and data streaming customer stories from BMW, Mercedes, Michelin, or Siemens.

Connected data streaming in automotive

The automotive industry requires B2B integration with suppliers, OT/IT integration from the shop floor to the cloud, and innovative customer-facing apps like mobility services. Many of these challenges can only be solved by combining low-latency networking and real-time data processing at scale with 5G and Apache Kafka.

Real-World Deployments of Kafka in the Automotive Industry explores various real-world deployments across several fields. Use cases include connected vehicles, smart manufacturing, and innovative mobility services. Case studies cover car makers such as Audi, BMW, Porsche, and Tesla, plus a few mobility services such as Uber, Lyft, and Here Technologies.

On-demand video: 5G and data streaming @ Dish Wireless

Here is the recording of an interactive discussion and presentation I gave together with Brian Mengwasser (Vice President, Head of Marketplace and App Design at DISH Wireless):

If you just want to see the high level story, watch the following 3min summary on YouTube exploring how Dish Wireless leverages Apache Kafka and Confluent Cloud in their 5G infrastructure:

Telco panel: From Telco to TechCo

If you want to learn more from other telco experts, here is an on-demand panel where I discuss the evolution of data streaming in the telecom sector with peers from several organizations: “How Telcos are Shaping the Future of Communication with Data Streaming“:

• Proximus (Service Provider): Antonietta Mastroianni, Telco Woman of the Year
• Telefónica (Service Provider): Mariam Kaynia, VP Mass Market
• Deloitte (System Integrator): Enterprise Architect and Telco expert Philip Parker
• TM Forum (Non-Profit Organization): Andy Tiller, Executive Vice President, Products & Services

5G and data streaming are a match made in heaven

Real-time data beats slow data. That’s true for almost every use case. Specifically, most innovative use cases in the telco industry require real-time accurate information, regardless of the scale. No matter if you build internal applications to monitor the networks or external new services for logistics, transportation, retail, or any other industry.

5G infrastructure provides low latency with critical SLAs (thanks to network slicing capabilities). Data streaming with Apache Kafka provides features for collecting, storing, processing, and sharing events. The combination of 5G and data streaming enables innovation, where the only limit is your imagination.

How do you use or plan to use 5G networks together with data to innovate? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post How Apache Kafka helps Dish Wireless building cloud-native 5G Telco Infrastructure appeared first on Kai Waehner.

I look at trends in the telecommunications sector to explore how data streaming helps as a business enabler, including customer stories from Dish Network, British Telecom, Globe Telecom, Swisscom, and more. A complete slide deck and on-demand video recording are included.

General trends in the telco industry

The Telco industry is fundamental for growth and innovation across all industries.

The global spending on telecom services is expected to reach 1.595 trillion U.S. dollars by 2024 (Source: Statista, Jul 2022).

Cloud-native infrastructure and digitalization of business processes are critical enablers. 5G network capabilities and telco marketplaces enable entirely new business models.

5G enables new business models

Presentation of Amdocs / Mavenir:

A report from McKinsey & Company says, “74 percent of customers have a positive or neutral feeling about their operators offering different speeds to mobile users with different needs”. The potential for increasing the revenue per user (ARPU) with 5G use cases is enormous for telcos:

Telco marketplace

Many companies across industries are trying to build a marketplace these days. But especially the telecom sector might shine here because of its interface between infrastructure, B2B, partners, and end users for sales and marketing.

tmforum has a few good arguments for why communication service providers (CSP) should build a marketplace for B2C and B2B2X:

• Operating the marketplace keeps CSP in control of the relationship with customers
• A marketplace is a great sales channel for additional revenue
• Operating the marketplace helps CSPs monetize third-party (over-the-top) content
• The only other option is to be relegated to connectivity provider
• Enterprise customers have decided this is their preferred method of engagement
• CPSs can take a cut of all sales
• Participating in a marketplace prevents any one company from owning the customer

Data streaming in the telco industry

Adopting trends like network monitoring, personalized sales and cybersecurity is only possible if enterprises in the telco industry can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means):

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

“Use Cases for Apache Kafka in Telco” is a good article for starting with an industry-specific point of view on data streaming. “Apache Kafka for Telco-OTT and Media Applications” explores over-the-top B2B scenarios.

Data streaming with the Apache Kafka ecosystem and cloud services are used throughout the supply chain of the telco industry. Search my blog for various articles related to this topic: Search Kai’s blog.

From Telco to TechCo: Next-generation architecture

Deloitte describes the target architecture for telcos very well:

Data streaming provides these characteristics: Open, scalable, reliable, and real-time. This unique combination of capabilities made Apache Kafka so successful and widely adopted.

Kafka decouples applications and is the perfect technology for microservices across a telco’s enterprise architecture. Deloitte’s diagram shows this transition across the entire telecom sector:

This is a massive shift for telcos:

• From purpose-built hardware to generic hardware and elastic scale
• From monoliths to decoupled, independent services

Digitalization with modern concepts helps a lot in designing the future of telcos.

Open Data Architecture (ODA)

tmforum describes Open Digital Architecture (ODA) as follows:

“Open Digital Architecture is a standardized cloud-native enterprise architecture blueprint for all elements of the industry from Communication Service Providers (CSPs), through vendors to system integrators. It accelerates the delivery of next-gen connectivity and beyond – unlocking agility, removing barriers to partnering, and accelerating concept-to-cash.

ODA replaces traditional operations and business support systems (OSS/BSS) with a new approach to building software for the telecoms industry, opening a market for standardized, cloud-native software components, and enabling communication service providers and suppliers to invest in IT for new and differentiated services instead of maintenance and integration.”

If you look at the architecture trends and customer stories for data streaming in the next section, you realize that real-time data integration and processing at scale is required to provide most modern use cases in the telecommunications industry.

Architecture trends for data streaming

The telco industry applies various trends for enterprise architectures for cost, flexibility, security, and latency reasons. The three major topics I see these days at customers are:

• Hybrid architectures with synchronization between edge and cloud in real-time
• End-to-end network and infrastructure monitoring across multiple layers
• Proactive service management and context-specific customer interactions

Let’s look deeper into some enterprise architectures that leverage data streaming for telco use cases.

Hybrid 5G architecture with data streaming

Most telcos have a cloud-first strategy to set up modern infrastructure for network monitoring, sales and marketing, loyalty, innovative new OTT services, etc. However, edge computing gets more relevant for use cases like pre-processing for cost reduction, innovative location-based 5G services, and other real-time analytics scenarios:

Learn about architecture patterns for Apache Kafka that may require multi-cluster solutions and see real-world examples with their specific requirements and trade-offs. That blog explores scenarios such as disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments, and global Kafka.

Edge deployments for data streaming are their own challenges. In separate blog posts, I covered use cases for Kafka at the edge and provided an infrastructure checklist for edge data streaming.

End-to-end network and infrastructure monitoring

Data streaming enables unifying telemetry data from various sources such as Syslog, TCP, files, REST, and other proprietary application interfaces:

End-to-end visibility into the telco networks allows massive cost reductions. And as a bonus, a better customer experience. For instance, proactive service management tells customers about a network outage:

Context-specific sales and digital lifestyle services

Customers expect a great customer experience across devices (like a web browser or mobile app) and human interactions (e.g., in a telco store). Data streaming enables a context-specific omnichannel sales experience by correlating real-time and historical data at the right time in the proper context:

“Omnichannel Retail and Customer 360 in Real Time with Apache Kafka” goes into more detail. But one thing is clear: Most innovative use cases require both historical and real-time data. In summary, correlating historical and real-time information is possible with data streaming out-of-the-box because of the underlying append-only commit log and replayability of events. A cloud-native Tiered Storage Kafka infrastructure to separate compute from storage makes such an enterprise architecture more scalable and cost-efficient.

The article “Fraud Detection with Apache Kafka, KSQL and Apache Flink” explores stream processing for real-time analytics in more detail, shows an example with embedded machine learning, and covers several real-world case studies.

New customer stories for data streaming in the telco industry

So much innovation is happening in the telecom sector. Automation and digitalization change how telcos monitor networks, build customer relationships, and create completely new business models.

Most telecommunication service providers use a cloud-first approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure. And elastic scalability gets even more critical with all the growing networks and 5G workloads.

Here are a few customer stories from worldwide telecom companies:

• Dish Network: Cloud-native 5G Network with Kafka as the central communications hub between all the infrastructure interfaces and IT applications. The standalone 5G infrastructure in conjunction with data streaming enables new business models for customers across all industries, like retail, automotive, or energy sector.
• Verizon: MEC use cases for low-latency 5G stream processing, such as autonomous drone-in-a-box-based monitoring and inspection solutions or vehicle-to-Everything (V2X).
• Swisscom: Network monitoring and incident management with real-time data at scale to inform customers about outages, root cause analysis, and much more. The solution relies on Apache Kafka and Apache Druid for real-time analytics use cases.
• British Telecom (BT): Hybrid multi-cloud data streaming architecture for proactive service management. BT extracts more value from its data and prioritizes real-time information and better customer experiences.
• Globe Telecom: Industrialization of event streaming for various use cases. Two examples: Digital personalized rewards points based on customer purchases. Airtime loans are made easier to operationalize (vs. batch, where top-up cash is already spent again).

Resources to learn more

This blog post is just the starting point. Learn more about data streaming in the telco industry in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-demand video recording

The video recording explores the telecom industry’s trends and architectures for data streaming. The primary focus is the data streaming case studies. Check out our on-demand recording:

Slides

If you prefer learning from slides, check out the deck used for the above recording:

Case studies and lightboard videos for data streaming in telco

The state of data streaming for telco is fascinating. New use cases and case studies come up every month. This includes better data governance across the entire organization, real-time data collection and processing data from network infrastructure and mobile apps, data sharing and B2B partnerships with OTT players for new business models, and many more scenarios.

We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories. Stay tuned; I will update the links in the next few weeks and publish a separate blog post for each story and lightboard video.

And this is just the beginning. Every month, we will talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, telcos, gaming, and so on…

Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post The State of Data Streaming for Telco appeared first on Kai Waehner.

Video Recording – Cloud-Native Telco for 5G, MEC and OSS/BSS/OTT with Kafka and Kubernetes

Here is the video recording:

Slide Deck – Kafka in the Telecom Sector (OSS/BSS/OTT)

Here is the related slide deck for the video recording:

Use Cases and Architectures for Apache Kafka in the Telecom Sector

This section shares various other blog posts about event streaming, cloud-native architectures, and use cases in the telecom sector powered by Apache Kafka.

Topics include:

• Use cases and real-world deployments
• Innovative OSS, BSS, and OTT scenarios
• Edge, hybrid, and multi-cloud architectures
• Low-latency cloud-native MEC (multi-access edge computing)
• Cybersecurity with situational awareness and threat intelligence
• Comparison of different event streaming frameworks and cloud services

Real-Time Data Beats Slow Data in the Telco Industry

Think about the use cases in your project, business unit, and company: Real-time data beats slow data in almost all use cases in the telco industry. That’s why so many next-generation telco service providers and business applications leverage event streaming powered by Apache Kafka.

Do you already leverage Apache Kafka in the telecom sector? What use cases did you or do you plan to implement with Kafka and Kubernetes? How does your (future) edge or hybrid architecture look like? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Cloud-Native 5G, MEC and OSS/BSS/OTT Telco with Apache Kafka and Kubernetes appeared first on Kai Waehner.

Apache Kafka + MQTT Blog Series

The first blog post explores the relation between MQTT and Apache Kafka. Afterward, the other four blog posts discuss various use cases, architectures, and reference deployments.

• Part 1 – Overview: Relation between Kafka and MQTT, pros and cons, architectures
• Part 2 – Connected Vehicles: MQTT and Kafka in a private cloud on Kubernetes; use case: remote control and command of a car
• Part 3 – Manufacturing: MQTT and Kafka at the edge in a smart factory; use case: Bidirectional OT-IT integration with Sparkplug between PLCs, IoT Gateways, Data Historian, MES, ERP, Data Lake, etc.
• Part 4 – Mobility Services: MQTT and Kafka leveraging serverless cloud infrastructure; use case: Traffic jam prediction service using machine learning
• Part 5 – Smart City (THIS POST): MQTT at the edge connected to fully-managed Kafka in the public cloud; use case: Intelligent traffic routing by combining and correlating 3rd party services

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Use Case: Smart City and 5G

A smart city is an urban area that uses different types of electronic Internet of Things (IoT) sensors to collect data and then use insights gained from that data to manage assets, resources, and services efficiently.

A smart city provides many benefits for civilization and city management. Some of the goals are:

• Improved Pedestrian Safety
• Improved Vehicle Safety
• Proactively Engaged First Responders
• Reduced Traffic Congestion
• Connected / Autonomous Vehicles
• Improved Customer Experience
• Automated Business Processes

I covered the use cases in more detail in the post “Event Streaming with Kafka as Foundation for a Smart City“. For a specific 5G example, check out “Building a Smart Factory with Apache Kafka and 5G Campus Networks“.

Let’s now explore the relation of Kafka and MQTT for smart city use cases.

Architecture: MQTT and Kafka for a Smart City

The following architecture shows an infrastructure deployed at a stadium:

In this example, both MQTT and Kafka are deployed close to the stadium. For instance, AWS Wavelength is an innovative infrastructure option to build low latency 5G use cases. The connected “regular AWS cloud region” is still used for use cases that do not require low latency.

The combination of Kafka and MQTT enables connectivity and real-time data processing for various use cases:

• Parking information and smart navigation.
• Location-based shopping and restaurant experiences, including innovative scenarios such as monitoring of queues and geofencing.
• Integration of loyalty platforms to earn rewards and points.
• Live information about the game or concert
• Lottery drawing experiences while watching a sports game.

The possibilities are endless. Integration with 1st and 3rd party applications will create completely new opportunities to improve the customer experience, increase safety, and improve operational efficiency.

The stadium example is a particular scenario to explore the added value of processing data in motion. Let’s take a look at other real-world examples that leverage MQTT and Kafka.

Example: Cloud-based Traffic Control Systems @Berlex

The Swedish company Berlex designs and manufactures new ways to improve traffic safety.

Berlex provides cloud-based portable traffic signals. Their innovative R6 traffic signal is one of the first mobile traffic signals controlled by a cloud-based service. Berlex’s connected solution allows customers to monitor the new traffic signals on a smartphone, computer, or tablet anytime and from anywhere. MQTT enables real-time information delivery and constant monitoring.

The cloud-based service reduces the time that their customers need to spend in dangerous traffic work zones. The system enables customers to carry out numerous tasks such as checking the battery status of a traffic signal or performing an inspection remotely, with no need for risky and time-consuming on-site intervention.

Each portable R6 traffic signal is equipped with a radar that allows the signal to see traffic. Sensors within the signals publish detailed information on the current status of the signal as MQTT data. The Berlex Connect cloud service captures the continuous stream of MQTT data from each signal and shares the information with the appropriate subscribers.

To prevent interruption of the traffic signal operation, high availability is essential for the system. Berlex customers monitor the real-time information on individual portals with customized user roles that fit their specific use case.

Read the complete case study from HiveMQ for more details about this successful smart city project.

Example: The Life of Citizens as a Stream of Events @ NAV

NAV (Norwegian Work and Welfare Department) currently distributes more than one-third of the national budget to Norway or abroad citizens. NAV assists people through all phases of life within work, family, health, retirement, and social security. Events happening throughout a person’s life determines which services we provide to them, how we provide them and when we provide them.

In most countries, each person has to apply for these services resulting in many tasks handled manually by various caseworkers in the organization. Their access to insight and useful information is limited and often hard to find, causing frustration to both our caseworkers and our users. By streaming a person’s life events through our Kafka pipelines, NAV revolutionized the way users experience government services and the way the employees work:

NAV and the government as a whole have access to vast amounts of data about the citizens, reported by health institutions, employers, various government agencies, or the users themselves. Some data is distributed by large batches, while others are available on-demand through APIs. The data is ingested into streams using Kafka, Streams API, and Java microservices. NAV distributes and acts on events about birth, death, relationships, employment, income, and business processes to vastly improve the user experience, provide real-time insight and reduce the need to apply for services the government already knows are needed.

NAV chose Confluent Platform to implement to get valuable insight from life and business events. Security is a key concern. Compliance with GDPR is essential for the success of this project.

More details about NAV’s Kafka usage in their Kafka Summit presentation.

Kafka + MQTT = Smart City

In conclusion, Apache Kafka and MQTT are a perfect combination for smart city and 5G use cases. Follow the blog series to learn about use cases such as connected vehicles, manufacturing, mobility services, and smart city. Every blog post also includes real-world deployments from companies across industries. It is key to understand the different architectural options to make the right choice for your project.

What are your experiences and plans in IoT projects? What use case and architecture did you implement? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka and MQTT (Part 5 of 5) – Smart City and 5G appeared first on Kai Waehner.

Event Streaming with Kafka at the Disconnected Edge

Apache Kafka is the new black at the edge.

This is true not just for obvious verticals such as manufacturing, oil&gas, and the automotive industry. Other industries, including retail, healthcare, government, financial services, and energy, leverage Apache Kafka to take advantage of IoT devices, sensors, smart machines, robotics, and connected data.

This post focuses on the autonomous (and sometimes disconnected) edge. This means the edge sites required good, stable network communication, but not necessarily stable and low latency connectivity to the remote data center or cloud. The autonomous or disconnected edge needs to operate continuously even if the connection to the internet is broken. The below example utilizes smart factories, but the same use cases are deployed across many other scenarios, including restaurants, retail stores, and hospitals.

This post does NOT explore the connected edge with use cases such as V2X (vehicle-to-everything) and standards such as C-V2X (Cellular / 5G) by 5GAA. V2X and all the use cases around mobility services and smart cities will be explored in another post. This topic is very different, e.g., because there is no stable internet connection and you (have to) leverage standards such as MQTT in conjunction with Kafka. Obviously,  plenty of very relevant use cases exist here, too. Subscribe to my newsletter to stay updated with new blog posts!

Why is 5G a Game Changer for Industrial IoT, Automotive, and Smart City?

5G is the fifth generation technology standard for broadband cellular networks. Many people wonder why there is such a hype around 5G.

What actually is 5G?

I cannot tell you all the technical details. But on a high level from a use case perspective, it is important to understand that 5G is much more than just higher speed and lower latency:

• Public 5G telco infrastructure: That’s what Verizon, AT&T, T-Mobile, and all the other telco providers talk about in their TV spots. The end consumer gets higher download speeds and lower latency (at least in theory). This infrastructure integrates vehicles (e.g., cars) and devices (e.g., mobile phones) to the 5G network (V2N).
• Private 5G campus networks: That’s what many enterprises are most interested in. The enterprise can setup private 5G networks with guaranteed quality of service (QoS) using acquired 5G slices from the 5G spectrum. Enterprise work with telco providers, telco hardware vendors, and sometimes also with cloud providers (e.g., AWS Wavelength). This infrastructure is used similarly to the public 5G but deployed, e.g., in a factory or hospital. The trade-offs are guaranteed SLAs and increased security vs. higher cost.
• Direct connection between devices: That’s for interlinking the communication between two or more vehicles (V2V) or vehicles and infrastructure (V2I) via unicast or multicast. There is no need for a network hop to the cell tower due to using a 5G technique called 5G sidelink communications. This enables new use cases, especially in safety-critical environments (e.g., autonomous driving) where Bluetooth, Wi-Fi, and similar network communications do not work well for different reasons.

As I mentioned before, this post focuses on architectures for private 5G campus networks and their relation to the public 5G infrastructure. V2X, including all the connected mobility services, will be covered in other posts.

5G for Wide-Area, Local-Area, and Personal-Area Communication

In conclusion about the 5G hype: “Instead of providing a different radio interface for every use case, device vendors could rely solely on 5G as the link for wide-area, local-area, and personal-area communications“, as explained in a great 5G blog post from Benny Vejlgaard (Nokia).

Let’s now see how 5G infrastructures are related to event streaming with Apache Kafka.

Multi-Access Edge Computing (MEC)

Multi-access edge computing (MEC) is another important term in this context. MEC was formerly called mobile edge computing. It is an ETSI-defined network architecture concept that enables cloud computing capabilities and an IT service environment at the edge of the cellular network. Hence, data processing n general is closer at the edge of any network.

The basic idea behind MEC is that by running applications and performing related processing tasks closer to the cellular customer, network congestion is reduced and applications perform better. MEC technology is designed to be implemented at the cellular base stations or other edge nodes. It enables flexible and rapid deployment of new applications and services for customers. Combining elements of information technology and telecommunications networking, MEC also allows cellular operators to open their radio access network (RAN) to authorized third parties, such as application developers and content providers.

The use cases overlap with what you can read about 5G. So I focus on the term 5G in this blog post. However, the concept of MEC is equally relevant.

Event Streaming in a Hybrid 5G Architecture

Industry 4.0 is all about processing high volumes of data in real-time. That’s obviously a perfect fit for Apache Kafka. Please note that Apache Kafka is NOT used for “hard real-time” but only for soft real-time. If you need zero latency for embedded systems, PLCs, and robots, that’s assembler or MISRA C, not Java and Kafka. Kafka is a perfect fit for any use case where an end-to-end latency of 10+ms is good enough. This is almost all IT use cases, but not OT use cases.

The following shows a high-level hybrid 5G architecture. It combines cloud computing with edge processing in 5G campus networks installed in smart factories:

Some notes on the picture:

• Most enterprise applications, such as the Kafka-based real-time location system (RTLS), run in a data center or public cloud. They use public 5G networks or any other stable internet connection.
• Each smart factory has a dedicated 5G campus network. These 5G slices provide guaranteed QoS. Various deployment options exist for 5G networks. All have their pros and cons regarding cost, bandwidth, latency, cost, and SLAs. In this example, the combination of a Telco provider and AWS Wavelength is used to enable an edge infrastructure with stable 5G processing and compute power to deploy Apache Kafka and other applications close to the production line in the plant within AWS EC2 instances.
• The integration between edge sites and the central data center or cloud is implemented with Kafka-native real-time technologies such as MirrorMaker 2 or Confluent’s Cluster Linking. This enables decoupled infrastructures and high throughput, guaranteed ordering, real-time replication, and out-of-the-box error handling. These are key characteristics: Each smart factory runs mission-critical workloads disconnected from the cloud.

Let’s now dig a little bit deeper into a smart factory to understand how edge computing works in this example.

Apache Kafka in Smart Factory at the Edge with a 5G Campus Network

The following picture shows the event streaming infrastructure inside a smart factory:

Some notes on this architecture:

• All the mission-critical workloads on the production line at the edge can operate without a connection to the internet. This includes processing on the production lines and analytics such as predictive maintenance or real-time dashboards for the on-site plant manager. The infrastructure runs 24/7, even if the location is offline and not connected to the public internet. This is not just about the outage of a data center or cloud! Often, applications in Industrial IoT(IIoT) are disconnected intentionally to provide a more secure environment.
• Some applications run in the remote data center or cloud. They continuously consume relevant data from the smart factory in real-time. After a disconnection, they fall behind. As soon as they get connected again, they consume all missed data and go back to real-time updates.
• In this example, Mojix, a Kafka-native supply chain management service, is deployed in the cloud. Obviously, if these supply chain processes are critical for the production line, the architecture would either include a direct, stable connection to the cloud (e.g., AWS Direct Connect or Azure ExpressRoute) or also be deployed in the smart factory. Kafka allows a flexible, hybrid architecture where applications can live where it makes the most sense from a technical and business perspective.
• A supply chain is complex. It includes much more than just the production lines and MES/ERP/APS systems in the smart factory. Integration to enterprise IT systems in the data center AND integration with suppliers and partners is key for success. Event Streaming with Apache Kafka plays a huge role in many postmodern supply chain architectures.

5G is the Future for many Edge and Hybrid Kafka Use Cases in Industry 4.0

Apache Kafka plays a key role in processing massive volumes of data in real-time in a reliable, scalable, and flexible way. This is relevant across industries for Industry 4.0 use cases. Public and private 5G networks enable the next generation of Industrial IoT, edge computing, and real-time use cases across verticals.

At the beginning of 2021, we are still in the early stage of 5G infrastructures. But first enterprises already work with telco providers to build great use cases with 5G and event streaming.

What are your experiences and plans with private and public 5G infrastructures? Do you plan to use Apache Kafka at the edge, too? Which approach works best for you? What is your strategy? Check out the “Infrastructure Checklist for Apache Kafka at the Edge” if you plan to go that direction!

Let’s connect on LinkedIn and discuss it! Also, stay informed about new blog posts by subscribing to my newsletter.

The post Building a Smart Factory with Apache Kafka and 5G Campus Networks appeared first on Kai Waehner.

Apache Kafka in the Telecom Sector (OSS, BSS, Middleware, OTT)

A previous blog post covered the use cases of Event Streaming and Apache Kafka in the telecom industry: “Event Streaming and Apache Kafka in Telco Industry“. Follow that post or the related “whiteboard on Youtube about Apache Kafka and the telecom sector” to learn about various use cases like the following:

Telco-OTT (Over-The-Top) – 3rd Party Telecom and Media Services

Telco-OTT (Over-The-Top, OTT) is a particular field in the overall architecture in Telco and Media enterprises:

• OSS (Operations Support System), as part of the telecommunication infrastructure, ensures services are working.
• BSS (Business Support System), as part of the telecommunication infrastructure, ensures that they can be provided to an actual customer
• OTT Applications use the existing telecommunication infrastructure and provide better cost and/or features and/or convenience.

An example is the easiest way to understand this better.

OTT Applications for Messaging and Chat (SMS, RCS, WhatsApp, WeChat)

Messaging applications are a great example of Telco-OTT. Easy to understand for everybody, but pretty powerful tools.

Here are some examples of messaging apps:

• SMS (short message service) by telco providers: Text format message.
• WhatsApp by Facebook: Above + group chat, gif/stickers, photos, videos, audio, location, contact information, and ‘walkie-talkie’ services.
• WeChat by Tencent: Above + payment + various other partner integrations

The telecom sector had huge revenue coming from simple text messages via SMS. No competition existed for a long time. Telcos were able to charge ~20 Cent per message. A billion-dollar business emerged.

The internet and app economy changed this massively:

People-to-People (P2P) SMS shrunk significantly. Application-to-people (A2P) is lower today, too.

The telecom sector has the following options to act when a competing service becomes successful:

• Partnership: No control over the service and potentially damage their reputation and customer relationships.
• Development of own services: No skills for building the service and too late anyway.
• Blocking OTT services: Losing revenue for traffic and customers.

Not good options, right? The best answer is to innovate early; before competitors take over.

In this case, the answer from the telco providers was RCS (rich communication services)… RCS enabled additional features to simple text, such as sending images and videos. But it came “a little bit late” and was much more expensive compared to “almost free” apps like WhatsApp and WeChat. (yes, I know, if the product is free, then you are the product – but this is another discussion…)

Next-Generation Telecom Architecture

The above example shows the main problem in the Telecommunications Industry. The telecom sector needs to completely change their strategy and overall infrastructure to stay competitive and innovative. This change is the only way to “bridge the gap with the Over-the-Top (OTT) internet providers and survive in the digital age“:

Microservices Architecture for the Telecom Sector to Compete with 3rd Party OTT Applications

The telco infrastructure grew over the decades. It is inflexible, monolithic, and complex. The trend at most telco companies (I talked to in the last 12 months across the world) only sees one direction for the future: Build a flexible, open, scalable Telco architecture to process massive volumes of data in real-time.

The capability to decouple applications is where Apache Kafka shines with its distributed architecture providing a combination of messaging and storage to enable real domain-driven design (DDD):

A microservices architecture decouples applications and allows the integration between legacy and modern applications, infrastructure, and technologies. This is once again where Apache Kafka shines as scalable real-time data integration and middleware with any Telco infrastructure, using low-level interfaces such as TCP, Syslog or SNMP, business integrations with a CRM, EMS, NMS, IMS, or even mainframe offloading and replacement.

Hybrid Architectures as Key for Success in the Telco Industry

Hybrid cloud infrastructure is key to success in the telecommunications industry. The network infrastructure will always be on-premise and at the edge, while big data analytics, customer relations, and other business applications can run in a central data center or at a cloud provider:

Apache Kafka and its ecosystem provide various architecture patterns for distributed, hybrid, edge, and global deployments.

Telco architectures often combine Kafka with other Telco-specific applications and services from leading providers in the telecom sector, such as Amdocs, Ericsson, or Huawei.

Disney Hotstar – OTT in Real-Time for Millions of Cricket Fans in India

Let’s now take a look at a specific example for building an OTT service with requirements for real-time processing at a massive scale.

Disney+ Hotstar (known as Hotstar outside India), is an Indian over-the-top streaming service. In 2018, during the Indian Premier League, Hotstar introduced “Watch N Play”, a real-time cricket prediction game in which over 33 million unique users answered over 2 billion questions, won more than 100 million rewards, built with Kafka as the backbone.

In the game, the user guesses the outcome of the next ball. If he/she guesses right before the actual result, they score points to climb up the ladder and receive rewards along the way. Supporting potentially millions of users with differing stream times & device latencies, Hotstar used topics to separate logical streams & partitions provide up to supporting 1M requests/second and more.

This is an impressive OTT service built with Apache Kafka and its ecosystem, isn’t it? Check out more details in Hotstar’s Kafka Summit talk “Scaling for India’s Cricket Hungry Population“.

OTT at Netflix and Tencent with Kafka

In addition to the above Disney+ Hotstar example, I want to quickly refer to two more OTT and media examples:

• Kafka at Netflix provides its business teams critical insights that help plan, determine spending, and account for all Netflix content. Kafka acts as a bridge for all point-to-point and Netflix Studio wide communications. The Netflix Studio Productions and Finance Team embraces distributed governance as the way of architecting systems.
• Kafka at Tencent is responsible for integrating Tencent’s internet, social, and content platforms. Kafka handles 10 trillion+ messages per day. This includes many major products – from the well-known QQ, QZone, video, App Store, news, and browser, to relatively new members of the ecosystem such as live broadcast, anime, and movies.

Time to Change for Traditional Players in the Telecom and Media Sector

Deloitte has defined four scenarios for 2030. Every company in the telco industry needs to think about they want to go:

This necessary change affects any company working in the telco industry. It is time to change – no matter if you work on projects in OSS, BSS, OTT, IMS, Middleware, 3rd party services, or any other telecom domain.

Open Source MANO (OSM) for NFV and OSS Interoperability

A great example of this change in the telecom sector is the project “Open Source MANO (OSM)” for management and orchestration in OSS environments. This open-source project formed by ETSI (European Telecommunication Standards Institute) to “address challenges associated with orchestration, interoperability and performance optimization between different NFV (Network Functions Virtualization) and OSS systems”:

“The unified message bus of Open Source MANO is implemented with Apache Kafka. This bus allows asynchronous communication between OSM components and enables the introduction of new modules that can be easily pluggable.”

(please note that I don’t like the term ‘bus’ because it depicts Kafka as a messaging bus even though it is an event streaming platform including messaging, storage, processing, and integration capabilities)

The Evolution of the Telco Industry with Apache Kafka

The following slide deck and video recording cover the evolution of Kafka in the telco industry, including use cases, architectures, and technologies (OSS, BSS, OTT, IMS, NFV, Middleware, Mainframe, etc.):

Apache Kafka and Event Streaming for Innovation in the Telecom Sector

Current IT architectures in the telecom sector are not able to satisfy business needs because of their high complexity, lack of flexibility, and low level of automation.

Event Streaming with Apache Kafka and its ecosystem provides a scalable, reliable, and flexible infrastructure to process massive volumes in real-time. It enables real decoupling, plus powerful data integration and processing capabilities. Many telco enterprises build new infrastructures around Kafka.

This blog post focused on next-generation architecture to build Telco-OTT services. Disney+ Hotstar is an imposing example. But no matter in which part of the Telco industry you work, change and innovation are essential to staying competitive and innovative in the telecom sector.

What are your experiences with modernizing the infrastructure and applications in the telco industry? Did you or do you plan to use Apache Kafka and its ecosystem? What is your strategy? Let’s connect on LinkedIn and discuss it!

The post Apache Kafka for Telco-OTT (Telecom Sector) and Media Applications appeared first on Kai Waehner.