Apache Kafka in the Telecom Sector (OSS, BSS, Middleware, OTT)

A previous blog post covered the use cases of Event Streaming and Apache Kafka in the telecom industry: “Event Streaming and Apache Kafka in Telco Industry“. Follow that post or the related “whiteboard on Youtube about Apache Kafka and the telecom sector” to learn about various use cases like the following:

Telco-OTT (Over-The-Top) – 3rd Party Telecom and Media Services

Telco-OTT (Over-The-Top, OTT) is a particular field in the overall architecture in Telco and Media enterprises:

• OSS (Operations Support System), as part of the telecommunication infrastructure, ensures services are working.
• BSS (Business Support System), as part of the telecommunication infrastructure, ensures that they can be provided to an actual customer
• OTT Applications use the existing telecommunication infrastructure and provide better cost and/or features and/or convenience.

An example is the easiest way to understand this better.

OTT Applications for Messaging and Chat (SMS, RCS, WhatsApp, WeChat)

Messaging applications are a great example of Telco-OTT. Easy to understand for everybody, but pretty powerful tools.

Here are some examples of messaging apps:

• SMS (short message service) by telco providers: Text format message.
• WhatsApp by Facebook: Above + group chat, gif/stickers, photos, videos, audio, location, contact information, and ‘walkie-talkie’ services.
• WeChat by Tencent: Above + payment + various other partner integrations

The telecom sector had huge revenue coming from simple text messages via SMS. No competition existed for a long time. Telcos were able to charge ~20 Cent per message. A billion-dollar business emerged.

The internet and app economy changed this massively:

People-to-People (P2P) SMS shrunk significantly. Application-to-people (A2P) is lower today, too.

The telecom sector has the following options to act when a competing service becomes successful:

• Partnership: No control over the service and potentially damage their reputation and customer relationships.
• Development of own services: No skills for building the service and too late anyway.
• Blocking OTT services: Losing revenue for traffic and customers.

Not good options, right? The best answer is to innovate early; before competitors take over.

In this case, the answer from the telco providers was RCS (rich communication services)… RCS enabled additional features to simple text, such as sending images and videos. But it came “a little bit late” and was much more expensive compared to “almost free” apps like WhatsApp and WeChat. (yes, I know, if the product is free, then you are the product – but this is another discussion…)

Next-Generation Telecom Architecture

The above example shows the main problem in the Telecommunications Industry. The telecom sector needs to completely change their strategy and overall infrastructure to stay competitive and innovative. This change is the only way to “bridge the gap with the Over-the-Top (OTT) internet providers and survive in the digital age“:

Microservices Architecture for the Telecom Sector to Compete with 3rd Party OTT Applications

The telco infrastructure grew over the decades. It is inflexible, monolithic, and complex. The trend at most telco companies (I talked to in the last 12 months across the world) only sees one direction for the future: Build a flexible, open, scalable Telco architecture to process massive volumes of data in real-time.

The capability to decouple applications is where Apache Kafka shines with its distributed architecture providing a combination of messaging and storage to enable real domain-driven design (DDD):

A microservices architecture decouples applications and allows the integration between legacy and modern applications, infrastructure, and technologies. This is once again where Apache Kafka shines as scalable real-time data integration and middleware with any Telco infrastructure, using low-level interfaces such as TCP, Syslog or SNMP, business integrations with a CRM, EMS, NMS, IMS, or even mainframe offloading and replacement.

Hybrid Architectures as Key for Success in the Telco Industry

Hybrid cloud infrastructure is key to success in the telecommunications industry. The network infrastructure will always be on-premise and at the edge, while big data analytics, customer relations, and other business applications can run in a central data center or at a cloud provider:

Apache Kafka and its ecosystem provide various architecture patterns for distributed, hybrid, edge, and global deployments.

Telco architectures often combine Kafka with other Telco-specific applications and services from leading providers in the telecom sector, such as Amdocs, Ericsson, or Huawei.

Disney Hotstar – OTT in Real-Time for Millions of Cricket Fans in India

Let’s now take a look at a specific example for building an OTT service with requirements for real-time processing at a massive scale.

Disney+ Hotstar (known as Hotstar outside India), is an Indian over-the-top streaming service. In 2018, during the Indian Premier League, Hotstar introduced “Watch N Play”, a real-time cricket prediction game in which over 33 million unique users answered over 2 billion questions, won more than 100 million rewards, built with Kafka as the backbone.

In the game, the user guesses the outcome of the next ball. If he/she guesses right before the actual result, they score points to climb up the ladder and receive rewards along the way. Supporting potentially millions of users with differing stream times & device latencies, Hotstar used topics to separate logical streams & partitions provide up to supporting 1M requests/second and more.

This is an impressive OTT service built with Apache Kafka and its ecosystem, isn’t it? Check out more details in Hotstar’s Kafka Summit talk “Scaling for India’s Cricket Hungry Population“.

OTT at Netflix and Tencent with Kafka

In addition to the above Disney+ Hotstar example, I want to quickly refer to two more OTT and media examples:

• Kafka at Netflix provides its business teams critical insights that help plan, determine spending, and account for all Netflix content. Kafka acts as a bridge for all point-to-point and Netflix Studio wide communications. The Netflix Studio Productions and Finance Team embraces distributed governance as the way of architecting systems.
• Kafka at Tencent is responsible for integrating Tencent’s internet, social, and content platforms. Kafka handles 10 trillion+ messages per day. This includes many major products – from the well-known QQ, QZone, video, App Store, news, and browser, to relatively new members of the ecosystem such as live broadcast, anime, and movies.

Time to Change for Traditional Players in the Telecom and Media Sector

Deloitte has defined four scenarios for 2030. Every company in the telco industry needs to think about they want to go:

This necessary change affects any company working in the telco industry. It is time to change – no matter if you work on projects in OSS, BSS, OTT, IMS, Middleware, 3rd party services, or any other telecom domain.

Open Source MANO (OSM) for NFV and OSS Interoperability

A great example of this change in the telecom sector is the project “Open Source MANO (OSM)” for management and orchestration in OSS environments. This open-source project formed by ETSI (European Telecommunication Standards Institute) to “address challenges associated with orchestration, interoperability and performance optimization between different NFV (Network Functions Virtualization) and OSS systems”:

“The unified message bus of Open Source MANO is implemented with Apache Kafka. This bus allows asynchronous communication between OSM components and enables the introduction of new modules that can be easily pluggable.”

(please note that I don’t like the term ‘bus’ because it depicts Kafka as a messaging bus even though it is an event streaming platform including messaging, storage, processing, and integration capabilities)

The Evolution of the Telco Industry with Apache Kafka

The following slide deck and video recording cover the evolution of Kafka in the telco industry, including use cases, architectures, and technologies (OSS, BSS, OTT, IMS, NFV, Middleware, Mainframe, etc.):

Apache Kafka and Event Streaming for Innovation in the Telecom Sector

Current IT architectures in the telecom sector are not able to satisfy business needs because of their high complexity, lack of flexibility, and low level of automation.

Event Streaming with Apache Kafka and its ecosystem provides a scalable, reliable, and flexible infrastructure to process massive volumes in real-time. It enables real decoupling, plus powerful data integration and processing capabilities. Many telco enterprises build new infrastructures around Kafka.

This blog post focused on next-generation architecture to build Telco-OTT services. Disney+ Hotstar is an imposing example. But no matter in which part of the Telco industry you work, change and innovation are essential to staying competitive and innovative in the telecom sector.

What are your experiences with modernizing the infrastructure and applications in the telco industry? Did you or do you plan to use Apache Kafka and its ecosystem? What is your strategy? Let’s connect on LinkedIn and discuss it!

The post Apache Kafka for Telco-OTT (Telecom Sector) and Media Applications appeared first on Kai Waehner.

The Evolution of the Telecommunications Industry

The telecommunications industries within the sector of information and communication technology is made up of all telecommunications / telephone companies and internet service providers. It plays the crucial role in the evolution of mobile communications and the information society.

Traditional telephone calls continue to be the industry’s biggest revenue generator in Telecommunications Industry. But thanks to advances in network technology, Telecom today is less about voice and increasingly about text (messaging, email) and images (e.g. video streaming).

High-speed internet access for computer-based data applications such as broadband information services and interactive entertainment, is pervasive. Digital subscriber line (DSL) is the main broadband telecom technology. The fastest growth comes from (value-added) services delivered over mobile networks.

Telco Architecture – OSS, BSS and Cloud

The traditional telco landscape is separated into OSS (Operations Support Systems), BSS (Business Support Systems) and OSS-BSS-Integration.

OSS track network inventory, assets and provisioning of services. BSS deal with customer relationship management (CRM) and processes such as taking orders, processing bills, and collecting payments. Here is an example of such a Telecommunications Industry landscape:

(Overview about OSS / BSS Landscape)

Modern architectures (have to) include one more aspect: Cloud. Like in almost all other traditional industries, telcos move towards cloud. The consequence is a hybrid architecture. Telco have to stay hybrid forever because a core part of their business is infrastructure. A combination of on premises data centers, edge processing and multi cloud architectures is the new normal in Telecommunications Industry.

Telco Business – CDR (Call Detail Record), Customer 360, Media and more

The business in Telecommunications Industry did not change that much from an end user perspective yet. Many basic concepts are still the same. OSS and BSS work together. Call Detail Records (CDR) metadata from phone calls is still processed and analysed to monitor the infrastructure and sell services to customers. Smartphone contracts and media offerings are  still pretty similar like ten years ago (you just get higher data limits per month).

However, the telecom business for the traditional telecommunications industry is changing significantly these days. It has to change. Otherwise, competitors like Apple or Netflix take over most of the business.

I like how this picture from Huawei describes these changes of the digital transformation in the telco industry:

The trend goes towards an open ecosystem, partnering, context-specific personalized offerings, multichannel, improved customer service and new innovative products.

These trends create a lot of challenges for the technical implementation. Traditional technologies which were used in the last 20 years do not work to realize this digital transformation. Requirements include real time processing at scale, decoupled and flexible applications and infrastructure, and still a reliable system with zero down time and zero data loss.

The combination of these requirements makes clear why more and more Telco projects and infrastructure are built with event streaming in mind and leveraging the de facto standard Apache Kafka and its ecosystem.

Use Cases for Apache Kafka in Telco

Apache Kafka is used in various very different telco projects in OSS, BSS, OSS-BSS-Integration. Many different architectures are used, included edge, data center, hybrid and multi cloud. Let’s take a look at some examples for use cases and architectures.

Middleware / Central Event Streaming Platform

• Central Event Streaming Platform as core infrastructure for integration and new applications
• Integration on all levels: OSS, BSS-OSS-Integration, BSS
• OSS – Separate clusters are used in each country / OpCo for integration with syslog etc.
• Integration with legacy and modern systems (including integration to existing legacy middleware)
• Move Data from OSS Fixed, Mobile, Cable into various systems
• Spanning different business units (TV, Analytics, Video Platform, Sales and Services)
• Integration with external partners (today via APIs, future: Kafka-native)
• Merger after acquisitions

Monitoring (Hardware, Performance, Security)

• Real time monitoring of their network devices (routers, switches, other network devices)
• Outside infrastructure: Million miles of fiber and coax, and over 40 million in home devices (severe weather to power grid outages to construction-related disruptions)
• Proactive monitoring: Real time monitoring, problem analysis, metrics reporting and action response
• Nationwide distributed IT infrastructure for all districts -> Fully-managed Security Information and Event Management
• Smarter network planning and deployment, as well as more precise ROI-based investment decisions, taking into account the profitability of each radio site based on customers’ actual and predicted profitability
• SIEM / SOAR: Mission critical cyber defence (typically a combination with other products like Elastic, Splunk, etc.)

Data Distribution

• Event Hub across the whole telco organisation for automation and virtualization of transports to enable data-driven infrastructure decisions
• Abstracted customer layer for all customer facing apps (Mobile App, Web Site, other CRM Systems)
• Next generation media distribution network. Provides alarm correlation to allow identifying network faults and reroute traffic in real time.
• Share information about network performance back to external customers
• API Gateway for network providers
• Accelerate the simplification and automation of standard processes, in both operational and support areas. These include IT and network operations, customer management back office functions and all other administrative activities
• Message archiving service for customers (traders specifically) that want fully searchable archive of all messages (delivered through all channels).

Data Processing

• CDR (Call Detail Record) processing: Technical ETL and business applications (range of scenarios like including upsell, churn analysis)
• Data consolidation issues with many data sources, late arriving events, data out of sync and high SLA guarantees
• Text messaging service to let you send and receive text and multimedia messages seamlessly on your smartphone, tablet, computer or the web using your mobile number; including rate limiting of SMS, MMS and RCS (Rich Communication Services) requests specifically for seasonal spikes; including redirect of RCS via SMS when delivery ability for the RCS message is not present
• Personalized messages and recommendations are built and then delivered to customer inboxes

Business Applications

• Microservices: Infrastructure for agile and decoupled app development
• BSS: Data engine, customer engagement engine, access broker, dynamic business engine, information backbone, back office systems, customer 360, master data management, …
• Customer 360: Track every interaction that the customer has with any channel
• Supply Chain Management (SCM): Real Time Inventory. Support real time information down to the store level
• Billing, Accounts, Lines of Service Data. Relieve Ops Burden
• Make the mobile app and digital marketing channels over time become the main customer acquisition and management platform
• B2B products like Inventory / Asset Tracking, Fleet Management
• Connected Car solutions to retailers for aftersales, including vehicle analytics for drivers, Infotainment, vehicle diagnostics, personal assistance, SOS, remote access, convenience features
• Connected Car solution platform for auto manufacturers to automate vehicle analytics for insurance companies and fleet management, tracking sales; vehicle performance; consumer demographics, etc.

Hybrid Infrastructure and Cloud Migration

• Consume data from on premises and monitor in cloud
• Aggregation cluster: Hub-and-spoke architecture with several regional data centers capturing data to local Kafka clusters and then sending metrics info to the main cluster
• Edge processing (on different levels: OpCo, data center, retail store, …)

Media

• Pass all data through a single source of truth (real time, highly scalable) for decoupling and quick innovation of building new services
• IP-to-MAC: Handle all incoming MAC address registration requests from set-top boxes (to check account entitlements to make sure they are a valid customer)
• Managing content metadata / epg
• Processing pay per view bookings
• Fraud use cases (MAC address cloning, sharing of streaming account details, SMS Spam detection, etc); leveraging machine learning in real time at scale for predictions
• Digital natives (like Netflix) built completely new businesses that were born digital. Everything in a digitally-native business is an event
• Context-specific advertising in real time based on user data (including product upsell, coupons, and other revenue creation with ad partners)

10 Reasons for Event Streaming (i.e. Apache Kafka) in Telco Industry

10 Reasons why telco projects leverage Event Streaming and Apache Kafka:

1. Real Time
2. Scalable
3. Cost Reduction
4. 24/7 – Zero downtime, zero data loss
5. Decoupling – Storage, Domain Driven Design (DDD), Reprocessing of events
6. Data processing and stateful client applications
7. Integration
8. Hybrid Architecture – On Premises, multi cloud, edge computing
9. Fully managed cloud
10. No vendor locking

These characteristics are not just for telco business but relevant in general, of course. However, in the video below, I cover how they map to the telco industry and OSS / BSS use cases.

Obviously, Apache Kafka alone has a lot of strong characteristics. But to realize the digital transformation successfully, you also need expertise (services and support) and tools (development, operations and monitoring) to set up, operate and scale such an infrastructure 24/7 mission-critical.

Slide Deck

The following slide deck covers the evolution of Apache Kafka in the telecom sector, including use cases, architectures and technologies (OSS, BSS, OTT, IMS, NFV, Middleware, Mainframe, etc.):

Live Whiteboard

The following whiteboard explains why and how different companies in the telco industry leverage Event Streaming and Apache Kafka in OSS, BSS, OSS-BSS-Integration and Cloud / Hybrid Architectures. I cover in detail how the above 10 characteristics can used in your next telco project:

Video Recording – Telco Use Cases and Architectures

Here is a webinar where I talk about use cases, architectures and technologies around Apache Kafka in the telecom sector:

What are your experiences with modernizing the infrastructure and applications in the telco industry? Did you or do you plan to use Apache Kafka and its ecosystem? What is your strategy? Let’s connect on LinkedIn and discuss it!

The post Event Streaming and Apache Kafka in Telco Industry appeared first on Kai Waehner.