Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch.

Data Streaming with Kafka and Flink in the Aviation and Airline Industry

Data streaming with Apache Kafka and Flink is revolutionizing aviation by enabling real-time data processing and integration across complex airline and airport ecosystems. Airlines rely on diverse systems for flight tracking, crew scheduling, baggage handling, and passenger services, all of which generate vast volumes of data.

Kafka’s event-driven architecture ensures seamless communication between these systems, allowing real-time updates and consistent data flows. Flink then processes this data to provide useful information.

IT Modernization, Cloud-native Middleware and Analytics with Apache Kafka at Lufthansa

For instance, Lufthansa leverages Apache Kafka as a cloud-native middleware to modernize its data integration and enable real-time analytics. Through its KUSCO platform, Kafka replaces legacy tools like TIBCO EMS, offering scalable, cost-efficient, and seamless data sharing across systems. Kafka also powers Lufthansa’s advanced analytics use cases, including:

• Anomaly Detection: Real-time alerts using ksqlDB to enhance safety and efficiency.
• Fleet Management: Machine learning models embedded in Kafka pipelines for real-time operational predictions.

This shift enables Lufthansa to decouple systems, accelerate innovation, and reduce costs, positioning the airline to meet the demands of a rapidly evolving industry with greater efficiency and agility.

Business Value of Data Streaming at Amsterdam Airport Schiphol Group)

The business value of data streaming in aviation is immense. Airlines gain operational efficiency by reducing delays and optimizing resource allocation. Real-time insights enhance the passenger experience with timely updates, better baggage handling, and personalized interactions.

Airport modernization and digitalization, with consistent real-time information, is another excellent trend. This includes data sharing with partners, such as GDS systems and airlines. Schiphol Group (Amsterdam Airport) presented various use cases for data streaming with Apache Kafka and Flink.

Scalable platforms like Kafka allow airlines to integrate new technologies, future-proofing their operations in an increasingly competitive industry. By leveraging data streaming, aviation companies are not just keeping pace—they’re redefining what’s possible in airline and airport management.

Virgin Australia: Business Overview and IT Strategy

Founded in 2000, Virgin Australia is a leading airline connecting Australia to key global destinations through domestic and international flights. Known for exceptional service and innovation, the airline serves a diverse range of passengers, from leisure travelers to corporate clients.

Virgin Australia’s IT strategy drives its success, focusing on digital transformation to modernize legacy systems and integrate real-time data solutions. The airline uses the latest technology, such as Apache Kafka, and focuses on efficiency to offer good value and new ideas in the airline industry.

This enables the airline to optimize operations, enhance on-time performance, and quickly adapt to disruptions. A customer-first approach is central, leveraging data insights to personalize every stage of the passenger journey and build lasting loyalty.

Virgin Australia partnered with Confluent and the IT consulting firm 4impact to implement Apache Kafka for event streaming, ensuring their systems could meet the airline’s evolving demands. The following is a summary of 4impact’s published success stories:

• Virgin Australia Flight State Engine (FSE) Replacement
• Virgin Australia Business Flyers (SME) Loyalty Program

Success Story 1: Real-Time Flight Schedule Updates with the Flight State Engine (FSE)

Virgin Australia’s Flight State Engine (FSE) creates a central, authoritative view of flight status and streams real-time updates to multiple internal and external systems. Initially built on Oracle SOA, the legacy FSE faced significant limitations:

• High costs and slow implementation of new features.
• Limited monitoring capabilities.
• Lack of scalability for additional event-streaming use cases.

The Solution

4impact replatformed the FSE with a Kafka-based architecture, introducing:

• Modern Event Streaming: Kafka replaced Oracle SOA, enabling real-time, high-throughput updates.
• Phased Rollout: To minimize disruption, the new FSE ran parallel to the legacy system during implementation.
• Future-Proofing: Patterns, templates, and blueprints were developed for future event-streaming applications.

Key Outcomes

• The new FSE went live in late 2022, delivering zero outages and exceeding performance expectations.
• Speed and cost efficiency for adding new features improved significantly.
• The platform became the foundation for other business units, enabling faster delivery of new services and innovations.

By replacing the legacy FSE with Kafka, Virgin Australia ensured real-time reliability and created a scalable event-streaming platform to support future projects.

Success Story 2: Transforming the Virgin Business Rewards Program

Virgin Business Rewards is a loyalty program designed to engage small and medium-sized enterprises (SMEs). Previously, the program relied on manual workflows and siloed systems, leading to:

• Inefficient processes prone to errors.
• Delayed updates on reward earnings and redemptions.
• High costs due to the lack of automated communication between systems like Salesforce, Amadeus, and iFly.

The Solution

To address these challenges, 4impact implemented Kafka to automate the program’s workflows:

• Event-Driven Architecture: Kafka topics handled asynchronous messaging between systems, avoiding point-to-point integrations.
• Custom Microservices: Developed to transform messages and interact with APIs on target systems.
• Monitoring and Logging: A centralized mechanism captured business events and system logs, ensuring observability.

Key Outcomes

The new reward and loyalty system went live in Q1 2023, processing thousands of messages daily with a minimal load on endpoint systems.

• Reward data was synchronized across all systems, eliminating manual intervention.
• Other business units began exploring Kafka’s potential to leverage data for faster, more cost-effective service enhancements.

With Apache Kafka, Virgin Australia transformed its loyalty program, ensuring real-time updates and creating a scalable platform for future business needs.

IT Modernization with Data Streaming using Apache Kafka: A Blueprint for Innovation in the Airline Industry

Virgin Australia’s success stories illustrate how data streaming with Apache Kafka, implemented with the help of Confluent and 4impact, can address critical challenges in the aviation industry. By replacing legacy systems with modern event-streaming architectures, the airline achieved:

• Real-Time Reliability: Ensuring up-to-date flight information and seamless customer interactions.
• Scalability: Creating platforms that support new features and services without high costs or delays.
• Customer-Centric Solutions: Enhancing loyalty programs and operational efficiency.

The blog post “Customer Loyalty and Rewards Platform with Apache Kafka” explores how enterprises across various industries use Apache Kafka to enhance customer retention and drive revenue growth through real-time data streaming. It presents case studies from companies like Albertsons, Globe Telecom, Virgin Australia, Disney+ Hotstar, and Porsche to show the value of data streaming in improving customer loyalty programs.

Stay ahead of the curve! Subscribe to my newsletter for insights into data streaming and connect with me on LinkedIn to continue the conversation.

The post Virgin Australia’s Journey with Apache Kafka: Driving Innovation in the Airline Industry appeared first on Kai Waehner.

Data streaming in the aviation industry

The future business of airlines and airports will be digitally integrated into the ecosystem of partners and suppliers. Companies will provide more personalized customer experiences and be enabled by a new suite of the latest technologies, including automation, robotics, and biometrics.

The entire aviation industry leverages data streaming powered by Apache Kafka already. This includes airlines, airports, global distribution systems (GDS), aircraft manufacturers, travel agencies, etc. Why? Because real-time data beats slow data across almost all use cases.

Learn more in my blog about “Apache Kafka in the Airline, Aviation and Travel Industry” covering companies like Singapore Airlines, Air France, and Amadeus.

This article focuses on data streaming in critical Lufthansa projects. Lufthansa is a major German airline and one of the largest in Europe. It is known for its extensive network of domestic and international flights. Lufthansa offers services ranging from passenger transportation to cargo logistics and is a member of the Star Alliance, one of the world’s largest airline alliances.

Apache Kafka as next-generation middleware replacing ETL, ESB, and iPaaS

Typically, an enterprise service bus (ESB) or other integration solutions like extract-transform-load (ETL) tools have been used trying to decouple systems. However, the sheer number of connectors, as well as the requirement that applications publish and subscribe to the data at the same time, mean that systems are always intertwined. As a result, development projects depend on other systems, and nothing can be truly decoupled.

Many enterprises leverage the ecosystem of Apache Kafka for successful integration of different legacy and modern applications. Data streaming differs but also complements existing integration solutions like ESB or ETL tools. Apache Kafka is unique because it combines the following characteristics into a single middleware platform:

• Real-time messaging at any scale
• Event store for true decoupling, backpressure handling, and replayability of historical events
• Data integration eliminating the need for additional integration tools
• Stream processing for stateless and stateful data correlation of real-time and historical data

“Apache Kafka vs. Enterprise Service Bus (ESB) – Friends, Enemies or Frenemies?” explores how data streaming with Kafka complements legacy middleware. If your workloads run mostly in the public cloud, you need to understand the difference between Integration Platform as a Service (iPaaS) and data streaming powered by fully-managed Kafka infrastructure.

Lufthansa uses Apache Kafka as cloud-native middleware for mission-critical integrations

Lufthansa leverages data streaming with Confluent as cloud-native middleware for its strategic integration project KUSCO (Kafka Unified Streaming Cloud Operations).

The team discussed the benefits of using Apache Kafka instead of traditional messaging queues (TIBCO EMS, IBM MQ) for data processing. My two favorite statements:

• “Scaling Kafka is really inexpensive”
• “Kafka adopted and integrated within 3 months”

Lufthansa’s Kafka architecture does not have any surprises. A key lesson learned from many companies: The real added value is created when you leverage Kafka not just for messaging, but its entire ecosystem, including different clients/proxies, connectors, stream processing, and data governance.

The result at Lufthansa: A better, cheaper, and faster infrastructure for real-time data processing at scale.

Watch the full talk from Marcos Carballeira Rodríguez from Lufthansa Group recorded at the Confluent Streaming Days 2020 to see all the architectures and quotes from Lufthansa. More and more projects are onboarded on the KUSCO platform. Here are a few statistics on the adoption from 2022 to 2023 of the KUSCO project that System Architect Krzysztof Torunski of Lufthansa Group presented:

I see this typical pattern in customers across industries: The first use case is the hardest to get live. Afterward, new business units tap into the data feeds and build their projects. It has never been easier to access data feeds in real-time and with good data quality at any scale. Just build a downstream application (with your favorite programming language, tool, or SaaS) and start innovating.

Apache Kafka for analytics and AI/machine learning

Apache Kafka serves thousands of enterprises as the mission-critical and scalable real-time data fabric for machine learning infrastructures. The evolution of Generative AI (GenAI) with large language models (LLM) like ChatGPT changed how people think about intelligent software and automation. In various blog posts, I explored the relationship between data streaming with the Kafka ecosystem and AI/machine learning.

My latest article shows the enormous opportunities and some early adopters combining Kafka and GenAI beyond the buzz.

Lufthansa uses Apache Kafka with AI/machine learning for real-time predictions

Lufthansa leverages the KUSCO platform to build new analytics use cases with real-time data for critical workloads. In the webinar, we learned about the following two projects from Lufthansa Groups’s Domain Architect Sebastian Weber: anomaly detection for alerts and fleet management for aircraft operations.

Anomaly detection with Apache Kafka and ksqlDB

Data is fed into the streaming platform from various data sources. Lufthansa consolidates and aggregates the data with stream processing before the analytics applications do real-time alerting.

Machine learning and Apache Kafka for real-time fleet management

Lufthansa leverages the streaming platform as data fabric for data ingestion, data processing, and model scoring.

Embedding analytic models into a Kafka application is a standard best practice. While the data lake or lakehouse (that receives data via Kafka) trains the model in batch, many use cases require real-time model scoring and predictions at scale with critical SLAs and low latency. That’s exactly the sweet spot of the Kafka ecosystem.

You can either directly embed a model into the Kafka app or leverage a model server that supporting streaming interfaces. I blogged about the trade-offs and use cases: “Streaming Machine Learning with Kafka-native Model Deployment“.

Interactive conversation with Lufthansa

Here is an on-demand video of my conversation with Lufthansa. We talk about use cases for data streaming in the aviation industry and how Lufthansa leverages Apache Kafka as cloud-native middleware and as the data fabric for analytics and machine learning:

Data streaming as cloud-native middleware and for mission-critical analytics

Lufthansa showed us how you can innovate in the airline industry with a fast time-to-market while still integrating with traditional technologies. The two projects show very different challenges and use cases solved with data streaming powered by the Apache Kafka ecosystem.

The aviation industry is changing rapidly. A good customer experience, valuable loyalty platforms, and competitive pricing (or better hard and soft products) require digitalization of the end-to-end supply chain. This includes topics like Industrial IoT (e.g., predictive maintenance), B2B communication with partners (like GDS, airports, and retailers), and customer 360 (including great mobile apps and omnichannel experiences).

How do you leverage data streaming with Apache Kafka in your projects and enterprise architecture? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post How Lufthansa uses Apache Kafka for Middleware and Analytics appeared first on Kai Waehner.

Airlines and Aviation are Changing – Beyond Covid-19!

Aviation and travel are notoriously vulnerable to social, economic, and political events. These months have been particularly testing one due to the global pandemic with Covid-19. But the upcoming change is coming not just due to the Coronavirus but because of the ever-changing expectations of consumers.

Right now is the time to lay the ground for the future of the aviation and travel industry.

Consumer behaviors and expectations are changing. Whole industries are being disrupted, and the aviation industry is not immune to these sweeping forces of change.

The future business of airlines and airports will be digitally integrated into the ecosystem of partners and suppliers. Companies will provide more personalized customer experiences and be enabled by a new suite of the latest technologies, including automation, robotics, and biometrics.

For instance, new customer notification mobile apps provide customers with relevant and timely updates throughout their journeys. Other major improvements support the front line service teams at various touchpoints throughout the airports and end to end travel journey.

Apache Kafka in the Airline Industry

Apache Kafka is the de facto standard for event streaming use cases across industries. Many use cases can be applied to the aviation industry, too. Concepts like payment, customer experience, and manufacturing differ in detail. But in the end, it is about integrating systems and processing data in real-time at scale.

For instance, omnichannel retail with Apache Kafka applies to airline, airports, global distribution systems (GDS), and other aviation industry sectors.

However, it is always easier to learn from other companies in the same industry. Therefore, the following explores a few public Apache Kafka success stories from the aviation industry.

Lufthansa – Kafka Unified Streaming Cloud Operations

Lufthansa talks about the benefits of using Apache Kafka instead of traditional messaging queues (TIBCO EMS, IBM MQ) for data processing.

The journey started with the question if Lufthansa can do data processing better, cheaper, and faster.

Lufthansa’s Kafka architecture does not have any surprises. A key lesson learned from many companies: The real added value is created when you leverage Kafka not just for messaging, but its entire ecosystem, including different clients/proxies, connectors, stream processing, and data governance.

The result at Lufthansa: A better, cheaper, and faster infrastructure for real-time data processing at scale.

My two favorite statements (once again: not really a surprise, as I see the same at many other customers):

• “Scaling Kafka is really inexpensive”
• “Kafka adopted and integrated within 3 months”

Watch the full talk from Marcos Carballeira Rodríguez from Lufthansa recorded at the Confluent Streaming Days 2020 to see all the architectures and quotes from Lufthansa.

And check out this exciting video recording of Lufthansa discussing their Kafka use cases for middleware modernization and machine learning:

Singapore Airlines – Predictive Maintenance with Kafka Connect, Kafka Streams, and ksqlDB

Singapore Airlines is an early adopter of KSQL to continuously process sensor data and apply analytic models to the events. They already talked about their Kafka ecosystem usage (including Kafka Connect, Kafka Streams, and KSQL) back in 2018. The use case is predictive maintenance with a scalable real-time infrastructure, as you can see in my summary slide:

Check out the complete slide deck from Singapore Airlines for more details.

Air France Hop – Scalable Real-Time Microservices

I really like the Kafka Summit talk title:  “Hop! Airlines Jets to Real-Time“. Air France Hop leverage Change Data Capture (CDC) with HVR and Kafka for real-time data processing and integration with legacy monoliths. A pretty common pattern to integrate the old and the new software and IT world:

The complete slide deck and on-demand video recording about this case study are available on the Kafka Summit page.

Amadeus – Real-Time and Batch Log Processing

As I said initially, Kafka is not just relevant for each airline, airport, and aircraft manufacturers. The global distribution system (GDS) from Amadeus is one of the world’s biggest (competing mainly with Sabre). Passenger name record (PNR) is a record in the computer reservation system (CRS) and a crucial part of any GDS vendor.  While many end-users don’t even know about Amadeus, the aviation industry could not survive without them. Their workloads are mission-critical and need to run 24/7 in real-time, plus connect to their partners’ systems (like an airline) in a very stable and mature manner!

Amadeus is relying on Apache Kafka for both real-time and batch data processing, as they explain on the official Apache Kafka website:

Streaming Data Exchange for the Travel Industry

After looking at some examples, let’s now cover one more key topic: Data integration and correlation between partners in the aviation industry. Airline, airports, GDS, travel companies, and many other companies need to integrate very well. Obviously, this is already implemented. Otherwise, there is no way to operate flights with passengers and cargo. At least in theory. Honestly, one of the most significant pain points of the travel industry for customers is bad integration across companies. Some examples:

• Late or (even worse) no notification about a delay or cancellation
• Issues with the display of available seats or upgrade
• Broken booking process on the website because of different flight numbers, connecting flights,
• Booking class issues for upgrades or rebookings
• Display of technical error messages instead of business information (for instance, I can’t count how often I had seen an “IBM WebSphere” error message when I tried to book a flight on the website of my most commonly used airline)
• The list goes on and on and on… No matter which airline you pick. That’s at my experience as a frequent traveler across all continents and timezones.

There are reasons for these issues. The aviation network is very complex. For instance, Lufthansa group sells tickets for all their own brands (like Swiss or Austrian Airlines), plus tickets from Star Alliance partners (such as United or Singapore Airlines). Hence, airline, airports, GDS, and many partner systems have to work together. 24/7. In real-time. For this reason, more and more companies in the aviation industry rely on Kafka internally.

But that’s only half of the story… How do you integrate with partners?

Event Streaming vs. REST / HTTP APIs

I explored the discussion around event streaming with Kafka vs. RESTful web services with HTTP in much more detail in another article: “Comparison: Apache Kafka vs. API Management / API Gateway tools like Mulesoft or Kong“. In short: Kafka and REST APIs have their trade-offs. Both are complementary and used together in many architectures. API Management is a great add-on for many applications and microservices, no matter if they are built with HTTP or Kafka under the hood.

But one point is clear: If you need a scalable real-time integration with a partner system, then HTTP is not the right choice. You can either pick gRPC as a request-response alternative or use Kafka natively for the integration with partners, as you use it internally already anyway:

Kafka-native replication between partners works very well. No matter what Kafka vendor and version you and your partner are running. Obviously, the biggest challenge is the security (not from a technical but an organizational perspective). Kafka requires TCP. That’s much harder to get approval for opening it to a partner than HTTP ports.

But from a technical point of view, streaming replication often makes much more sense. I have seen the first customers implementing integration via tools like Confluent Replicator. I am sure that we will see this pattern much more in the future and with better out-of-the-box tool support from vendors.

Data Integration and Correlation at an Airport with Airline Data using KSQL

So, let’s assume that you have the data streams connected at an airport. No matter if just internal data or also partner data. Data correlation adds the business value. Sönke Liebau from OpenCore presented a great airport demo with Kafka and KSQL at a Kafka Summit.

Let’s take a look at some events at an airport:

These events exist in various structures and with different technologies and formats. Some data streams arrive in real-time. However, some other data sets come from a monolithic mainframe in batch via a file integration. Kafka Connect is a Kafka-native middleware to implement this integration.

Afterward, all this data needs to be correlated with historical data from a loyalty system or relational database. This is where stream processing comes into play: This concept enables the continuous data correlation in real-time at scale. Kafka-native technologies like Kafka Streams or ksqlDB exist to build streaming ETL pipelines or business applications.

The following example correlates the gate information from the airport with the airline flight information to send a delay notification to the customer who is waiting for the connection flight:

Tons of use cases exist to leverage event streams from different systems (and partners) in real-time. Some examples from an airport perspective:

• Location-based services while the customer is walking through the airport and waiting for the flight. Example: Coupons for a restaurant (with many empty seats or food reserves to thrash if not sold during the day)
• Airline services such as free or points-based discounted lounge entrance (because the lounge tracking systems knows that it is almost empty right now anyway)
• Partner services like notifying the airport hotel that the guest can stay longer in the room because of a long delay of the upcoming flight

The list of opportunities is almost endless. However, most use cases are only possible if all systems are integrated and data is continuously correlated in real-time. If you need some more inspiration, check out the two blogs “Kafka at the Edge in a Smart Retail Store” and “Kafka in a Train for Improved Customer Experience“. All these use cases are a perfect fit for airline, airports, and their partner ecosystem.

Slides – Apache Kafka in the Aviation, Airline and Travel Industry

The following slide deck goes into more detail:

Kafka for Improved Operations and Customer Experience in the Aviation Industry

This post explored various use cases for event streaming with Apache Kafka in the aviation industry. Airline, airports, aerospace, flight safety, manufacturing, GDS, retail, and many more partners rely on Apache Kafka.

No question: Kafka is getting mainstream these months in the aviation industry. Serverless and consumption-based offerings such as Confluent Cloud boost the adoption even more. A streaming data exchange between partners is the next step I see on the horizon. I am looking forward to Kafka-native interfaces from Open APIs of enterprises, better support for streaming interfaces in API Management tools, and COTS solution from software vendors.

What are your experiences and plans for event streaming in the aviation industry? Did you already build applications with Apache Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka in the Airline, Aviation and Travel Industry appeared first on Kai Waehner.