The Background: My Role and Travel Lifestyle

My work as Field CTO at a global software company involves meeting customers, partners, analysts and the broader tech community worldwide. I give presentations at conferences, validate business value and use cases of our products in workshops, and enable colleagues globally. Explore my in-depth article on the job role of a Field CTO to gain insights into this fascinating and challenging position.

I’ve embraced a lifestyle where I work efficiently from wherever I am—whether it’s an airport lounge, a plane, a hotel room, or a conference venue. Over the years, I’ve mastered the art of productive travel.

My home airport is Nürnberg (NUE), a small German airport with frequent daily connections through Frankfurt (FRA), Amsterdam (AMS), (Paris (CDG) and Istanbul (IST). Unfortunately, the excellent, short flight to Munich (MUC) was killed after the pandemic for political reasons. Therefore, sometimes, I begin my travel to Munich (MUC) by car. I never take the annoying „Lufthansa Bus Transfer“ from NUE to MUC for various reasons.

My travel is a mix of international long-haul (10-12 trips annually, primarily to the U.S.,  but also to the Middle East and Asia) and short-haul flights across Europe. The total is around 100 flight segments per year.

While my company typically covers economy (short haul) or premium economy (long haul), I’ve personally upgraded most of my long-haul flights with cash to business class for comfort. This is essential for such a high-travel lifestyle. Mile upgrades, while an option, are often too risky and annoying—they don’t always come through, and at Lufthansa Group (unlike many U.S. airlines), changes or refunds are not easily managed if plans shift. Worse, you typically lose the upgrade entirely in the process, making it an unreliable and costly choice for frequent and flexible travelers like me.

Reassessing Loyalty: Balancing Quality, Cost, and Comfort

I’ve been a Lufthansa Senator for years, enjoying perks like lounge access, priority boarding, and better upgrade opportunities. However, the pandemic changed my travel habits.

To save costs and access better business class experiences, I began flying with airlines like Qatar Airways, Turkish Airlines, and KLM. My primary focus is on the business class seat, and unfortunately, Lufthansa’s offering has been very outdated compared to competitors. For instance, while United Airlines may have worse service and food than Lufthansa, their business class seats are far superior, and for me, the seat—providing comfort and rest on long-haul flights—is what matters most.

While this approach worked initially, it came with challenges:

Declining Quality and Service Over Time

While affordable, my experiences with Turkish Airlines have been mixed recently. The airline, which once stood out for its premium service and comfort, has shown a noticeable decline in several areas. Outdated seats, inconsistent in-flight service, and overcrowded lounges have made the overall experience less appealing. These issues, combined with challenges like poor lounge Wi-Fi and reduced food options, highlight a significant drop in quality compared to previous years.

KLM, while often available at a similarly “cheap” price for long haul business class, offers a consistent but less premium experience. The service is reliable but never reaches the high standards of the Middle Eastern carriers or Lufthansa Group when it comes to overall quality and attention to detail. While KLM remains a solid option for budget-conscious business class travelers, it lacks the exceptional service and comfort that make certain airlines stand out in the business class experience.

Longer Connection Times and Inconvenient Layovers

Flying with non-Lufthansa airlines often comes with the challenge of inconvenient layovers at hubs far from Germany, which has become increasingly frustrating over time. For example, connections via Istanbul require a roughly 3-hour flight from Germany, while reaching Doha for Qatar Airways takes about 6 hours—often scheduled in the middle of the night.

These long feeder flights, combined with awkward timings, make it difficult to get any meaningful rest, exacerbating jet lag when crossing multiple time zones. Adding to the discomfort, lounges at these hubs are frequently overcrowded during rush hours, offering little respite between flights. This contrasts sharply with the more convenient and less stressful connections offered by Lufthansa at hubs like FRA, MUC, ZRH or VIE.

Moreover, if a feeder flight is canceled—a relatively common occurrence, especially in winter—traveling via FRA or MUC offers the relatively efficient option of taking a train or rental car to get home. In contrast, when connecting even in relatively close hubs like AMS or CDG, there are no such alternatives. Rebooking to the next day becomes the only option, significantly extending travel time and adding to the inconvenience.

Fragmented Benefits and the Challenges of Multiple Alliance

Switching between different loyalty programs has diluted the perks I could consistently enjoy, making airlines like KLM less appealing despite their affordability and the convenience of a short flight to Amsterdam. While KLM is often a cheaper option, it belongs to a different alliance, creating challenges for earning and redeeming points, securing upgrades, and maintaining status benefits. Additionally, KLM lacks a First Class option, which further limits its appeal for premium travel.

After the pandemic, when earning status became easier, I even (unintentionally) achieved gold status in all three major alliances—Star Alliance, SkyTeam, and oneworld. While this gave me broader access to lounges and benefits, it also fragmented my rewards and made it harder to concentrate on one program to maximize the (premium) perks.

And let’s not even start on the connecting experience at CDG airport in Paris—it’s easily one of the worst in Europe. Only LHR in London can rival it for sheer chaos and inefficiency. These factors make sticking with Lufthansa and Star Alliance far more attractive for frequent and seamless travel.

Why Lufthansa HON Circle (Miles & More)?

The introduction of Lufthansa’s new loyalty program, based on points rather than revenue, reignited my interest in sticking with the Lufthansa Group. It rewards true frequent flyers over those who simply buy the most expensive tickets, making HON Circle a feasible goal.

While Lufthansa Senator status provides great benefits, HON Circle takes it to another level, offering a first-class experience that caters specifically to frequent long-haul travelers like me. The most significant benefits for me include:

• First-Class Lounges: Exclusive access to quieter, less crowded lounges with premium amenities in hubs like FRA, MUC, and ZRH.
• Limousine Service: Seamless transfers from the plane to the terminal in a luxury car, avoiding long walks or bus rides.
• Top Upgrade Priority: The highest chances of securing upgrades to first class, especially valuable on ultra-long-haul flights.
• Enhanced Comfort for Connecting Flights: Traveling via FRA, MUC or ZRH becomes a pleasant experience rather than a hassle.
• Simplified Travel: Sticking with one airline group simplifies bookings, loyalty points, and service consistency.

An extra benefit of achieving HON Circle status is that you can earn many points toward becoming a Lifetime Senator, which gives you Star Alliance Gold status with Lufthansa for for your entire life. Under the new system, you need 40,000 qualifying points to achieve this. Since you earn at least 6,000 points per year by reaching HON Circle, it becomes quite easy to reach this goal.

The Unmatched Luxury of Lufthansa HON Circle (Miles & More) vs. United Global Services

For international readers, HON Circle is somewhat comparable to United Global Services. However, HON takes it a step further with its unparalleled First-Class experience. Even on domestic economy flights, the access to First-Class lounges and the limousine service elevate the travel experience to a level that is simply unmatched. This premium service is exclusive to Lufthansa Group and a select few luxury carriers in the Middle East and Asia, making it one of the most sought-after statuses in the world of frequent flying.

Lufthansa Business Class: Competitive Again with Allegris

Looking ahead, Lufthansa’s Allegris cabin upgrade is slowly being introduced. While it won’t be widely available in 2025, it is expected to roll out more extensively in 2026 ahead. This upgrade finally makes Lufthansa’s business class seats competitive with other leading airlines. While I, like many, am not particularly enthusiastic about the new First Class with the double bed in the middle, I anticipate that HON Circle members will be prioritized for the best Allegris business class seats as complimentary upgrades, adding even more value to the status.

The Challenges: Cost and Commitment

Pursuing HON Circle status comes at a cost, both financially and in terms of flexibility. Here’s what I anticipate:

Increased Costs

Achieving HON Circle status requires accruing 6,000 HON Circle points annually. HON Circle points can only be collected by flying Lufthansa Group airlines in business or first class. While my company covers travel expenses for economy on short-haul flights and premium economy on long-haul trips, I personally cover the difference to upgrade to business class or book directly in a higher fare class. These upgrades are essential not only for comfort but also to meet the points requirement for HON Circle status.

In 2025, I estimate that my personal expenses for upgrades and ticket premiums will increase significantly to past years. This includes the higher costs of flying exclusively with the Lufthansa Group instead of mixing in cheaper alternatives like Turkish Airlines, KLM, or Qatar Airways, as well as the premium fares required for certain routes and schedules.

While this is a substantial financial commitment, I see it as an investment in improving my travel quality and overall experience. The enhanced comfort, time savings, and first-class perks provided by HON status make the additional expense worthwhile, especially given the amount of time I spend in the air and at airports for my job.

Reduced Flexibility

Sticking with the Lufthansa Group means foregoing cheaper or more convenient flights with other airlines like KLM or Turkish Airlines. This commitment to maintaining loyalty often requires compromises in terms of flexibility. For instance, KLM has excellent direct flights to Las Vegas while Turkish Airlines is perfect for many Asian countries where Lufthansa usually requires a stopover in San Francisco/Los Angeles/Denver/Chicago respectively Singapore/Bangkok for many less important travel destinations.

While my home airport is Nürnberg (NUE), Lufthansa’s pricing or flight schedules sometimes make it necessary for me to start my journey from Munich (MUC), which involves a two-hour car drive. In contrast, from NUE, I could easily connect to hubs like Amsterdam (AMS), Paris (CDG) or Istanbul (IST) with multiple daily flights—up to four times a day.

This convenience with other airlines often offers not only better prices but also shorter overall travel times. However, prioritizing Lufthansa Group for HON Circle status means sacrificing these options to consolidate points and loyalty benefits.

Strategic Planning

Achieving HON Circle status requires not only frequent flying but also careful planning to optimize both costs and points. To meet the requirements efficiently, I rely on two key strategies:

Flexible Travel Dates (If meetings and events allow)

I plan my travel strategically, often avoiding the most expensive and high-demand days for business travel. For example, instead of flying Sunday to Friday like many executives, I schedule my trips for Wednesday to Tuesday whenever possible. This flexibility allows me to book more affordable business class fares while still aligning with my work commitments.

Additionally, I sometimes extend my travel by a day or two of vacation to take advantage of significantly lower fares (and relax) or incorporate a weekend to transition between cities or countries where I have meetings. This not only optimizes costs but also makes the travel experience smoother and less stressful.

Leveraging “Cheap Airports” (if business class is too expensive from my home airport)

If business class fares from Germany are prohibitively expensive, I sometimes start long-haul Lufthansa trips from “cheap airports” like AMS (Amsterdam) or CPH (Copenhagen), where fares can often be €1,000–€2,000 lower than from German airports (for Lufthansa flights, not for direct flights with the home airline). However, this strategy introduces additional complexity. To reach these airports, I need to book a feeder flight or drive to the departure city, adding travel time and personal expenses. These feeder flights are not covered by my company and must be paid for out of my pocket.

While this approach can save significantly on the overall cost of long-haul business class tickets, it requires extra coordination and effort. Planning routes, ensuring seamless connections, and managing additional logistics can be time-consuming. Despite these challenges, these strategies are essential trade-offs to achieve my HON Circle status goal in 2025 while keeping the financial impact manageable.

My Strategy for Lufthansa HON Circle Status

Achieving HON Circle status in 2025 will require careful and deliberate planning. Here’s the approach I’m taking to make it a reality:

Prioritize Lufthansa Group Flights

Whenever possible, I will fly Lufthansa Group flights. This includes Lufthansa, Swiss, Austrian Airlines, Brussels Airlines, LOT Polish Airlines, and a few others. I fly business class, especially on long haul flights. This ensures that most of my flights contribute to HON Circle points and align with my strategy to accumulate the required status points efficiently.

For short-haul flights, I’ll cash upgrade to business class when feasible, even though the value may not be as strong as in the U.S. or Asia, where domestic business class provides a true premium seat. The key is to prioritize accumulating HON points effectively. I will upgrade most of the time in 2025 to reach HON Circle. But beginning 2026, I can be relaxed as my long haul flights should be sufficient to keep the HON Circle status for the next year.

Optimize Booking Strategies

To manage costs, I’ll book routes and dates strategically. This sometimes means choosing flights with better fares from neighboring airports like AMS (Amsterdam) or PRG (Prague) instead of starting from my home airport in Germany. These routes sometimes offer significantly lower business class fares, making them an essential part of my strategy. I will avoid this option whenever possible. But I am ready to do this two or three times a year if needed to “save” a few thousand Euros.

Conduct Quarterly Status Checks

I’ll regularly monitor my progress toward accumulating 6,000 HON Circle points by reviewing my points balance and upcoming travel plans. I’m fully committed to achieving HON Circle status in 2025, but I’ll reevaluate whether to continue pursuing it in 2026 and beyond. Ultimately, the decision will depend on weighing the costs against the value and comfort provided by the benefits. But honestly, as long as I travel as much as I do right now, I am confident that it is worth the spent for the better comfort.

Leverage Cash Upgrades

I’ll frequently book premium economy for long-haul flights and cash upgrade to business class when the cost is reasonable. However, as I’ve explained earlier, this approach can be risky because upgrades are forfeited if the flight needs to be rebooked. To avoid losing both the upgrade and flexibility, I’ll consider this option carefully for each trip. This is mainly possible if dates are fixed. For instance, if I fly to a country for a week long conference. In that case, the travel dates are fix and will not change after the flight booking.

Work Around Travel Needs

I plan my travel to align with my ability to work anywhere, taking advantage of flexible travel dates and layovers to secure cheaper fares without sacrificing productivity. For instance, I use time in transit to work extensively—whether on flights or in lounges—preparing presentations, writing, or catching up on emails.

However, I balance this travel-heavy schedule by minimizing work commitments when I’m at home. For example, if I’m home on a Friday, I typically dedicate just two hours in the morning to emails before focusing on family, friends, and hobbies for the weekend. I also do longer travel breaks, e.g., during the German summer or around the Christmas break.

This combination of strategic planning, flexibility, and balancing cost versus value will help me achieve my goal of HON Circle status in 2025 while maintaining a healthy work-life balance.

My Lufthansa HON Circle Road Ahead: Updates in 2025

Reaching HON Circle status will be a significant milestone in my frequent flyer journey. Throughout 2025, I’ll share updates on my progress, including:

• Points and Bookings: How close I am to achieving the required 6,000 points and insights into my booking strategies.
• Cost Analysis: A breakdown of the additional expenses incurred for HON status and whether the benefits justify the investment.
• Lessons Learned: Challenges, surprises, and tips for others pursuing similar goals.

For me, pursuing HON Circle is not about achieving a status. It’s about enhancing the quality of life by making frequent travel more comfortable, efficient, and enjoyable. Although it involves meticulous planning and a financial investment, the rewards of a seamless and elevated travel experience are invaluable for someone who spends as much time traveling as I do.

Now is the right time. Although I still miss having flights to ZRH and VIE from my home airport NUE, my biggest concern is losing FRA as a feeder flight. Green politicians have been debating this for a while. If this happens, I would have to change my entire travel strategy again, connecting at non-German hubs like AMS or IST. Like most frequent travelers, I will NOT take a three-hour train ride every week as a feeder option. However, I don’t think this will happen because Lufthansa and green politicians understand that it would just drive loyal German customers to international airlines.

Stay tuned for HON Circle updates on my blog as I navigate the world of travel and aim to elevate my Lufthansa experience in 2025. You can subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. But please keep in mind that this is mainly a software and technology blog, not about travel…

The post My Road to Lufthansa HON Circle Status in 2025 appeared first on Kai Waehner.

The Digitalization of Airports and the Aviation Industry

Digitalization transforms airport operations and improves the experience of employees and passengers. It affects various aspects of airport operations, passenger experiences, and overall efficiency.

Schiphol Group is a Dutch company that owns and operates airports in the Netherlands. The company is primarily known for operating Amsterdam Airport Schiphol, which is one of the busiest and most important airports in Europe. The Schiphol Group is involved in a range of activities related to airport management, including aviation and non-aviation services.

Schiphol Group describes its journey of becoming a leading autonomous airport until 2050:

Data Streaming with Kafka and Flink for Airport Operations and Passenger Experience

Data streaming with Apache Kafka and Apache Flink enables airport and aviation systems to process and analyze real-time data from various sources, such as flight information, passenger movements, and baggage tracking, enhancing operational efficiency and passenger experience.

These technologies facilitate predictive maintenance, personalized services, and improved security measures through the continuous flow and immediate processing of critical data at any scale reliably.

Continuous Stream Processing in an Event-driven Architecture with Kafka Streams, KSQL, or Flink

Continuous processing of incoming events in real-time enables transparency and context-specific decision making. OpenCore, an IT consultancy in Germany, presented already in 2018 at Kafka Summit San Francisco how stream processing with technologies like Kafka Streams, KSQL or Apache Flink serves the real-time needs of an airport.

Think about the technical IoT events ingested from aircraft, gates, retail stores, passenger mobile apps, and many other interfaces…

… and how continuous correlation of data in real-time enables use cases such as predictive forecasting, planning, maintenance, plus scenarios like cross-organization loyalty platforms, advertisement, and recommendation engines for improving the customer experience and increasing revenue:

Airport Digitalization with Data Streaming using Apache Kafka and Flink

Real-time data beats slow data. That’s true for almost any use in the aviation industry, including airports, airlines, and other involved organizations. Additionally, data consistency matters across organizations.

Here are key areas where digitalization affects airports. While compiling this list, I realized I wrote about many of these scenarios in the past because other industry already deployed these use cases. Hence, each section includes a reference to another article where data streaming with Kafka and Flink is already applied in this context.

1. Passenger Experience

As frequent traveller myself, I put this at the beginning of the list. Examples:

• Self-service Kiosks: Check-in, baggage drop, and boarding processes have become faster and more efficient.
• Mobile Applications: Passengers can book tickets, receive real-time flight updates, and access boarding passes.
• Biometric Systems: Facial recognition and fingerprint scanning expedite security checks and boarding.

The past decade already significantly improved the passenger experience. But it still needs to get better. And data consistency matters. Today, a flight delay or cancellation is not shared consistently across the customer mobile app, airport screens, and customer service of the airline and airport.

Reference to data streaming in financial services: Operational and transactional systems leverage Kafka for data consistency, not because of its real-time capabilities. Apache Kafka ensures data consistency with its durable commit log, timestamps, and guaranteed ordering. Kafka connects to real-time and non-real-time systems (files, batch, HTTP/REST APIs).

2. Operational Efficiency

Automation with IoT sensors, paperless processes, and software innovation enables more cost-efficient and reliable airport operations. Examples:

• Automated Baggage Handling: RFID tags and automated systems track and manage luggage, reducing errors and lost baggage).
• Predictive Maintenance: IoT sensors and data analytics predict equipment failures before they occur, ensuring smoother operations.
• Air Traffic Management: Advanced software systems enhance the coordination and efficiency of air traffic control.

Reference to data streaming in manufacturing: Condition monitoring and predictive maintenance leverage stream processing with Apache Kafka and Flink for many years already, either in the cloud or at the edge and shop floor level for Industrial IoT (IIoT) use cases.

3. Security, Safety and Health Enhancements

Safety and health are one of the most important aspects at any airport. Airports continuously improved security, monitoring, and surveillance because of terrorist attacks, the Covid pandemic, and many other dangerous scenarios.

• Advanced Screening Technologies: AI-powered systems and improved scanning technologies detect threats more effectively.
• Cybersecurity: Protecting sensitive data and systems from cyber threats is crucial, requiring robust digital security measures.
• Health Monitoring: Temperature measurements and people tracking were introduced during the Covid pandemic in many airports.

Reference to data streaming in Real Estate Management: Apache Kafka and Flink improve real estate maintenance and operations, optimize space usage, provide better employee experience, and better defense against cyber attacks. Check out “IoT Analytics with Kafka and Flink for Real Estate and Smart Building” and “Apache Kafka as Backbone for Cybersecurity” for more details.

4. Sustainability and Energy Management

Sustainability and energy management in airports involve optimizing energy use and reducing environmental impact through efficient resource management and implementing eco-friendly technologies. Examples:

• Smart Lighting and HVAC Systems: Automated systems reduce energy consumption and enhance sustainability.
• Data Analytics: Monitoring and optimizing resource usage helps reduce the carbon footprint of airports.

Sustainability and energy management in an airport can be significantly enhanced by using Apache Kafka and Apache Flink to stream and analyze real-time data from smart meters and HVAC systems, optimizing energy consumption and reducing environmental impact.

Reference to data streaming in Environmental, Social, and Governance (ESG) across industries: Kafka and Flink’s real-time data processing capabilities build a powerful alliance with ESG principles. Beyond just buzzwords, I wrote about real-world deployments with Kafka and Flink and architectures across industries to show the value of data streaming for better ESG ratings.

5. Customer Service and Communication

Customer service is crucial for each airport. While lots of information comes from airlines (like delays, cancellations, seat changes, etc.), the airport provides the critical communication backend with display, lounges, service personal, and so on.  Examples to improve the customer experience:

• AI Chatbots: Provide 24/7 customer support for inquiries and assistance with Generative AI (GenAI) embedded into the existing business processes.
• Digital Signage: Real-time updates on flight information, gate changes, and other announcements improve communication.
• Loyalty Integration: Airports do not provide a loyalty platform, but they integrate more and more with airlines (e.g., to reward miles for shopping).

Reference to data streaming in retail: The retail industry is years ahead with providing a hyper-personalized customer experience. “Omnichannel Retail and Customer 360 in Real Time with Apache Kafka” and “Customer Loyalty and Rewards Platform with Data Streaming” tell you more. GenAI is a fundamental change for customer services. Kafka and Flink play a critical role for GenAI to provide contextual, up-to-date information from transactional systems into the large language model (LLM).

6. Revenue Management

Airport revenue management involves optimizing income from aviation and non-aviation sources through demand forecasting and strategic resource allocation. Examples:

• Dynamic Pricing: Algorithms adjust prices for parking, retail spaces, and other services based on demand and other factors.
• Personalized Marketing: Data analytics help target passengers with tailored offers and promotions.

Reference to data streaming in retail: While the inventory looks different for an airport, the principles from retail can be adopted one-to-one. Instead of TVs or clothes, the inventory is the parking lot, lounge seat, and similar. Advertising is another great example. Airports can learn from many digital natives how they built a real-time digital ads platform with Kafka and Flink. This can be adopted to retail media in the airport, but also to any physical inventory management.

7. Emergency Response and Safety

Emergency response and safety at the airport involve coordinating real-time monitoring, quick decision-making, and efficient resource deployment to ensure the safety and security of passengers, staff, and infrastructure during emergencies. Examples:

• Real-time Monitoring: IoT devices and sensors provide live data on airport conditions, aiding in faster response times.
• Digital Simulation and Training: Virtual reality and simulation technologies enhance training for emergency scenarios.
• Seamless Connectivity: Stable Wi-Fi and 5G Networks with good latency and network slicing for safety-critical use cases.

Reference to data streaming in Industrial IoT: Safety-critical applications require hard real-time. This is NOT Kafka, Flink, or any similar IT technology. Instead, this is embedded systems, robotics, and programming languages like C or Rust. However, data streaming integrates the OT/IT world for near real-time data correlation and analytics in edge or hybrid cloud architectures. Every relevant data set from aircraft, gates, and other equipment is continuously monitored to ensure a safe airport environment.

Data Sharing with Kafka between Airport, Airlines and other B2B Partners like Retail Stores

Cross-organization data sharing is crucial for any airport and airline. Today, most integrations are implemented with APIs (usually HTTP/REST) or still even file-based systems. This works well for some use cases. But data streaming – by nature – is perfect for sharing streaming data like transactions, sensor data, location-based services, etc. in real-time between organizations:

As Apache Kafka is the de facto standard for data streaming, many companies directly replicate data to partners using the Kafka protocol. AsyncAPI as an open standard (beyond Kafka) and integration via HTTP on top of Kafka (via Kafka Connect API connectors) are other common patterns.

Real-World Success Stories for Data Streaming in the Aviation Industry

Several real world success stories exist for deployments of data streaming with Apache Kafka and Flink in airports and airlines. Let’s explore a few case studies and refer to further material.

Schiphol Group (Amsterdam Airport)

Roel Donker and Christiaan Hoogendoorn from Schiphol Group presented at the Data in Motion Tour 2024 in Utrecht, Netherlands. This was an excellent presentation with various data streaming use cases across fields like application integration, data analytics, internet of things, and artificial intelligence.

On its journey to an autonomous airport until 2025, the digitalization involves many technologies and software/cloud services. Schiphol Group transitioned from open source Apache Kafka to Confluent Cloud for cost-efficiency, elasticity, and multi-tenancy.

The company runs operational and analytical data streaming workloads with different SLAs. The integration team uses the data streaming platform to integrate with both the legacy and the new world, also 3rd party like airlines, GDS, police, etc (all point-to-point and with different interfaces).

Here are a few examples of the scenarios Schiphol Group explored:

Schiphol Group: Data Platform with Apache Kafka

Schiphol uses Apache Kafka as a core integration platform. The various use cases require different Kafka clusters depending on the uptime SLA, scalability, security, and latency requirements. Confluent Cloud fully manages the data streaming platform, including connectors to various data sources and sinks:

Kafka connects critical PostgreSQL databases, Databricks analytics platform, applications running in containers on Red Hat OpenShift, and others.

3Scale is used as complementary API gateway for request-response communication. The latter is not a surprise, but very common. HTTP/REST APIs and Apache Kafka complement each other. API Management solutions such as 3Scale, MuleSoft, Apigee or Kong connect to Kafka via HTTP or other interfaces.

Schiphol Group: IoT with Apache Kafka

Some use cases at Schiphol Group require connectivity and processing of IoT data. That’s not really a big surprise in the aviation industry, where airports and airlines rely on data-driven business processes:

Kafka Connect and stream processing connect and combine IoT data and feed relevant context into other IT applications.

Connectivity covers various infrastructures and networks, including:

• Private LoRa networks
• Passenger flow management system(FMS)
• BLIP (the supplier delivering IoT devices in the terminal measuring real-time how crowded areas are so people can be redirected when needed)
• Wi-Fi location services (like heatmaps for crowd management)

Schiphol Group: AI and Machine Learning with Apache Kafka

Artificial Intelligence (AI) requires various technologies and concepts to add business value. Predictive analytics, active learning, batch model training, debugging and testing the entire pipeline, and many other challenges need to be solved. Apache Kafka is the data fabric of many AI/ML infrastructures.

Here is how Kafka provides the foundation of an event-driven AI architecture at Schiphol Group:

The combination of Apache Kafka and AI/ML technologies enables various valuable use cases at Schiphol Group, including:

• Analysis of historical data (root cause analysis, critical path & process analysis, reporting)
• Insights on real-time data (insight on turnaround process with one shared truth, real time insight on ramp capacity and turnaround progress per ramp, real-time insight on ramp safety, input for E2E insight Airside
• Predictions (input for dynamic gate management, input for autonomous vehicles, input for predicting delays)

Lufthansa, Southwest, Cathay Pacific, and many other Airlines…

I met plenty of airlines that already use data streaming in production for different scenarios. Fortunately, a few of these airlines were happy to share their stories in the public:

• Southwest Airlines (Data in Motion Tour 2024 in Dallas): Single pane of glass with the ability to view all flight operations and sync their three key schedules: aircraft, passengers, workforce.
• Cathay Pacific (Data in Motion Tour 2024 in Singapore): Rebranded to Cathay because of transitioning from focus on passenger transport to adding cargo and lifestyle / shopping experiences.
• Lufthansa (Webinar 2023): Operations steering, IT modernization (from MQ and ESB to Confluent), and real-time analytics with AI/ML.

The Lufthansa success story is available in its own blog post (including video recording). For even more examples, including Singapore Airlines, Air France, and Amadeus, check out the overview article “Apache Kafka in the Airline, Aviation and Travel Industry“.

Apache Kafka and Flink as Data Fabric for Operational and Analytical Airport Use Cases

Schiphol Group’s vision of an autonomous Amsterdam Airport in 2050 shows where the aviation industry is going: Automated business processes, continuous monitoring and processing of IoT infrastructure, and data-driven decision making and passenger experiences.

Airports like Amsterdam, similarly like airlines such as Lufthansa, Southwest or Cathay, modernize existing IT infrastructure, transition to hybrid cloud architectures, and innovate with new use cases (often learning from other industries like financial services, retail or manufacturing).

Data Streaming with Apache Kafka and Flink plays a crucial role in this journey. Data processing at any scale to provide consistent and good quality data in real-time enables any downstream application (including batch and API) to build reliable operational and analytical systems.

How do you leverage data streaming with Kafka and Flink in the aviation industry? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post The Digitalization of Airport and Airlines with IoT and Data Streaming using Kafka and Flink appeared first on Kai Waehner.

Data streaming in the aviation industry

The future business of airlines and airports will be digitally integrated into the ecosystem of partners and suppliers. Companies will provide more personalized customer experiences and be enabled by a new suite of the latest technologies, including automation, robotics, and biometrics.

The entire aviation industry leverages data streaming powered by Apache Kafka already. This includes airlines, airports, global distribution systems (GDS), aircraft manufacturers, travel agencies, etc. Why? Because real-time data beats slow data across almost all use cases.

Learn more in my blog about “Apache Kafka in the Airline, Aviation and Travel Industry” covering companies like Singapore Airlines, Air France, and Amadeus.

This article focuses on data streaming in critical Lufthansa projects. Lufthansa is a major German airline and one of the largest in Europe. It is known for its extensive network of domestic and international flights. Lufthansa offers services ranging from passenger transportation to cargo logistics and is a member of the Star Alliance, one of the world’s largest airline alliances.

Apache Kafka as next-generation middleware replacing ETL, ESB, and iPaaS

Typically, an enterprise service bus (ESB) or other integration solutions like extract-transform-load (ETL) tools have been used trying to decouple systems. However, the sheer number of connectors, as well as the requirement that applications publish and subscribe to the data at the same time, mean that systems are always intertwined. As a result, development projects depend on other systems, and nothing can be truly decoupled.

Many enterprises leverage the ecosystem of Apache Kafka for successful integration of different legacy and modern applications. Data streaming differs but also complements existing integration solutions like ESB or ETL tools. Apache Kafka is unique because it combines the following characteristics into a single middleware platform:

• Real-time messaging at any scale
• Event store for true decoupling, backpressure handling, and replayability of historical events
• Data integration eliminating the need for additional integration tools
• Stream processing for stateless and stateful data correlation of real-time and historical data

“Apache Kafka vs. Enterprise Service Bus (ESB) – Friends, Enemies or Frenemies?” explores how data streaming with Kafka complements legacy middleware. If your workloads run mostly in the public cloud, you need to understand the difference between Integration Platform as a Service (iPaaS) and data streaming powered by fully-managed Kafka infrastructure.

Lufthansa uses Apache Kafka as cloud-native middleware for mission-critical integrations

Lufthansa leverages data streaming with Confluent as cloud-native middleware for its strategic integration project KUSCO (Kafka Unified Streaming Cloud Operations).

The team discussed the benefits of using Apache Kafka instead of traditional messaging queues (TIBCO EMS, IBM MQ) for data processing. My two favorite statements:

• “Scaling Kafka is really inexpensive”
• “Kafka adopted and integrated within 3 months”

Lufthansa’s Kafka architecture does not have any surprises. A key lesson learned from many companies: The real added value is created when you leverage Kafka not just for messaging, but its entire ecosystem, including different clients/proxies, connectors, stream processing, and data governance.

The result at Lufthansa: A better, cheaper, and faster infrastructure for real-time data processing at scale.

Watch the full talk from Marcos Carballeira Rodríguez from Lufthansa Group recorded at the Confluent Streaming Days 2020 to see all the architectures and quotes from Lufthansa. More and more projects are onboarded on the KUSCO platform. Here are a few statistics on the adoption from 2022 to 2023 of the KUSCO project that System Architect Krzysztof Torunski of Lufthansa Group presented:

I see this typical pattern in customers across industries: The first use case is the hardest to get live. Afterward, new business units tap into the data feeds and build their projects. It has never been easier to access data feeds in real-time and with good data quality at any scale. Just build a downstream application (with your favorite programming language, tool, or SaaS) and start innovating.

Apache Kafka for analytics and AI/machine learning

Apache Kafka serves thousands of enterprises as the mission-critical and scalable real-time data fabric for machine learning infrastructures. The evolution of Generative AI (GenAI) with large language models (LLM) like ChatGPT changed how people think about intelligent software and automation. In various blog posts, I explored the relationship between data streaming with the Kafka ecosystem and AI/machine learning.

My latest article shows the enormous opportunities and some early adopters combining Kafka and GenAI beyond the buzz.

Lufthansa uses Apache Kafka with AI/machine learning for real-time predictions

Lufthansa leverages the KUSCO platform to build new analytics use cases with real-time data for critical workloads. In the webinar, we learned about the following two projects from Lufthansa Groups’s Domain Architect Sebastian Weber: anomaly detection for alerts and fleet management for aircraft operations.

Anomaly detection with Apache Kafka and ksqlDB

Data is fed into the streaming platform from various data sources. Lufthansa consolidates and aggregates the data with stream processing before the analytics applications do real-time alerting.

Machine learning and Apache Kafka for real-time fleet management

Lufthansa leverages the streaming platform as data fabric for data ingestion, data processing, and model scoring.

Embedding analytic models into a Kafka application is a standard best practice. While the data lake or lakehouse (that receives data via Kafka) trains the model in batch, many use cases require real-time model scoring and predictions at scale with critical SLAs and low latency. That’s exactly the sweet spot of the Kafka ecosystem.

You can either directly embed a model into the Kafka app or leverage a model server that supporting streaming interfaces. I blogged about the trade-offs and use cases: “Streaming Machine Learning with Kafka-native Model Deployment“.

Interactive conversation with Lufthansa

Here is an on-demand video of my conversation with Lufthansa. We talk about use cases for data streaming in the aviation industry and how Lufthansa leverages Apache Kafka as cloud-native middleware and as the data fabric for analytics and machine learning:

Data streaming as cloud-native middleware and for mission-critical analytics

Lufthansa showed us how you can innovate in the airline industry with a fast time-to-market while still integrating with traditional technologies. The two projects show very different challenges and use cases solved with data streaming powered by the Apache Kafka ecosystem.

The aviation industry is changing rapidly. A good customer experience, valuable loyalty platforms, and competitive pricing (or better hard and soft products) require digitalization of the end-to-end supply chain. This includes topics like Industrial IoT (e.g., predictive maintenance), B2B communication with partners (like GDS, airports, and retailers), and customer 360 (including great mobile apps and omnichannel experiences).

How do you leverage data streaming with Apache Kafka in your projects and enterprise architecture? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post How Lufthansa uses Apache Kafka for Middleware and Analytics appeared first on Kai Waehner.

Airlines and Aviation are Changing – Beyond Covid-19!

Aviation and travel are notoriously vulnerable to social, economic, and political events. These months have been particularly testing one due to the global pandemic with Covid-19. But the upcoming change is coming not just due to the Coronavirus but because of the ever-changing expectations of consumers.

Right now is the time to lay the ground for the future of the aviation and travel industry.

Consumer behaviors and expectations are changing. Whole industries are being disrupted, and the aviation industry is not immune to these sweeping forces of change.

The future business of airlines and airports will be digitally integrated into the ecosystem of partners and suppliers. Companies will provide more personalized customer experiences and be enabled by a new suite of the latest technologies, including automation, robotics, and biometrics.

For instance, new customer notification mobile apps provide customers with relevant and timely updates throughout their journeys. Other major improvements support the front line service teams at various touchpoints throughout the airports and end to end travel journey.

Apache Kafka in the Airline Industry

Apache Kafka is the de facto standard for event streaming use cases across industries. Many use cases can be applied to the aviation industry, too. Concepts like payment, customer experience, and manufacturing differ in detail. But in the end, it is about integrating systems and processing data in real-time at scale.

For instance, omnichannel retail with Apache Kafka applies to airline, airports, global distribution systems (GDS), and other aviation industry sectors.

However, it is always easier to learn from other companies in the same industry. Therefore, the following explores a few public Apache Kafka success stories from the aviation industry.

Lufthansa – Kafka Unified Streaming Cloud Operations

Lufthansa talks about the benefits of using Apache Kafka instead of traditional messaging queues (TIBCO EMS, IBM MQ) for data processing.

The journey started with the question if Lufthansa can do data processing better, cheaper, and faster.

Lufthansa’s Kafka architecture does not have any surprises. A key lesson learned from many companies: The real added value is created when you leverage Kafka not just for messaging, but its entire ecosystem, including different clients/proxies, connectors, stream processing, and data governance.

The result at Lufthansa: A better, cheaper, and faster infrastructure for real-time data processing at scale.

My two favorite statements (once again: not really a surprise, as I see the same at many other customers):

• “Scaling Kafka is really inexpensive”
• “Kafka adopted and integrated within 3 months”

Watch the full talk from Marcos Carballeira Rodríguez from Lufthansa recorded at the Confluent Streaming Days 2020 to see all the architectures and quotes from Lufthansa.

And check out this exciting video recording of Lufthansa discussing their Kafka use cases for middleware modernization and machine learning:

Singapore Airlines – Predictive Maintenance with Kafka Connect, Kafka Streams, and ksqlDB

Singapore Airlines is an early adopter of KSQL to continuously process sensor data and apply analytic models to the events. They already talked about their Kafka ecosystem usage (including Kafka Connect, Kafka Streams, and KSQL) back in 2018. The use case is predictive maintenance with a scalable real-time infrastructure, as you can see in my summary slide:

Check out the complete slide deck from Singapore Airlines for more details.

Air France Hop – Scalable Real-Time Microservices

I really like the Kafka Summit talk title:  “Hop! Airlines Jets to Real-Time“. Air France Hop leverage Change Data Capture (CDC) with HVR and Kafka for real-time data processing and integration with legacy monoliths. A pretty common pattern to integrate the old and the new software and IT world:

The complete slide deck and on-demand video recording about this case study are available on the Kafka Summit page.

Amadeus – Real-Time and Batch Log Processing

As I said initially, Kafka is not just relevant for each airline, airport, and aircraft manufacturers. The global distribution system (GDS) from Amadeus is one of the world’s biggest (competing mainly with Sabre). Passenger name record (PNR) is a record in the computer reservation system (CRS) and a crucial part of any GDS vendor.  While many end-users don’t even know about Amadeus, the aviation industry could not survive without them. Their workloads are mission-critical and need to run 24/7 in real-time, plus connect to their partners’ systems (like an airline) in a very stable and mature manner!

Amadeus is relying on Apache Kafka for both real-time and batch data processing, as they explain on the official Apache Kafka website:

Streaming Data Exchange for the Travel Industry

After looking at some examples, let’s now cover one more key topic: Data integration and correlation between partners in the aviation industry. Airline, airports, GDS, travel companies, and many other companies need to integrate very well. Obviously, this is already implemented. Otherwise, there is no way to operate flights with passengers and cargo. At least in theory. Honestly, one of the most significant pain points of the travel industry for customers is bad integration across companies. Some examples:

• Late or (even worse) no notification about a delay or cancellation
• Issues with the display of available seats or upgrade
• Broken booking process on the website because of different flight numbers, connecting flights,
• Booking class issues for upgrades or rebookings
• Display of technical error messages instead of business information (for instance, I can’t count how often I had seen an “IBM WebSphere” error message when I tried to book a flight on the website of my most commonly used airline)
• The list goes on and on and on… No matter which airline you pick. That’s at my experience as a frequent traveler across all continents and timezones.

There are reasons for these issues. The aviation network is very complex. For instance, Lufthansa group sells tickets for all their own brands (like Swiss or Austrian Airlines), plus tickets from Star Alliance partners (such as United or Singapore Airlines). Hence, airline, airports, GDS, and many partner systems have to work together. 24/7. In real-time. For this reason, more and more companies in the aviation industry rely on Kafka internally.

But that’s only half of the story… How do you integrate with partners?

Event Streaming vs. REST / HTTP APIs

I explored the discussion around event streaming with Kafka vs. RESTful web services with HTTP in much more detail in another article: “Comparison: Apache Kafka vs. API Management / API Gateway tools like Mulesoft or Kong“. In short: Kafka and REST APIs have their trade-offs. Both are complementary and used together in many architectures. API Management is a great add-on for many applications and microservices, no matter if they are built with HTTP or Kafka under the hood.

But one point is clear: If you need a scalable real-time integration with a partner system, then HTTP is not the right choice. You can either pick gRPC as a request-response alternative or use Kafka natively for the integration with partners, as you use it internally already anyway:

Kafka-native replication between partners works very well. No matter what Kafka vendor and version you and your partner are running. Obviously, the biggest challenge is the security (not from a technical but an organizational perspective). Kafka requires TCP. That’s much harder to get approval for opening it to a partner than HTTP ports.

But from a technical point of view, streaming replication often makes much more sense. I have seen the first customers implementing integration via tools like Confluent Replicator. I am sure that we will see this pattern much more in the future and with better out-of-the-box tool support from vendors.

Data Integration and Correlation at an Airport with Airline Data using KSQL

So, let’s assume that you have the data streams connected at an airport. No matter if just internal data or also partner data. Data correlation adds the business value. Sönke Liebau from OpenCore presented a great airport demo with Kafka and KSQL at a Kafka Summit.

Let’s take a look at some events at an airport:

These events exist in various structures and with different technologies and formats. Some data streams arrive in real-time. However, some other data sets come from a monolithic mainframe in batch via a file integration. Kafka Connect is a Kafka-native middleware to implement this integration.

Afterward, all this data needs to be correlated with historical data from a loyalty system or relational database. This is where stream processing comes into play: This concept enables the continuous data correlation in real-time at scale. Kafka-native technologies like Kafka Streams or ksqlDB exist to build streaming ETL pipelines or business applications.

The following example correlates the gate information from the airport with the airline flight information to send a delay notification to the customer who is waiting for the connection flight:

Tons of use cases exist to leverage event streams from different systems (and partners) in real-time. Some examples from an airport perspective:

• Location-based services while the customer is walking through the airport and waiting for the flight. Example: Coupons for a restaurant (with many empty seats or food reserves to thrash if not sold during the day)
• Airline services such as free or points-based discounted lounge entrance (because the lounge tracking systems knows that it is almost empty right now anyway)
• Partner services like notifying the airport hotel that the guest can stay longer in the room because of a long delay of the upcoming flight

The list of opportunities is almost endless. However, most use cases are only possible if all systems are integrated and data is continuously correlated in real-time. If you need some more inspiration, check out the two blogs “Kafka at the Edge in a Smart Retail Store” and “Kafka in a Train for Improved Customer Experience“. All these use cases are a perfect fit for airline, airports, and their partner ecosystem.

Slides – Apache Kafka in the Aviation, Airline and Travel Industry

The following slide deck goes into more detail:

Kafka for Improved Operations and Customer Experience in the Aviation Industry

This post explored various use cases for event streaming with Apache Kafka in the aviation industry. Airline, airports, aerospace, flight safety, manufacturing, GDS, retail, and many more partners rely on Apache Kafka.

No question: Kafka is getting mainstream these months in the aviation industry. Serverless and consumption-based offerings such as Confluent Cloud boost the adoption even more. A streaming data exchange between partners is the next step I see on the horizon. I am looking forward to Kafka-native interfaces from Open APIs of enterprises, better support for streaming interfaces in API Management tools, and COTS solution from software vendors.

What are your experiences and plans for event streaming in the aviation industry? Did you already build applications with Apache Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka in the Airline, Aviation and Travel Industry appeared first on Kai Waehner.