Healthcare Archives - Kai Waehner https://www.kai-waehner.de/blog/category/healthcare/ Technology Evangelist - Big Data Analytics - Middleware - Apache Kafka Fri, 07 Feb 2025 03:39:08 +0000 en-US hourly 1 https://wordpress.org/?v=6.7.2 https://www.kai-waehner.de/wp-content/uploads/2020/01/cropped-favicon-32x32.png Healthcare Archives - Kai Waehner https://www.kai-waehner.de/blog/category/healthcare/ 32 32 How Siemens Healthineers Leverages Data Streaming with Apache Kafka and Flink in Manufacturing and Healthcare https://www.kai-waehner.de/blog/2024/12/17/how-siemens-healthineers-leverages-data-streaming-with-apache-kafka-and-flink-in-manufacturing-and-healthcare/ Tue, 17 Dec 2024 05:58:17 +0000 https://www.kai-waehner.de/?p=7036 Siemens Healthineers, a global leader in medical technology, delivers solutions that improve patient outcomes and empower healthcare professionals. A significant aspect of their technological prowess lies in their use of data streaming to unlock real-time insights and optimize processes. This blog post explores how Siemens Healthineers uses data streaming with Apache Kafka and Flink, their cloud-focused technology stack, and the use cases that drive tangible business value, such as real-time logistics, robotics, SAP ERP integration, AI/ML, and more.

The post How Siemens Healthineers Leverages Data Streaming with Apache Kafka and Flink in Manufacturing and Healthcare appeared first on Kai Waehner.

]]>
Siemens Healthineers, a global leader in medical technology, delivers solutions that improve patient outcomes and empower healthcare professionals. As part of the Siemens AG family, Siemens Healthineers stands out with innovative products, data-driven solutions, and services designed to optimize workflows, improve precision, and enhance efficiency in healthcare systems worldwide. A significant aspect of their technological prowess lies in their use of data streaming to unlock real-time insights and optimize processes. This blog post explores how Siemens Healthineers uses data streaming with Apache Kafka and Flink, their cloud-focused technology stack, and the use cases that drive tangible business value such as real-time logistics, robotics, SAP ERP integration, AI/ML, and more.

Data Streaming with Apache Kafka and Flink in Healthcare and Manufacturing at Siemens Healthineers

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch.

Siemens Healthineers: Shaping the Future of Healthcare Technology

Who They Are

Siemens AG, a global powerhouse in industrial manufacturing, energy, and technology, has been a leader in innovation for over 170 years. Known for its groundbreaking contributions across sectors, Siemens combines engineering expertise with digitalization to shape industries worldwide. Within this ecosystem, Siemens Healthineers stands out as a pivotal player in healthcare technology.

Siemens Healhineers Company Overview
Source: Siemens Healthineers

With over 71,000 employees operating in 70+ countries, Siemens Healthineers supports critical clinical decisions in healthcare. Over 90% of leading hospitals worldwide collaborate with them, and their technologies influence over 70% of critical clinical decisions.

Their Vision

Siemens Healthineers focuses on innovation through data and AI, aiming to streamline healthcare delivery. With more than 24,000 technical intellectual property rights, including 15,000 granted patents, their technological foundation enables precision medicine, enhanced diagnostics, and patient-centric solutions.

Smart Logistics and Manufacturing at Siemens
Source: Siemens Healthineers

Siemens Healthineers and Data Streaming for Healthcare and Manufacturing

Siemens is a large conglomerate. I already covered a few data streaming use cases at other Siemens divisions. For instance, the integration project from SAP ERP on-premise to Salesforce CRM in the cloud.

At the Data in Motion Tour 2024 in Frankfurt, Arash Attarzadeh (“Apache Kafka Jedi“) from Siemens Heathineers presented various very interesting success stories that leverage data streaming using Apache Kafka, Flink, Confluent, and its entire ecosystem.

Healthcare and manufacturing processes generate massive volumes of real-time data. Whether it’s monitoring devices on production floors, analyzing telemetry data from hospitals, or optimizing logistics, Siemens Healthineers recognizes that data streaming enables:

  • Real-time insights: Immediate and continuously action on events as they happen.
  • Improved decision-making: Faster and more accurate responses.
  • Cost efficiency: Reduced downtime and optimized operations.

Healthineers Data Cloud

The Siemens Healthineers Data Cloud serves as the backbone of their data strategy. Built on a robust technology stack, it facilitates real-time data ingestion, transformation, and analytics using tools like Confluent Cloud (including Apache Kafka and Flink) and Snowflake.

Siemens Healthineers Data Cloud Technology Stack with Apache Kafka and Snowflake for Healthcare
Source: Siemens Healthineers

This combination of leading SaaS solutions enables seamless integration of streaming data with batch processes and diverse analytics platforms.

Technology Stack: Healthineers Data Cloud

Key Components

  • Confluent Cloud (Apache Kafka): For real-time data ingestion, data integration and stream processing.
  • Snowflake: A centralized warehouse for analytics and reporting.
  • Matillion: Batch ETL processes for structured and semi-structured data.
  • IoT Data Integration: Sensors and PLCs collect data from manufacturing floors, often via MQTT.
Machine Monitoring and Streaming Analytics with MQTT Confluent Kafka and TensorFlow AI ML in Healthcare and Manufacturing
Source: Siemens Healthineers

Many other solutions are critical for some use cases. Siemens Healthineers also uses Databricks, dbt, OPC-UA, and many other systems for the end-to-end data pipelines.

Diverse Data Ingestion

  • Real-Time Streaming: IoT data (sensors, PLCs) is ingested within minutes.
  • Batch Processing: Structured and semi-structured data from SAP systems.
  • Change Data Capture (CDC): Data changes in SAP sources are captured and available in under 30 minutes.

Not every data integration process is or can be real-time. Data consistency is still one of the most underrated capabilities of data streaming. Apache Kafka supports real-time, batch and request-response APIs communicating with each other in a consistent way.

Use Cases for Data Streaming at Siemens Healthineers

Siemens Healthineers described six different use cases that leverage data streaming together with various other IoT, software and cloud services:

  1. Machine monitoring and predictive maintenance
  2. Data integration layer for analytics
  3. Machine and robot integration
  4. Telemetry data processing for improved diagnostics
  5. Real-time logistics with SAP events for better supply chain efficiency
  6. Track and Trace Orders for improved customer satisfaction and ensured compliance

Let’s take a look at them in the following subsections.

1. Machine Monitoring and Predictive Maintenance in Manufacturing

Goal: To ensure the smooth operation of production devices through predictive maintenance.

Using data streaming, real-time IoT data from drill machines is ingested into Kafka topics, where it’s analyzed to predict maintenance needs. By using a TensorFlow machine learning model for infererence with Apache Kafka, Siemens Healthineers can:

  • Reduce machine downtime.
  • Optimize maintenance schedules.
  • Increase productivity in manufacturing CT scanners.

Business Value: Predictive maintenance reduces operational costs and prevents production halts, ensuring timely delivery of critical medical equipment.

2. IQ-Data Intelligence from IoT and SAP to Cloud

Goal: Develop an end-to-end data integration layer for analytics.

Data from various lifecycle phases (e.g., SAP systems, IoT interfaces via MQTT using Mosquitto, external sources) is streamed into a consistent model using stream processing with ksqlDB. The resulting data backend supports the development of MLOps architectures and enables advanced analytics.

AI MLOps with Kafka Stream Processing Qlik Tableau BI at Siemens Healthineers
Source: Siemens Healthineers

Business Value: Streamlined data integration accelerates the development of AI applications, helping data scientists and analysts make quicker, more informed decisions.

3. Machine Integration with SAP and KUKA Robots

Goal: Integrate machine data for analytics and real-time insights.

Data from SAP systems (such as SAP ME and SAP PCO) and machines like KUKA robots is streamed into Snowflake for analytics. MQTT brokers and Apache Kafka manage real-time data ingestion and facilitate predictive analytics.

Siemens Machine Integration with SAP KUKA Jungheinrich Kafka Confluent Cloud Snowflake
Source: Siemens Healthineers

Business Value: Enhanced machine integration improves production quality and supports the shift toward smart manufacturing processes.

4. Digital Healthcare Service Operations using Data Streaming

Goal: Stream telemetry data from Siemens Healthineers products for analytics.

Telemetry data from hospital devices is streamed via WebSockets to Kafka and combined with ksqlDB for continuous stream processing. Insights are fed back to clients for improved diagnostics.

Business Value: By leveraging real-time device data, Siemens Healthineers enhances the reliability of its medical equipment and improves patient outcomes.

5. Real-Time Logistics with SAP Events and Confluent Cloud

Goal: Stream SAP logistics event data for real-time packaging and shipping updates.

Using Confluent Cloud, Siemens Healthineers reduces delays in packaging and shipping by enabling real-time insights into logistics processes.

SAP Logistics Integration with Apache Kafka for Real-Time Shipping Points
Source: Siemens Healthineers

Business Value: Improved packaging planning reduces delivery times and enhances supply chain efficiency, ensuring faster deployment of medical devices.

6. Track and Trace Orders with Apache Kafka and Snowflake

Goal: Real-time order tracking using streaming data.

Data from Siemens Healthineers orders is streamed into Snowflake using Kafka for real-time monitoring. This enables detailed tracking of orders throughout the supply chain.

Business Value: Enhanced order visibility improves customer satisfaction and ensures compliance with regulatory requirements.

Real-Time Data as a Catalyst for Healthcare and Manufacturing Innovation at Siemens Healthineers

Siemens Healthineers’ innovative use of data streaming exemplifies how real-time insights can drive efficiency, reliability, and innovation in healthcare and manufacturing. By leveraging tools like Confluent (including Apache Kafka and Flink), MQTT and Snowflake and transitiing some workloads to the cloud, they’ve built a robust infrastructure to handle diverse data streams, improve decision-making, and deliver tangible business outcomes.

From predictive maintenance to enhanced supply chain visibility, the adoption of data streaming unlocks value at every stage of the production and service lifecycle. For Siemens Healthineers, these advancements translate into better patient care, streamlined operations, and a competitive edge in the dynamic healthcare industry.

To learn more about the relationship between these key technologies and their applications in different use cases, explore the articles below:

Do you have similar use cases and architectures like Siemens Healthineers to leverage data streaming with Apache Kafka and Flink in the healthcare and manufacturing sector? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post How Siemens Healthineers Leverages Data Streaming with Apache Kafka and Flink in Manufacturing and Healthcare appeared first on Kai Waehner.

]]>
Data Streaming in Healthcare and Pharma: Use Cases and Insights from Cardinal Health https://www.kai-waehner.de/blog/2024/11/28/data-streaming-in-healthcare-and-pharma-use-cases-cardinal-health/ Thu, 28 Nov 2024 04:12:15 +0000 https://www.kai-waehner.de/?p=7047 This blog explores Cardinal Health’s journey, exploring how its event-driven architecture and data streaming power use cases like supply chain optimization, and medical device and equipment management. By integrating Apache Kafka with platforms like Apigee, Dell Boomi and SAP, Cardinal Health sets a benchmark for IT modernization and innovation in the healthcare and pharma sectors.

The post Data Streaming in Healthcare and Pharma: Use Cases and Insights from Cardinal Health appeared first on Kai Waehner.

]]>

The post Data Streaming in Healthcare and Pharma: Use Cases and Insights from Cardinal Health appeared first on Kai Waehner.

]]>
The State of Data Streaming for Healthcare with Apache Kafka and Flink https://www.kai-waehner.de/blog/2023/11/27/the-state-of-data-streaming-for-healthcare-in-2023/ Mon, 27 Nov 2023 13:52:35 +0000 https://www.kai-waehner.de/?p=5841 This blog post explores the state of data streaming for the healthcare industry powered by Apache Kafka and Apache Flink. IT modernization and innovation with pioneering technologies like sensors, telemedicine, or AI/machine learning are explored. I look at enterprise architectures and customer stories from Humana, Recursion, BHG (former Bankers Healthcare Group), and more. A complete slide deck and on-demand video recording are included.

The post The State of Data Streaming for Healthcare with Apache Kafka and Flink appeared first on Kai Waehner.

]]>
This blog post explores the state of data streaming for the healthcare industry. The digital disruption combined with growing regulatory requirements and IT modernization efforts require a reliable data infrastructure, real-time end-to-end observability, fast time-to-market for new features, and integration with pioneering technologies like sensors, telemedicine, or AI/machine learning. Data streaming allows integrating and correlating legacy and modern interfaces in real-time at any scale to improve most business processes in the healthcare sector much more cost-efficiently.

I look at trends in the healthcare industry to explore how data streaming helps as a business enabler, including customer stories from Humana, Recursion, BHG (former Bankers Healthcare Group), Evernorth Health Services, and more. A complete slide deck and on-demand video recording are included.

The State of Data Streaming for Healthcare in 2023 with Apache Kafka and Flink

The digitalization of the healthcare sector and disruptive use cases is exciting. Countries where healthcare is not part of the public administration innovate quickly. However, regulation and data privacy are crucial across the world. And even innovative technologies and cloud services need to comply with law and in parallel connect to legacy platforms and protocols.

Regulation and interoperability

Healthcare does often not have a choice. Regulations by the government must be implemented by a specific deadline. IT modernization, adoption of new technologies, and integration with the legacy world are mandatory. Many regulations demand Open APIs and interfaces. But even if not enforced, the public sector does itself a favour adopting open technologies for data sharing between different sectors and the members.

A concrete example: Interoperability and Patient Access final rule (CMS-9115-F), as explained by a US government, “aims to put patients first, giving them access to their health information when they need it most and, in a way, they can best use it.

  • Interoperability = Interoperability is the ability of two or more systems to exchange health information and use the information once it is received.
  • Patient Access = Patient Access refers to the ability of consumers to access their health care records.

Lack of seamless data exchange in healthcare has historically detracted from patient care, leading to poor health outcomes, and higher costs. The CMS Interoperability and Patient Access final rule establishes policies that break down barriers in the nation’s health system to enable better patient access to their health information, improve interoperability and unleash innovation while reducing the burden on payers and providers.

Patients and their healthcare providers will be more informed, which can lead to better care and improved patient outcomes, while reducing burden. In a future where data flows freely and securely between payers, providers, and patients, we can achieve truly coordinated care, improved health outcomes, and reduced costs.”

Digital disruption and automated workflows

Gartner has a few interesting insights about the evolution of the healthcare sector. The digital disruption is required to handle revenue reduction and revenue reinvention because of economic pressure, scarce and extensive talent, and supply challenges:

Challenges for the Digital Disruption in the Health System

Gartner points out that real-time workflows and automation are critical across the entire health process to enable an optimal experience:

Real Time Automated Interoperable Data and Workflows

Therefore, data streaming is very helpful in implementing new digitalized healthcare processes.

Data streaming in the healthcare industry

Adopting healthcare trends like telemedicine, automated member service with Generative AI (GenAI), or automated claim processing are only possible if enterprises in the games sector can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later:

Use Cases for Real-Time Data Streaming in the Healthcare Industry with Apache Kafka and Flink

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

The following blog series about data streaming with Apache Kafka in the healthcare industry is a great starting point to learn more about data streaming in the health sector, including a few industry-specific and Kafka-powered case studies:

The healthcare industry applies various software development and enterprise architecture trends for cost, elasticity, security, and latency reasons. The three major topics I see these days at customers are:

  • Event-driven architectures (in combination with request-response communication) to enable domain-driven design and flexible technology choices
  • Data mesh for building new data products and real-time data sharing with internal platforms and partner APIs
  • Fully managed SaaS (whenever doable from compliance and security perspective) to focus on business logic and faster time-to-market

Let’s look deeper into some enterprise architectures that leverage data streaming for healthcare use cases.

Event-driven architecture for integration and IT modernization

IT modernization requires integration between legacy and modern applications. The integration challenges include different protocols (often proprietary and complex), various communication paradigms (asynchronous, request-response, batch), and SLAs (transactions, analytics, reporting).

Here is an example of a data integration workflow combining clinical health data and claims in EDI / EDIFACT format, data from legacy databases, and modern microservices:

Public Healthcare Data Automation with Data Streaming

One of the biggest problems in IT modernization is data consistency between files, databases, messaging platforms, and APIs. That is a sweet spot for Apache Kafka: Providing data consistency between applications no matter what technology, interface or API they use.

Data mesh for real-time data sharing and consistency

Data sharing across business units is important for any organization. The healthcare industry has to combine very interesting (different) data sets, like big data game telemetry, monetization and advertisement transactions, and 3rd party interfaces.

 

Data Streaming with Apache Kafka and Flink in the Healthcare Sector

Data consistency is one of the most challenging problems in the games sector. Apache Kafka ensures data consistency across all applications and databases, whether these systems operate in real-time, near-real-time, or batch.

One sweet spot of data streaming is that you can easily connect new applications to the existing infrastructure or modernize existing interfaces, like migrating from an on-premise data warehouse to a cloud SaaS offering.

New customer stories for data streaming in the healthcare sector

The innovation is often slower in the healthcare sector. Automation and digitalization change how healthcare companies process member data, execute claim processing, integrate payment processors, or create new business models with telemedicine or sensor data in hospitals.

Most healthcare companies use a hybrid cloud approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating all IT infrastructure on premises. The integration between legacy protocols like EDIFACT and modern applications is still one of the toughest challenges.

Here are a few customer stories from healthcare organizations for IT modernization and innovation with new technologies:

  • BHG Financial (formerly: Bankers Healthcare Group): Direct lender for healthcare professionals offering loans, credit card, insurance
  • Evernorth Health Services: Hybrid integration between on-premise mainframe and microservices on AWS cloud
  • Humana: Data integration and analytics at the point of care
  • Recursion: Accelerating drug discovery with a hybrid machine learning architecture

Resources to learn more

This blog post is just the starting point. Learn more about data streaming with Apache Kafka and Apache Flink in the healthcare industry in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-demand video recording

The video recording explores the healthcare industry’s trends and architectures for data streaming. The primary focus is the data streaming architectures and case studies.

I am excited to have presented this webinar in my interactive light board studio:

Lightboard Video about Apache Kafka and Flink in Healthcare

This creates a much better experience, especially in a time after the pandemic, where many people are “Zoom fatigue”.

Check out our on-demand recording:

Video: Data Streaming in Real Life in Healthcare

Slides

If you prefer learning from slides, check out the deck used for the above recording:

Fullscreen Mode

Case studies and lightboard videos for data streaming in the healthcare industry

The state of data streaming for healthcare in 2023 is interesting. IT modernization is the most important initiative across most healthcare companies and organizations. This includes cost reduction by migrating from legacy infrastructure like the mainframe, hybrid cloud architectures with bi-directional data sharing, and innovative new use cases like telehealth.

We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories. Here is an example of cost reduction through mainframe offloading.

Healthcare is just one of many industries that leverages data streaming with Apache Kafka and Apache Flink.. Every month, we talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, telcos, gaming, and so on… Check out my other blog posts.

How do you modernize IT infrastructure in the healthcare sector? Do you already leverage data streaming with Apache Kafka and Apache Flink? Maybe even in the cloud as a serverless offering? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post The State of Data Streaming for Healthcare with Apache Kafka and Flink appeared first on Kai Waehner.

]]>
Open API and Omnichannel with Apache Kafka in Healthcare https://www.kai-waehner.de/blog/2022/05/18/open-api-and-omnichannel-with-apache-kafka-in-healthcare/ Wed, 18 May 2022 13:34:41 +0000 https://www.kai-waehner.de/?p=4505 IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part five: Open API and Omnichannel. Examples include Care.com and Invitae.

The post Open API and Omnichannel with Apache Kafka in Healthcare appeared first on Kai Waehner.

]]>
IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part five: Open API and Omnichannel. Examples include Care.com and Invitae.

Kafka in Healthcare - Open API and Omnichannel Data Streaming

Blog Series – Kafka in Healthcare

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

Stay tuned for blog posts around these topics. Subscribe to my newsletter to get an email after each publication (no spam or ads).

Open API and Omnichannel in Healthcare

An open API is a publicly available application programming interface that provides developers with programmatic access to a proprietary software application or web service. APIs are sets of requirements that govern how one application can communicate and interact with another. Also, keep in mind that the Open API is not necessarily for B2B communication. Internal divisions in larger organizations often leverage this approach for business and technical reasons, too.

Omnichannel in healthcare connects the fragmentation between health providers, hospitals, pharmaceutical companies, and patients. Open APIs enable services and applications to improve the customer experience in the healthcare industry.

Omnichannel and customer 360 are already very prevalent and thriving in the retail industry leveraging Apache Kafka. Healthcare is way behind. People still have to use paper and fax for many use cases.

Streaming Data Exchange and Data Sharing with Apache Kafka

Currently, many APIs are provided by organizations for access via HTTP. However, synchronous request-response communication is an anti-pattern for many data streaming use cases around Apache Kafka. For that reason, data streaming with Apache Kafka is complementary to traditional API management tools like MuleSoft Anypoint, IBM API Connect, Apigee, or Kong.

The market is changing, though. On the one side, API management tools adopt data streaming APIs (native Kafka APIs, AsyncAPI as schema standard, or proprietary interfaces). On the other side, native streaming replication between Kafka clusters for data sharing is coming up more regularly in real-time at any scale:

Streaming Data Exchange with Apache Kafka and Data Mesh in Motion

Both HTTP API integration and native data streaming are valid options, depending on the use case. Let’s explore two healthcare case studies where data sharing and APIs are used for different use cases.

Care.com – Trusted Caregivers via Kafka and Open API

Care.com is an online marketplace for care services, including senior care and housekeeping. Their cloud-native Bravo Platform provides a simple, unified IT architecture to streamline go-to-market initiatives.

Care.com moved from a monolithic architecture into a truly decoupled, scalable microservices platform powered by the Apache Kafka ecosystem. The migration from Confluent Platform to Confluent Cloud allowed the focus on business problems. The data streaming infrastructure is serverless and consumed as a service.

The Schema Registry enables data governance across different applications built with various technologies like Java, .NET, Go, etc. The “Care APIs” (inspired by Google APIs) define all of their data and service contracts with Protobuf to communicate between different stakeholders with an Open API and enforced schemas. Additional capabilities enhance security for PII data with fine-grained RBAC and data lineage.

Invitae – Omnichannel for 24/7 Production and Data Science

Invitae is a biotechnology company that provides DNA-based testing to detect genetic abnormalities beyond what can be identified through traditional methodologies. Gene panels and single-gene testing are used for a broad range of clinical areas, including hereditary cancer, cardiology, neurology, pediatric genetics, metabolic disorders, immunology, and hematology.

While I have no idea what the above means, I like the technical details of how Invitae leverages a data streaming platform to provide a tremendous omnichannel experience from research to end-users. Their platform brings comprehensive genetic information into mainstream medical practice to improve the quality of healthcare for billions of people.

Invitae’s Kafka Summit talk, “From Zero to Streaming Healthcare in Production,” went into detail about their data streaming architecture:

The omnichannel chain of Invitae’s data flow does not stop after the DNA testing: Genetic results are often just the beginning. Invitae’s interactive, educational portal and caring genetic counselors can help patients understand their results and what to do next.

The truly decoupled infrastructure and Open API enables other stakeholders to join in and consume the data. That’s a considerable paradigm shift: Building an application entirely of streams is now possible. Additionally, data science teams consume the data for AI and Machine Learning use cases.

Open API and Omnichannel with Kafka for Improved Patient and Customer Experience

Open API and data sharing are significant game-changers for the healthcare industry. Enabling the same principles not just via “normal APIs” (= REST/HTTP) but also via data streaming empowers innovative new use cases.

Data streaming with Kafka and Open APIs can be built natively with a streaming data exchange or complementary API Management solutions to add capabilities like advanced reporting or monetization of the service consumption.

How do you leverage data streaming with Apache Kafka in the healthcare industry? What architecture does your platform use? Which products do you combine with data streaming? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Open API and Omnichannel with Apache Kafka in Healthcare appeared first on Kai Waehner.

]]>
Machine Learning and Data Science with Kafka in Healthcare https://www.kai-waehner.de/blog/2022/04/18/machine-learning-data-science-with-kafka-in-healthcare-pharma/ Mon, 18 Apr 2022 11:44:09 +0000 https://www.kai-waehner.de/?p=4446 IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part five: Machine Learning and Data Science. Examples include Recursion and Humana.

The post Machine Learning and Data Science with Kafka in Healthcare appeared first on Kai Waehner.

]]>
IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part five: Machine Learning and Data Science. Examples include Recursion and Humana.

Machine Learning and Data Science with Apache Kafka in Healthcare

Blog Series – Kafka in Healthcare

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

Stay tuned for a dedicated blog post for each of these topics as part of this blog series. I will link the blogs here as soon as they are available (in the next few weeks). Subscribe to my newsletter to get an email after each publication (no spam or ads).

Machine Learning and Data Science with Data Streaming using Apache Kafka

The relationship between Apache Kafka and machine learning (ML) is getting more and more traction for data engineering at scale and robust model deployment with low latency.

The Kafka ecosystem helps in different ML use cases for model training, model serving, and model monitoring. The core of most ML projects requires reliable and scalable data engineering pipelines across

  • different technologies
  • communication paradigms (REST, gRPC, data streaming)
  • programming languages (like Python for the data scientist or Java/Go/C++ for the production engineer)
  • APIs
  • commercial products
  • SaaS offerings

Here is an architecture diagram that shows how Kafka helps in data science projects:

The beauty of Kafka is that it combines real-time data processing with extreme scalability and true decoupling between systems.

Tiered Storage adds cost-efficient storage of big data sets and replayability with guaranteed ordering.

I’ve written about this relationship between Kafka and Machine Learning in various articles:

Let’s look at a few real-world deployments for Apache Kafka and Machine Learning in the healthcare sector.

Humana – Real-Time Interoperability at the Point of Care

Humana Inc. is a for-profit American health insurance company. They leverage data streaming with Apache Kafka to improve real-time interoperability at the point of care.

The interoperability platform to transition from an insurance company with elements of health to truly a health company with elements of insurance.

Their core principles include:

  • Consumer-centric
  • Health plan agnostic
  • Provider agnostic
  • Cloud resilient
  • Elastic scale
  • Event-driven and real-time

A critical characteristic is inter-organization data sharing (known as “data exchange/data sharing”).

Humana’s use cases include

  • real-time updates of health information, for instance
  • connecting health care providers to pharmacies
  • reducing pre-authorizations from 20-30 minutes to 1 minute
  • real-time home healthcare assistant communication

The Humana interoperability platform combines data streaming (= the Kafka ecosystem) with artificial intelligence and machine learning (= IBM Watson) to correlate data, train analytic models, and act on new events in real-time.

Humana’s data journey is described in this diagram from their Kafka Summit talk:

Real-Time Healthcare Insurance at Humana with Apache Kafka Data Streaming

Learn more details about Humana’s use cases and architecture in the keynote of another Kafka Summit session.

Recursion – Industrial Revolution of Drug Discovery with Kafka and Deep Learning

Recursion is a clinical-stage biotechnology company that built the “industrial revolution of drug discovery“. They decode biology by integrating technological innovations across biology, chemistry, automation, machine learning, and engineering to industrialize drug discovery.

Industrial pharma revolution - accelerate drug discovery at recursion

Kafka-powered data streaming speeds up the pharma processes significantly. Recursion has already made significant strides in accelerating drug discovery, with over 30 disease models in discovery, another nine in preclinical development, and two in clinical trials.

With serverless Confluent Cloud and the new data streaming approach, the company has built a platform that makes it possible to screen much larger experiments with thousands of compounds against hundreds of disease models in minutes and less expensive than alternative discovery approaches.

From a technical perspective, Recursion finds drug treatments by processing biological images. A massively parallel system combines experimental biology, artificial intelligence, automation, and real-time data streaming:

Apache Kafka and Machine Learning at Recursion for Drug Discovery in Pharma

Recursion went from ‘drug discovery in manual and slow, not scalable, bursty BATCH MODE’ to ‘drug discovery in automated, scalable, reliable REAL-TIME MODE’.

Recursion leverages Dagger, an event-driven workflow and orchestration library for Kafka Streams that enables engineers to orchestrate services by defining workloads as high-level data structures. Dagger combines Kafka topics and schemas with external tasks for actions completed outside of the Kafka Streams applications.

Drug Discovery in automated, scalable, reliable real time Mode

In the meantime, Recursion did not just migrate from manual batch workloads to Kafka but also migrated to serverless Kafka, leveraging Confluent Cloud to focus its resources on business problems instead of infrastructure operations.

Machine Learning and Data Science with Kafka for Intelligent Healthcare Applications

Think about IoT sensor analytics, cybersecurity, patient communication, insurance, research, and many other domains. Real-time data beats slow data in the healthcare supply chain almost everywhere.

This blog post explored the capabilities of the Apache Kafka ecosystem for machine learning infrastructures. Real-world deployments from Humana and Recursion showed how enterprises successfully deploy Kafka together with Machine Learning frameworks like TensorFlow for use cases.

How do you leverage data streaming with Apache Kafka in the healthcare industry? What architecture does your platform use? Which products do you combine with data streaming? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Machine Learning and Data Science with Kafka in Healthcare appeared first on Kai Waehner.

]]>
Real Time Analytics with Apache Kafka in the Healthcare Industry https://www.kai-waehner.de/blog/2022/04/04/real-time-analytics-machine-learning-with-apache-kafka-in-the-healthcare-industry/ Mon, 04 Apr 2022 10:31:47 +0000 https://www.kai-waehner.de/?p=4414 IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. This is part four: Real-Time Analytics. Examples include Cerner, Celmatix, CDC/Centers for Disease Control and Prevention.

The post Real Time Analytics with Apache Kafka in the Healthcare Industry appeared first on Kai Waehner.

]]>
IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part four: Real-Time Analytics. Examples include Cerner, Celmatix, CDC/Centers for Disease Control and Prevention.

Real Time Analytics and Machine Learning with Apache Kafka in Healthcare

Blog Series – Kafka in Healthcare

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

Stay tuned for a dedicated blog post for each of these topics as part of this blog series. I will link the blogs here as soon as they are available (in the next few weeks). Subscribe to my newsletter to get an email after each publication (no spam or ads).

Real-Time Analytics with Apache Kafka

Real-time analytics (aka stream processing, streaming analytics, or complex event processing) is a data processing technology used to collect, store, and manage continuous data streams when produced or received.

Stream processing has many use cases. Examples include the backend process for claim processing, billing, logistics, manufacturing, fulfillment, or fraud detection. Data processing may need to be decoupled from the frontend, where users click buttons and expect things to happen.

The de facto standard for real-time analytics is Apache Kafka. Kafka is like a central data hub that holds shared events and keeps services in sync. Its distributed cluster technology provides availability, resiliency, and performance properties that strengthen the architecture. It leaves the programmer to write and deploy client applications that will run load balanced and be highly available.

Real Time Analytics with Data Streaming Stream Processing and Apache Kafka

Technologies for real-time analytics with the Kafka ecosystem include Kafka-native stream processing with Kafka Streams or ksqlDB, or 3rd party add-ons like Apache Flink, Spark Streaming, or commercial streaming analytics cloud services.

The critical difference with the Kafka ecosystem is that you leverage a single platform for data integration and processing at scale in real-time. There is no need to combine several platforms to achieve this. The result is a Kappa architecture that enables real-time but also batch workloads with a single integration architecture.

Let’s look at a few real-world deployments in the healthcare sector.

Cerner – Sepsis Alerting in Real-Time

Cerner is a supplier of health information technology services, devices, and hardware. ~30% of all US healthcare data in a Cerner solution.

Sepsis kills. In fact, it kills up to 52,000 people every year in the UK alone. With sepsis alerting, the key to saving lives is early identification, especially the need to administer antibiotics within that first critical ‘golden hour’. Quick alerts make a significant impact. Cerner’s sepsis alert, coupled with the care plans developed with the big room approach, means that patients are now 71% more likely to receive timely antibiotics.

Cerner leverages a Kafka-powered central event streaming platform for sepsis alerting in real-time to save lives. Legacy systems hit a wall preventing going faster (and missed SLAs). The data processing with Kafka progressed from minutes to seconds.

Real Time Sepsis Alerting at Cerner with Apache Kafka

Cerner is a long-term Kafka user and early adopter in the healthcare sector. Learn more about this use case in their Kafka Summit talk from 2016.

Celmatix – Reproductive Health Care

Celmatix is a preclinical-stage biotech company that provides digital tools and genetic insights focused on fertility. They offer personalized information to disrupt how women approach their lifelong reproductive health journey.

The streaming platform provides real-time aggregation of heterogeneous data collected from Electronic Medical Records (EMRs) and genetic data collected from partners through their Personalized Reproductive Medicine (PReM) Initiative.

Proactive reproductive health decisions are enabled by real-time genomics data and by applying technologies such as big data analytics, machine learning, A/I, and whole-genome DNA sequencing.

Celmatix Reproductive Health Care Eletronical Medical Records EMR Processing with Apache Kafka

Data governance for security and compliance is critical in such a healthcare application. “Apache Kafka and Confluent are invaluable investments to scale the way we want to and future-proof our business,” says the lead data architect at Celmatix. Learn more in the Confluent case study.

CDC – Covid-19 Electronic Lab Reporting

The Centers for Disease Control and Prevention (CDC) built Covid-19 Electronic Lab Reporting (CELR) with the Kafka ecosystem. Use cases include case notifications, lab reporting, and healthcare interoperability.

The threat of the COVID-19 virus is tracked in real-time to provide comprehensive data for local, state, and federal responses. The application allows them to understand locations with an increase in incidence better.

With the true decoupling of the data streaming platform, the CDC can rapidly aggregate, validate, transform, and distribute laboratory testing data submitted by public health departments and other partners:

Centers for Disease Control and Prevention CDC Covid Analytics with Kafka

Real-Time Analytics with Kafka for Smart Healthcare Applications at any Scale

Think about IoT sensor analytics, cybersecurity, patient communication, insurance, research, and many other domains. Real-time data beats slow data in the healthcare supply chain almost everywhere.

This blog post explored the capabilities of the Apache Kafka ecosystem for real-time analytics. Real-world deployments from Cerner, Celmatix and the Centers for Disease Control and Prevention showed how enterprises successfully deploy Kafka for different enterprise architecture use cases.

How do you leverage data streaming with Apache Kafka in the healthcare industry? What architecture does your platform use? Which products do you combine with data streaming? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Real Time Analytics with Apache Kafka in the Healthcare Industry appeared first on Kai Waehner.

]]>
Legacy Modernization and Hybrid Multi-Cloud with Kafka in Healthcare https://www.kai-waehner.de/blog/2022/03/30/legacy-modernization-and-hybrid-multi-cloud-with-kafka-in-healthcare/ Wed, 30 Mar 2022 08:10:25 +0000 https://www.kai-waehner.de/?p=4393 IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. This is part two: Legacy modernization and hybrid multi-cloud. Examples include Optum / UnitedHealth Group, Centene, and Bayer.

The post Legacy Modernization and Hybrid Multi-Cloud with Kafka in Healthcare appeared first on Kai Waehner.

]]>
IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part two: Legacy modernization and hybrid multi-cloud. Examples include Optum / UnitedHealth Group, Centene, and Bayer.

Legacy Modernization and Hybrid Multi Cloud with Apache Kafka in Healthcare

Blog Series – Kafka in Healthcare

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

Stay tuned for a dedicated blog post for each of these topics as part of this blog series. I will link the blogs here as soon as they are available (in the next few weeks). Subscribe to my newsletter to get an email after each publication (no spam or ads).

Legacy Modernization and Hybrid Multi-Cloud with Kafka

Application modernization benefits from the Apache Kafka ecosystem for hybrid integration scenarios.

Most enterprises require a reliable and scalable integration between legacy systems such as IBM Mainframe, Oracle, SAP ERP, and modern cloud-native applications like Snowflake, MongoDB Atlas, or AWS Lambda.

I already explored “architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments” some time ago:

Hybrid Cloud Architecture with Apache Kafka

TL;DR: Various alternatives exist to deploy Apache Kafka across data centers, regions, and continents. There is no single best architecture. It always depends on characteristics such as RPO / RTO, SLAs, latency, throughput, etc.

Some deployments focus on on-prem to cloud integration. Others link Kafka clusters on multiple cloud providers. Technologies such as Apache Kafka’s  MirrorMaker 2, Confluent Replicator, Confluent Multi-Region-Clusters, and Confluent Cluster Linking help build such an infrastructure.

Let’s look at a few real-world deployments in the healthcare sector.

Optum (United Health Group) – Cloud-native Kafka-as-a-Service

Optum is an American pharmacy benefit manager and health care provider. It is a subsidiary of UnitedHealth Group. The Apache Kafka infrastructure is provided as an internal service, centrally managed, and used by over 200 internal application teams.

Optum built a repeatable, scalable, cost-efficient way to standardize data. They leverage the whole Kafka ecosystem:

  • Data ingestion from multiple resources (Kafka Connect)
  • Data enrichment (Table Joins & Streaming API)
  • Aggregation and metrics calculation (Kafka Streams API)
  • Sinking data to database (Kafka Connect)
  • Near real-time APIs to serve the data

Optum’s Kafka Summit talk explored the journey and maturity curve for their data streaming evolution:

Optum United Healthcare Apache Kafka Journey

As you can see, the journey started with a self-managed Kafka cluster on-premises. Over time, they migrated to a cloud-native Kubernetes environment and built an internal Kafka-as-a-Service offering. Right now, Optum works on multi-cloud enterprise architecture to deploy across multiple cloud service providers.

Centene – Data Integration for M&A across Infrastructures

Centene is the largest Medicaid and Medicare Managed Care Provider in the US. The healthcare insurer acts as an intermediary for government-sponsored and privately insured healthcare programs. Centene’s mission is to “help people live healthier lives and to help make the health system work better for everyone”.

The critical challenge of Centene is interesting: Growth! Many mergers and acquisitions happened in the last decade: Envolve, HealthNet, Fidelis, and Wellcare.

Data integration and processing at scale in real-time between various systems, infrastructures, and cloud environments is a considerable challenge. Kafka provides Centene with valuable capabilities, as they explained in an online talk:

  • Highly scalable
  • High autonomy/decoupling
  • High availability & data resiliency
  • Real-time data transfer
  • Complex stream processing

The event-driven integration architecture leverages Apache Kafka with MongoDB:

Centene Cloud Architecture with Kafka MongoDB ETL Pipeline

Bayer – Hybrid Multi-Cloud Data Streaming

Bayer AG is a German multinational pharmaceutical and life sciences company and one of the largest pharmaceutical companies in the world. They leverage Kafka in various use cases and business domains. The following scenario is from Monsanto.

Bayer adopted a cloud-first strategy and started a multi-year transition to the cloud to provide real-time data flows across hybrid and multi-cloud infrastructures.

The Kafka-based cross-data center DataHub facilitated migration and the shift to real-time stream processing. It offers strong enterprise adoption and supports a myriad of use cases. The Apache Kafka ecosystem is the “middleware” to build a bi-directional streaming replication and integration architecture between on-premises data centers and multiple cloud providers:

From legacy on premise to hybrid multi cloud at Bayer with Apache Kafka

The Kafka journey of Bayer started on AWS. Afterward, some project teams worked on GCP. In parallel, DevOps and cloud-native technologies modernized the underlying infrastructure. Today, Bayer operates a multi-cloud infrastructure with mature, reliable, and scalable stream processing use cases:

Bayer AG using Apache Kafka for Hybrid Cloud Architecture and Integration

Learn about Bayer’s journey and how they built their hybrid and multi-cloud Enterprise DataHub with Apache Kafka and its ecosystem: Bayer’s Kafka Summit talk.

Data Streaming with Kafka across Hybrid and Multi-cloud Infrastructures

Think about IoT sensor analytics, cybersecurity, patient communication, insurance, research, and many other domains. Real-time data beats slow data in the healthcare supply chain almost everywhere.

This blog post explored the value of data streaming with Apache Kafka to modernize IT infrastructure and build hybrid multi-cloud architectures. Real-world deployments from Optum, Centene, and Bayer showed how enterprises deploy Kafka successfully for different use cases in the enterprise architecture.

How do you leverage data streaming with Apache Kafka in the healthcare industry? What architecture does your platform use? Which products do you combine with data streaming? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Legacy Modernization and Hybrid Multi-Cloud with Kafka in Healthcare appeared first on Kai Waehner.

]]>
Apache Kafka in the Healthcare Industry https://www.kai-waehner.de/blog/2022/03/28/apache-kafka-data-streaming-healthcare-industry/ Mon, 28 Mar 2022 20:40:24 +0000 https://www.kai-waehner.de/?p=4359 IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores real-world examples of data streaming with Apache Kafka to increase efficiency, reduce cost, and improve the human experience across the healthcare value chain including pharma, insurance, providers, retail, and manufacturing. This is part one: Overview.

The post Apache Kafka in the Healthcare Industry appeared first on Kai Waehner.

]]>
IT modernization and innovative new technologies change the healthcare industry significantly. This blog series explores how data streaming with Apache Kafka enables real-time data processing and business process automation. Real-world examples show how traditional enterprises and startups increase efficiency, reduce cost, and improve the human experience across the healthcare value chain, including pharma, insurance, providers, retail, and manufacturing. This is part one: Overview.

Data Streaming with Apache Kafka in the Healthcare Industry

Here is the entire blog series:

Healthcare – A broad spectrum of very different domains

Health care is the maintenance or improvement of health via the prevention, diagnosis, treatment, amelioration, or cure of disease, illness, injury, and other physical and mental impairments.

Health professionals and allied health fields deliver health care. Medicine, dentistry, pharmacy, midwifery, nursing, optometry, audiology, psychology, occupational therapy, physical therapy, athletic training, and other health professions are all part of health care. It includes work done in providing primary care, secondary care, tertiary care, and public health. Access to health care varies across countries, communities, and individuals, influenced by social and economic conditions, as well as health policies.

The Healthcare Industry

The healthcare industry (also called the medical industry or health economy) is one of the world’s largest industries. It aggregates and integrates sectors within the economic system that provide goods and services to treat patients with curative, preventive, rehabilitative, and palliative care.

This industry includes the generation and commercialization of goods and services lending themselves to maintaining and re-establishing health. The modern healthcare industry has three essential branches: Services, products, and finance, and may be divided into many sectors and categories.

The blog series explores the technical architectures and use cases relevant across the healthcare supply chain. The slide deck shows various real-world deployments from different domains.

Real-time data beats slow data in the healthcare sector

Processing information in the proper context at the right time is crucial for most use cases across the healthcare value chain. Real-time data processing with the Kafka ecosystem reduces risks, improves efficiency, and decreases cost in many domains. Here are some examples:

Data Streaming with Apache Kafka in Healthcare Insurance Pharma Cybersecurity

A real-time Kappa architecture beats batch processing with Lambda architectures and adds business value in almost all use cases.

Apache Kafka to process data in motion across the healthcare supply chain

The beauty of the Kafka ecosystem is the capability to provide a truly decoupled infrastructure for workloads at any scale, including transactional and analytical use cases.

Data flows in and out of various systems. Some are real-time. Others are batch or web service APIs. Some are modern and cloud-native microservices. Others are monolithic proprietary on-premise applications. Some use open standards like Health Level Seven (HL7) with FHIR, others use open data formats like JSON, and some use proprietary data formats.

Here is an example of public health data automation. It leverages Apache Kafka to connect claims and clinical data from proprietary legacy systems with modern cloud-native microservices:

Public Health Data Automation with Data Streaming

Real-World Deployments of Apache Kafka in the Healthcare Industry

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

Stay tuned for a dedicated blog post for each of these topics as part of this blog series. I will link the blogs here as soon as they are available (in the next few weeks). Subscribe to my newsletter to get an email after each publication (no spam or ads).

Slide Deck – Apache Kafka in Healthcare

Here is a slide deck that covers an introduction, use cases, and architectures for data streaming with Apache Kafka in the healthcare sector:

The other blogs of the series take a deeper look into the use cases and architectures.

Data Streaming as Game Changer in Healthcare

Think about IoT sensor analytics, cybersecurity, patient communication, insurance, research, and many other domains. Real-time data beats slow data in the healthcare supply chain almost everywhere.

The above slide deck is an excellent overview of the added value of data streaming to modernize IT infrastructure and build innovative new applications in the cloud or hybrid scenarios. Several real-world deployments show how game-changing the technology is for this very traditional and often still paper-driven sector.

How do you leverage data streaming with Apache Kafka in the healthcare industry? What architecture does your platform use? Which products do you combine with data streaming? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka in the Healthcare Industry appeared first on Kai Waehner.

]]>