As threats grow more sophisticated, the technologies powering cybersecurity solutions must advance to stay ahead. Data streaming technologies like Apache Kafka and Apache Flink have become foundational in this evolution, enabling real-time threat detection, prevention, and rapid response. These tools transform cybersecurity from static defenses to dynamic systems capable of identifying and neutralizing threats as they occur.

A notable example is McAfee, a global leader in cybersecurity, which has embraced data streaming to revolutionize its operations. By transitioning to an event-driven architecture powered by Apache Kafka, McAfee processes massive amounts of real-time data from millions of endpoints, ensuring instant threat identification and mitigation. This integration has enhanced scalability, reduced infrastructure complexity, and accelerated innovation, setting a benchmark for the cybersecurity industry.

Real-time data streaming is not just an advantage—it’s now a necessity for organizations aiming to safeguard digital environments against ever-evolving threats.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch.

Antivirus is NOT Enough: Supply Chain Attack

A supply chain attack occurs when attackers exploit vulnerabilities in an organization’s supply chain, targeting weaker links such as vendors or service providers to indirectly infiltrate the target.

For example, an attacker compromises Vendor 1, a software provider, by injecting malicious code into their product. Vendor 2, a service provider using Vendor 1’s software, becomes infected. The attacker then leverages Vendor 2’s connection to the Enterprise to access sensitive systems, even though Vendor 1 has no direct interaction with the enterprise.

Traditional antivirus software is insufficient to prevent such complex, multi-layered attacks. Ransomware often plays a role in supply chain attacks, as attackers use it to encrypt data or disrupt operations across compromised systems.

Modern solutions focus on real-time monitoring and event-driven architecture to detect and mitigate risks across the supply chain. These solutions utilize behavioral analytics, zero trust policies, and proactive threat intelligence to identify and stop anomalies before they escalate.

By providing end-to-end visibility, they protect organizations from cascading vulnerabilities that traditional endpoint security cannot address. In today’s interconnected world, comprehensive supply chain security is critical to safeguarding enterprises.

The Role of Data Streaming in Cybersecurity

Cybersecurity platforms must rely on real-time data for detecting and mitigating threats. Data streaming provides a backbone for processing massive amounts of security event data as it happens, ensuring swift and effective responses. My blog series on Kafka and cybersecurity looks deeply into these use cases.

To summarize:

• Data Collection: A data streaming platforms powered by Apache Kafka collect logs, telemetry, and other data from devices and applications in real time.
• Data Processing: Stream processing frameworks like Kafka Streams and Apache Flink continuously process this data with low latency at scale for analytics, identifying anomalies or potential threats.
• Actionable Insights: The processed data feeds into Security Information and Event Management (SIEM) and Security Orchestration, Automation, and Response (SOAR) systems, enabling automated responses and better decision-making.

This approach transforms static, batch-driven cybersecurity operations into dynamic, real-time processes.

McAfee: A Real-World Data Streaming Success Story

McAfee is a global leader in cybersecurity, providing software solutions that protect computers, networks, and devices. Founded in 1987, the company has evolved from traditional antivirus software to a comprehensive suite of products focused on threat prevention, identity protection, and data security.

McAfee’s products cater to both individual consumers and enterprises, offering real-time protection through partnerships with global integrated service providers (ISPs) and telecom operators.

Mahesh Tyagarajan (VP, Platform Engineering and Architecture at McAfee) spoke with Confluent and Forrester about McAfee’s transition from a monolith to event-driven Microservices leveraging Apache Kafka in Confluent Cloud.

Data Streaming at McAfee with Apache Kafka Leveraging Confluent Cloud

As cyber threats have grown more complex, McAfee’s reliance on real-time data streaming has become essential. The company transitioned from a monolithic architecture to a microservices-based ecosystem with the help of Confluent Cloud, powered by Apache Kafka. The fully managed data streaming platform simplified infrastructure management, boosted scalability, and accelerated feature delivery for McAfee

Use Cases for Data Streaming

1. Real-Time Threat Detection: McAfee processes security events from millions of endpoints, ensuring immediate identification of malware or phishing attempts.
2. Subscription Management: Data streaming supports real-time customer notifications, updates, and billing processes.
3. Analytics and Reporting: McAfee integrates real-time data streams into analytics systems, providing insights into user behavior, threat patterns, and operational efficiency.

Transition to an Event-Driven Architecture and Microservices

By moving to an event-driven architecture with Kafka using Confluent Cloud, McAfee:

• Standardized its data streaming infrastructure.
• Decoupled systems using microservices, enabling scalability and resilience.
• Improved developer productivity by reducing infrastructure management overhead.

This transition to data streaming with a fully managed, complete and secure cloud service empowered McAfee to handle high data ingestion volumes, manage hundreds of millions of devices, and deliver new features faster.

Business Value of Data Streaming

The adoption of data streaming delivered significant business benefits:

• Improved Customer Experience: Real-time threat detection and personalized updates enhance trust and satisfaction.
• Operational Efficiency: Automation and reduced infrastructure complexity save time and resources.
• Scalability: McAfee can now support a growing number of devices and data sources without compromising performance.

Data Streaming as the Backbone of an Event-Driven Cybersecurity Evolution in the Cloud

McAfee’s journey showcases the transformative potential of data streaming in cybersecurity. By leveraging Apache Kafka as fully managed cloud service as the backbone of an event-driven microservices architecture, the company has enhanced its ability to detect threats, respond in real time, and deliver exceptional customer experiences.

For organizations looking to stay ahead in the cybersecurity race, investing in real-time data streaming technologies is not just an option—it’s a necessity. To learn more about how data streaming can revolutionize cybersecurity, explore my cybersecurity blog series and follow me for updates on LinkedIn or X (formerly Twitter).

The post The Role of Data Streaming in McAfee’s Cybersecurity Evolution appeared first on Kai Waehner.

Blog series: Apache Kafka for Cybersecurity

This blog series explores why security features such as RBAC, encryption, and audit logs are only the foundation of a secure event streaming infrastructure. Learn about use cases,  architectures, and reference deployments for Kafka in the cybersecurity space:

• Part 1: Data in Motion as cybersecurity backbone
• Part 2: Situational awareness
• Part 3: Threat intelligence
• Part 4: Forensics
• Part 5: Air-gapped and zero trust environments
• Part 6 (THIS POST): SIEM / SOAR modernization

What are SIEM and SOAR?

SIEM (Security information and event management) and SOAR (security orchestration, automation and response) are terms coined by Gartner (like so often in the industry).

SIEM combines security information management (SIM) and security event management (SEM). They provide analysis of security alerts generated by applications and network hardware. Vendors sell SIEM as software, as appliances, or as managed services; these products are also used for logging security data and generating reports for compliance purposes.

SOAR tools automate security incident management investigations via a workflow automation workbook. The cyber intelligence API enables the playbook to automate research related to the ticket (lookup potential phishing URL, suspicious hash, etc.). The first responder determines the criticality of the event. At this level, it is either a normal or an escalation event. SOAR includes security incident response platforms (SIRPs), Security orchestration and automation (SOA), and threat intelligence platforms (TIPs).

In summary, SIEM and SOAR are key pieces of a modern cybersecurity infrastructure. The capabilities, use cases, and architectures are different for every company.

SIEM and SOAR Vendors

In practice, many products in this area will mix these functions, so there will often be some overlap. Many commercial vendors also promote their own terminology.

The leaders in Gartner’s Magic Quadrant for SIEM 2021 are Exabeam, IBM, Securonix, Splunk, Rapid, LogRhythm. Elastic is a niche player for SIEM but very prevalent in Kafka architectures for other use cases.

The Gartner Market Guide for SOAR 2020 includes Anomali, Cyware, D3 Security, DFLabs, EclecticIQ, FireEye, Fortinet (CyberSponse), Honeycomb.

These are obviously not complete lists of SIEM and SOAR vendors. Even more complex: Gartner says, “SIEM vendors are adopting and acquiring/integrating SOAR solutions in their ecosystems”. Now, if you ask another research analyst or the vendors themselves, you will get even more different opinions

Hence, as always in the software business, do your own evaluation to solve your business problems. Instead of evaluating vendors, you might first check your pain points and capabilities that solve the problems. Capabilities include data aggregation, correlation, dashboards, alerting, compliance, forensic analysis to implement log analytics, threat detection, and incident management.

If you read some of the analyst reports or vendor websites/whitepapers, it becomes clear that the Kafka ecosystem also has many overlaps regarding capabilities.

The Challenge with SIEM / SOAR Platforms

SIEM and SOAR platforms provide you with various challenges:

• Proprietary forwarders can only send data to a single tool
• Data is locked from being shared
• Difficult to scale with growing data volumes
• High indexing costs of proprietary tools hinder wide adoption
• Filtering out noisy data is complex and slows response
• No one tool can support all security and SIEM / SOAR requirements

The consequence is a complex and expensive spaghetti architecture with different proprietary protocols:

The first post of this blog series explored why Kafka helps a central streaming backbone to avoid such a costly and complex spaghetti architecture. But let’s dig deeper into this discussion.

Kafka for SIEM and SOAR Modernization

A modern cybersecurity architecture has the following requirements:

• Real-time data access to all your security experts
• Historical and contextual data access for forensic reporting
• Rapid detection of vulnerabilities and malicious behavior
• Predictive modeling of security incidents using newer capabilities like ML/AI

Flexible Enterprise Architecture including Kafka, SIEM and SOAR:

Kafka is NOT a SIEM or SOAR. But it enables an open, real-time, and portable data architecture:

• Ingest diverse, voluminous, and high-velocity data at scale
• Scalable platform that grows with your data needs
• Reduce indexing costs and OPEX managing legacy SIEM
• Enable data portability to any SIEM tool or downstream application

The following diagram how the event streaming platform fits into the enterprise architecture together with SIEM/SOAR and other applications:

Here are the benefits of this approach for an enterprise architecture:

• Data integration just once, ingest and aggregate events from everywhere: Web / Mobile, SaaS, Applications, Datastores, PoS Systems, loT Sensors, Legacy Apps, and Systems, Machine data
• Data correlation independent from specific products: Join, enrich, transform and analyze data in real-time to convert raw events into clean, usable data, avoid the “shit in shit out” issue from raw data lake architectures
• Standardized, open, and elastic integration layer: Standardize schemas to ensure data compatibility to all downstream apps
• Long-term storage: Store and persist events in a highly available and scalable platform for real-time analytics and forensics
• Integration with one or more SIEM/SOAR tools
• Additional truly decoupled consumers (real-time, batch, request-response): Choose the right tool or technology for the job with a domain-driven design

Reference Architectures for Kafka-powered SIEM and SOAR Infrastructures

Kafka-powered enterprise architecture is open and flexible—no need to try a big bang change. Business units change solutions based on their current need. Often, a combination of different technologies and migration from legacy to modern tools is the consequence. Streaming ETL as ingestion and pre-processing layer for SIEM/SOAR is one use case for Kafka.

However, ingestion and preprocessing for SIEM/SOAR is a tiny fraction of what Kafka can do with the traffic. Other business applications, ML platforms, BI tools, etc., can also consume the data from the event-based central nervous system. At their own speed. With their own technology. That’s what makes Kafka so unique and successful.

This section shows a few reference architectures and migration scenarios for SIEM and SOAR deployments.

SIEM Modernization: Kafka and Elasticsearch

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine. Elasticsearch is developed in Java and dual-licensed under the source-available Server Side Public License and the Elastic license. Other parts fall under the proprietary (source-available) Elastic License.

The open-source Elasticsearch project is NOT a complete SIEM but often a key building block. That’s why it comes up in SIEM discussions regularly.

Most scalable Elasticsearch architectures leverage Kafka as the ingestion layer. Other tools like Logstash or Beam are okay for small deployments. But Kafka provides various advantages:

• Scalable: Start small, but scale up without changing the architecture
• Decoupling: Elastic is one sink, but the integration layer provides the events to every consumer
• Out-of-the-box integration: The Kafka Connect connector for Elastic is battle-tested and used in hundreds of companies across the globe.
• Fully managed: In the cloud, the complete integration pipeline including Kafka, Kafka Connect, and Elastic is truly serverless end-to-end.
• Backpressure handling: Elastic is not built for real-time ingestion but is a so-called slow consumer. Kafka handles the backpressure. Elastic indexes the events at its own speed, no matter how fast the data source produces events. Elastic Data Streams is actually improving this to provide more native streaming ingestion.

The following diagram shows the reference architecture for an end-to-end integration from data sources via Kafka into Elastic:

SIEM Modernization: Kafka and Splunk

Splunk provides proprietary software. It is a leading SIEM player in the market. Splunk makes machine data accessible across an organization by identifying data patterns, providing metrics, diagnosing problems, and providing intelligence for business operations. The biggest complaint I hear about Splunk regularly is the costly licensing model. Also, the core of Splunk (like almost every other SIEM) is based on batch processing.

The combination of Kafka and Splunk reduces costs significantly. Here is the Confluent reference architecture:

A prominent example of combining Confluent and Splunk is Intel’s Cyber Intelligence Platform (CIP), which I covered in part 3 of this blog series.

The above architecture shows an open, flexible architecture. Splunk provides several integration points. But at its core, Splunk uses the proprietary S2S (“Splunk-to-Splunk”) protocol. All universal forwarders (UF) broadcast directly to indexers or heavy forwarders (HF). The Confluent Splunk S2S Source Connector provides a way to integrate Splunk with Apache Kafka. The connector receives data from Splunk UFs.

This approach allows customers to cost-effectively & reliably read data from Splunk Universal Forwarders to Kafka. It enables users to forward data from universal forwarders into a Kafka topic to unlock the analytical capabilities of the data.

The direct S2S integration is beneficial if a company does not have Kafka out front and the data goes straight to Splunk indexers. To leverage Kafka, the connector “taps” into the Universal Forwarder infrastructure. Often, companies have 10,000s of UFs. If you are in the lucky situation of not having hundreds or thousands of Splunk UFs in place, then the regular Splunk sink connector for Kafka Connect might be sufficient for you.

The data is processed, filtered, aggregated in real-time at scale with Kafka-native tools such as Kafka Streams or ksqlDB. The processed data is ingested into Splunk; and potentially many other real-time or batch consumers that are completely decoupled from Splunk.

Using the right SIEM and SOAR for the Job

Most customers I talk to don’t use just one tool for solving their cybersecurity challenges. For that reason, Kafka is the perfect backbone for true decoupling and ingestion layer for different SIEM and SOAR tools:

Kafka’s commit log stores the incoming data. Each consumer consumes the data as it can. In this example, different SIEM tools consume the same events. Elasticsearch and Splunk consume the same raw data in different near real-time or batch processes. Both are completely independent in how they consume the events.

IBM QRadar cannot process high volumes of data. Hence, ksqlDB continuously preprocesses the raw data and puts it into a new Kafka topic. QRadar consumes the aggregated data.

Obviously, SOAR can consume data similarly. As discussed earlier in this article, the SOAR functionality could also be part of one of the SIEMs. However, then it would (have to) consume the data in real-time to provide true situational awareness.

Legacy SIEM Replacement and Hybrid Cloud Migration

In the field, I see several reasons to migrate workloads away from a deployed SIEM:

• Very high costs, usually with a throughput-based license model
• Scalability issues for the growing volumes of data in the enterprise
• Processing speed (batch) is not sufficient for real-time situational awareness and threat intelligence
• Migration from on-premise to (multi) cloud across regions and data centers

Groupon published an exciting success story: “We Replaced Splunk at 100TB Scale in 120 Days“:

The new platform leverages Kafka for high volume processing in real-time, migration, and backpressure handling. Elasticsearch provides reports, analytics, and dashboards.

The article also covers the repeating message that Logstash is not ideal for these kinds of workloads. Hence, the famous ELK stack with Elasticsearch, Logstash, and Kibana in most real-world deployments is actually an EKK stack with Elasticsearch, Kafka, and Kibana.

This story should make you aware that Logstash, FluentD, Sumo Logic, Cribl, and other log analytics platforms are built for exactly this use case. Kafka-native processing enables the same but also many other use cases.

Another key advantage of Kafka is the ability to operate as a resilient, hybrid migration pathway from on-premise to one or multiple clouds. Confluent can be deployed everywhere to coordinate log traffic across multiple data centers and cloud providers. I explored hybrid Kafka architectures in another blog post in detail.

Kafka-native SOAR: Cortex Data Lake from Palo Alto Networks (PANW)

I covered various SIEMs in this post. SOAR is a more modern concept than SIEM. Hence, the awareness and real-world deployments are still limited. I am glad that I found at least one public example I can show you in this post.

Cortex Data Lake is a Kafka-native SOAR that collects, transforms, and integrates enterprise security data at scale in real-time. Billions of messages pass through their Kafka clusters. Confluent Schema Registry enforces data governance. Palo Alto Networks (PANW) has multiple Kafka clusters in production with a size from 10 to just under 100 brokers each. Check out Palo Alto Networks’ engineering blog for more details.

Here is the architecture of Cortex Data Lake:

PANW’s design principles overlap significantly with the unique characteristics of Apache Kafka:

• Cloud agnostic infrastructure
• Massively scalable
• Aggressive ETA on integrations
• Schema versioning support
• Microservices architecture
• Operational efficiency

If you look at these design principles, it is obvious why the backbone of PANW’s product is Kafka.

Most Enterprises have more than one SIEM/SOAR

SIEM and SOAR are critical for every enterprise’s cybersecurity strategy. One SIEM/SOAR is typically not good enough. It is either not cost-efficient or includes scalability/performance issues.

Kafka-native SIEM/SOAR modernization is prevalent across industries. The central event-based backbone enables the integration with different SIEM/SOAR products. Other consumers (like ML platforms, BI tools, business applications) can also access the data. Innovative Kafka integrations like Confluent’s S2S connector enable the modernization of monolithic Splunk deployments and significantly reduce costs. Many next-generation SOARs such as PANW’s Cortex Data Lake are even based on top of Kafka.

Do you use any SIEM / SOAR? How and why do you use or plan to use the Kafka ecosystem together with these tools? How does your (future) architecture look like? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Kafka for Cybersecurity (Part 6 of 6) – SIEM / SOAR Modernization appeared first on Kai Waehner.

Blog series: Apache Kafka for Cybersecurity

This blog series explores why security features such as RBAC, encryption, and audit logs are only the foundation of a secure event streaming infrastructure. Learn about use cases,  architectures, and reference deployments for Kafka in the cybersecurity space:

• Part 1: Data in Motion as cybersecurity backbone
• Part 2: Situational awareness
• Part 3: Threat intelligence
• Part 4 (THIS POST): Forensics
• Part 5: Air-gapped and zero trust environments
• Part 6: SIEM / SOAR modernization

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Digital Forensics

Let’s start with the definition of the term “Digital Forensics”. In the IT world, we can define it as analytics of historical data sets to find insights. More specifically, digital forensics means:

• Application of science to criminal and civil laws, mainly during a criminal investigation.
• It is applied to internal corporate investigations in the private sector or, more generally, to intrusion investigations in the public and private sector (a specialist probe into the nature and extent of an unauthorized network intrusion).
• Forensic scientists collect, preserve, and analyze scientific evidence during the course of investigating digital media in a forensically sound manner.
• Identify, preserve, recover, analyze and present facts and opinions about digital information.

The technical aspect is divided into several sub-branches relating to the type of digital devices involved: Computer forensics, network forensics, forensic data analysis, and mobile device forensics.

A digital forensic investigation commonly consists of three stages: acquisition, analysis, and reporting. The final goal is to reconstruct digital events. Let’s see what role Kafka and its ecosystem play here.

Digital Forensics with Kafka’s Long Term Storage and Replayability

Kafka stores data in its distributed commit log. The log is durable and persists events on the disk with guaranteed order. The replication mechanism guarantees no data loss even if a node goes down. Exactly-once semantics (EOS) and other features enable transactional workloads. Hence, more and more deployments leverage Kafka as a database for long-term storage.

Forensics on Historical Events in the Kafka Log

The ordered historical events enable Kafka consumers to do digital forensics:

• Capture the complete attack vector
• Playback of an attack for the training of humans or machines
• Create threat surface simulations
• Compliance / regulatory processing
• Etc.

The forensics consumption is typically a batch process to consume all events from a specific timeframe. As all consumers are truly decoupled from each other, the “normal processing” can still happen in real-time. There is no performance impact due to the concepts of Kafka’s decoupling to enable a domain-driven design (DDD). The forensics teams use different tools to connect to Kafka. For instance, data scientists usually use the Kafka Python client to consume historical data.

Challenges with Long-Term Storage in Kafka

Storing data long-term in Kafka is possible since the beginning. Each Kafka topic gets a retention time. Many use cases use a retention time of a few hours or days as the data is only processed and stored in another system (like a database or data warehouse). However, more and more projects use a retention time of a few years or even -1 (= forever) for some Kafka topics (e.g., due to compliance reasons or to store transactional data).

The drawback of using Kafka for forensics is the huge volume of historical data and its related high cost and scalability issues. This gets pretty expensive as Kafka uses regular HDDs or SDDS as the disk storage. Additionally, data rebalancing between brokers (e.g., if a new broker is added to a cluster) takes a long time for huge volumes of data sets. Hence, rebalancing takes hours can impact scalability and reliability.

But there is a solution to these challenges: Tiered Storage.

Tiered Storage for Apache Kafka via KIP-405

Tiered Storage for Kafka separates compute and storage. This solves both problems described above:

• Significant cost reduction by using a much cheaper storage system.
• Much better scalability and elasticity as rebalancing is only needed for the brokers (that only store the small hot data sets)

KIP-405 is the assigned open-source task that describes the plan and process for adding Tiered Storage to Apache Kafka. Confluent is actively working on this with the open-source community. Uber is leading the initiative for this KIP and works on HDFS integration. Check out Uber’s Kafka Summit APAC talk about Tiered Storage for more details.

Confluent Tiered Storage for Kafka

Confluent Tiered Storage is generally available for quite some time in Confluent Platform and used under the hood in Confluent Cloud in thousands of Kafka clusters. Certified object stores include cloud object stores such as AWS S3 or Google Cloud Storage and on-premise object storage such as Pure Storage FlashBlade.

The architecture of Confluent Tiered Storage looks like this:

Benefits of Confluent Tiered Storage for Kafka include:

• Store data forever in a cost-efficient way using your favorite object storage (cloud and on-premise)
• The separation between computing and storage (hot storage attached to the brokers and cold storage via the cheap object store)
• Easy scale up/down as only the hot storage requires rebalancing – most deployments only store the last few hours in hot storage
• No breaking code changes in Kafka clients as it is the same regular Kafka API as before
• Battle-tested in Confluent Cloud in thousands of Kafka clusters
• No impact on performance for real-time consumers as these consume from page cache/memory anyway, not from the hot or cold storage

As you can see, Tiered Storage is a huge benefit to provide long-term storage for massive volumes of data. This allows rethinking your data lake strategy.

True Decoupling for SIEM, SOAR, and other Kafka Consumers

Kafka’s Distributed Commit Log captures the running history of signals. This

• enables true decoupling and domain-driven design
• absorbs velocity and volume to protect and stabilize slow consumers
• allows organic truncation via the right retention time per Kafka topic

Various producers continuously ingest new events into Kafka without knowing or caring about slow consumers. Kafka handles the backpressure. Different consumer applications use their own capable speed and communication paradigm for data ingestion:

Affordability at Scale for Real-Time and Replay

The Role of AI and Machine Learning in Digital Forensics

Digital Forensics is all about collecting, analyzing, and acting on historical events. SIEM / SOAR and other cybersecurity applications are great for many use cases. However, they are often not real-time and do not cover all scenarios. In an ideal world, you can act in real-time or even in a predictive way to prevent threats.

In the meantime, Kafka plays a huge role in AI / Machine Learning / Deep Learning infrastructures. A good primer to this topic is the post “Machine Learning and Real-Time Analytics in Apache Kafka Applications“. To be clear: Kafka and Machine Learning are different concepts and technologies. However, they are complementary and a great combination to build scalable real-time infrastructures for predicting attacks and other cyber-related activities.

The following sections show how machine learning and Kafka can be combined for model scoring and/or model training in forensics use cases.

Model Deployment with ksqlDB and TensorFlow

Analytics models enable predictions in real-time if they are deployed to a real-time scoring application. Kafka natively supports embedding models for real-time predictions at scale:

This example uses a trained TensorFlow model. A ksqlDB UDF embeds the model. Of course, Kafka can be combined with any AI technology. An analytic model is just a binary. No matter if you train it with an open-source framework, a cloud service, or a proprietary analytics suite.

Another option is to leverage a streaming model server to connect a deployed model to another streaming application via the Kafka protocol. Various model servers already provide a Kafka-native interface in addition to RPC interfaces such as HTTP or gRPC.

Kafka-native Model Training with TensorFlow I/O

Embedding a model into a Kafka application for low latency scoring and decoupling is an obvious approach. However, in the meantime, more and more companies also train models via direct consumption from the Kafka log:

Many AI products provide a native Kafka interface. For instance, TensorFlow I/O offers a Kafka plugin. There is no need for another data lake just for model training! The model training itself is still a batch job in most cases. That’s the beauty of Kafka: The heart is real-time, durable, and scalable. But the consumer can be anything: Real-time, near real-time, batch, request-response. Kafka truly decouples all consumers and all producers from each other.

We have built a demo project on Github that shows the native integration between Kafka and TensorFlow for model training and model scoring.

Kafka and Tiered Storage as Backbone for Forensics

Digital Forensics collects and analyzes historical digital information to find and present facts about criminal actions. The insights help to reconstruct digital events, find the threat actors, and build better situational awareness and threat detection in the future. This post showed what role Apache Kafka and its ecosystem play in digital forensics.

Often, Kafka is the integration pipeline that handles the backpressure for slow consumers such as SIEM / SOAR products. Additionally, the concept of Tiered Storage for Kafka enables long-term storage and digital forensics use cases. This can include Kafka-native model training. All these use cases are possible parallel to any unrelated real-time analytics workloads as Kafka truly decouples all producers and consumers from each other.

Do you use Kafka for forensics or any other long-term storage use cases? Does the architecture leverage Tiered Storage for Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Kafka for Cybersecurity (Part 4 of 6) – Digital Forensics appeared first on Kai Waehner.

Blog series: Apache Kafka for Cybersecurity

This blog series explores why security features such as RBAC, encryption, and audit logs are only the foundation of a secure event streaming infrastructure. Learn about use cases,  architectures, and reference deployments for Kafka in the cybersecurity space:

• Part 1: Data in Motion as cybersecurity backbone
• Part 2: Situational awareness
• Part 3 (THIS POST): Threat intelligence
• Part 4: Forensics
• Part 5: Air-gapped and zero trust environments
• Part 6: SIEM / SOAR modernization

Subscribe to my newsletter to get updates immediately after the publication. Besides, I will also update the above list with direct links to this blog series’s posts as soon as published.

Cyber Threat Intelligence

Threat intelligence, or cyber threat intelligence, reduces harm by improving decision-making before, during, and after cybersecurity incidents reducing operational mean time to recovery, and reducing adversary dwell time for information technology environments.

Threat intelligence is evidence-based knowledge, including context, mechanisms, indicators, implications, and action-oriented advice about an existing or emerging menace or hazard to assets. This intelligence can be used to inform decisions regarding the subject’s response to that menace or hazard.

Threat intelligence solutions gather raw data about emerging or existing threat actors & threats from various sources. This data is then analyzed and filtered to produce threat intel feeds and management reports that contain information that automated security control solutions can use.

Threat intelligence keeps organizations informed of the risks of advanced persistent threats, zero-day threats and exploits, and how to protect against them.

Situational Awareness is Not Enough…

… but the foundation to collect and pre-process data in real-time at scale. Only real-time situational awareness enables real-time threat intelligence to provide huge benefits to the enterprise:

• Mitigate harmful events in cyberspace
• Proactive cybersecurity posture that is predictive, not just reactive
• Bolster overall risk management policies
• Improved detection of threats
• Better decision-making during and following the detection of a cyber intrusion

In summary, threat intelligence allows to:

• See the whole board. And see it more quickly.
• See around corners.
• See the enemy before they see you.

Threat Intelligence for Prevention or Mitigation across the Cyber Kill Chain

Threat intelligence is the knowledge that allows you to prevent or mitigate cyberattacks. It covers all the phases of the so-called “Cyber Kill Chain“:

Threat intelligence provides several benefits:

• Empowers organizations to develop a proactive cybersecurity posture and to bolster overall risk management policies
• Drives momentum toward a cybersecurity posture that is predictive, not just reactive
• Enables improved detection of threats
• Informs better decision-making during and following the detection of a cyber intrusion

Transactional Data vs. Analytics Data

Most use cases around data-in-motion are about all the data. This is true for all transactional use cases and even for many analytical use cases. Each event is valuable: A sale, an order, a payment, an alert from a machine, etc.

However, data is often full of noise. As I discussed earlier in this blog series, the goal in the cybersecurity space is to find the needle in the haystack and to reduce false-positive alerts.

SIEM, SOAR, OT, and ICS are almost always analytic processing regimes, BUT knowing when they are not is important. Kafka can configure topics to be tuned for transactions or analytics.  That is unprecedented in the history of data processing. Threat intelligence (= awareness-in-motion) assumes the PATTERN is valuable, not the data.

Analytics in Motion powered by Kafka Streams / ksqlDB

As you can hopefully imagine from the above requirements and characteristics, event streaming with Apache Kafka and its streaming analytics ecosystem is a perfect fit for the technical infrastructure for threat intelligence.

Threat detection makes sense of the signal and the noise of the data by continuously processing signatures. This enables to detect, contain and neutralize threats proactively:

Analytics can be many things in such a scenario:

• Simple business logic such as stateless filtering or stateful aggregations
• Complex business rules with custom code or with an integrated separate rules engine like Drools
• Machine Learning / Deep Learning by embedding any analytic model (TensorFlow, H2O, DataRobot, whatever) into the Kafka streaming application
• Information from external systems applied in real-time in the right context

On a high level, the advantages of using Kafka Streams or ksqlDB for threat intelligence can be described as follows:

• A single scalable and reliable real-time infrastructure for end-to-end data integration and data processing
• Flexibility to write custom rules and embed other rules engines, frameworks, or trained models
• Integration with other threat detection systems like IDS, SIEM, SOAR

The business logic for cyber threat detection looks different for every use case. Known attack patterns like MITRE ATT&ACK help with the implementation. However, situational awareness and threat detection also need to detect unknown anomalies.

Let’s now take a look at a concrete example.

Intel’s Cyber Intelligence Platform

Let me quote Intel themselves:

“As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. At Intel, we’ve addressed this need by implementing a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Confluent. We believe that CIP positions us for the best defense against cyber threats well into the future.

Our CIP ingests tens of terabytes of data each day and transforms it into actionable insights through streams processing, context-smart applications, and advanced analytics techniques. Kafka serves as a massive data pipeline within the platform. It provides us the ability to operate on data in-stream, enabling us to reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately leads to better prevention.”

Let’s explore Intel’s CIP for threat intelligence in more detail.

Detecting Vulnerabilities with Stream Processing

Intel’s CIP leverages the whole Kafka ecosystem provided by Confluent:

• Ingestion: Kafka producer clients for various sources such as databases, scanning engines, IP address management, asset management inventory, etc.
• Streaming analytics: Kafka Streams for filtering vulnerabilities by business unit, joining asset ownership with vulnerable assets, etc.
• Egress: Kafka Connect sink connectors for data lakes, IT partners, other business units, SIEM, SOAR, etc.
• High availability: Multi-Region Clusters (MRC) for high availability across regions
• And much more…

Here is a high-level architecture:

Intel’s Kafka Maturity Timeline

Building a cybersecurity infrastructure is not a big bang. A step-by-step approach starts with integrating the first sources and sinks, some simple stream processing, and deployment as a pilot project. Over time, more and more data sources and sinks are added, the business logic gets more powerful, and the scale increases.

Intel’s Kafka maturity timeline shows their learning curve:

Kafka Benefits to Intel

Intel describes their benefits for leveraging event streaming as follows:

• Economies of scale
• Operate on data in the stream
• Reduce technical debt and downstream costs
• Generates contextually rich data
• Global scale and reach
• Always on
• Modern architecture with a thriving community
• Kafka leadership through Confluent expertise

That’s pretty much the same reasons I use in many of my other blog posts to explain the rise of data in motion powered by Apache Kafka across industries and use cases…

For more intel on Intel’s Cyber Intelligence Platform powered by Confluent and Splunk, check out their whitepaper and Kafka Summit talk.

Scalable Real-time Cyber Threat Intelligence with Kafka

Kafka is not just used as a backbone and source of truth for data. It also monitors, correlates, and proactively acts on events from various real-time and batch data sources to implement cyber threat intelligence.

The Cyber Intelligence Platform from Intel is a great example of a Kafka-powered cybersecurity solution. It leverages the whole Kafka ecosystem to build a scalable and reliable real-time integration and processing layer. The streaming analytics logic depends on the use case. It can cover simple business logic but also external rules engines or analytic models.

How do you fight against cybersecurity risks? What technologies and architectures do you use to implement cyber threat intelligence? How does Kafka complement other tools? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Kafka for Cybersecurity (Part 3 of 6) – Cyber Threat Intelligence appeared first on Kai Waehner.

Blog series: Apache Kafka for Cybersecurity

This blog series explores why security features such as RBAC, encryption, and audit logs are only the foundation of a secure event streaming infrastructure. Learn about use cases,  architectures, and reference deployments for Kafka in the cybersecurity space:

• Part 1 (THIS POST): Data in Motion as cybersecurity backbone
• Part 2: Situational awareness
• Part 3: Threat intelligence
• Part 4: Forensics
• Part 5: Air-gapped and zero trust environments
• Part 6: SIEM / SOAR modernization

Why should you care about Cybersecurity?

Cybersecurity is the protection of computer systems and networks from information disclosure, theft of, or damage to their hardware, software, or electronic data, as well as from the disruption or misdirection of the services they provide.

The field is becoming increasingly significant due to the increased reliance on computer systems, the internet, and wireless network standards such as Bluetooth and Wi-Fi, and due to the growth of “smart” devices, including smartphones, televisions, and the various devices that constitute the “Internet of Things”. Owing to its complexity, cybersecurity is also one of the major challenges in the contemporary world in terms of politics and technology.

Various actors can be involved in cybersecurity attacks. This includes web scraping, hackers, criminals, terrorists, state-sponsored and state-initiated actors.

Examples of recent successful attacks

Most successful attacks have a financial and brand impact. However, it depends on the organization and the kind of attack or data breach. Here are a few recent examples quoted from news articles that you have probably heard of in the tv, newspaper, or internet:

• 533 million Facebook users’ phone numbers and personal data have been leaked online
• 500 Million LinkedIn Users’ Data Were Allegedly Hacked A Tale O
• A Tale of Two Hacks: From SolarWinds to Microsoft Exchange Ransomware
• Ransomware attack shuts down biggest U.S. gasoline pipeline

Privacy, safety, and cost are huge factors in these successful attacks. The craziest part is that most of the successful cyber attacks don’t even get public as companies prefer to keep them secret.

Supply Chain Attacks

You are not even safe if your own infrastructure is secure. A supply chain attack is a cyber-attack that seeks to damage an organization by targeting less-secure elements in the supply chain.

A supply chain attack can occur in any industry, from the financial, oil, or government sectors.  For instance, cybercriminals tamper with the manufacturing process by installing a rootkit or hardware-based spying components.

The supply chain involves hardware, software, and humans.

Norton Rose Fulbright shows how a supply chain attack looks like:

A well-known example of a supply chain attack is the Experian breach, where the data of millions of T-Mobile customers was exposed by the world’s biggest consumer credit monitoring firm.

The SolarWinds breach is another famous example. SolarWinds’ network management system has over 300,000 customers. Many of them are heavy hitters, much of the US Federal government, including the Department of Defense, 425 of the US Fortune 500, and many customers worldwide.

Impact of Cybersecurity attacks

“It takes 20 years to build a reputation and few minutes of cyber-incident to ruin it.” (Stephane Nappo)

Security attacks are exploding with high costs. The average cost of a data breach is $3.86 MILLION. It takes 280 DAYS on average time to identify and contain a breach. Additionally, the brand impact is huge but very hard to quantify.

Cybersecurity as a key piece of the security landscape

As you can see in the above examples: The threat is real!

The digital transformation requires IT capabilities, even in the OT world. Internet of Things (IoT), Industrial IoT (IIoT), Industry 4.0 (I40), connected vehicles, smart city, social networks, and similar game-changing trends are only possible with modern technology: Networking, communication, connectivity, open standards, ”always-on”, billions of devices, and so on.

The security landscape gets more and more important. And Cybersecurity is a key piece of that:

The other security components are very relevant and complimentary, of course. But let’s also talk about some challenges:

• Access control: Complex and error-prone
• Encryption: Very important in most cases, sometimes not needed (e.g., in DMZ / air-gapped environments)
• Hardware security: No help against insiders
• OT security: Avoid risk (change operations) vs. transfer some risk (buy insurance)

Therefore, in addition to the above security factors, each organization requires good cybersecurity infrastructure (including SIEM and SOAR components).

Continuous real-time data correlation across various data sources is mandatory for a good cybersecurity strategy to have a holistic view and understanding of all the events and potential abuses that are taking place. This is a combination of data collection of different activities happening on critical networks followed by data correlation in real-time (so-called stream processing/streaming analytics).

Cybersecurity challenges – The threat is real!

Plenty of different attacks exist: Stealing intellectual property (IP), Denial-of-service attacks (DDoS), ransomware, wiperware, and so on. WannaCry, NotPetya, and SolarWinds are a few famous examples of successful and impactful cyberattacks. The damage is often billions of dollars.

The key challenge for cybersecurity experts: Find the Needle(s) in the Haystack.

Systems need to detect true positives in real-time automatically. This includes capabilities such as:

• Threat detection
• Intrusion prevention
• Anomaly detection
• Compliance auditing
• Proactive response

The haystack is typically huge, i.e., massive volumes of data. Often, it is not just one haystack but many. Hence, a key task is to reduce false positives. This is possible via:

• Automation
• Process big volumes of data in real-time
• Integration of all sources
• No ‘ignore’ on certain events
• Creation of filters and correlated event rules
• Improve signal-to-noise ratio (SNR)
• Correlate a “collection of needles” into a “signature needle”

TL;DR: Correlate massive volumes of data and act in real-time to threats. That’s why Kafka comes into play…

Kafka and Cybersecurity

Real-time data beats slow data. That’s true (almost) everywhere and the main reason for Kafka’s success and huge adoption across use cases and industries. Real-time sensor diagnostics and track&trace in transportation, instant payment and trade processing in banking, real-time inventory and personalized offers in retail, that’s just a few of the examples.

But: Real-time data beats slow data in the (cyber)security space, too. Actually, it is even more critical here. A few examples:

• Security: Access control and encryption, regulatory compliance, rules engine, security monitoring, surveillance.
• Cybersecurity: Risk classification, threat detection, intrusion detection, incident response, fraud detection

The main goal in cybersecurity is to reduce the Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately leads to better prevention. For this reason, modern enterprise architectures adopt Apache Kafka and its ecosystem as the backbone for cybersecurity:

Kafka and its ecosystem provide low-latency performance at high throughput in conjunction with data integration and data processing capabilities. That’s exactly what you need for most cybersecurity systems.

Having said this, let’s be clear:

Kafka was NOT built for Cybersecurity

From a technology perspective, cybersecurity includes the following features, products, and services:

• SIEM / SOAR
• Situational Awareness
• Operational Awareness
• Intrusion Detection
• Signals and Noise
• Signature Detection
• Incident Response
• Threat Hunting & Intelligence
• Vulnerability Management
• Digital Forensics
• …

Kafka is an event streaming platform built to process data in motion, not to solve cybersecurity issues. So, why are we talking about Kafka in this context then?

Most existing cybersecurity platforms contain the same characteristics: Batch, proprietary, inflexible, not scalable, expensive. Think about this with your favorite SIEM in mind.

Kafka as the backbone for Cybersecurity

Kafka has different characteristics: Real-time, open, flexible, scalable, cost-efficient. Hence, Kafka is the ideal backbone for a next-generation cybersecurity infrastructure. This enables the following capabilities:

• Integrate with all legacy and modern interfaces
• Record, filter, curate a broad set of traffic streams
• Let analytic sinks consume just the right amount of data
• Drastically reduce the complexity of the enterprise architectures
• Drastically reduce the cost of SIEM / SOAR deployments
• Add new analytics engines
• Add stream-speed detection and response at scale in real-time
• Add mission-critical (non-) security-related applications

Kafka complements SIEM/SOAR and other Cybersecurity and Network Monitoring Tools

Every enterprise is different… Flexibility is key for your cybersecurity initiative! That’s why I see many customers adopting Confluent as an independent but enterprise-grade and hybrid foundation for the cybersecurity enterprise architecture.

Kafka or Confluent do not replace but complement other security products such as IBM QRadar, HP ArcSight, or Splunk. The same is true for other network security monitoring and cybersecurity tools.

High Velocity and (ridiculous) volume of Netflow / PCAP data are processed via tools such as Zeek / Corelight. Open-source frameworks such as TensorFlow or AutoML products such as DataRobot provide modern analytics (machine learning/deep learning) to enhance the intrusion detection systems (IDS) to respond to incidents proactively.

Kafka provides a flexible and scalable real-time backplane for the cybersecurity platform. Its storage capabilities truly decouple different systems. For instance, Zeek handles the incoming ridiculous volume of PCAP data before Kafka handles the backpressure for slow batch consumers such as Splunk or Elasticsearch:

In the real world, there is NOT just one SIEM, SOAR, or IDS in the enterprise architecture. Different applications solve different problems. Kafka shines due to its true decoupling while still providing real-time consumption for consumers that can handle it.

Cybersecurity is required everywhere

Running a cybersecurity suite in one location is not sufficient. Of course, it depends on the enterprise architecture, use cases, and many other factors. But most companies have hybrid deployments, multi-cloud, and/or edge scenarios.

Therefore, many companies choose Confluent to deploy event streaming everywhere, including uni- or bi-directional integration, edge aggregation setups, and air-gapped environments. This way, one open and flexible template architecture enables streaming applications and real-time cybersecurity everywhere.

Here is an example of an end-to-end cybersecurity infrastructure leveraging serverless Kafka in the cloud (Confluent Cloud) and self-managed, cloud-native Kafka (Confluent Platform) on-premise, connected Kafka at the edge (a Confluent cluster on the ships), and disconnected edge (a single broker in the drone):

Let’s now take a look at a real-world example for a Kafka-powered cybersecurity platform.

Crowdstrike’s Kafka backbone for cybersecurity

Crowdstrike is a cybersecurity cloud solution for endpoint security, threat intelligence, and cyberattack response services. Kafka is the backbone of their infrastructure. They ingest ~5 trillion events per week into the cloud platform.

A cybersecurity platform needs to be up and responsive 24/7. It must be available, operational, reliable, and maintainable all the time. Crowdstrike defined four critical roles for operating their streaming data infrastructure: Observability, Availability, Operability, Data Quality.

Check out Crowdstrike’s tech blog to learn more about their Kafka infrastructure.

Kafka is (not) all you need for cybersecurity

This introductory post explored the basics of cybersecurity and how it relates respectively why it requires data in motion powered by Apache Kafka. The rest of the series will go deeper into specific topics that partly rely on each other.

Threat intelligence is only possible with situational awareness. Forensics is complementary. Deployments differ depending on security, safety, and compliance requirements.

I will also give a few more concrete Kafka-powered examples and discuss a few success stories for some of these topics. Last but not least, I will show different reference architectures where Kafka complements existing tools such as Zeek or Splunk within the enterprise architecture.

How do you solve cybersecurity risks? What technologies and architectures do you use? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Kafka for Cybersecurity (Part 1 of 6) – Data in Motion as Backbone appeared first on Kai Waehner.