Real-Time Data Streaming in 2026: When It Pays and Which Platform to Choose

Data streaming means moving each event, such as a payment, a click, a sensor reading or a row changed in a database, to the systems that need it within seconds of it happening, instead of collecting events and moving them on a schedule. Most streaming estates are built on a log: Apache Kafka or a service that speaks its protocol, where events are kept in order, can be read by many consumers and can be replayed. A stream processor such as Apache Flink or Spark then filters, joins and aggregates the events as they arrive. The question for a business is rarely whether streaming works. It is which decisions are worth the extra cost and operational care, and which platform carries them most cheaply.

• ~1%of items Instacart scores in real time, a split that cut the compute for availability scoring by about 80%¹
• $0.02what AWS charges for each GiB of data that crosses an availability zone, the line that dominates busy Kafka bills⁷
• $11Bthe enterprise value at which IBM bought Confluent, closed on March 17, 2026⁵

Where streaming earns its cost

The documented returns come from decisions that cannot wait for the next batch because money or a customer is already in motion. Instacart moved fraud features from batch to real time so it could act before a loss occurs, which it says "directly reduces millions of fraud-related costs annually", and moved item availability scores from hours old to seconds old.¹² DoorDash scaled its Kafka and Flink event platform to hundreds of billions of events a day with four nines of delivery, and cut the delay before events reach its Snowflake warehouse from a day to a few minutes.¹³ Stripe says its Radar fraud models, trained on more than 70 trillion data points, reduce fraud by 32% on average.¹⁴

Payments regulation is the most durable driver, because it moves fraud losses onto the payment provider. Since October 9, 2025, euro-area payment providers must verify the payee before any credit transfer, with other member states following by July 9, 2027.¹⁵ Payment fraud across the European Economic Area reached €4.2 billion in 2024, and payment users bore about 85% of credit transfer fraud losses.¹⁶ In the UK, mandatory reimbursement of scam victims returned £112 million in its first nine months, 88% of the money lost.¹⁷ FedNow had 1,725 participating banks and credit unions by the first quarter of 2026.¹⁸ None of these rules sets a screening latency in milliseconds; the pressure comes through liability, and a payee check is a synchronous lookup that needs a fast database more than a streaming platform.

Survey figures are best read as sentiment. In Confluent's 2025 survey of 4,175 IT leaders, 44% reported a fivefold return on data streaming.¹⁹ Its 2026 edition, with 4,625 respondents, dropped the return figure and led instead with 72% saying weak real-time infrastructure is stalling AI, 66% unsure of the lineage, timeliness or quality of their data, and only 32% with agentic AI in production.²⁰ Both come from a company that sells streaming, and we found no independent survey of streaming returns.

How fresh each decision needs its data

The companies with the best evidence tier their freshness. Instacart scores every item daily, the trending items hourly, and only about 1% of items in real time, and reports that this cut compute for the model by about 80% while improving its online metrics.¹ An InfoQ case study from May 2026 describes an ads index pipeline that tried per-record streaming, found it left the index in partial-update states, and settled on Spark micro-batches every thirty seconds. Its worst-case delay fell from about ten minutes to thirty seconds, and the gain came from removing scheduling gaps.²

AI agents are the newest argument for streaming everything, and the evidence supports a narrower claim. A peer-reviewed benchmark found that outdated retrieved information "substantially reduces response accuracy" in retrieval-augmented generation.²¹ A September 2026 preprint found that the refresh schedule, more than the age of a cache, governs how stale an agent's answers become.²² The practical rule is to refresh each source at the rate it changes.

What changed in the platform market in 2025 and 2026

The open-source core changed more in eighteen months than in the five years before. Apache Kafka 4.0, released on March 18, 2025, is the first major release to run entirely without ZooKeeper, and it made the new consumer rebalance protocol generally available.³ Kafka 4.2, in February 2026, made queues (share groups) production-ready, so Kafka can now hand work items to competing consumers as well as keep an ordered log.⁴ Diskless topics, which keep data in object storage to avoid paying for traffic between zones, were accepted as a design direction in KIP-1150,⁷ and as of October 2026 no Apache Kafka release ships them.²³ Buying object-storage Kafka today means buying it from a vendor.

The commercial map consolidated. IBM completed its purchase of Confluent on March 17, 2026, at $31 a share in cash, an enterprise value of about $11 billion, describing Confluent as used by more than 6,500 enterprises including 40% of the Fortune 500.⁵ Confluent had bought WarpStream, an object-storage Kafka engine, in 2024,²⁴ so that also now sits with IBM. CoreWeave bought Bufstream from Buf on May 8, 2026.²⁵ The independents responded: Redpanda added object-storage Cloud Topics in March 2026, claiming over 90% savings on cross-zone networking fees,²⁶ Aiven moved Kafka to consumption pricing in August 2026,²⁷ and Aiven, Confluent, Redpanda, StreamNative and Ververica formed a Streamhouse Working Group in September 2026.²⁸ Google's low-cost Pub/Sub Lite has been closed to new customers since September 24, 2024.²⁹

What streaming costs

List prices differ by more than ten times for the same byte, and they only compare through a workload model, because each vendor splits fixed fees, compute, throughput and storage differently. These are the published US list rates we read on October 6, 2026.

On AWS and Google Cloud the biggest line on a busy cluster is usually data moving between availability zones, more than the brokers themselves. AWS charges $0.02 for each GiB that crosses a zone, Google Cloud charges $0.01 per GiB for replication between zones, and Azure does not charge for it.⁷ Google's own pricing page states that "for clusters with utilization above 20%, the cost of data transfer between zones is the largest component of the total cost".⁶ By our arithmetic from AWS list rates, a self-run cluster with three replicas spread over three zones and one consumer group pays about $170 a month in cross-zone fees for every megabyte per second produced, or about $200,000 a year at 100 MB/s before any broker or disk. Amazon MSK does not charge for replication between its brokers, which removes the largest share of that.³⁰

Google's worked example is the cleanest comparison of managed against self-run: at 100 MiB/s, Kafka on Compute Engine comes to about $9,100 a month and the managed service about $11,000.⁶ The larger savings figures come from vendors selling the fix. WarpStream's founders wrote that inter-zone fees are "70-90% of the cost" of a substantial Kafka workload,³⁵ and Confluent's own 2025 cost comparison lists operations at about $566,000 a month in a column whose total reads about $19,300, so it should not be reused without correction.³⁶ Staffing is the other large cost, and the only published estimates come from vendors: Confluent assumes two full-time engineers for a small production estate and seven to ten at the scale of Lyft's streaming team.³⁷

The cost levers follow from those rates: zone-aware producers and follower fetching so clients read from their own zone, a service that does not bill replication or an object-storage tier for data that can tolerate higher latency, compression, lower replication for data that can be rebuilt such as logs and telemetry, and tiered storage for long retention.

Processing, change data capture and the lakehouse

Apache Flink remains the default open-source engine for stateful, low-latency processing. Flink 2.0, released in March 2025, added a state backend that keeps state in remote storage and, in the project's own tests on heavy queries, ran at 75% to 120% of the throughput of local state; it also removed the DataSet API and dropped Java 8.³⁸ Flink 2.3, in June 2026, added control over where a materialized table resumes when its query changes.³⁹ Managed Flink costs $0.21 per CFU-hour on Confluent Cloud⁴⁰ and $0.11 per KPU-hour on AWS, plus one extra KPU per application for orchestration.⁴¹ Confluent recommends Flink for new work and keeps ksqlDB fully supported for existing applications.⁴²

Spark now competes for low-latency work. Open-source Spark 4.1 shipped a real-time mode for continuous, sub-second processing,⁴³ and Spark 4.2 added real-time support in PySpark and a SQL CHANGES clause for reading change data.⁴⁴ Databricks reports p99 latencies as low as single-digit milliseconds on its platform, a vendor figure.⁴⁵ Whatever the engine, the guarantees deserve testing: a peer-reviewed study that injected faults into Kafka Streams, Storm and Flink found duplicate events in the first outputs after Flink restarted, and that joins "often result in unreliable outputs".⁴⁶

Change data capture, reading changes from a database's own log, is how most operational systems join the stream. Debezium, the main open-source tool, reached version 3.7 in September 2026.⁴⁷ Its PostgreSQL documentation warns that disk space used by write-ahead log files can spike while a connector falls behind,⁴⁸ so replication slot lag belongs on the first page of alerts. Managed tools read the same log on a schedule: Fivetran's one-minute syncs are limited to its Enterprise and higher plans.⁴⁹

If the only consumer is a warehouse or lakehouse, a broker has become optional. Confluent's Tableflow turns Kafka topics into Apache Iceberg tables and became generally available on March 19, 2025,⁵⁰ and Redpanda's Iceberg Topics did the same in its 25.1 release.⁵¹ Snowflake has billed Snowpipe at a flat 0.0037 credits per GB since December 2025,⁵² and says its Snowpipe Streaming elastic channels make data queryable in as little as five seconds.¹⁰ Databricks' Zerobus Ingest, generally available since February 2026, streams into the lakehouse in under five seconds, by Databricks' own figures.¹¹ Real-time analytics on a lakehouse means seconds to minutes.

What breaks in production

Several of Kafka's durability defaults still sit on the unsafe side. In the Kafka 4.3 broker configuration, min.insync.replicas defaults to 1, default.replication.factor to 1 and automatic topic creation to true; unclean leader election, by contrast, defaults to false.⁸ A producer waiting for every in-sync replica therefore loses nothing only when topics have three replicas and at least two must confirm a write. Exactly-once processing is narrower than it sounds: Flink's documentation warns that if the Kafka transaction timeout is shorter than the longest checkpoint plus restart, "data loss may happen".⁵³ The safer design makes every sink idempotent, keyed on an event ID.

The public incident record points at clients and capacity more than at the log itself. At PagerDuty on August 28, 2025, a new feature created a Kafka producer for every API request, nearly 4.2 million extra producers an hour or 84 times normal; about 95% of events were rejected at the peak, and about 23% of notifications were delayed by five minutes or more over 209 minutes.⁵⁴ Inngest lost span data in October 2025 when one Kafka cluster's disks filled up.⁵⁵ An Amazon Kinesis Data Streams event in US-EAST-1 on July 30, 2024 caused errors across AWS services for almost seven hours.⁵⁶ Cloudflare found that consumers can stay connected and pass health checks while they have stopped making progress, and moved to checks on committed offsets.⁵⁷

Security has to be switched on. Kafka's documentation states that "security is optional", and it describes encryption in transit, with no native encryption at rest.⁹ The most serious recent flaw, CVE-2026-33557, let brokers on Kafka 4.1.0 to 4.1.1 configured for OAuth accept any JWT token without validating its signature, issuer or audience; earlier 2025 CVEs abused client login and JAAS settings.⁵⁸ Monitoring standards are still settling: OpenTelemetry's messaging conventions remain at Development status in version 1.44.0.⁵⁹ A useful dashboard covers consumer lag in records and seconds, committed-offset progress, rebalance counts, producer counts, broker heap and disk, partitions below their in-sync minimum, and an end-to-end latency probe.

Contracts, schemas and personal data

A stream other teams depend on is an interface and needs the same care as an API. A schema registry rejects incompatible changes; Confluent's defaults to backward compatibility, so consumers upgrade before producers.⁶⁰ AsyncAPI documents event interfaces the way OpenAPI documents request-response ones, and its 3.1.0 release in January 2026 kept the 3.0 model.⁶¹ A data contract adds the owner, quality rules and freshness target for each stream. Confluent's 2026 survey found 66% of IT leaders unsure of the lineage, timeliness or quality of their data,²⁰ which is the problem contracts address.

Personal data in a replayable log is the governance problem with no settled answer. Under UK GDPR an organization has one month to respond to an erasure request, and data that cannot be overwritten at once, such as backups, must be put "beyond use" until it is.⁶² No regulator has published guidance written for event streams. Retention settings also behave less simply than they look: Datadog found a broker keeping messages well beyond a topic's 36-hour retention because retention applies to closed log segments.⁶³ The defensible design keeps personal data in separate topics with short retention, sends pseudonymized events to long-lived analytics streams, and sets segment size and time so retention means what the policy says.

How to choose a platform

Start by asking whether you need a log at all. A log earns its place when several independent consumers need the same events in order, with replay. For work items that one worker must handle once, a queue is simpler: PostgreSQL's SKIP LOCKED is documented for avoiding lock contention on a queue-like table,⁶⁴ and Kafka itself now offers queues. To keep a service's database and its events consistent, the standard pattern is a transactional outbox published by change data capture.⁶⁵

Migrations take careful preparation and short downtime windows. Michelin moved a Kafka cluster of 15 brokers, more than 1,500 topics and 5 TB used by more than 25 teams to a managed service, and moved its largest application with under two hours of downtime after extensive preparation. It did not use Confluent's Cluster Linking, which was in preview at the time and did not fit its security constraints.⁶⁶

The streaming market of 2026 has conceded much of its old pitch. The same vendors that sold real-time as a replacement for batch now sell slower object-storage tiers, tables with a freshness setting and direct ingestion into the lakehouse, because most data does not need milliseconds. The companies that get value stream the decisions that cannot wait, schedule the rest, and run each stream as a product with an owner, a contract and a retention policy.

This is how we approach streaming in our data engineering work: write down a freshness target for each decision, stream only the slice that needs it, price the options on real traffic including the bytes that cross zones, and set safe defaults, idempotent sinks, contracts and alerts before the first consumer goes live.