Unified batch and streaming: traditional vs. Deephaven

Unified API eliminates complexity of separate batch and real-time systems

Traditional: Complex Multi-System

Batch Layer

• Historical data

• Parquet, HDFS

• Spark/Hadoop

• Hours/days latency

Speed Layer

• Real-time streams

• Kafka, Flink

• Storm, Samza

• ms-second latency

Serving Layer

• Merges batch + real-time views

• Complex data reconciliation

• Different APIs for each layer

• Manual consistency management

Challenges:

• Multiple systems to learn & maintain

• Different APIs for batch vs streaming

• Complex data reconciliation logic

• Infrastructure overhead

Deephaven: Unified Single System

Deephaven Query Engine

Batch Data

• Parquet, CSV

• Historical tables

• Static sources

Real-time Data

• Kafka streams

• Live tables

• Ticking sources

Unified DAG-based Update Model

Same API • Same semantics • Automatic consistency

Single Table API

.where() .aggBy() .join() - works on all data

Benefits:

• One system, one API to learn

• Seamless batch + real-time integration

• Automatic consistency via DAG

• Lower operational complexity

Code example: same API for both

Groovy code works identically for batch and real-time data:

Batch (Historical):

historical = readCsv("trades_2024.csv")

result = historical.where("Price > 100")

.aggBy([AggAvg("Price")], "Symbol")

Real-time (Live):

live = KafkaTools.consumeToTable(kafkaProps, "trades",

KafkaTools.ALL_PARTITIONS, KafkaTools.ALL_PARTITIONS_DONT_SEEK,

KafkaTools.Consume.IGNORE, valueSpec, KafkaTools.TableType.append())

result = live.where("Price > 100")

.aggBy([AggAvg("Price")], "Symbol")

Identical operations! The only difference is the data source.

Both use the same DAG, both maintain consistency, both support the same table operations.