Deephaven architecture overview

From Python/Java code to real-time web UIs

Deephaven query engine (Java)

Directed Acyclic Graph (DAG)

Tables and operations form a dependency graph with automatic update propagation

Source

Filter

Join

• Incremental updates

• Logical clock for consistency

• Garbage collection

Update Graph (UG) Cycles:

Batch changes per cycle (1000ms target, configurable) →

Propagate in topological order →

Notify dependents →

UI updates

Table Data Structures

RowSet

Sparse 64-bit keys

RSP / Roaring Bitmaps

ColumnSources

Column data by row key

Shared across tables

Zero-copy optimization: filters/views share ColumnSources → 50-90% memory savings

Chunk-Oriented Processing

Chunks (4096 elements)

Bulk data transfer: getChunk(), fillChunk()

Enables SIMD vectorization, reduces method calls 1000x

Performance: 1M rows filtered with ~244 calls vs 1M individual calls

Formula Engine

JavaParser → Compiled classes

Numba for Python expressions

Parameter substitution & reuse

Table Operations

Joins • Aggregations • Sorts

Columnar hash tables

Incremental algorithms

Data Sources

Parquet • Kafka • CSV

Streaming & Batch

Unified API for both

Key performance characteristics

Incremental updates: only the rows a change affects are recomputed

Shared structures: 50-90% memory reduction for filtered/joined views

Chunk processing: ~1000x fewer method calls, 4-8x SIMD speedup

Language Integration

JPY (Python ↔ Java)

JNI bridge, minimal crossings

Native Java API

Direct engine access

Network Protocols

gRPC / Arrow

Flight service

Barrage

Ticking data

WebSocket

Real-time updates

deephaven.ui (Python UI)

Pure Python components

@ui.component decorator

Server-side rendering

→ React in browser

No JavaScript required

JavaScript / TypeScript Client

web-client-ui components

Direct gRPC-Web

Grid, charts, dashboards

Custom integration

Maximum flexibility

Both approaches leverage same DAG engine, gRPC protocols, and real-time capabilities