lakestream/kafka repository overview

⁠Ursa for Apache Kafka (UFK)

A Kafka distribution with opt-in diskless topics on object storage — and a one-pass path from topic to Iceberg table.

Built on Apache Kafka · Apache-2.0 · openlakestream/kafka⁠

docker pull lakestream/kafka:latest

Each image is built from an Apache Kafka release, and latest always points to the most recent build. See the Tags tab if you need to pin a specific Kafka version.

⁠What it is

UFK is Apache Kafka plus a storage option. Diskless topics keep their records on object storage through Ursa; every other topic stays Kafka. Nothing changes until you turn it on — ursa.storage.enable defaults to false, so a stock configuration behaves exactly like the Apache Kafka release it is built from.

• Diskless topics — durability comes from S3 / GCS / Azure Blob instead of broker disks. Failover moves no bytes, scaling is a CPU-and-network decision, and a zone-aware client never pays for cross-AZ replication.
• Lakehouse tables — the compaction pass that consolidates the write-ahead log can also materialize an Iceberg table, queryable from DuckDB, Trino or Spark. No copy job, no Connect sink.
• No protocol changes — existing producers, consumers, Kafka Streams, Connect, admin clients and the bin/ tools work unmodified.

⁠One cluster, two storage modes

Both modes share one controller quorum, one metadata log, one set of ACLs, one consumer-group coordinator, and the same client protocol.

⁠What's in the image

The brokers and controllers, the Kafka CLI, and the Ursa compactor — one image for every role in the stack.

⁠Quick start

Three brokers, Oxia, MinIO and the compactor, using the Compose files from the source repository:

git clone https://github.com/openlakestream/kafka.git && cd kafka
cd docker/examples/docker-compose-files/cluster/ursa
export IMAGE=lakestream/kafka:latest

make up            # Oxia + MinIO + 3 brokers + compactor
make create-topic  # create a diskless topic
make demo          # producer/consumer perf demo
make destroy       # tear everything down

Adding the Iceberg catalog is one more command:

make lakehouse-demo   # Kafka -> Parquet -> Iceberg -> DuckDB, asserted end to end

⁠Enabling it on your own cluster

Broker and controller configuration:

ursa.storage.enable=true

# Metadata and producer state
ursa.catalog.oxia.service.url=oxia://oxia:6648/default
ursa.oxia.service.url=oxia://oxia:6648/default

# Record storage
ursa.storage.backend.type=S3
ursa.storage.path=ursa/wal
ursa.storage.s3.bucket=my-kafka-bucket
ursa.storage.s3.region=us-east-1

# Compacted objects
ursa.storage.compaction.bucket=my-kafka-bucket
ursa.storage.compaction.prefix=ursa/compacted

Then create a diskless topic:

bin/kafka-topics.sh --create \
  --topic events --partitions 12 --replication-factor 1 \
  --config ursa.storage.enable=true \
  --bootstrap-server localhost:9092

You need an object storage bucket, an Oxia service, and a running compaction service. Compaction is required: retention on a diskless topic only soft-trims the log, and compaction is what advances the watermark that lets storage actually be reclaimed. An Iceberg catalog is optional on top of that.

⁠Good to know

Diskless topics reject transactional producers, do not implement key-based log compaction, require replication-factor 1, and their mode is fixed at creation. Internal topics such as __consumer_offsets always stay classic.

⁠Links

• Source, full configuration reference and design docs: openlakestream/kafka⁠
• LIP-162: Diskless Storage with Ursa Integration⁠
• LIP-163: Ursa Zone-Aware Owner Selection⁠
• Apache Kafka documentation⁠ for everything inherited from upstream

Apache License 2.0, the same as Apache Kafka.