Install via Docker

For a single-node dev install:

docker run -d --name redpanda \
    --restart unless-stopped \
    -p 9092:9092 -p 9644:9644 -p 8081:8081 -p 8082:8082 \
    -v redpanda-data:/var/lib/redpanda/data \
    docker.redpanda.com/redpandadata/redpanda:latest \
    redpanda start \
        --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:9093 \
        --advertise-kafka-addr internal://redpanda:9092,external://localhost:9092 \
        --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:8083 \
        --advertise-pandaproxy-addr internal://redpanda:8082,external://localhost:8082 \
        --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 \
        --rpc-addr 0.0.0.0:33145 \
        --advertise-rpc-addr redpanda:33145 \
        --mode dev-container \
        --smp 1 \
        --memory 1G \
        --reserve-memory 0M \
        --check=false

For production: 3+ nodes, real Seastar tuning (one thread per CPU core, hugepages, NIC IRQ affinity). The official Helm chart or Operator handles Kubernetes; the apt repo gives .deb packages for bare metal.

The rpk CLI

rpk (Redpanda Keeper) is the unified CLI for cluster ops, topic management, produce/consume, and ACL management. Replaces Kafka's various shell scripts (kafka-topics.sh, kafka-console-producer.sh) with one tool:

docker exec -it redpanda rpk version
docker exec -it redpanda rpk cluster info

# Create a topic
docker exec -it redpanda rpk topic create events --partitions 6 --replicas 1

# List topics
docker exec -it redpanda rpk topic list

# Produce
echo '{"user": "alice", "event": "signup"}' | \
    docker exec -i redpanda rpk topic produce events

# Consume (from offset 0, follow new messages)
docker exec -it redpanda rpk topic consume events --offset start

# Topic info
docker exec -it redpanda rpk topic describe events

Connect from a real app

From any Kafka client (Java, Python's confluent-kafka, Node's kafkajs, Go's franz-go, Rust's rdkafka), point at the broker exactly as you would Kafka:

# Python
from confluent_kafka import Producer, Consumer

p = Producer({"bootstrap.servers": "localhost:9092"})
p.produce("events", key="user-42", value=b'{"event":"signup"}')
p.flush()

c = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "my-consumer-group",
    "auto.offset.reset": "earliest",
})
c.subscribe(["events"])
while True:
    msg = c.poll(1.0)
    if msg is not None and not msg.error():
        print(msg.value())

Same code works against Apache Kafka and Redpanda. Migration is the connection string.

What Redpanda does better than Kafka

  • No JVM, no GC pauses. Tail latency at the broker is consistently low; no surprise 200 ms stop-the-world events.
  • No ZooKeeper / KRaft cluster. Modern Kafka has KRaft (no ZooKeeper) but it's its own thing to operate; Redpanda's Raft layer is built-in and invisible.
  • Schema Registry built-in. No separate Confluent Schema Registry process; rpk + the included endpoint at :8081 covers it.
  • Pandaproxy built-in. HTTP-to-Kafka REST proxy without an extra Confluent REST Proxy install.
  • Tiered storage to S3 / object storage as a first-class feature (in the enterprise tier; OSS supports cold-tier reads).
  • Footprint. One ~100 MB binary; ~1-2 GB RAM per broker at light load. Apache Kafka's JVM tax is real.

Wire compatibility caveats

The Kafka API is huge. Redpanda implements the production-relevant parts; some edge-case admin APIs and very old client versions may differ:

  • Most modern clients (Confluent Java, kafkajs, confluent-kafka-python, franz-go) work perfectly.
  • Some Confluent ecosystem tools (ksqlDB, Kafka Connect with certain connectors) work but check per-version compatibility.
  • Older Kafka clients (<0.10) are not supported.

Production deployment shape

# A 3-node cluster (one Redpanda instance per host)
# /etc/redpanda/redpanda.yaml on each node

redpanda:
  data_directory: /var/lib/redpanda/data
  node_id: 1                  # 1, 2, 3 on the three nodes
  seed_servers:
    - host: { address: redpanda-1.lab, port: 33145 }
    - host: { address: redpanda-2.lab, port: 33145 }
    - host: { address: redpanda-3.lab, port: 33145 }
  rpc_server:
    address: 0.0.0.0
    port: 33145
  kafka_api:
    - address: 0.0.0.0
      port: 9092
  admin:
    - address: 0.0.0.0
      port: 9644

rpk:
  coredump_dir: /var/lib/redpanda/coredump
  tune_network: true
  tune_disk_scheduler: true
  tune_cpu: true
  tune_aio_events: true
  enable_memory_locking: true

Then systemd:

sudo systemctl enable --now redpanda
rpk cluster info       # all 3 nodes visible

Tiered storage

For "keep cold topics in S3, hot topics on local NVMe" patterns:

redpanda:
  cloud_storage_enabled: true
  cloud_storage_credentials_source: config_file
  cloud_storage_region: us-east-1
  cloud_storage_bucket: my-redpanda-tier
  cloud_storage_access_key: ...
  cloud_storage_secret_key: ...

  # Per-topic configurable
  cloud_storage_segment_max_upload_interval_sec: 3600

# Then on the topic
rpk topic alter-config events --set redpanda.remote.read=true --set redpanda.remote.write=true

Old segments auto-upload to S3; new consumers can read history back through the broker transparently.

When Redpanda isn't the right tool

  • If you're already deep in Confluent's ecosystem (Confluent Cloud, ksqlDB, full Connect ecosystem with managed connectors), the lift to swap is real even though the API is compatible.
  • For "we need a message queue, not a streaming platform" (work queue with task acking), NATS JetStream (see that tutorial), RabbitMQ, or a Postgres + background-job library is simpler.
  • For very small workloads, plain Postgres LISTEN/NOTIFY or Redis Streams covers the use case with less moving infrastructure.

For "I want the Kafka API at production scale without the operational tax of Apache Kafka," Redpanda is the cleanest choice in 2026.