Garage: lightweight distributed S3-compatible storage

Install (3-node example)

# On each of 3 nodes (or just one for testing)
GARAGE_VER=v1.0.1
wget https://garagehq.deuxfleurs.fr/_releases/${GARAGE_VER}/x86_64-unknown-linux-musl/garage
sudo install garage /usr/local/bin/

# Each node has its own config (with a unique RPC secret)
sudo mkdir -p /etc/garage /var/lib/garage/{meta,data}

/etc/garage.toml:

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"

db_engine = "lmdb"

replication_factor = 3

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "10.0.1.10:3901"
rpc_secret = "<random 64-char hex from openssl rand -hex 32>"

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = ".s3.example.com"

[s3_web]
bind_addr = "[::]:3902"
root_domain = ".web.example.com"
index = "index.html"

[k2v_api]
api_bind_addr = "[::]:3904"

[admin]
api_bind_addr = "[::]:3903"
admin_token = "<random>"
metrics_token = "<random>"

Same rpc_secret on all 3 nodes. rpc_public_addr changes per node.

Start + connect the nodes

# systemd unit
sudo tee /etc/systemd/system/garage.service <<'EOF'
[Unit]
Description=Garage
After=network-online.target

[Service]
ExecStart=/usr/local/bin/garage -c /etc/garage.toml server
Restart=always
User=garage

[Install]
WantedBy=multi-user.target
EOF

sudo useradd -r garage
sudo chown -R garage:garage /var/lib/garage
sudo systemctl enable --now garage

# Verify the node
garage node id
# Output: a node ID + the rpc_public_addr

# On node 1, connect the other nodes (paste each node's ID@addr from "garage node id")
garage node connect <node-2-id>@10.0.1.11:3901
garage node connect <node-3-id>@10.0.1.12:3901

garage status
# Shows all 3 nodes as Connected

Define the cluster layout

Garage uses a Russian-doll layout: each node belongs to a zone (typically a datacenter or city); replication ensures copies in different zones.

# Tag each node with a zone + capacity (in bytes-style human units)
garage layout assign <node-1-id> -z home -c 500G -t homelab
garage layout assign <node-2-id> -z cloud -c 200G -t hetzner
garage layout assign <node-3-id> -z office -c 300G -t office

# Show pending changes
garage layout show

# Apply
garage layout apply --version 1

Now writes are placed so each object has one copy in each zone (with replication_factor=3). Lose a zone, the others still serve reads + writes. Critical for the geo-distributed use case.

Create an S3 bucket + access key

# Create a key
garage key create my-app-key
# Output: AKEY + SKEY

# Create a bucket
garage bucket create my-app-data

# Grant the key access to the bucket
garage bucket allow --read --write --owner my-app-data --key my-app-key

Use with any S3 client

# aws-cli
aws s3 --endpoint-url http://10.0.1.10:3900 ls

# rclone
rclone config create garage s3 \
    provider Other \
    access_key_id AKEY \
    secret_access_key SKEY \
    endpoint http://garage.example.com:3900 \
    acl private \
    region garage

rclone copy /local/data garage:my-app-data

# restic (see /tutorials/restic-s3-encrypted-backups.html)
export AWS_ACCESS_KEY_ID=AKEY
export AWS_SECRET_ACCESS_KEY=SKEY
restic -r s3:http://garage.example.com:3900/my-app-data backup ~/photos

Web (static site hosting)

Garage's [s3_web] section turns any bucket into a static-site host. Upload an index.html + assets; Garage serves them via path-style URLs at the configured root domain.

# Upload a site
aws s3 cp ./public/ s3://my-site/ --recursive --endpoint http://garage:3900

# Browse to http://my-site.web.example.com/

The web subsystem honors index + standard 404 page; useful for self-hosted Hugo / Astro / Eleventy / SilverBullet builds.

Reverse proxy

# Caddy
s3.example.com {
    reverse_proxy 127.0.0.1:3900
}
*.s3.example.com {
    reverse_proxy 127.0.0.1:3900
}

web.example.com {
    reverse_proxy 127.0.0.1:3902
}
*.web.example.com {
    reverse_proxy 127.0.0.1:3902
}

Wildcards needed because S3 uses subdomain bucket-naming (<bucket>.s3.example.com). With root_domain set to .s3.example.com, Garage routes requests to the right bucket based on the subdomain.

WAN-friendly design

What makes Garage geo-friendly:

• No strict consistency required across nodes. Writes commit when a majority of zones have it; readers see eventually-consistent results within a few seconds.
• Compact RPC. Inter-node traffic is minimal; you can run replicas on cheap VPSes with consumer bandwidth.
• No bandwidth-amplifying erasure coding by default. Garage uses plain replication; predictable WAN cost.

MinIO's design assumes LAN-speed inter-node communication; running it across the WAN works but doesn't scale as gracefully. Garage explicitly targets the "3 nodes in 3 different cities" case.

Garage vs alternatives

• MinIO — high-throughput, LAN-first, K8s-native, erasure-coded. Heavier on bandwidth across the WAN. Pick for "I want fast S3 on a single cluster."
• Ceph RGW (see that tutorial) — production-grade distributed storage; way more setup; tightly LAN-coupled.
• SeaweedFS — small Go S3 store; LAN-focused; less geo-distributed-aware.
• Backblaze B2 / Cloudflare R2 — commercial; cheap; no self-hosting.

When Garage is the right pick

• Multi-location homelab / small business with home + office + cloud VPS storage to combine.
• Geo-redundant backups (restic to Garage; same data lives in 3 cities).
• "I want S3 that survives one of my homes burning down without paying per-GB cloud rates."

When it isn't

• For "single-DC fast S3 for K8s," MinIO is faster + more K8s-native.
• For "store petabytes," Ceph is more battle-tested at that scale.
• For "I only have one place to store this and one disk," plain disk is simpler.