Production Recommendations
This page covers best practices for running S4 in production.
Security
Enable IAM
Always enable IAM in production:
openssl rand -hex 32 > /etc/s4/jwt_secret # once, then keep the file
export S4_ROOT_PASSWORD=<strong-random-password>
export S4_JWT_SECRET_FILE=/etc/s4/jwt_secret
Important: Do not use the default password. Generate a strong random password and JWT secret.
S4_JWT_SECRET signs the admin tokens, and the admin API trusts the role inside a validly signed
token, so treat it like the root password. The server refuses to start when it is shorter than
32 bytes, and in cluster and gateway mode when IAM is in use and it is not set; there it must be
the same on every node. On a single node an unset secret is replaced by a random one at every
start, which is safe but ends admin sessions on restart. Details:
Admin JWT Secret.
Keep Credentials out of the Environment
Every credential variable also has a _FILE form (S4_ROOT_PASSWORD_FILE=/run/secrets/s4_pass)
that reads the value from a file, for Docker and Kubernetes secrets. See
Secrets from Files and the
Docker example.
Enable TLS
All production deployments should use TLS:
export S4_TLS_CERT=/etc/ssl/certs/s4.pem
export S4_TLS_KEY=/etc/ssl/private/s4-key.pem
Use certificates from a trusted CA (e.g., Let's Encrypt) or your organization's PKI.
Bind Address
Bind to 0.0.0.0 to accept connections from all interfaces, or restrict to a specific interface:
export S4_BIND=0.0.0.0 # All interfaces
Use a firewall or reverse proxy to control access.
Storage
Data Directory
Choose a dedicated partition or volume for S4 data:
export S4_DATA_DIR=/var/lib/s4
Filesystem
- ext4 or XFS are recommended
- Ensure the filesystem has enough inodes (though S4 uses very few)
- Use a filesystem with journaling for metadata integrity
NVMe for Metadata
The fjall database benefits from low-latency NVMe storage. Keep metadata, volumes and auxiliary directories on the same filesystem: local capacity accounting requires it and startup rejects incompatible mounts.
Disk Space Monitoring
Monitor disk usage and set alerts before running out of space. S4 needs space for:
- Volume files (object data)
- fjall database directory (metadata)
- Temporary files (multipart uploads in progress)
In single and cluster modes, writes preserve a configurable reserve and return 507 InsufficientStorage
before consuming it. DELETE, lifecycle and multipart abort use the reserve for metadata changes;
they cannot create space merely by dropping references. Free physical space, then allow the next
measurement to cross the recovery boundary. Do not delete storage files manually.
Tune disk settings for the filesystem size,
and monitor disk state, available bytes, sample freshness and s4_disk_write_refused_total. Reserve
is also needed by background metadata work. In a cluster, a full node stays alive and serves reads;
remaining replicas can acknowledge writes when they meet quorum. Fresh disk announcements can
prevent an impossible write before data is sent. Unknown or stale announcements do not prohibit
an attempt: each receiver checks its own physical budget. Cluster 507 can follow a partial write,
so retry according to the operation's existing semantics rather than assuming no replica accepted it.
With multi-set EC placement enabled, full erasure sets stop accepting new EC data.
A write acknowledged by RF=3 quorum still returns 200 and stays replicated, consuming
more space until capacity and enabled auto-tiering allow encoding. This differs from a
physical disk-space refusal that prevents write quorum and returns 507.
See EC capacity and fallback.
Updated senders retain hints and defer repair and pool movement on disk-space refusals. After
physical space is released, delivery and repair resume; a full new owner cannot justify cutover.
During rolling upgrades, older coordinators may return 503 and retain their previous hint behavior.
Deleting objects frees their space only when compaction reclaims the volume that holds them.
Volumes are never rewritten in place: compaction copies the live objects of a volume elsewhere and
then removes the whole file, and a volume with nothing live left is removed without copying. Copying
needs room of its own, and compaction never leaves less than half the reserve free: a volume whose
live data would not fit is skipped and retried by a later run. The skip is logged once per run as a
warning and counted in s4_disk_write_refused_total{path="compaction"}. On a nearly full node, delete
whole sets of older objects first: their volumes then empty completely and come back without any copy.
Volumes still held by an active download are reclaimed on a later compaction pass after that read
finishes, so a download already started can complete even if its object is deleted meanwhile.
If the disk did run out (the reserve was disabled, or another process took the space), the metadata
index may refuse every write until the node restarts. The node then logs an error saying so,
reports poisoned as its disk state in its health and s4_storage_index_poisoned 1, answers writes
with 503 and Retry-After, and its readiness check (/minio/health/ready) answers 503, so that a
load balancer stops sending it requests. Reads keep working. Free space, then restart the node:
freeing space alone does not clear the state. Everything acknowledged before reads back afterwards.
Performance
Upload Size
Set the maximum upload size based on your workload:
export S4_MAX_UPLOAD_SIZE=10GB
Lifecycle Worker
In production, keep the lifecycle worker enabled with a reasonable interval:
export S4_LIFECYCLE_ENABLED=true
export S4_LIFECYCLE_INTERVAL_HOURS=24
Monitoring
Prometheus
Enable Prometheus metrics and configure scraping:
export S4_METRICS_ENABLED=true
Set up alerts for:
- High error rates (5xx responses)
- Disk usage approaching capacity
- Request latency spikes
- Server uptime resets (unexpected restarts)
Health Checks
Use separate endpoints for process liveness and readiness to receive client traffic:
curl -f http://localhost:9000/health # Liveness
curl -f http://localhost:9000/minio/health/ready # Readiness
/health and /minio/health/live return 200 OK once the HTTP server is running.
Use these endpoints for liveness; waiting for peers or metadata recovery must not
cause a healthy process to be restarted.
/minio/health/ready returns 503 Service Unavailable with Retry-After: 1 while
a cluster node has required local recovery to finish: bucket configuration adoption,
single-to-cluster migration, or IAM adoption or freshness confirmation. It returns
200 OK when those local conditions allow client traffic. The X-S4-Readiness header
on a refusal gives a fixed reason code; the protected Admin API and server log give
details. Recovery does not stop inter-node RPC, gossip or the diagnostic API.
Configure your load balancer to send client traffic only to ready nodes, including at startup before the first check. The supplied HAProxy configuration starts backends unavailable until their readiness checks succeed. Readiness is a per-node check, separate from cluster read/write quorum availability; a background repair or pool growth does not by itself make a node unready. If adoption requires an operator's decision, connect directly to the node's Admin API even while the client load balancer excludes it. Do not use readiness failure as a reason to restart it.
For Kubernetes, use separate probes:
livenessProbe:
httpGet:
path: /health
port: 9000
periodSeconds: 10
timeoutSeconds: 2
readinessProbe:
httpGet:
path: /minio/health/ready
port: 9000
periodSeconds: 5
timeoutSeconds: 2
Do not probe /api/stats: it counts every object, and on a cluster node every object of the pool.
Backup
Data Backup
Back up the entire S4_DATA_DIR directory:
# Stop the server for a consistent backup
systemctl stop s4-server
tar -czf s4-backup-$(date +%Y%m%d).tar.gz /var/lib/s4
systemctl start s4-server
For zero-downtime backups, use filesystem-level snapshots (LVM, ZFS, or cloud volume snapshots).
Metadata Backup
The fjall database directory can be backed up independently for faster metadata recovery.
High Availability
S4 supports two deployment models for high availability:
Federation (Recommended)
S4 has built-in leaderless quorum replication (S4_MODE=cluster). A 3-node cluster with default quorum (N=3, W=2, R=2) tolerates 1 node failure with zero downtime. See Cluster Deployment for setup instructions and Federation for architecture details.
Single-Node HA (Alternative)
For single-node deployments (S4_MODE=single, the default), you can use infrastructure-level replication:
- Filesystem-level replication (DRBD, ZFS send/receive)
- Active-Passive failover using a shared volume
- Cloud provider features (EBS snapshots, Azure Managed Disks)
Reverse Proxy
For load balancing, TLS termination, or rate limiting, place S4 behind a reverse proxy:
Nginx Example
upstream s4 {
server 127.0.0.1:9000;
}
server {
listen 443 ssl;
server_name s4.example.com;
ssl_certificate /etc/ssl/certs/s4.pem;
ssl_certificate_key /etc/ssl/private/s4-key.pem;
client_max_body_size 10G;
location / {
proxy_pass http://s4;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
Note: Set client_max_body_size to match or exceed S4_MAX_UPLOAD_SIZE.