Production Recommendations

This page covers best practices for running S4 in production.

Security

Enable IAM

Always enable IAM in production:

openssl rand -hex 32 > /etc/s4/jwt_secret   # once, then keep the file
export S4_ROOT_PASSWORD=<strong-random-password>
export S4_JWT_SECRET_FILE=/etc/s4/jwt_secret

Important: Do not use the default password. Generate a strong random password and JWT secret.

S4_JWT_SECRET signs the admin tokens, and the admin API trusts the role inside a validly signed token, so treat it like the root password. The server refuses to start when it is shorter than 32 bytes, and in cluster and gateway mode when IAM is in use and it is not set; there it must be the same on every node. On a single node an unset secret is replaced by a random one at every start, which is safe but ends admin sessions on restart. Details: Admin JWT Secret.

Keep Credentials out of the Environment

Every credential variable also has a _FILE form (S4_ROOT_PASSWORD_FILE=/run/secrets/s4_pass) that reads the value from a file, for Docker and Kubernetes secrets. See Secrets from Files and the Docker example.

Enable TLS

All production deployments should use TLS:

export S4_TLS_CERT=/etc/ssl/certs/s4.pem
export S4_TLS_KEY=/etc/ssl/private/s4-key.pem

Use certificates from a trusted CA (e.g., Let's Encrypt) or your organization's PKI.

Bind Address

Bind to 0.0.0.0 to accept connections from all interfaces, or restrict to a specific interface:

export S4_BIND=0.0.0.0  # All interfaces

Use a firewall or reverse proxy to control access.

Storage

Data Directory

Choose a dedicated partition or volume for S4 data:

export S4_DATA_DIR=/var/lib/s4

Filesystem

  • ext4 or XFS are recommended
  • Ensure the filesystem has enough inodes (though S4 uses very few)
  • Use a filesystem with journaling for metadata integrity

NVMe for Metadata

The fjall database benefits from low-latency NVMe storage. Keep metadata, volumes and auxiliary directories on the same filesystem: local capacity accounting requires it and startup rejects incompatible mounts.

Disk Space Monitoring

Monitor disk usage and set alerts before running out of space. S4 needs space for:

  • Volume files (object data)
  • fjall database directory (metadata)
  • Temporary files (multipart uploads in progress)

In single and cluster modes, writes preserve a configurable reserve and return 507 InsufficientStorage before consuming it. DELETE, lifecycle and multipart abort use the reserve for metadata changes; they cannot create space merely by dropping references. Free physical space, then allow the next measurement to cross the recovery boundary. Do not delete storage files manually.

Tune disk settings for the filesystem size, and monitor disk state, available bytes, sample freshness and s4_disk_write_refused_total. Reserve is also needed by background metadata work. In a cluster, a full node stays alive and serves reads; remaining replicas can acknowledge writes when they meet quorum. Fresh disk announcements can prevent an impossible write before data is sent. Unknown or stale announcements do not prohibit an attempt: each receiver checks its own physical budget. Cluster 507 can follow a partial write, so retry according to the operation's existing semantics rather than assuming no replica accepted it.

With multi-set EC placement enabled, full erasure sets stop accepting new EC data. A write acknowledged by RF=3 quorum still returns 200 and stays replicated, consuming more space until capacity and enabled auto-tiering allow encoding. This differs from a physical disk-space refusal that prevents write quorum and returns 507. See EC capacity and fallback.

Updated senders retain hints and defer repair and pool movement on disk-space refusals. After physical space is released, delivery and repair resume; a full new owner cannot justify cutover. During rolling upgrades, older coordinators may return 503 and retain their previous hint behavior.

Deleting objects frees their space only when compaction reclaims the volume that holds them. Volumes are never rewritten in place: compaction copies the live objects of a volume elsewhere and then removes the whole file, and a volume with nothing live left is removed without copying. Copying needs room of its own, and compaction never leaves less than half the reserve free: a volume whose live data would not fit is skipped and retried by a later run. The skip is logged once per run as a warning and counted in s4_disk_write_refused_total{path="compaction"}. On a nearly full node, delete whole sets of older objects first: their volumes then empty completely and come back without any copy. Volumes still held by an active download are reclaimed on a later compaction pass after that read finishes, so a download already started can complete even if its object is deleted meanwhile.

If the disk did run out (the reserve was disabled, or another process took the space), the metadata index may refuse every write until the node restarts. The node then logs an error saying so, reports poisoned as its disk state in its health and s4_storage_index_poisoned 1, answers writes with 503 and Retry-After, and its readiness check (/minio/health/ready) answers 503, so that a load balancer stops sending it requests. Reads keep working. Free space, then restart the node: freeing space alone does not clear the state. Everything acknowledged before reads back afterwards.

Performance

Upload Size

Set the maximum upload size based on your workload:

export S4_MAX_UPLOAD_SIZE=10GB

Lifecycle Worker

In production, keep the lifecycle worker enabled with a reasonable interval:

export S4_LIFECYCLE_ENABLED=true
export S4_LIFECYCLE_INTERVAL_HOURS=24

Monitoring

Prometheus

Enable Prometheus metrics and configure scraping:

export S4_METRICS_ENABLED=true

Set up alerts for:

  • High error rates (5xx responses)
  • Disk usage approaching capacity
  • Request latency spikes
  • Server uptime resets (unexpected restarts)

Health Checks

Use separate endpoints for process liveness and readiness to receive client traffic:

curl -f http://localhost:9000/health               # Liveness
curl -f http://localhost:9000/minio/health/ready   # Readiness

/health and /minio/health/live return 200 OK once the HTTP server is running. Use these endpoints for liveness; waiting for peers or metadata recovery must not cause a healthy process to be restarted.

/minio/health/ready returns 503 Service Unavailable with Retry-After: 1 while a cluster node has required local recovery to finish: bucket configuration adoption, single-to-cluster migration, or IAM adoption or freshness confirmation. It returns 200 OK when those local conditions allow client traffic. The X-S4-Readiness header on a refusal gives a fixed reason code; the protected Admin API and server log give details. Recovery does not stop inter-node RPC, gossip or the diagnostic API.

Configure your load balancer to send client traffic only to ready nodes, including at startup before the first check. The supplied HAProxy configuration starts backends unavailable until their readiness checks succeed. Readiness is a per-node check, separate from cluster read/write quorum availability; a background repair or pool growth does not by itself make a node unready. If adoption requires an operator's decision, connect directly to the node's Admin API even while the client load balancer excludes it. Do not use readiness failure as a reason to restart it.

For Kubernetes, use separate probes:

livenessProbe:
  httpGet:
    path: /health
    port: 9000
  periodSeconds: 10
  timeoutSeconds: 2
readinessProbe:
  httpGet:
    path: /minio/health/ready
    port: 9000
  periodSeconds: 5
  timeoutSeconds: 2

Do not probe /api/stats: it counts every object, and on a cluster node every object of the pool.

Backup

Data Backup

Back up the entire S4_DATA_DIR directory:

# Stop the server for a consistent backup
systemctl stop s4-server
tar -czf s4-backup-$(date +%Y%m%d).tar.gz /var/lib/s4
systemctl start s4-server

For zero-downtime backups, use filesystem-level snapshots (LVM, ZFS, or cloud volume snapshots).

Metadata Backup

The fjall database directory can be backed up independently for faster metadata recovery.

High Availability

S4 supports two deployment models for high availability:

S4 has built-in leaderless quorum replication (S4_MODE=cluster). A 3-node cluster with default quorum (N=3, W=2, R=2) tolerates 1 node failure with zero downtime. See Cluster Deployment for setup instructions and Federation for architecture details.

Single-Node HA (Alternative)

For single-node deployments (S4_MODE=single, the default), you can use infrastructure-level replication:

  • Filesystem-level replication (DRBD, ZFS send/receive)
  • Active-Passive failover using a shared volume
  • Cloud provider features (EBS snapshots, Azure Managed Disks)

Reverse Proxy

For load balancing, TLS termination, or rate limiting, place S4 behind a reverse proxy:

Nginx Example

upstream s4 {
    server 127.0.0.1:9000;
}

server {
    listen 443 ssl;
    server_name s4.example.com;

    ssl_certificate /etc/ssl/certs/s4.pem;
    ssl_certificate_key /etc/ssl/private/s4-key.pem;

    client_max_body_size 10G;

    location / {
        proxy_pass http://s4;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Note: Set client_max_body_size to match or exceed S4_MAX_UPLOAD_SIZE.