Ceph and ZFS storage for businesses in Pittsburgh and Cleveland

ZFS: storage inside one server

ZFS combines a file system and a volume manager and is built into Proxmox VE. A typical design is one Proxmox host on ZFS, or two hosts that replicate to each other, with backups to a separate system.

• Pools: disks grouped as mirrors or RAID-Z. ZFS needs direct access to each disk, not a hardware RAID volume.
• Checksums and scrubs: ZFS detects data a disk has silently damaged and repairs it from the redundant copy.
• Snapshots: a point-in-time copy of a virtual machine disk, taken in seconds before an update or risky change.
• Replication: snapshots sent to a second server on a schedule. If the first host fails, the second starts its machines, missing only changes since the last copy.

Ceph: storage across several servers

Ceph pools the disks in several servers into one storage system and keeps copies of each piece of data on different machines. If a drive or a whole server fails, the data stays available and the cluster rebuilds the missing copies on its own. No single storage controller can take everything down. Ceph presents that pool in three ways:

• Block (RBD): virtual disks for virtual machines. This is what Proxmox uses.
• File (CephFS): a shared file system that many servers can mount at once.
• Object: an S3-compatible gateway for applications and backup software.

Ceph needs at least three servers, a dedicated storage network, and someone who understands its health states. A warning is usually early notice of a failed drive or a pool nearing its limit, and we investigate it when it appears.

Which one fits

• One server: ZFS. Ceph is built to spread data across machines and gains nothing on a single server.
• Two servers: ZFS on each, replicating to each other. A two-server Proxmox cluster also needs a small third device to break ties, so both hosts agree on which one is in charge.
• Three or more servers, and machines must restart elsewhere with no lost changes: Ceph, with a storage network between the hosts.
• Shared files or S3 storage across many servers: Ceph.

ZFS is the simpler system to run. Our Proxmox consulting page covers the server side of the decision.

Capacity and health

Both systems need free space. A ZFS pool slows down as it gets close to full. A Ceph cluster with three copies of each object has roughly a third of its raw space usable, and it needs more on top because it rebuilds lost copies into the space that remains. A Ceph cluster that fills up stops accepting writes. We size storage so it can lose a drive, or a Ceph node, and still have room to rebuild.

Recovering degraded storage

The first rule is to avoid making things worse. On ZFS that usually means confirming which disk failed, replacing it and letting the pool rebuild. On Ceph the usual causes are failed drives, a node that is down, a misconfigured network, or a cluster that is too full. We find the cause and work in small, verified steps.

Neither system protects against deletion, ransomware or a bad change. Pair your storage with tested backups and recovery planning. If the storage serves containers, see Linux and Kubernetes consulting.

How an engagement works

1. Assess. We review hardware, networking, and storage health.
2. Scope. You get a written plan with risks and order of work.
3. Implement and verify. We build or repair, test failure scenarios, and document the storage.
4. Operate. Optional monitoring and upgrades, no lock-in.

We reply within one business day. Cleveland-area on-site work is arranged around the project.

Questions about business storage

Should we use ZFS or Ceph?

Mostly it depends on how many servers you have. One or two point to ZFS. Three or more, where machines must keep running through the loss of a server, point to Ceph.

How many servers does Ceph need?

At least three, with a proper storage network between them. Three is the smallest cluster that can lose a server and keep serving data.

Can you take over ZFS or Ceph storage someone else built?

Yes. We review health and configuration, fix what is wrong, and document it.

Can snapshots or Ceph replicas replace our backups?

No. ZFS snapshots sit on the same disks as the data, and Ceph copies every mistake to every replica. Backups on a separate system let you recover from mistakes and attacks.