<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Miles,<div><br></div><div>I’ve been a ZFS guy from long before Linux had heard of it. The data integrity in ZFS, performance with the ARC and L2ARC on modern machines with excess RAM and fast storage, and snapshots that can be taken and shoved around the network or to external disks are just part of why I rarely think about other filesystems.</div><div><br></div><div>But I’ve looked at Ceph a few times over the last few years since Proxmox has it largely built in. I’m loath to reduce the level of data integrity I have now with ZFS. How do you find Ceph compares?</div><div><br></div><div>Also how well does Ceph work with nodes with only one storage device in them? For instance we have some that only support one NVMe and one SATA, so we could put the boot/OS on a SATA SSD and then use the NVMe for Ceph.</div><div><br></div><div>Cheers,</div><div>Andrew<br><div><br><blockquote type="cite"><div>On 6 Jun 2023, at 10:45 am, Miles Goodhew via linux-aus <linux-aus@lists.linux.org.au> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Hi Anestis,<br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"> I used to be "The Ceph guy" at a large an annoying government department. I think the nutshell differences I see are:<br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"> <br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Gluster:<br></div><ul style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><li>Smaller scale (5-ish nodes max, I think)<br></li><li>Network filesystem only<br></li><li>Integrated services (storage and control/mgmt on the same boxes)<br></li><li>Limited redundancy and failure-domain options<br></li><li>A little simpler to set up on its own<br></li></ul><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Ceph:<br></div><ul style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><li>Scales up to gigantic, multi-region clusters<br></li><li>Block storage (RBD), File storage (CephFS) and Object storage (RGW) options available<br></li><li>Control/mgmt can be on separate nodes (And should be unless you have a really small cluster)<br></li><li>Any speed, redundancy (replication or erasure coding) or failure-domain setup you can think of. You can have multiple setups for different storage pools within the cluster.<br></li><li>Takes a bit more planning and implementation to deploy<br></li></ul><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Like Neill said: Openstack uses the RBD application to present "disk like" virtual storage devices to the compute nodes for the VMs to use. The old Redhat Enterprise Virtualisation (OVirt)<span class="Apple-converted-space"> </span><i>used</i><span class="Apple-converted-space"> </span>to use Gluster as its network storage system (putting disk images as files on top of it). However I'm not sure this is still the case.<br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">CephFS works really well as an NFS replacement (it's just a lot more fiddly to set up). RGW can present itself as either S3 or Swift protocol (Or a "weird" NFS version too - but don't go there).<br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Hope that's enough, but not too much info,<br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">M0les.<br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br></div><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">On Tue, 6 Jun 2023, at 04:55, Anestis Kozakis via linux-aus wrote:<br></div><blockquote type="cite" id="qt" style="font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div dir="ltr"><div>I was wondering fi people could summarize me the difference as well s the pros and cons fo GlusterFS vs CephFS inr regards to the following uses:<br></div><div><br></div><div>File Server/System and creating Virtual Machines and Containers.<br></div><div><br></div><div>I will, of course, do my own research, but I am looking to get other people's experiences and opinions.<br></div><div><br></div><div><div>Anestis.<br></div><div><span class="qt-gmail_signature_prefix">--</span><br></div><div dir="ltr" class="qt-gmail_signature"><div dir="ltr"><div>Anestis Kozakis |<span class="Apple-converted-space"> </span><a href="mailto:kenosti@gmail.com" target="_blank">kenosti@gmail.com</a><br></div><div><br></div></div></div></div></div><div>_______________________________________________<br></div><div>linux-aus mailing list<br></div><div><a href="mailto:linux-aus@lists.linux.org.au">linux-aus@lists.linux.org.au</a><br></div><div><a href="http://lists.linux.org.au/mailman/listinfo/linux-aus">http://lists.linux.org.au/mailman/listinfo/linux-aus</a><br></div><div><br></div><div>To unsubscribe from this list, send a blank email to<br></div><div><a href="mailto:linux-aus-unsubscribe@lists.linux.org.au">linux-aus-unsubscribe@lists.linux.org.au</a><br></div></blockquote><div style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br></div><span style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">linux-aus mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><a href="mailto:linux-aus@lists.linux.org.au" style="font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">linux-aus@lists.linux.org.au</a><br style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><a href="http://lists.linux.org.au/mailman/listinfo/linux-aus" style="font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://lists.linux.org.au/mailman/listinfo/linux-aus</a><br style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">To unsubscribe from this list, send a blank email to</span><br style="caret-color: rgb(0, 0, 0); font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><a href="mailto:linux-aus-unsubscribe@lists.linux.org.au" style="font-family: HelveticaNeue; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">linux-aus-unsubscribe@lists.linux.org.au</a></div></blockquote></div><br></div></body></html>