*<https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/727>\- gitlab evaluated Ceph and moving to metal in 2016 and decided to stick with the cloud
alternatively, we could go with a SAN, home-grown or commercial, but i would
rather avoid proprietary stuff, which means we'd have to build our own, and
i'm not sure how we would do _that_. ZFS replication maybe? and that would
only solve the Ganeti storage problems. we'd still need an S3 storage, but we
could use something like MinIO for that specifically.
oh, and we could fix the backup problems by ditching bacula and switching to
something like borg. we'd need an offsite server to "pull" the backups,
however (because borg is push, which means a compromised backup server can
trash its own backups). we could build this with ZFS/BTRFS replication, again.
> another caveat with borg is that restores are kind of slow. bacula
> seems to be really fast at restores, at least it's my experience
> restoring websites in #40501 (closed) today, really positive
> feeling.
[openio]:https://www.openio.io/
## TODO: triage Ceph war stories from GitLab and SO
more war stories, this time from gitlab:
* when they were saying they would move to bare metal and ceph: <https://about.gitlab.com/blog/2016/11/10/why-choose-bare-metal/>
* when they subsequently tried and failed and switched back to the cloud and not ceph, see <https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/727><https://gitlab.com/gitlab-com/operations/-/issues/1> quote from [this deployment issue][]:
> While it's true that we lean towards PostgreSQL, our usage of CephFS was not
> for the database server, but for the git repositories. In the end we
> abandoned our usage of CephFS for shared storage and reverted back to a
> sharded NFS design.
and StackOverflow's (presumably) Jeff Atwood:
* "We had disastrous experiences with Ceph and Gluster on bare metal. I think this says more about the immaturity (and difficulty) of distributed file systems than the cloud per se." <https://news.ycombinator.com/item?id=12940042> in response to the first article from GitLab.com above (which ended up being correct: the went back to the cloud)
about this, one key thing to keep in mind is that GitLab were looking
for an NFS replacement.
we don't use NFS anywhere right now (thank god) so that is not a
requirement.
the above "horror stories" might not be the same with other storage
mechanisms. indeed, there's a big difference between using Ceph as a
filesystem (ie. CephFS) and an object storage (RadosGW) or block
storage (RBD), which might be better targets for us.
In particular, we're likely to use Ceph as a block device -- for
Ganeti instance disks, which Ganeti has good support for -- or object
storage -- for GitLab's "things", which it is now also designed
for. And indeed, "NFS" (ie. real filesystem) is now (14.x?) deprecated
in GitLab, so shared data storage is expected to go through S3-like
*[Grafana](https://packages.debian.org/unstable/ceph-grafana-dashboards) and [Prometheus dashboards](https://packages.debian.org/unstable/ceph-prometheus-alerts), both packaged in Debian
More features:
* block device snapshots and mirroring
* erasure coding
* self-healing
* used at CERN, OVH, and Digital Ocean
*[yearly release cycle with two-year support lifetime](https://docs.ceph.com/en/latest/releases/general/)
* cache tiering (e.g. use SSDs as caches)
* also provides a networked filesystem (CephFS) with an optional NFS
frontend
Downsides:
* complexity: at least 3-4 daemons to manager a cluster, although
this could might be easier to live with thanks to the Debian
packages
* high hardware requirements (quad-core, 64-128GB RAM, 10gbps),
although their [minimum requirements](https://docs.ceph.com/en/latest/start/hardware-recommendations/#minimum-hardware-recommendations) are actually quite
attainable
### Scalability promises
CERN started with a 3PB Ceph deployment [around 2015][]. It seems it's
still in use:
*[2017][], 65PB
*[2018][], 300PB?
*[2019][], 1PB/day, 115PB/year?
*[2021][], 65PB?
... although, as you can see, it's not exactly clear to me how much data is
managed by ceph. they seem to have a good experience with Ceph in any case,
with three active committers, and they say it's a "great community", which is
certainly a plus.
On the other hand, managing lots of data is part of their core
mission, in a sense, so they can probably afford putting more people
GitLab tried to [move from the cloud to bare metal](https://about.gitlab.com/blog/2016/11/10/why-choose-bare-metal/). [Issue 727](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/727)
and [issue 1](https://gitlab.com/gitlab-com/operations/-/issues/1) track their attempt to migrate to Ceph which
failed. They moved back to the cloud. A choice quote from [this
deployment issue][]:
> While it's true that we lean towards PostgreSQL, our usage of CephFS was not
> for the database server, but for the git repositories. In the end we
> abandoned our usage of CephFS for shared storage and reverted back to a
> sharded NFS design.
Jeff Atwood also described his experience, presumably from
StackOverflow's attempts:
> We had disastrous experiences with Ceph and Gluster on bare metal. I
> think this says more about the immaturity (and difficulty) of
> distributed file systems than the cloud per se.
This was a [Hacker News comment](https://news.ycombinator.com/item?id=12940042) in response to the first article
from GitLab.com above, which ended up being correct as GitLab went
back to the cloud.
One key thing to keep in mind is that GitLab were looking for an NFS
replacement, but we don't use NFS anywhere right now (thank god) so
that is not a requirement for us. So those issues might be less of a
problem, as the above "horror stories" might not be the same with
other storage mechanisms. Indeed, there's a big difference between
using Ceph as a filesystem (ie. CephFS) and an object storage
(RadosGW) or block storage (RBD), which might be better targets for
us.
In particular, we could use Ceph as a block device -- for Ganeti
instance disks, which Ganeti has good support for -- or object storage
-- for GitLab's "things", which it is now also designed for. And
indeed, "NFS" (ie. real filesystem) is now (14.x?) deprecated in
GitLab, so shared data storage is expected to go through S3-like
