Garage, an open-source distributed storage service you can self-host to fullfill many needs

[robi]

Too bad this is only replicated storage. (from what I can tell with a look of the front page)

Minio can use error correcting codes instead of replication.

[infogulch]

@robi Yeah unfortunately.

Non-goals include:

• ...
• Erasure coding (our replication model is simply to copy the data as is on several nodes, in different datacenters if possible)

[jdaviescoates]

@robi said in Garage, an open-source distributed storage service you can self-host to fullfill many needs:

Too bad this is only replicated storage. (from what I can tell with a look of the front page)

Minio can use error correcting codes instead of replication.

@infogulch said in Garage, an open-source distributed storage service you can self-host to fullfill many needs:

@robi Yeah unfortunately.

Non-goals include:

• ...
• Erasure coding (our replication model is simply to copy the data as is on several nodes, in different datacenters if possible)

What's the difference? or advantage of error correcting / erasure codiing?

[robi]

@jdaviescoates You can read about it in many places online comparing the two architectures and implementations.

EC is more space efficient, especially at large scale (Petabyte, Exabyte, etc), while replication always uses 100% more per replica and doesn't guarantee bit flip protection, etc. hence it doesn't scale well.

EC can always compute any missing pieces and self-heal or regenerate.

That's why you can hear all the words speaking with someone on the phone from across the planet, as many lost packets are recomputed to fill in gaps instead of long retransmits.

This applies to huge storage on continuously failing hardware under it too.

[LoudLemur]

@robi said in Garage, an open-source distributed storage service you can self-host to fullfill many needs:

@jdaviescoates You can read about it in many places online comparing the two architectures and implementations.

EC is more space efficient, especially at large scale (Petabyte, Exabyte, etc), while replication always uses 100% more per replica and doesn't guarantee bit flip protection, etc. hence it doesn't scale well.

EC can always compute any missing pieces and self-heal or regenerate.

That's why you can hear all the words speaking with someone on the phone from across the planet, as many lost packets are recomputed to fill in gaps instead of long retransmits.

This applies to huge storage on continuously failing hardware under it too.

You have a gift for explaining things.

[LoudLemur]

@ruihildt said in Garage, an open-source distributed storage service you can self-host to fullfill many needs:

https://garagehq.deuxfleurs.fr/

Alternative to Minio, specially made for selfhosting setup.

See: https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/
Code: https://git.deuxfleurs.fr/Deuxfleurs/garage

I haven't checked out this suggestion yet, but if it like what I have in mind, it could be brilliant for Cloudron.

Cloudron is all about self-hosting. and there are a huge number of games with outdated gaming systems that could be repurposed to Cloudron self-hosting. These machines would have large amounts of RAM (for games), a very fast SSD (for running the system) and also a large storage drive (to hold the games).

That sort of legacy hardware is, I think, absolutely ripe for Cloudron! The Garage software could make those old storage disks have a new life!

[MooCloud_Matt]

@LoudLemur

Garage still doesn't support 100% the s3 protocol.
And due to how is written, modern NVMe will not get a BIG boost if you don't have a powerful CPU (it CPU intensive for big files like cloudron Backup, and is still not well-performing if you have small files like in a Nextcloud with small docs.)

MooCloud is testing Garage, and we have an agreement with Garage leading developer to publish the data collected using garage in a modern cloud environment.

[robi]

@MooCloud_Matt Have you looked at SeaweedFS yet?

[MooCloud_Matt]

@robi
We did, but i don't recall the reason we excluded as an alternate to ceph(that we are using in prod currently)

[jdaviescoates]

@MooCloud_Matt said in Garage, an open-source distributed storage service you can self-host to fullfill many needs:

Garage still doesn't support 100% the s3 protocol.

Seems they got most of it covered now:
https://garagehq.deuxfleurs.fr/documentation/reference-manual/s3-compatibility/

[jadudm]

I started work on a package; this was under a different thread, but it is probably more appropriate to mention here:

https://forum.cloudron.io/post/116583

[jadudm]

update

I've poked the package with a stick, added more to the README, and have run into a few interesting things about this package.

s3 functionality

I was able to create a bucket and put stuff in it. This seems to be a core function of an S3-compatible object server.

administrative server

Garage has a notion of having an administrative API at one port. I can use httpPorts to bind this port, and I can use it. For example, if I have

s3.example.com

as the root domain, then

admin.s3.example.com

can be the home for the Admin API. And, I was able to create an administrative API token, and using it, access the admin API.

static websites

This one is tricky. In theory, if I configure part of the garage.toml correctly:

[s3_web]
bind_addr = "[::]:3902"
# This wants to be set dynamically in the startup.
# That way, it can grab a Cloudron variable.
root_domain = ".web.s3.example.com"
index = "index.html"

I can serve static sites out of buckets. However, this implies domain name manipulation. And, possibly, a wildcard. As in, I'd like

*.web.example.com to resolve to s3.example.com, and for Garage to pick it up (internally) on port 3902.

I have explored this manually (by manipulating DNS settings in Cloudflare), but even though I have configured a bucket to serve static content, I can't (yet) convince it to serve something up.

While it might be that this functionality has to be sacrificed, I think it would be a nice way (if it was baked in) to manage 100% static sites. However, that would be new machinery: a way to map domains to buckets.

backups

I'm not convinced the backups are good. Specifically

<path>/meta/db.sqlite

is the metadata database for the Garage instance. This is, as far as I can tell, all of the information about where all of the files is stored. Losing this database is tantamount to losing all of the data. I think. So, making sure it backs up correctly matters. And, it is clear that updates will need to do things like garage repair and garage migrate, in the event of migrations/changes to this metadata database.

Ah:

Since Garage v0.9.4, you can use the garage meta snapshot --all command to take a simultaneous snapshot of the metadata database files of all your nodes. This avoids the tedious process of having to take them down one by one before upgrading. Be careful that if automatic snapshotting is enabled, Garage only keeps the last two snapshots and deletes older ones, so you might want to disable automatic snapshotting in your upgraded configuration file until you have confirmed that the upgrade ran successfully. In addition to snapshotting the metadata databases of your nodes, you should back-up at least the cluster_layout file of one of your Garage instances (this file should be the same on all nodes and you can copy it safely while Garage is running).

(Emphasis mine.)

So, the backup process is something I'll need to investigate further. It might be that some manual/scripted management of this database file---and dumping it---is going to be a bit of a thing in terms of having it be a robust process.

(Given that Cloudron does backups before upgrades, as long as the SQLite DB is snapshotted correctly on backup, I think it will be fine.) I suspect that a cron will need to be installed for this package that---daily?---runs the snapshot command, rotates DBs, and those are part of the backup. (I have a suspicion that Cloudron packages handle this kind of thing in the start.sh scripts?)

healthcheck URL

The manifest assumes that the health check URL is on the main app. In this case, if I have

s3.example.com

and the Admin API is at

admin.s3.example.com

(defined in httpPorts), I want the health check URL to be

https://admin.s3.example.com/health

because that is where Garage put it. I don't think I can do that with the manifest as-designed.

summary

I think the package is off to a good start. I have questions, but most of them are described above, and I'll probably figure things out. The health check and static site subdomains, though, might not be easily solved.

[girish]

@jadudm great work. I think it will be great to offer garage as an addon i.e another options to https://docs.cloudron.io/packaging/addons/ . What is needed is a way to provision buckets and restore/backup them on the fly . From what I can make out this is doable from what you have posted.

Does Garage have web interface? For that matter, do any of the other alternatives have one? If not, we have to make one.

[jdaviescoates]

@girish said in Garage, an open-source distributed storage service you can self-host to fullfill many needs:

Does Garage have web interface?

Seems so https://github.com/khairul169/garage-webui

[girish]

@jdaviescoates i think that one is just an unofficial web ui, no? (similar to the many that exist for wireguard).

[jdaviescoates]

@girish could be the result of this https://nlnet.nl/project/Garage-AdminUI/ ?

Hard to tell its status, but I guess at worst it could be used a base/ inspiration for making one?

[robi]

@girish said in Garage, an open-source distributed storage service you can self-host to fullfill many needs:

Does Garage have web interface? For that matter, do any of the other alternatives have one? If not, we have to make one.

Why do you think you need one? You are just going to use the API and one time setup can be automated or manual, no?

Or could it become a Cloudron UI exposed in a new menu via API?

[girish]

@robi I mixed two topics For an addon, no UI is required. But if we package as an app, a UI is required.

[jadudm]

@girish Glad to serve.

If the goal is an addon, then...

1. I think the backup piece should be straight-forward? (I have some more info in the packaging thread about this.) Essentially, you want to make sure the SQLite DB is backed up, and then you backup the filesystem. I have questions about what happens if (say) a restore happens (e.g. what if the filesystem path changes?), but those things can be explored.
2. There is an administrative API that (once you generate a secret/trusted admin API token) lets you do everything administratively via that API (bucket creation, etc.).

As to a web interface, I would not recommend you create one for end-users. (I suspect this is not what you mean.) You have NextCloud, XBackBone, and other apps that can talk to an S3-compatible backend for file browsing. What people might need/want is a way to:

• Create/remove buckets (which, on the backend, you'd use your secret admin key)
• Create/remove keys and attach them to buckets
• Create/remove administrative keys (for superusers who want to script things against the backend)
• Bonus: the ability to designate a bucket as a static site, and then you do the DNS work on the backend to point either a subsubdomain at it (e.g. site.s3.example.com) or a whole new domain (e.g. someothersite.com -> site.s3.example.com)

I suspect you could iterate towards this, if you wanted to. Release it with terminal-only management to start, and work towards an admin interface for common bucket creation/removal-type tasks.

There are things that Garage does not do (e.g. lifecycle rules), so this is not a "full" S3 clone (the way Minio aspired to be). In this regard, SeaweedFS might offer more features (and a built-in admin UI)... so, it might be worth doing a spike to explore it as well. At a glance, it is similar, but it also is intended to scale (e.g. it does Raft consensus and Reed-Solomon EC if you want it) in a way that Garage does not. This might also be a reason to not use Seaweed.

I can poke it with a stick if it would be valuable.

Update, a few hours later: I would recommend sticking with Garage.