Addon request: blob storage (s3)

[infogulch]

Some apps, especially recently, store and access data via an s3-compatible api. I think Cloudron should add s3 storage as a new addon that can be requested in CloudronManifest.json to better support packaging such apps.

Some potential apps that would use this addon:

• Outline
• Notea
• Nextcloud

Here are some numbered opinions so you can refer to (refute) them:

1. I think it should be implemented as a Minio server which is set up like the postgresql or mysql addons in that the one instance is configured automatically and can serve a number of different apps. Say, each app that has the s3 addon gets a new random bucket and creds for that bucket.
2. Ideally it can be configured in either local or gateway mode. Users can use local if there is enough storage on their server, or configure a gateway to point to any S3-compatible, Azure, or NAS service. Note that many service providers offer an S3-compatible storage API, so the list of supported providers is already much bigger than just those three, and includes hosts like Backblaze, Linode, DigitalOcean among many others.
3. Figuring out how backups should work will probably be the biggest issue. Here are a few potential solutions:
   a. Just clone / replicate the whole bucket on every backup. Might cost a lot of storage, but that's how backups work today so... See: Bucket Replication Guide
   b. Something-something object versions? Though it might be difficult to, say, take a snapshot of a whole bucket. Bucket Versioning Guide
   c. Minio contributors don't seem very receptive to the general idea of bucket-level shapshots. Taking Backups of MinIO #4398, Snapshots #9376
   d. Just don't support backups at all? (not a fan of this option, but it's technically an option)

Thoughts?

[girish]

@infogulch Ah, that's a good idea! AFAIK, the minio cli can also be used for automatic provision and deprovision of buckets. Backups should be quite straightforward since minio just stores the data as-is inside folders with the same name as the bucket from what I remember.

As to whether it's worth implementing, I don't know. In fact, it's sort of an overkill already that some apps wants S3 storage. Especially for a note taking app like notea

[infogulch]

I agree, the weakest part of this request is that there's only one somewhat-legitimate app use case for it in Outline (silly Notea..), and it doesn't seem very efficient to build an addon for just one app. Perhaps the community can scrounge up some additional examples of useful apps whose packaging for Cloudron was complicated or thwarted by the lack of s3.

---

I'm curious what you think about using Minio in gateway mode (#2 above). One apprehension that's preventing me from going all-in on using some media management apps (photos, videos, music) hosted on Cloudron is that I'm worried my server will run out of disk space; with externally-hosted object storage that problem is solved (with easy to grok pay-for-what-you-use billing). My guess is this is the reason why some apps are starting to lean towards object stores like s3.

Perhaps the best way to cut it is: v1 only supports local (disk) storage; if some proportion of users have a specific need for externally-hosted object stores then we cross that gateway bridge when we get there.

[robi]

I think gateway mode by default is best, which can then point to any local or remote S3 endpoint as needed.

Again, MultiCloudron will make this very useful.

The apps will come once the capability is available, even NextCloud in S3 only mode.

[infogulch]

Nice thanks for the Nextcloud example. I added it to the list in the OP.

Gateway mode would be wonderful but it also adds some complications, in particular if done incorrectly it could throw a spanner into the currently very clean backup/restore system which is actually my favorite Cloudron feature. With that in mind I would be happy to get a version-1 implementation that only supports the local filesystem configuration.

[robi]

@infogulch It's an additional capability, not a replacement of what already works brilliantly.

[infogulch]

I would say what works brilliantly in Cloudron is the tight, seamless integration between the different components which enables admins to not have to think about huge problems like taking backups.

[MooCloud_Matt]

@robi said in Addon request: blob storage (s3):

gateway

Yes but in that case, cloudron will have to provide support for multiple NIC, 1 Gbit's NIC can't offer good performance with remote s3 + nextcloud + fire redundancy for minio for example.

[infogulch]

@moocloud_matt I'm confused how "multiple Network Interface Card (NIC)" support is relevant here. Are you familiar with the minio gateways? They are orthogonal to networking configuration.

https://docs.min.io/docs/minio-gateway-for-s3.html

[MooCloud_Matt]

@infogulch
what I mean, is that you need more than 1 gbit's connection for 1 server, and not always 2.5/10gbits are available.
Because Minio Gateway, and Minio Backend if it's used they still pass through data, and a lot of it, especially if you have a big install of NextCloud.

We need to be able to support multiple NIC, so that we can dedicate one for Minio, or other backend file transfer without impacting the second connection that is dedicated for Apps.

[infogulch]

@moocloud_matt The number of servers that use a 1GB NIC and could saturate it because using a gateway doubles the bandwidth used to serve minio requests would probably fit on one hand if there are any at all. This concern is like a third or fourth level extrapolation, I don't think it's worth anyone's time to consider this eventuality right now.

[MooCloud_Matt]

@infogulch

for experience with CEPH, and reading how Minio works, and their requirement.

Multi NICs, are needed or highly performant one, like 2.5 at least, for decent installation with Gateway or Minio storage cluster.
https://docs.min.io/minio/vsphere/reference/production-recommendations.html

A normal NextCloud server on one single SSD, not NVMe; can saturate easily one single Gbit's NIC.

That without considering that with Minio Gateway you have to split the 1Gbits NIC in incoming S3 traffic and outgoing WebDAV traffic.
I know that NICs are duplex, but they don't perform full capacity if used to the max.

It will not be for everyone but is needed for those who use Cloudron for heavily used Server.

[infogulch]

@moocloud_matt As soon as someone comes in reporting a real case that their NIC is saturated and it's slowing down their server I'd be interested in pursuing a solution. I've never seen such a real case on these forums (though I may just be misinformed) so I'd have to tilt towards YAGNI until one appears.

Object storage is not magic, it's just data like everything else. If it's just used to serve file attachments for an app with 25 users I wouldn't expect it to be a bottleneck.