Nextcloud External Storage - mounting external services with Rclone?

[iqweb]

Hi,
I am looking for a solution for the following scenario:

We need to upload & download approx 250GB of large video files per week. There would probably be about a maximum of 500GB but could reach more, on the server at any one time. This is in addition to our normal Nextcloud use for internal files. I don't really want to add this as Block storage to our server as that would be too expensive.

I tried adding External Storage to a Hetzner Storage box (that's what we use for backups via SMB) but we have found that it is a bit slow. Our users are used to sharing our large unedited video files via mega or google drive & their upload/download speeds are much faster with those services. The Google Drive External Storage app for Nextcloud doesn't work anymore with the current versions of Nextcloud.

I was thinking of maybe using RClone to locally mount a mega or google drive account & then add that as local External Storage. I noticed in the Cloudron docs that:

Mounting an existing server directory as 'external storage' on Nextcloud is currently not supported.

So will this not work?

I have also tried out Wasabi S3 storage which is very competitively priced but they have a model that you are charged for any upload for a full 90 days even if you delete the file within 90 days. This wouldn't really suit us as we are generally deleting the files within a few days once a client has downloaded them.

Does anyone have in mind any suitable solution for our needs?

[nebulon]

Not sure how much that has changed with recent Nextcloud versions, but generally we have had quite bad experience with that. Many non-block storage types are simply not well suited for regular file usage, while network mounts often are, as you also mentioned, quite slow.

What could help you here is to get some kind of storage box in the same datacenter as where your server is and then experiment between sshfs/samba/nfs, depending on what is supported and see which one works best.

[iqweb]

Thanks for the quick reply nebulon. Regarding what I quoted from the docs about mounting an existing server directory as 'external storage' not being supported, - is it not recommended or is that it simply can't be done with a Cloudron Nextcloud install?

What I mean is - let's say if I use Rclone to mount a Google Drive or Mega account to /mnt/webstorage could I then use the 'local' option in External Storage to 'attach' that drive as External Storage to the relevant users. This can work on a non-Cloudron Nextcloud, is there something that stops it on Cloudron?

[nebulon]

The local mount would not work, since Nextcloud runs in its own container and does not have access to folders on the host system. This requires more changes on the platform and is unlikely to happen any time soon.

The only option is to try the mount options from the external storage addon in nextcloud.

[girish]

@iqweb Mounting a local volume into an app is part of next release - https://forum.cloudron.io/topic/3205/what-s-coming-in-6-0-take-2

[iqweb]

Actually @nebulon - that's what I meant all along - mounting a local folder in the External Storage app - would that work or only as @girish said - after the 6.0 release because of Nextcloud being in a container?

[girish]

@iqweb said in Nextcloud External Storage - mounting external services with Rclone?:

because of Nextcloud being in a container?

Yes, correct. It's sandboxed and can't see the host file system.

[jagan]

@iqweb If you get a solution going, can you please share here?

[dkdkdkk]

@girish

Though when I look in the documentation (specifically the disk layout): https://docs.cloudron.io/storage/#app-data-directory after installation there is a new folder made in the appsdata folder. Does this mean that the data put in the nextcloud docker instance are stored on the disk/folder itself and are not stored within the docker instance? This would be a good thing, storing all the data inside of the nextcloud instance would seem like bad practice

[girish]

@dkdkdkk Yes, correct, on Cloudron, all app data is stored outside the container (for persistence). In Cloudron 6, a new data storage concept is coming up - https://docs.cloudron.io/storage/#volumes

[dkdkdkk]

@girish Interesting, any ETA on when 6.0 is coming?

Concerning the volume concept. I read another post (maybe you or another dev around here) mentioning that a "shared" folder/volume is not working as it should be. E.g. nextcloud is ignoring files when added to the volume and don't show up inside of the Nextcloud web UI. Is that still the case or is that fixed?

[necrevistonnezr]

@dkdkdkk 10 days ago:
@girish said in What's coming in 6.0 (take 2):

Another brief update:

• We added auth wall functionality that apps without any notion of login can use
• i18n is mostly done, just lots of testing before we let translators do their work. @nebulon can give some updates on how the language is selected etc.
• We added httpPaths functionality so that @MooCloud_Matt can work on his OLS app. Also, see https://forum.cloudron.io/topic/3755/apache-ols-and-nginx-custom-benchmarks/5
• Reworked a lot of the internal dns stuff, so that we can have apps as dns servers (ad blocker apps)
• solr fts search was merged today.

Looks like we are set for a release next week!