Does it make sense to use s3 (Backblaze/Wasabi) as a backend storage ?

[JOduMonT]

A long time ago , I tried to mount a s3/webdav directory on Cloudron via fstab to use is as a storage for apps like Nextcloud, but it failed mainly because of a DNS situation. That idea always stuck around but never truly searched for an alternative until now.

Basically today I was looking how to configure/mount wasabi via the plugin External Storage and I found this article where it explain how to use an s3 storage as backend storage for Nextcloud. The beauty of it, it's managed by the Nextcloud App itself so you have no modification to make inside nor outside of the container.

So inside the container you simply have to add a storage.config.php inside the config directory

<?php
$CONFIG = array (
'objectstore' => array(
        'class' => '\\OC\\Files\\ObjectStore\\S3',
        'arguments' => array(
                'bucket' => 'my-nextcloud-bucket',
                'autocreate' => true,
                'key'    => 'BYQLD4LZGEXAMPLE',
                'secret' => 'dycOcoq6R9YwTVEXAMPLE',
                'hostname' => 's3.us-west-1.wasabisys.com',
                'region' => 'us-west-1',
                'port' => 443,
                'use_ssl' => true,
                'use_path_style'=>true
        ),
),
);

They recommend to do it, before the installation, but, I did it after the installation (only with the default data in the Nextcloud) and it didn't complain.

Now to make it more suitable, Cloudron team might think of a way to implementing that in the storage area

[robi]

When Nextcloud is being deployed at scale and the S3 storage layer is on the same or adjacent local network, it makes sense.

If your VPS is at one provider and your S3 is on another, it's generally not a good idea

[nebulon]

@jodumont I think what is worth mentioning here is that block storage and object storage are fundamentally two different things and using them interchangeably might cause unexpected side-effects. With S3 for example a common side-effect are high costs due to I/O operations, which are charged by some providers like AWS.

So it is unlikely in my opinion that we add object storage support in places where a block storage is required or expected by the app.

To give one obvious example. If you edit a text file within Nextcloud, changing one character, then usually Nextcloud would have to create a whole new object at your storage and purge the old one instead of only uploading the block containing that character of the file. Further seeking within larger files usually require downloading large chunks or whole objects. The reduced access and modify restrictions on object stores though allow providers to offer them more cost effectively. Which I think is the main reason often people try to use them in the first place.

Of course some smarter filesystem like implementations can be done on top of object stores, no doubt, but those usually depend on the use-case. For example an application like peertube or Emby could make great use of cheap but fast on read object storage, if the backend supports proper seeking or the app can split up video blobs for that use-case.

[JOduMonT]

@nebulon I don't fully understand your explanation but understand it is not a great usage of this kind of storage and I'll align myself with your judgement and your expertise

but, I'm still curious about why Nextcloud implement it and propose it then ?

I means it was already possible to interact with s3 via external storage and to use it as a vault for archiving.

[robi]

When Nextcloud is being deployed at scale and the S3 storage layer is on the same or adjacent local network, it makes sense.

If your VPS is at one provider and your S3 is on another, it's generally not a good idea

[JOduMonT]

@robi said in Nextcloud with s3 as a backend storage:

When Nextcloud is being deployed at scale and the S3 storage layer is on the same or adjacent local network, it makes sense.

got it, so using it with minio, would make sense

[JOduMonT]

@robi said in Does it make sense to use s3 (Backblaze/Wasabi) as a backend storage ?:

When Nextcloud is being deployed at scale and the S3 storage layer is on the same or adjacent local network, it makes sense.
If your VPS is at one provider and your S3 is on another, it's generally not a good idea

I thought about it and so if I follow you it would make sense to use the s3 for folders/contains like pictures, video, music and pdf because it is less likely to have micro/small interaction (edition and collaboration) on them and keep documents, notes and talk on a local drive right ??

[robi]

@jodumont No, nextcloud has a completely different install mode which is S3 only for everything.