Backups to Storj suddenly fails: "Error listing objects in snapshot/app_0c3fb7c9-77c7-4d96-a90a-b75b5d1562bf. Message: ListObjects(V2): listing too many items for gateway-side filtering"

[thoresson]

As of this morning, my backups to Storj suddenly fails. This is the error message I get.

Error listing objects in snapshot/app_0c3fb7c9-77c7-4d96-a90a-b75b5d1562bf. Message: ListObjects(V2): listing too many items for gateway-side filtering using arbitrary delimiter/prefix HTTP Code: NotImplemented

Any ideas?

[nebulon]

oh seems like Storj does not implement some S3 feature for filtering. Can you check how many objects you have in that path in your bucket?

Can you maybe contact them to let them know and i maybe get some info if they will add this?

[girish]

If you are using rsync, maybe you can try tgz. Looks like S3 API has some incompatibility with too many files?

[thoresson]

@nebulon Sure – if I only can figure out how to see how many objects there are. Do you know?

@girish Yeah, using rsync. Can try tgz once I've checked in with Storj's support.

[thoresson]

Got this reply from Storj @girish @nebulon

Could you change some settings in this software to use a delimiter for prefixes? By default it's `/` (a forward slash). Without delimiting you will hit the listing limit if you would use S3.  
As an alternative you may use a native Storj protocol, it doesn't have this limit, but I'm not sure that you can use it in this software.  
What's supposed to do? if backups, then I can suggest to use a normal ones, who can use prefixes correctly (and may reduce your storage costs by using a larger segments (less or equal to 64MiB)): [https://docs.storj.io/dcs/third-party-tools#backups](https://docs.storj.io/dcs/third-party-tools#backups)  
   
Our S3 compatibility you can see there: [https://docs.storj.io/dcs/api/s3/s3-compatibility](https://docs.storj.io/dcs/api/s3/s3-compatibility)  
However, there is a limitation, how much objects you can list using the not limited prefix (like a prefix without a trailing `/`).

[girish]

@thoresson mm, ok. Usually, each storage provider has their own quirk (like the one you reported). I think we have to test with storj properly to see all the issues. I think this was requested in https://forum.cloudron.io/topic/5171/add-storj-and-filebase-as-backup-options/ . We have to test this when we add that storage provider.

[girish]

I put storj separately at https://forum.cloudron.io/topic/11670/add-storj-as-backup-option