v7.10.3 / 1.75.1 lost/erased all files from S3

[potemkin_ai]

@girish both - the bucket is empty and S3 connection is broken.

Yeah - I don’t consider that could be Cloudron issue in any way - I’ve seen the only thing that’s changed on the packaging - it’s a patch version update.

Retention policy - not on Mattermost side, but I didn’t think of that, thanks for a new vector of thoughts!

[robi]

I had something similar happen at a different S3 provider, and after 2 months, the solution was to delete that bucket and create a new one, as their infrastructure has upgraded past the support for the original bucket code version, heh.

[potemkin_ai]

@robi wtf ?!
Could possibly share the providers names? And how on earth could it be that s3 content can get lost because of the version’s update?!

[robi]

@potemkin_ai nah, too much drama

[girish]

I would also consider contacting Exoscale about it and asking them if they have any audit logs of who deleted the files. Maybe the originating IP and such.

[potemkin_ai]

@robi I understand…
Is it something you had with Mattermost as well? Or it was something completely different?

I’m just trying to figure out the dangerous use case.

@girish thank you! I did!

[robi]

@potemkin_ai nope just regular backups

[potemkin_ai]

@robi who cares about backups, right?
I would appreciate though if you could share (here or any other way) the providers names; not a problem if not, but I would feel better knowing what to avoid.

[potemkin_ai]

In my specific case, Exoscale's support was quite helpful - they checked the logs of the script that calculates the bucket size and they pointed out that the bucket was always empty.

After additional checks, I realized that Mattermost doesn't respect file's storage settings and doesn't upload files to s3, even if S3 connection works; I believe something goes wrong with using those settings which leads Mattermost to silently default to work with local storage. I understand it shall be reported upstream, but I have no capacity for that right now, so just leaving this note here for now - in case if someone will encounter this issue as well.

[scooke]

So, where did your files go?? They just stayed local, but appeared as remote?

[potemkin_ai]

@scooke yes, the files were stored locally all of the time, despite my settings were quite opposite (S3)