Help! Updating Nextcloud deleted my user account (but files are still present on-disk)

[FalconFour]

Cloudron has a few issues I've had to work around... (1) lack of support for Cloudflare tunnel led me to needing a complex workaround for Cloudron trying desperately and angrily to generate its own certificates (not needed, just let me handle it!). And (2) file-storage being a primary purpose of my Cloudron system, leading to me disabling auto-backups and using btrfs snapshots onto an external device instead (tar.gz and rsync are both unacceptable options for storage space & efficiency reasons). There is no way the backup system would be capable of handling 3+ TB backups... nor do I want to spend the kind of money on storage to flatten my de-duplicated BTRFS array and slog it all out the front door as a *.tar.gz. The system I have with BTRFS snapshots/clones works beautifully and I do have a backup of the whole volume the Nextcloud app is "homed" in - from before these updates.

Today I embarked upon "realizing that auto-updates were not working at all in Cloudron", and processing a number of Nextcloud updates: 30.0.3.3(?) -> 30.0.4 -> 30.0.5 -> (Now we're gonna break your accounts with LDAP to OIDC) -> everything broke.

I don't use "log in with Cloudron" - it wasn't offered during setup, so I used the "admin" account to create my own account, then made that an admin and deleted the "admin" account, as recommended. My Nextcloud account (with all my data) never had an association with an external ID provider (like Cloudron).

After the update, I tried logging in - and got an error that some error had occurred, and try again. Next time I tried, "unknown username/password".

I used the console "occ user:list" command to discover: only the admin account exists now. My own user account is just gone.

Trying to "log in with Cloudron" just gives an error about OpenID:

First thing on my mind: can we not make updates that, without explicit warning and options, may lead to deleting user accounts? The update that I suspect broke everything, wasn't even a version update - it explcitly said it would make changes to accounts and login, but this seemed to be a necessary intermediate step that, I had hoped/imagined, wouldn't affect me (not using SSO).
Second thing on my mind: how the hell do I get my user account back and all its files/settings/functionality?

[FalconFour]

Happy news: back to my (unhappy) old web-admin roots, I dug into the PostgreSQL database and found that the "oc_users" entry for me had completely disappeared - but I still existed as a "ghost" all over the system (e.g. "recent admins", I'm there). Something seems to have just deleted me from "oc_users" in a "dirty" way. Simply re-adding myself back (directly to "oc_users" with an INSERT, with a garbage string as a password, the same UID as before, and the same UID as "uid_lower") and then "occ user:resetpassword" on my account... I'm back in! Everything is back.

God let me never have to deal with a scare like that again. Now I'm afraid to apply any updates...

[FalconFour]

Boo.

Didn't think it warranted another post, so I went to edit my last reply to add:

But because I love staring death in the face to take a leap into a better place, I finished the remaining hop, hop, hop, hop, hop, hop... each one taking 2-3 minutes to process... and finally sitting at NextCloud 31.0.6 (package 5.5.2) with my user account still intact after the fix. Yay.

[james]

Hello @FalconFour

@FalconFour said in Help! Updating Nextcloud deleted my user account (but files are still present on-disk):

can we not make updates that, without explicit warning and options, may lead to deleting user accounts?

The update for Nextcloud that switched LDAP to OIDC was a manual update that required manual intervention.
It informed every Cloudron administrator via a system notification and via the app update screen.

@FalconFour said in Help! Updating Nextcloud deleted my user account (but files are still present on-disk):

"realizing that auto-updates were not working at all in Cloudron"

Exactly why it was not auto updating, since it was manual upgrade to inform everyone.

@FalconFour said in Help! Updating Nextcloud deleted my user account (but files are still present on-disk):

so I used the "admin" account to create my own account, then made that an admin and deleted the "admin" account, as recommended. My Nextcloud account (with all my data) never had an association with an external ID provider (like Cloudron).

Does your local Nextcloud account, that you have created manually, the same username as your user in Cloudron?
If so, this might lead to issues with OIDC migration and might be the issue why this happened.
By default, all local accounts within Nextcloud should not be affected by this migration.

---

@FalconFour said in Help! Updating Nextcloud deleted my user account (but files are still present on-disk):

how the hell do I get my user account back and all its files/settings/functionality?

You can always restore the app from a backup but since =>

@FalconFour said in Help! Updating Nextcloud deleted my user account (but files are still present on-disk):

leading to me disabling auto-backups

This is no option for you.
Since you have your own SNAPSHOT/Backup Solution with BTRFS we can't really help you with the restore issue.

---

It is great to read that you were able to restore you local account and are now back on track with the updates.
I would really like to investigate this issue.
A local account should not have been touched in any way with this migration update.

Could you please share a little more details about your Nextcloud setup and update history?
When installing the app, did you choose "let all Cloudron users login" or "let the app handle user management"?

You are completely correct that this type of scare is unacceptable, so I would appreciate it immensely if you could assist me in improving Cloudron.

[FalconFour]

Thanks for the offer to help! Actually, the update that triggered the scare was buried a few layers further - it seems it hadn't been updated since installation (or near after) - with the version I was running released late 2024, and the system installed October '24. When I first hit the "update" button, it took me a Nextcloud version newer, then once again, then after those two upgrades (to .5, I think), I was offered the LDAP change - but it seemed totally normal.

I definitely can't say the system did much to notify me of the updates or the change (I don't really hear much from Cloudron itself, but the Nextcloud app pings me about "app updates" available). When clicking through it, it presented the usual changelog, which I read - and the bit about the auth change sounded like a "that should be fine?" by the way it was presented.

Unfortunately I can't recall for sure what settings I chose at the time of install. Is there any way to determine that after-the-fact?

As far as diagnosis, I did download the logs at the time (during my repair attempts). 15 MB of logs, spanning back with lots of personal activity and filenames, etc... maybe I can try and sanitize that (e.g. trim to just the upgrade points) and send it along?

As for restoring from a backup, is it okay to just restore the app volume's structure (containing "apps", "config", "data", etc) but omit the file tree (/data/myname/*)? That is, is there additional metadata about the app that might get corrupted by doing so? If I were able to do that, I believe it contains the database and everything should just snap into place if I "stop" the app, restore everything, then "start" it, I'd imagine...

(that is, if a future situation puts me in a situation where I need to restore such a backup... I'm good at the moment; it seems everything is working!)

Let me know about that log, and maybe how to determine its user config, and I'll give you what I can

edit to add: my Cloudron username is, indeed, the same as my Nextcloud username.

[FalconFour]

I'm also just realizing the database might not be in the volume I think it is... that means I'd better figure out where it is, and fast. I expected everything to be stored on the volume I installed the app to If it were, a btrfs snapshot (with "btrfs send | btrfs receive" cloning and management to the external backup drive) would just Thanos-snap it into persistent security... (btrfs is pretty freaking awesome, imo... 10/10 would store life on it)

Guidance on where to manually back-up/otherwise protect the database would be appreciated

[nebulon]

Cloudron uses addons (services) for apps when it comes to the database. This means there is a postgres service which Nextcloud uses for its database. The volume or storage of an app is just the filesystem (for an app /app/data) which is the writable location within the app. Thus the database is not part of the volume. One core reason is, because database systems need to trigger a dump or a commit point to flush data to disk. One cannot safely rely on a filesystem for database files to be consistent (thus a btrfs snapshot will not reliably work for a database like postgres, btrfs has no clue about the flush state of the database)

For the backups of large amounts, the rsync method with a backend capable of using hardlinks like a sshfs backend should work just fine, as it does not duplicate files. Overall the Cloudron backup/restore tasks work only with onboard backup logic, which as mentioned with databases, cannot just be filesystem snapshots.

[FalconFour]

Or you could just, Place a copy of the database backup on the app's volume mayhaps Seems the need/request for alternate backup strategies has been out there for a long time (seen a post from 2020 asking for it -> https://forum.cloudron.io/topic/2503/nextcloud-backup-without-data -> kinda left at an unsatisfactory resolve; settling for rsync with some hackarounds). My backup today with an incremental btrfs snapshot just took a couple minutes to transfer to the USB drive - and in fact, I later realized I didn't need it at all! The prior snapshot already existed (as it has to increment from one shot to the next, the previous snapshot remains in place), so I could've just browsed that - even if I lost all my files.

Regardless, I found my friend in this at the "Terminal" for the Nextcloud app:

PGPASSWORD=${CLOUDRON_POSTGRESQL_PASSWORD} pg_dump -h postgresql -U ${CLOUDRON_POSTGRESQL_USERNAME} -d ${CLOUDRON_POSTGRESQL_DATABASE} -f /your/backup/location/database.sql

Takes just a couple seconds, bang-bang and done, database secured. Would be nice to have a button to "export database" or the like, for alternate backup modes like this. I don't expect to be able to one-click restore it without support for this btrfs mechanism, but I know the data is there - and in case of disaster, there's a lot bigger things to worry about.

The current backup mechanism might work for non-data-centric apps, but for Nextcloud, it's enormous and needs outside-the-box thinking for efficiently managing terabytes of data in backup.

(Not going to use it, but I think I also located the database itself, at /home/yellowtent/platformdata/postgresql - not something to be directly backed-up or manipulated of course )

[BrutalBirdie]

@FalconFour
Since you have a very technical custom backup / snapshot solution with BTRFS and you wrote you are missing the Databases in your backup. (hope I did not fly over the text and missunderstood?¿)
You can always create a cron-job on the root server itself to dump all Databases into your backup destination.

NOTE:
I also use BTRFS on my Notebook for fast snapshots before each system update or package installation and also now love the blazing fast snapshot time.
But when something breaks with BTRFS the fixing becomes a real pain.
And still I see BTRFS snapshots as such, snapshots.
Even to a different disk, I would not consider it a real backup.
On the other hand when using btrfs send and btrfs receive in conjunction this could be a solid approach.
Maybe I should add a btrfs send/receive weekly task to my notebook.
Right now I use restic as an incremental backup solution to a Hetzner storage box.

Also, here is my approach of bricking my BTRFS snapshots

[USER STORY] ICU blocks system upgrade? - I just ignore it! NO, DO NOT!