Unable to restore

[ivan-petro]

All applications are down. I need to fully restore the server((
Auto-update will be permanently disabled. I don’t understand why I enabled it—I always disable it, and it only updates after several weeks.

[joseph]

@ivan-petro you shouldn't have to restore fully, all the data is on the server. Do you have any error in the logs? In fact, if you write to support@cloudron.io, we can check if this is the same as issue has here. (8.3.1 is already out with a fix)

[ivan-petro]

Hi Joseph,

I'm experiencing an issue where I can't start the database, and it's urgent as users are contacting me on multiple channels. I've been troubleshooting for about 30 minutes, and a restart didn't help. I attempted to restore the server, but now I'm facing additional issues. My Default Data Directory was migrated according to the Cloudron documentation (https://docs.cloudron.io/storage/#default-data-directory), and as a result, I'm unable to start some of the Cloudron services.

Could you please help me resolve these issues?

Thank you for your assistance.

[joseph]

@ivan-petro We need some logs or some error messages to help debug . Is the restore issue and default data directory change related? Why do you have to start Cloudron services when restoring?

[ivan-petro]

@joseph ,

I'm having difficulty understanding what is happening. I'm in the process of preparing logs for you. Could you please advise on how to force an update to version 8.3.1? Currently, I'm unable to update because the apps are stuck in the starting state.

Thank you for your assistance.

[joseph]

@ivan-petro try this:

• Edit the file /home/yellowtent/box/src/infra_version.js
• Change the one line as per https://git.cloudron.io/platform/box/-/commit/e3982e48ea57e67b391e6ec4e95e3a863dd1e647 . The postgres line should become like below:

'postgresql': 'registry.docker.com/cloudron/postgresql:6.0.1@sha256:c24dc6ff43cc4f7dc080836adf5e9fd0d1e0e65b44b4e2d107faa379cb0185ab',

• systemctl restart box . after this postgres container will automatically update
• I think depending on what has happenned previously the apps may error or start in some fresh state. In such a case, just go to restore of each app and restore from the last backup.

[ivan-petro]

Hi Joseph,

I believe a hard update to version 8.3.1 is necessary—not just updating PostgreSQL. Here are the details of the actions I've taken so far:

1. I encountered an issue after the automatic update to 8.3.0.
2. I wasn’t able to resolve the PostgreSQL service error, so I performed a “hard” restore using a Hetzner backup of the entire server.
3. However, my Cloudron data is stored on a separate volume (as per the documentation: https://docs.cloudron.io/storage/#default-data-directory), so I restored to version v8.2.4. There might be issues because I restored the main drive but not the volume files.
4. On version v8.2.4, I’m now unable to start all Redis services.
5. I’m unsure whether to continue troubleshooting version 8.2.4 or proceed with a hard update to 8.3.1.

I've attached the latest Cloudron logs and Redis logs to this message. Could you please advise on the best course of action?

Thank you for your assistance.

P.S. Can't attach files to this message. some errors from forum.

please check on google drive:

https://drive.google.com/drive/folders/1pRt7KKP4Sjynmf-vvmDrmc1164QYL82u?usp=sharing

[joseph]

@ivan-petro said in Unable to restore:

My Default Data Directory was migrated according to the Cloudron documentation (https://docs.cloudron.io/storage/#default-data-directory),

Did you move platformdata, appsdata and the boxdata (or just some of them?) .

Thing is HEtzner backup of full server will of course not include any of these if it's in some other disk .

Since the update to 8.3.0 was initiated previously, the data in platformdata atleast is already post 8.3.0 (or in some intermediate state) and the database versions of 8.2.4 won't be able to handle it. I am guess redis doesn't start because 8.3.0 has redis 7 and 8.2.4 had older redis.

Do you have Cloudron backups? Might be easiest to restore from Cloudron backups of 8.2.4 .

[ivan-petro]

@joseph ,

Did you move platformdata, appsdata and the boxdata (or just some of them?) .

yes, I moved everything according docs: https://docs.cloudron.io/storage/#default-data-directory
this is my setup for a long time already.

Do you have Cloudron backups? Might be easiest to restore from Cloudron backups of 8.2.4 .

do you mean restore of every apps? or whole cloudron? I can't see backup of whole Cloudron. only apps backups.

--
is there are way to ignore that apps can't start and make hard update to 8.3.0 to fix issues? or to 8.3.1?

[ivan-petro]

@joseph , I can't do anything now, and can't restores apps from backups.
all apps in state "Restarting - Waiting for platform to initialize"
due to redis issue.

How to force fix this? really need help 8 hours of outage

[joseph]

@ivan-petro from what i understand, the setup is not correct.

I am afraid to give suggestions which might result in loss of data. Can you give the contents of files below:

• /home/yellowtent/platformdata/VERSION
• /home/yellowtent/box/VERSION

Can you also tell me where you saved Cloudron Backups. This is the Backups view. Are they saved outside the server or were they in the default /var/backups ?

[ivan-petro]

@joseph , please check output:

root@tc-cloudron:~# cat /home/yellowtent/box/VERSION
8.2.4
root@tc-cloudron:~# cat /home/yellowtent/platformdata/VERSION
8.2.4root@tc-cloudron:~#

---

backups stored on AWS.
but I can't understand should I do manual restore of every app?) I have around 100 apps.

can I just make hard update of cloudron?

and I can't make restore of apps due to status "Restarting - Waiting for platform to initialize"
all apps in this state now.

[joseph]

@ivan-petro you said you moved the platformdata to somewhere else. the VERSION file is from that moved location, correct ?

So, if I understood correctly, you have reverted back both the main volume and the storage where you moved the (platformdata, appsdata, ...) storage .

If the above is correct, I am not even sure why it says "Waiting for platform to initialize" . Haven't you reverted everything to the old state? What am I missing? The confusion is that if everything is reverted to 8.2.4, that message should never appear.

[ivan-petro]

@joseph , my data is moved.
I did it several month ago according docs:
https://docs.cloudron.io/storage/#default-data-directory

it's symlinks. you may check:

root@tc-cloudron:/home/yellowtent# ls -lah
total 28K
drwxr-xr-x 4 yellowtent yellowtent 4.0K Feb 6 11:04 .
drwxr-xr-x 6 root root 4.0K Mar 9 2023 ..
lrwxrwxrwx 1 root root 34 Jan 23 00:58 appsdata -> /mnt/tc-cloudron-volume-1/appsdata
-rw-r--r-- 1 yellowtent yellowtent 220 Jan 6 2022 .bash_logout
-rw-r--r-- 1 yellowtent yellowtent 3.7K Jan 6 2022 .bashrc
drwxr-xr-x 8 yellowtent yellowtent 4.0K Feb 2 10:16 box
lrwxrwxrwx 1 root root 33 Jan 23 00:58 boxdata -> /mnt/tc-cloudron-volume-1/boxdata
-rw-r--r-- 1 yellowtent yellowtent 0 Dec 11 2022 .cloud-locale-test.skip
drwx------ 2 yellowtent yellowtent 4.0K Apr 2 2023 .gnupg
lrwxrwxrwx 1 root root 38 Jan 23 01:05 platformdata -> /mnt/tc-cloudron-volume-1/platformdata
-rw-r--r-- 1 yellowtent yellowtent 807 Jan 6 2022 .profile

[ivan-petro]

@joseph
"The confusion is that if everything is reverted to 8.2.4, that message should never appear"
yes it's interesting. can't understand wha's going on. I just can told you that I have setup like in your docs here:

https://docs.cloudron.io/storage/#default-data-directory

My platformdata , boxdata, appsdata on seperate volume.

I did restore from hetzner backup of main server disk.

[joseph]

@ivan-petro yes, so what I am wondering is if the separate volume was also restored to 8.2.4 state (i.e alongside the main server disk).

Think of it this way: you have mysql v6 (your main disk) . you store mysql's data in another volume. now you restored mysql with v4 (i.e restored your main disk). it won't start with the data in another volume. this is because the data is of a newer version i.e v6. this is what I am asking. this is possibly why redis is also not starting. most software won't work with old code and new data .

[ivan-petro]

@joseph , ok, how to resolve this?
is there are way to make hard update of cloudron? ignoring state of apps

[joseph]

@ivan-petro depending on the state maybe we have to hotfix the server with 8.3 . but this is mostly a developer tool. if you write to me on support@cloudron.io , i can try to hotfix the server.

[ivan-petro]

@joseph , Ok, I will write. just interesting if I can stop all apps by one command and make hard update to 8.3?

[joseph]

@ivan-petro it's also quite easy to just start afresh using the AWS backup . See https://docs.cloudron.io/backups/#restore-cloudron .