my.cloudron.tld is not accessible (mysql down but apps are online)

[james]

Hello @hpz24
Can you please check the file /var/log/mysql/error.log for logs?

[hpz24]

Sure,
here is the output:

2025-12-01T08:48:55.564514Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.44-0ubuntu0.22.04.1) starting as process 14253
2025-12-01T08:48:55.582334Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-12-01T08:48:56.300337Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-12-01T08:48:56.423692Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: fut0lst.ic:82:addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA thread 140714909271616
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2025-12-01T08:48:56Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=1d7b54cce103c7eb7743f877f7974b44575bd8ee
Thread pointer: 0x5595afeed3b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ffabe2e7bd0 thread_stack 0x100000
 #0 0x5595ac3d504d _Z8my_abortv
 #1 0x5595ac5853f2 _Z23ut_dbg_assertion_failedPKcS0_m
 #2 0x5595acc0933f <unknown>
 #3 0x5595ac6c5691 _Z18flst_insert_beforePhS_S_P5mtr_t
 #4 0x5595ac5a189b _Z36trx_purge_add_update_undo_to_historyP5trx_tP14trx_undo_ptr_tPhbmP5mtr_t
 #5 0x5595ac5a1af2 _Z23trx_undo_update_cleanupP5trx_tP14trx_undo_ptr_tPhbmP5mtr_t
 #6 0x5595ac5a1ead <unknown>
 #7 0x5595ac5a25ad _Z14trx_commit_lowP5trx_tP5mtr_t
 #8 0x5595ac5a2cc7 _Z10trx_commitP5trx_t
 #9 0x5595ac5a41a4 _Z31trx_rollback_or_clean_recoveredb
 #10 0x5595ac55f4e4 _Z27srv_dict_recover_on_restartv
 #11 0x5595ac41dcff <unknown>
 #12 0x5595ac286f1b _ZN2dd9bootstrap7restartEP3THD
 #13 0x5595ac3e4345 <unknown>
 #14 0x5595ac3f531d _ZN2dd10upgrade_5731do_pre_checks_and_initialize_ddEP3THD
 #15 0x5595aba95915 <unknown>
 #16 0x5595ac7d484d <unknown>
 #17 0x7ffac153dac2 <unknown>
 #18 0x7ffac15cf8bf <unknown>
 #19 0xffffffffffffffff <unknown>

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

[james]

Hello @hpz24
Do you have access to your latest Cloudron backups?

[hpz24]

Sure, I have backups.
Should I follow the instructions in https://docs.cloudron.io/troubleshooting/#mysql?

What is, if I need a backup which is idk, 2-3 weeks old ... could that cause other problems?

[james]

Hello @hpz24
Let's try something else first.

Make sure the MySQL service is stopped (I know it is not even starting, but just to make sure).
Create a full backup of the MySQL data with the following command:

rsync -avrlP /var/lib/mysql /root/mysql_backup_$(date +%Y%m%d-%H%M%S)

Edit the file /etc/mysql/my.cnf and under [mysqld] add this line:

innodb_force_recovery = 1

Try to start the service again with:

systemctl start mysql.service

If the start is still failing, edit the same file again and increase innodb_force_recovery to 2 and repeat when failing up to 6.
Details on innodb_force_recovery and what 1 to 6 does https://dev.mysql.com/doc/refman/8.4/en/forcing-innodb-recovery.html
Warning! Do not jump straight to 6. Values ≥4 disable background page-cleaning and make the server effectively read-only; they are last-resort and risky

This is also documented here: https://docs.cloudron.io/guides/community/mysql-recovery/ but missing the backup first step, I will update this community guide later.

---

If this is still not resulting in a solution we can proceed to https://docs.cloudron.io/troubleshooting/#mysql

[hpz24]

So, the forced recovery did not help, still an error when I tried to start the mysql daemon.

Now I wanted to try out the restore function and ran into another problem.

I do have backups (as I pompously wrote above), but the password for decryption doesn't work (for whatever reason).

Is there a way to read the password in the file system of the running Cloudron instance?
Or:
Can the database be regenerated? Basically, everything is working, except the dashboard won't load...

[james]

Hello @hpz24

@hpz24 said in my.cloudron.tld is not accessible (mysql down but apps are online):

Is there a way to read the password in the file system of the running Cloudron instance?

Reading the encryption/decryption password from a running Cloudron is not possible and would be a major security flaw.

@hpz24 said in my.cloudron.tld is not accessible (mysql down but apps are online):

but the password for decryption doesn't work (for whatever reason)

Where does the password not work?
Are you trying to restore the full Cloudron or trying to decrypt a single backup with the cloudron cli?

[hpz24]

I tried it with

$ cloudron backup decrypt --password=passphrase backupid.tar.gz.enc backupid.tar.gz

My password contains special characters, so i tried with ' but I get

Error: Could not decrypt: Invalid password or tampered file (mac mismatch)

The backup itself is only 55mb, i`m afraid, there is not everthing in it ...

app_forum.xxx.yy_v2.22.0.tar.gz.enc

[james]

Hello @hpz24
For some reason, either the decryption password being incorrect or some other reason caused by the cloudron cli you are unable to decrypt the backup.
Now we run into the risk of data loss.

Before we continue with anything, please create a full server backup.
Since you are using netcup, see the following guide for creating an offline snapshot https://helpcenter.netcup.com/en/wiki/server/media/#create-and-export-snapshot

After that, please write a mail to support@cloudron.io, explain the issue and link this topic.

[hpz24]

thanks, done!

[msbt]

@hpz24 I have one machine that has quite a lot of special characters in the encryption password. What I do is this:

cloudron backup decrypt --password="123\$123" ...

So the backslash is not part of the password and is required to escape the $ sign, if that's something that you're using.

[joseph]

The mysql db was corrupt beyond repair. I reimported from a back up and things are good.