my.cloudron.tld is not accessible (mysql down but apps are online)

hpz24

Description

The dashboard has recently become inaccessible, but all apps (e. g. NodeBB) works without problems.

Steps to reproduce

Several reboots, no changes in the behaviour

Logs

2025-12-01T08:45:37.296Z box:server ==========================================
2025-12-01T08:45:37.296Z box:server            Cloudron 8.3.2
2025-12-01T08:45:37.296Z box:server ==========================================
2025-12-01T08:45:37.296Z box:platform initialize: start platform
2025-12-01T08:45:37.329Z box:tasks stopAllTasks: stopping all tasks
2025-12-01T08:45:37.332Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2025-12-01T08:45:37.396Z box:shell All tasks stopped

2025-12-01T08:45:37.409Z Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"connect ECONNREFUSED 127.0.0.1:3306","nestedError":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":3306,"fatal":true},"code":"ECONNREFUSED","sqlMessage":null}
2025-12-01T08:45:37.409Z     at main (/home/yellowtent/box/box.js:58:41)
2025-12-01T08:45:37.409Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Output from systemctl status mysql.service:

root@server:~# systemctl status mysql.service
× mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2025-12-01 08:48:56 UTC; 2min 57s ago
    Process: 14245 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
    Process: 14253 ExecStart=/usr/sbin/mysqld (code=exited, status=2)
   Main PID: 14253 (code=exited, status=2)
     Status: "Server startup in progress"
        CPU: 876ms

Dez 01 08:48:56 v220230337933223533 systemd[1]: Failed to start MySQL Community Server.
Dez 01 08:48:56 v220230337933223533 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Dez 01 08:48:56 v220230337933223533 systemd[1]: Stopped MySQL Community Server.
Dez 01 08:48:56 v220230337933223533 systemd[1]: mysql.service: Start request repeated too quickly.
Dez 01 08:48:56 v220230337933223533 systemd[1]: mysql.service: Failed with result 'exit-code'.
Dez 01 08:48:56 v220230337933223533 systemd[1]: Failed to start MySQL Community Server.

Troubleshooting Already Performed

Reboot done, same again

System Details

Cloudron Version

Cloudron 8.3.2

Ubuntu Version

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

Cloudron installation method

Manual with ./cloudron-setup

Output of `cloudron-support --troubleshoot`

Vendor: netcup Product: KVM Server
Linux: 5.15.0-161-generic
Ubuntu: jammy 22.04
Processor: QEMU Virtual CPU version 2.5+ x 2
RAM: 2010712KB
Disk: /dev/vda3        19G
[OK]	node version is correct
[OK]	IPv6 is enabled and public IPv6 address is working
[OK]	docker is running
[OK]	docker version is correct
	MySQL is down. Trying to restart MySQL ...
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xeu mysql.service" for details.

Thank you for any hints!

james

Hello @hpz24
Can you please check the file /var/log/mysql/error.log for logs?

hpz24

Sure,
here is the output:

2025-12-01T08:48:55.564514Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.44-0ubuntu0.22.04.1) starting as process 14253
2025-12-01T08:48:55.582334Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-12-01T08:48:56.300337Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-12-01T08:48:56.423692Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: fut0lst.ic:82:addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA thread 140714909271616
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2025-12-01T08:48:56Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=1d7b54cce103c7eb7743f877f7974b44575bd8ee
Thread pointer: 0x5595afeed3b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ffabe2e7bd0 thread_stack 0x100000
 #0 0x5595ac3d504d _Z8my_abortv
 #1 0x5595ac5853f2 _Z23ut_dbg_assertion_failedPKcS0_m
 #2 0x5595acc0933f <unknown>
 #3 0x5595ac6c5691 _Z18flst_insert_beforePhS_S_P5mtr_t
 #4 0x5595ac5a189b _Z36trx_purge_add_update_undo_to_historyP5trx_tP14trx_undo_ptr_tPhbmP5mtr_t
 #5 0x5595ac5a1af2 _Z23trx_undo_update_cleanupP5trx_tP14trx_undo_ptr_tPhbmP5mtr_t
 #6 0x5595ac5a1ead <unknown>
 #7 0x5595ac5a25ad _Z14trx_commit_lowP5trx_tP5mtr_t
 #8 0x5595ac5a2cc7 _Z10trx_commitP5trx_t
 #9 0x5595ac5a41a4 _Z31trx_rollback_or_clean_recoveredb
 #10 0x5595ac55f4e4 _Z27srv_dict_recover_on_restartv
 #11 0x5595ac41dcff <unknown>
 #12 0x5595ac286f1b _ZN2dd9bootstrap7restartEP3THD
 #13 0x5595ac3e4345 <unknown>
 #14 0x5595ac3f531d _ZN2dd10upgrade_5731do_pre_checks_and_initialize_ddEP3THD
 #15 0x5595aba95915 <unknown>
 #16 0x5595ac7d484d <unknown>
 #17 0x7ffac153dac2 <unknown>
 #18 0x7ffac15cf8bf <unknown>
 #19 0xffffffffffffffff <unknown>

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

james

Hello @hpz24
Do you have access to your latest Cloudron backups?

hpz24

Sure, I have backups.
Should I follow the instructions in https://docs.cloudron.io/troubleshooting/#mysql?

What is, if I need a backup which is idk, 2-3 weeks old ... could that cause other problems?

james

Hello @hpz24
Let's try something else first.

Make sure the MySQL service is stopped (I know it is not even starting, but just to make sure).
Create a full backup of the MySQL data with the following command:

rsync -avrlP /var/lib/mysql /root/mysql_backup_$(date +%Y%m%d-%H%M%S)

Edit the file /etc/mysql/my.cnf and under [mysqld] add this line:

innodb_force_recovery = 1

Try to start the service again with:

systemctl start mysql.service

If the start is still failing, edit the same file again and increase innodb_force_recovery to 2 and repeat when failing up to 6.
Details on innodb_force_recovery and what 1 to 6 does https://dev.mysql.com/doc/refman/8.4/en/forcing-innodb-recovery.html
Warning! Do not jump straight to 6. Values ≥4 disable background page-cleaning and make the server effectively read-only; they are last-resort and risky

This is also documented here: https://docs.cloudron.io/guides/community/mysql-recovery/ but missing the backup first step, I will update this community guide later.

If this is still not resulting in a solution we can proceed to https://docs.cloudron.io/troubleshooting/#mysql

hpz24

So, the forced recovery did not help, still an error when I tried to start the mysql daemon.

Now I wanted to try out the restore function and ran into another problem.

I do have backups (as I pompously wrote above), but the password for decryption doesn't work (for whatever reason).

Is there a way to read the password in the file system of the running Cloudron instance?
Or:
Can the database be regenerated? Basically, everything is working, except the dashboard won't load...

james

Hello @hpz24

@hpz24 said in my.cloudron.tld is not accessible (mysql down but apps are online):

Is there a way to read the password in the file system of the running Cloudron instance?

Reading the encryption/decryption password from a running Cloudron is not possible and would be a major security flaw.

@hpz24 said in my.cloudron.tld is not accessible (mysql down but apps are online):

but the password for decryption doesn't work (for whatever reason)

Where does the password not work?
Are you trying to restore the full Cloudron or trying to decrypt a single backup with the cloudron cli?

hpz24

I tried it with

$ cloudron backup decrypt --password=passphrase backupid.tar.gz.enc backupid.tar.gz

My password contains special characters, so i tried with ' but I get

Error: Could not decrypt: Invalid password or tampered file (mac mismatch)

The backup itself is only 55mb, i`m afraid, there is not everthing in it ...

app_forum.xxx.yy_v2.22.0.tar.gz.enc

james

Hello @hpz24
For some reason, either the decryption password being incorrect or some other reason caused by the cloudron cli you are unable to decrypt the backup.
Now we run into the risk of data loss.

Before we continue with anything, please create a full server backup.
Since you are using netcup, see the following guide for creating an offline snapshot https://helpcenter.netcup.com/en/wiki/server/media/#create-and-export-snapshot

After that, please write a mail to support@cloudron.io, explain the issue and link this topic.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

my.cloudron.tld is not accessible (mysql down but apps are online)

Description

Steps to reproduce

Logs

Troubleshooting Already Performed

System Details

Cloudron Version

Ubuntu Version

Cloudron installation method

Output of `cloudron-support --troubleshoot`

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

my.cloudron.tld is not accessible (mysql down but apps are online)

Description

Steps to reproduce

Logs

Troubleshooting Already Performed

System Details

Cloudron Version

Ubuntu Version

Cloudron installation method

Output of cloudron-support --troubleshoot

Output of `cloudron-support --troubleshoot`