MySQL binlogs growing unbounded — 145 GB, no expiry set?

[james]

Hello @jayonrails

I will have to check some things first, please give me 10-15 minutes.

[james]

Hello @jayonrails

Since 26.02.2021 all Cloudron installation ship the MySQL system server with:

skip-log-bin

in the /etc/mysql/my.cnf file.

Is your Cloudron installation older than 26.02.2021?

Please check your /etc/mysql/my.cnf file if skip-log-bin is set or if for example:

expire_logs_days = 10
max_binlog_size = 100M

or

binlog_expire_logs_seconds = 864000
max_binlog_size = 100M

You can also run the following command to get the live configuration:

mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@expire_logs_days, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"

[jayonrails]

hey @james,

thanks for your investigation. Let met get straight to your questions:

@james sagte:

Is your Cloudron installation older than 26.02.2021?

No, never has been.

@james sagte:

/etc/mysql/my.cnf

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

@james sagte:

You can also run the following command to get the live configuration:

root@my:~# mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@expire_logs_days, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1193 (HY000) at line 1: Unknown system variable 'expire_logs_days'

[james]

Hello @jayonrails
Your MySQL config looks empty and that is wrong.
Is also assume there is nothing in the folders listed in the config.

Since that command also failed for you please try again without the expire_logs_days:

mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"

[CRBear]

Hello,

I seem to have the same or a very similar issue as reported here:

https://forum.cloudron.io/topic/15988/mysql-binlogs-growing-unbounded-145-gb-no-expiry-set/18

My system:

Cloudron: 10.0.5
Ubuntu:   Ubuntu 26.04.1 LTS
Kernel:   Linux 7.0.0-31-generic

The root filesystem is currently at 93%:

/dev/sda1   150G   134G   11G   93% /

The unexpected usage is mainly:

/var/lib/mysql   76G

There are hundreds of binlog.xxxxxx files, mostly about 101 MB each.

MySQL version:

8.4.11-0ubuntu0.26.04.1

Current MySQL settings:

log_bin                       = 1
binlog_expire_logs_seconds    = 2592000
max_binlog_size               = 104857600
binlog_expire_logs_auto_purge = 1

SHOW REPLICA STATUS\G returns no replica.

/etc/mysql/my.cnf only contains:

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

There is no skip-log-bin entry.

I have not deleted any binlogs or changed the MySQL configuration yet.

Could you please advise what I should do next?

In particular, is it safe/recommended in this situation to use:

RESET BINARY LOGS AND GTIDS;

and should skip-log-bin then be restored manually, or is there another Cloudron-supported fix?

The server currently has only about 11 GB free, so I would prefer to resolve this before the root filesystem fills up.

Thanks in advance for your help!

[nebulon]

So just to confirm here, this is about the Ubuntu host system mysql, not the mysql addon service running in docker?

If so how old is that Cloudron? The skip-log-bin was added in Cloudron around 2021 to /etc/mysql/mysql.cnf and a Cloudron update should theoretically ensure that this file gets updated accordingly.

[james]

This indeed does look like the same issue as https://forum.cloudron.io/topic/15988/mysql-binlogs-growing-unbounded-145-gb-no-expiry-set
So maybe the upgrade from Ubuntu 24 to Ubuntu 26 resets the /etc/mysql.cnf.
The guide https://docs.cloudron.io/guides/upgrade-ubuntu-26/ explictly states to:

For all packages (mime, nginx, timesyncd, logrotate, journald, etc.), select N or O to keep your currently-installed version (default).

I will try to reproduce this.

[jayonrails]

Good morning @james,

root@my:~# mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-------------------------+-----------+------------------------------+-------------------+---------------------------------+
| VERSION()               | @@log_bin | @@binlog_expire_logs_seconds | @@max_binlog_size | @@binlog_expire_logs_auto_purge |
+-------------------------+-----------+------------------------------+-------------------+---------------------------------+
| 8.4.11-0ubuntu0.26.04.1 |         1 |                      2592000 |         104857600 |                               1 |
+-------------------------+-----------+------------------------------+-------------------+---------------------------------+

@james sagte:

Is also assume there is nothing in the folders listed in the config.

Yes.

As there have been two notes to this thread, I would like to add:

When upgrading from Ubuntu 24.04 to Ubuntu 26.04. I kept all the config files and didn't let it create new ones.

[james]

Hello @jayonrails
I am currently trying to reproduce if the Ubuntu 24 to Ubuntu 26 upgrade resets the my.cnf.

[CRBear]

Yes, this is the Ubuntu host system MySQL under /var/lib/mysql, not the MySQL addon service running in Docker.

The Cloudron dashboard shows:

Cloudron activation time: 2 years ago
Cloudron version: 10.0.5
Ubuntu: 26.04.1 LTS

So the activation is approximately from 2024 and therefore well after skip-log-bin was added in 2021.

Regarding the Ubuntu 24 → 26 upgrade: I used the default N/O option to keep the currently installed configuration files when prompted.

I also checked /var/log/apt/term.log. For the MySQL 8.4 installation itself I cannot see any configuration-file prompt. Instead, the relevant part is:

Setting up mysql-server (8.4.11-0ubuntu0.26.04.1)…
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode

Cloudron's own template still contains:

/home/yellowtent/box/setup/start/mysql.cnf

including:

# disable bin logs. they are only useful in replication mode
skip-log-bin

but skip-log-bin is no longer present anywhere under /etc/mysql.

I have not changed or deleted anything yet. I will wait for your guidance before making any changes, so please let me know what you would like me to do next.

I can also provide any further diagnostics that may help with reproducing this.

Thank you very much.

[james]

Hello @jayonrails and @crbear
I have merged the two topics into this one.

---

I have reproduced the issue, and it comes from upgrading Ubuntu 24 to Ubuntu 26.
A documentation update is deployed as of writing this.

The issue is that Ubuntu upgrade does uninstall the MySQL server on the system and both of you did correctly follow the guide and run:

systemctl stop box
apt install mysql-server
systemctl start box

This installed the MySQL server with the apt package default configuration and not the Cloudron configurations.
To fix this please run the following command from your ssh root/sudo session:

bash /home/yellowtent/box/setup/start.sh

This will ensure all services on the root system including MySQL are configured correctly.

---

Thank you again for bringing this to our attention.

[james]

Hello @jayonrails and @crbear
After running the command please report if this resolved the issue for you.

note: binlogs might not be purged automatically

If that is the case for you please run:

mysql -uroot -ppassword -e "RESET BINARY LOGS AND GTIDS;"

if that also does not clean up the binlog files in /var/lib/mysql/ stop the box.service and mysql.service and delete the files manually.

Example:

systemctl stop box.service mysql.service
# this will ask you for every file for confirmation
rm -rfi /var/lib/mysql/binlog.*
rm: remove regular file '/var/lib/mysql/binlog.000003'? y
rm: remove regular file '/var/lib/mysql/binlog.000004'? y
rm: remove regular file '/var/lib/mysql/binlog.index'? y
systemctl status mysql.service box.service

[CRBear]

Thank you very much - the issue is fully resolved now.

I ran bash /home/yellowtent/box/setup/start.sh, confirmed that skip-log-bin is present again and @@log_bin = 0.

RESET BINARY LOGS AND GTIDS; did not remove the existing files, so I followed your fallback procedure, stopped box.service and mysql.service, removed /var/lib/mysql/binlog.*, and started both services again.

Current status:

/var/lib/mysql: 337M
/: 58G used / 86G available / 41%
@@log_bin = 0

Everything is running normally again.

Thank you again for the incredibly fast and excellent support, and especially for reproducing the issue and updating the documentation so quickly. Much appreciated!

[jayonrails]

@james I cannot maintain this until the weekend, so I would leave FreeScout stopped and report back in a few days.

root@my:~# 
root@my:~# mysql -uroot -ppassword -e "RESET BINARY LOGS AND GTIDS;"
mysql: [Warning] Using a password on the command line interface can be insecure.
root@my:~# mysql -uroot -ppassword -e "SHOW BINARY LOGS;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000001 |   3783544 | No        |
+---------------+-----------+-----------+

This seems fine for now, let see if it blows up again...

[james]

Hello @jayonrails
This is not a Freescout issue.
Please run the instructed commands so the MySQL system server has the correct configuration again.
If you don't do this, it could also lead to future issues like a wrong character set and a wrong collation for databases created in the future by apps, which in turn could lead to backup restore issues.

It is highly advised to run the commands I have provided to ensure that the MySQL system server is running as intended by Cloudron.

[jayonrails]

The topic has been merged and I haven’t seen the post you’re referring to. Thanks, will take care of it.

[jayonrails]

This seems fine now, @james ?

root@my:~# systemctl stop box
root@my:~# apt install mysql-server
mysql-server is already the newest version (8.4.11-0ubuntu0.26.04.1).
The following package was automatically installed and is no longer required:
  libevent-pthreads-2.1-7t64
Use 'apt autoremove' to remove it.

Summary:
  Upgrading: 0, Installing: 0, Removing: 0, Not Upgrading: 15
root@my:~# systemctl start box
root@my:~# bash /home/yellowtent/box/setup/start.sh
2026-09-23T10:43:05 ==> start: Cloudron Start
media:x:500:
2026-09-23T10:43:05 ==> start: Configuring docker
Synchronizing state of apparmor.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable apparmor
2026-09-23T10:43:06 ==> start: Ensuring directories
2026-09-23T10:43:06 ==> start: Configuring journald
2026-09-23T10:43:06 ==> start: Setting up unbound
2026-09-23T10:43:06 ==> start: Adding systemd services
Synchronizing state of unbound.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable cron
Synchronizing state of rpcbind.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install disable rpcbind
2026-09-23T10:43:12 ==> start: Configuring sudoers
2026-09-23T10:43:12 ==> start: Configuring logrotate
2026-09-23T10:43:12 ==> start: Adding motd message for admins
2026-09-23T10:43:12 ==> start: Configuring nginx
2026-09-23T10:43:13 ==> start: Stopping mysql
2026-09-23T10:43:17 ==> start: Starting mysql
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
2026-09-23T10:43:17 ==> start: Migrating data
[INFO] No migrations to run
[INFO] Done
2026-09-23T10:43:18 ==> start: Changing ownership
2026-09-23T10:43:18 ==> start: Starting cloudron-syslog
2026-09-23T10:43:18 ==> start: Starting Cloudron
2026-09-23T10:43:20 ==> start: Almost done
root@my:~# mysql -uroot -ppassword -e "SHOW BINARY LOGS;"
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1381 (HY000) at line 1: You are not using binary logging

[james]

Hello @jayonrails

The only instruction was:

james said:

To fix this please run the following command from your ssh root/sudo session:

bash /home/yellowtent/box/setup/start.sh

But you also did run:

systemctl stop box
apt install mysql-server
systemctl start box

From the log it looks like everything worked out correctly.
Please make sure to carefully read provided instructions in the future since you did run commands that where not instruced to be executed.

Please also read https://forum.cloudron.io/post/129886 which explains that the binlogs might not be purged correctly and how to clean it up.
Otherwise some binlog files might remain and block some diskspace.