MySQL binlogs growing unbounded — 145 GB, no expiry set?

[james]

Hello @jayonrails

@jayonrails said:

After adding the changes you suggested, I noticed that it does not match the styling of the other entries. After starting, I got the following error:

The code block I have posted with the URLs are source code and not environment variables.
I should have been more clear, sorry.
The environment variables names are ACTIVITY_LOGGER_ENABLED and ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS.

---

In your /app/data/env you have shared

@jayonrails said:

export FILE_STORAGE=local
export FILE_STORAGE_UPLOAD_MAX_SIZE=26214400

there are these two entries which look incorrect.
No other entries in the /app/data/env have the export $SOME_ENVIRONMENT_VARIABLE=$VALUE style.
Also I did not find anything in the upstream Freescout code that uses these two environment variables.
You might want to look into that as well.

---

@jayonrails said:

how can I remove the data from the mySQL binlogs?

Can you please share where exactly you have found these binlogs?
Are they inside the freescout container or on the host system?
When on the host system please connect to your server with ssh and share the output of the following command:

ls -lah /home/yellowtent/platformdata/mysql/
# and
du -hd 0 /home/yellowtent/platformdata/mysql

---

@jayonrails said:

I would like to know when "ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS" has been implemented.

We Cloudron have never implemented ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS.
This is a default from Freescout and has been like that as far as I can tell for 8 years.

@jayonrails said:

My FreeScout installtion probably didn't have that when I first installed it.

With my statement above, no, it has always been like that unless configured otherwise.

[jdaviescoates]

@james said:

No other entries in the /app/data/env have the export $SOME_ENVIRONMENT_VARIABLE=$VALUE style.

If I'm not mistaken, that style used to be how things were routinely done on Cloudron(?), so this is probably just left over from it being an old install?

[jayonrails]

@james sagte:

The code block I have posted with the URLs are source code and not environment variables.
I should have been more clear, sorry.

no problem at all :))

@james sagte:

there are these two entries which look incorrect.
No other entries in the /app/data/env have the export $SOME_ENVIRONMENT_VARIABLE=$VALUE style.
Also I did not find anything in the upstream Freescout code that uses these two environment variables.
You might want to look into that as well.

I will check this!

@james sagte:

Can you please share where exactly you have found these binlogs?
Are they inside the freescout container or on the host system?

root@my:~# # Biggest items inside /var/lib/mysql, one level deep
du -xh --max-depth=1 /var/lib/mysql | sort -h | tail -15
36K	/var/lib/mysql/mysql
116K	/var/lib/mysql/sys
804K	/var/lib/mysql/#innodb_temp
1.7M	/var/lib/mysql/performance_schema
101M	/var/lib/mysql/#innodb_redo
213M	/var/lib/mysql/box
173G	/var/lib/mysql
root@my:~# 
root@my:~# # Binlogs vs everything else, side by side
du -ch /var/lib/mysql/binlog.* | tail -1
du -sh --exclude='binlog.*' /var/lib/mysql
173G	total
424M	/var/lib/mysql
root@my:~# # Oldest and newest, with timestamps
ls -lh --time-style=long-iso /var/lib/mysql/binlog.* | head -5
ls -lh --time-style=long-iso /var/lib/mysql/binlog.* | tail -5

# The index MySQL maintains — one line per binlog it knows about
wc -l /var/lib/mysql/binlog.index
head -3 /var/lib/mysql/binlog.index
-rw-r----- 1 mysql mysql  180 2026-09-01 14:36 /var/lib/mysql/binlog.000001
-rw-r----- 1 mysql mysql  180 2026-09-01 14:38 /var/lib/mysql/binlog.000002
-rw-r----- 1 mysql mysql  180 2026-09-01 14:38 /var/lib/mysql/binlog.000003
-rw-r----- 1 mysql mysql  441 2026-09-11 17:40 /var/lib/mysql/binlog.000004
-rw-r----- 1 mysql mysql 101M 2026-09-11 17:49 /var/lib/mysql/binlog.000005
-rw-r----- 1 mysql mysql 101M 2026-09-22 17:23 /var/lib/mysql/binlog.001764
-rw-r----- 1 mysql mysql 101M 2026-09-22 17:32 /var/lib/mysql/binlog.001765
-rw-r----- 1 mysql mysql 101M 2026-09-22 17:41 /var/lib/mysql/binlog.001766
-rw-r----- 1 mysql mysql  79M 2026-09-22 17:48 /var/lib/mysql/binlog.001767
-rw-r----- 1 mysql mysql  28K 2026-09-22 17:41 /var/lib/mysql/binlog.index
1767 /var/lib/mysql/binlog.index
./binlog.000001
./binlog.000002
./binlog.000003
root@my:~# docker exec mysql ls -lh /var/lib/mysql/ | head -5
docker exec mysql df -h /var/lib/mysql
total 295M
-rw-r----- 1 mysql mysql 6.0M Sep 22 17:48 #ib_16384_0.dblwr
-rw-r----- 1 mysql mysql  14M Sep 21 22:47 #ib_16384_1.dblwr
drwxr-x--- 2 mysql mysql 4.0K Sep 22 17:22 #innodb_redo
drwxr-x--- 2 mysql mysql 4.0K Sep 11 17:40 #innodb_temp
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       295G  290G  4.7G  99% /var/lib/mysql

@james sagte:

ls -lah /home/yellowtent/platformdata/mysql/

and

du -hd 0 /home/yellowtent/platformdata/mysql

root@my:~# ls -lah /home/yellowtent/platformdata/mysql/
total 295M
-rw-r-----  1 102        uuidd      6.0M Sep 22 17:45 '#ib_16384_0.dblwr'
-rw-r-----  1 102        uuidd       14M Sep 21 22:47 '#ib_16384_1.dblwr'
drwxr-x---  2 102        uuidd      4.0K Sep 22 17:22  '#innodb_redo'
drwxr-x---  2 102        uuidd      4.0K Sep 11 17:40  '#innodb_temp'
drwxr-xr-x 14 102        uuidd      4.0K Sep 11 17:40  .
drwxr-xr-x 23 yellowtent yellowtent 4.0K Sep  1 21:03  ..
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:14  a627b901205146af
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:33  a78a01e8ce480b13
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:32  a79f97d58ed20a71
-rw-r-----  1 102        uuidd        56 Sep  1 21:00  auto.cnf
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:31  c894b11c3cecf0d3
-rw-------  1 102        uuidd      1.7K Sep  1 21:00  ca-key.pem
-rw-r--r--  1 102        uuidd      1.1K Sep  1 21:00  ca.pem
-rw-r--r--  1 102        uuidd      1.1K Sep  1 21:00  client-cert.pem
-rw-------  1 102        uuidd      1.7K Sep  1 21:00  client-key.pem
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:31  df2322f70b8c510e
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:29  fc0ce74fabc98ba9
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:32  fd74cc63fc19b991
-rw-r-----  1 102        uuidd      3.9K Sep 11 17:39  ib_buffer_pool
-rw-r-----  1 102        uuidd       12M Sep 22 17:45  ibdata1
-rw-r-----  1 102        uuidd       12M Sep 11 17:40  ibtmp1
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:00  mysql
-rw-r-----  1 102        uuidd       26M Sep 22 17:44  mysql.ibd
-rw-r-----  1 102        uuidd       125 Sep  1 21:00  mysql_upgrade_history
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:00  performance_schema
-rw-------  1 102        uuidd      1.7K Sep  1 21:00  private_key.pem
-rw-r--r--  1 102        uuidd       452 Sep  1 21:00  public_key.pem
-rw-r--r--  1 102        uuidd      1.1K Sep  1 21:00  server-cert.pem
-rw-------  1 102        uuidd      1.7K Sep  1 21:00  server-key.pem
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:00  sys
-rw-r-----  1 102        uuidd      208M Sep 22 17:45  undo_001
-rw-r-----  1 102        uuidd       16M Sep 22 17:45  undo_002
root@my:~# du -hd 0 /home/yellowtent/platformdata/mysql
12G	/home/yellowtent/platformdata/mysql

[jayonrails]

@jdaviescoates sagte:

If I'm not mistaken, that style used to be how things were routinely done on Cloudron(?), so this is probably just left over from it being an old install?

Yes, the installation is around 3 years old I would guess.

[james]

Hello @jayonrails
Since you did run the commands I have asked for against /var/lib/mysql and not /home/yellowtent/platformdata/mysql/ I now understand that this is about the Cloudron MySQL server on the root system itself and not the MySQL service.

This changes things and I will have to look into it again.

[jayonrails]

Perfect. Since I am running out of space now, is there a safe way to do a backup (which I would share with you to investigate) and get rid of the bullshit data so I don't have to move to another server tonight?

[james]

Hello @jayonrails

Before we do a clean-up, please run the following command and share the output with me.

mysql -uroot -ppassword -e "SHOW BINARY LOGS;"

[jayonrails]

I cannot attach a textfile nor add the whole lines of output here because this is too big.

https://paste.cloudron.io/p/bat-cobra-emu

[james]

Hello @jayonrails
Thanks for that.

For a quick clean-up you can run the following command:

mysql -uroot -ppassword -e "RESET BINARY LOGS AND GTIDS;"

After that please check again with the following command if they got removed:

mysql -uroot -ppassword -e "SHOW BINARY LOGS;"

After that we need to understand why your MySQL server even has bin logging enabled.
But let's attempt the quick clean-up first.

[jayonrails]

Seems fine for now. Would you like to aggree on a meeting so we could analyze this together?

root@my:~# mysql -uroot -ppassword -e "RESET BINARY LOGS AND GTIDS;"
mysql: [Warning] Using a password on the command line interface can be insecure.
root@my:~# mysql -uroot -ppassword -e "SHOW BINARY LOGS;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000001 |   1895520 | No        |
+---------------+-----------+-----------+

[james]

Hello @jayonrails

I will have to check some things first, please give me 10-15 minutes.

[james]

Hello @jayonrails

Since 26.02.2021 all Cloudron installation ship the MySQL system server with:

skip-log-bin

in the /etc/mysql/my.cnf file.

Is your Cloudron installation older than 26.02.2021?

Please check your /etc/mysql/my.cnf file if skip-log-bin is set or if for example:

expire_logs_days = 10
max_binlog_size = 100M

or

binlog_expire_logs_seconds = 864000
max_binlog_size = 100M

You can also run the following command to get the live configuration:

mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@expire_logs_days, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"

[jayonrails]

hey @james,

thanks for your investigation. Let met get straight to your questions:

@james sagte:

Is your Cloudron installation older than 26.02.2021?

No, never has been.

@james sagte:

/etc/mysql/my.cnf

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

@james sagte:

You can also run the following command to get the live configuration:

root@my:~# mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@expire_logs_days, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1193 (HY000) at line 1: Unknown system variable 'expire_logs_days'

[james]

Hello @jayonrails
Your MySQL config looks empty and that is wrong.
Is also assume there is nothing in the folders listed in the config.

Since that command also failed for you please try again without the expire_logs_days:

mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"

[CRBear]

Hello,

I seem to have the same or a very similar issue as reported here:

https://forum.cloudron.io/topic/15988/mysql-binlogs-growing-unbounded-145-gb-no-expiry-set/18

My system:

Cloudron: 10.0.5
Ubuntu:   Ubuntu 26.04.1 LTS
Kernel:   Linux 7.0.0-31-generic

The root filesystem is currently at 93%:

/dev/sda1   150G   134G   11G   93% /

The unexpected usage is mainly:

/var/lib/mysql   76G

There are hundreds of binlog.xxxxxx files, mostly about 101 MB each.

MySQL version:

8.4.11-0ubuntu0.26.04.1

Current MySQL settings:

log_bin                       = 1
binlog_expire_logs_seconds    = 2592000
max_binlog_size               = 104857600
binlog_expire_logs_auto_purge = 1

SHOW REPLICA STATUS\G returns no replica.

/etc/mysql/my.cnf only contains:

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

There is no skip-log-bin entry.

I have not deleted any binlogs or changed the MySQL configuration yet.

Could you please advise what I should do next?

In particular, is it safe/recommended in this situation to use:

RESET BINARY LOGS AND GTIDS;

and should skip-log-bin then be restored manually, or is there another Cloudron-supported fix?

The server currently has only about 11 GB free, so I would prefer to resolve this before the root filesystem fills up.

Thanks in advance for your help!

[nebulon]

So just to confirm here, this is about the Ubuntu host system mysql, not the mysql addon service running in docker?

If so how old is that Cloudron? The skip-log-bin was added in Cloudron around 2021 to /etc/mysql/mysql.cnf and a Cloudron update should theoretically ensure that this file gets updated accordingly.

[james]

This indeed does look like the same issue as https://forum.cloudron.io/topic/15988/mysql-binlogs-growing-unbounded-145-gb-no-expiry-set
So maybe the upgrade from Ubuntu 24 to Ubuntu 26 resets the /etc/mysql.cnf.
The guide https://docs.cloudron.io/guides/upgrade-ubuntu-26/ explictly states to:

For all packages (mime, nginx, timesyncd, logrotate, journald, etc.), select N or O to keep your currently-installed version (default).

I will try to reproduce this.

[jayonrails]

Good morning @james,

root@my:~# mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-------------------------+-----------+------------------------------+-------------------+---------------------------------+
| VERSION()               | @@log_bin | @@binlog_expire_logs_seconds | @@max_binlog_size | @@binlog_expire_logs_auto_purge |
+-------------------------+-----------+------------------------------+-------------------+---------------------------------+
| 8.4.11-0ubuntu0.26.04.1 |         1 |                      2592000 |         104857600 |                               1 |
+-------------------------+-----------+------------------------------+-------------------+---------------------------------+

@james sagte:

Is also assume there is nothing in the folders listed in the config.

Yes.

As there have been two notes to this thread, I would like to add:

When upgrading from Ubuntu 24.04 to Ubuntu 26.04. I kept all the config files and didn't let it create new ones.

[james]

Hello @jayonrails
I am currently trying to reproduce if the Ubuntu 24 to Ubuntu 26 upgrade resets the my.cnf.

[CRBear]

Yes, this is the Ubuntu host system MySQL under /var/lib/mysql, not the MySQL addon service running in Docker.

The Cloudron dashboard shows:

Cloudron activation time: 2 years ago
Cloudron version: 10.0.5
Ubuntu: 26.04.1 LTS

So the activation is approximately from 2024 and therefore well after skip-log-bin was added in 2021.

Regarding the Ubuntu 24 → 26 upgrade: I used the default N/O option to keep the currently installed configuration files when prompted.

I also checked /var/log/apt/term.log. For the MySQL 8.4 installation itself I cannot see any configuration-file prompt. Instead, the relevant part is:

Setting up mysql-server (8.4.11-0ubuntu0.26.04.1)…
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode

Cloudron's own template still contains:

/home/yellowtent/box/setup/start/mysql.cnf

including:

# disable bin logs. they are only useful in replication mode
skip-log-bin

but skip-log-bin is no longer present anywhere under /etc/mysql.

I have not changed or deleted anything yet. I will wait for your guidance before making any changes, so please let me know what you would like me to do next.

I can also provide any further diagnostics that may help with reproducing this.

Thank you very much.