MySQL binlogs growing unbounded — 145 GB, no expiry set?

[james]

Hello @jayonrails

@jayonrails said:

Is it intended that the MySQL addon ships without a binlog expiry?
On installations without replication, the logs will inevitably
fill the disk.

The Cloudron MySQL Service is shipped with skip-log-bin and I just checked there are no bin logs stored for multiple Cloudron servers.

@jayonrails said:

What is the supported way to set binlog_expire_logs_seconds
persistently, so it survives container restarts and app updates?

Since skip-log-bin is the default I am currently unsure where these are coming from.
I would suspect Freescout to be the culprit here.
The default configuration for Freescout and the activity_log is:

# https://github.com/freescout-help-desk/freescout/blob/88734eec10a898682a8dc0e846459f477d60600a/config/activitylog.php#L8
'enabled' => env('ACTIVITY_LOGGER_ENABLED', true),

# https://github.com/freescout-help-desk/freescout/blob/88734eec10a898682a8dc0e846459f477d60600a/config/activitylog.php#L14
'delete_records_older_than_days' => env('ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS', 30),

So by default it is enabled and should remove records older than 30 days.
Did you configure a different value for ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS in your /app/data/env file?

Also, could it be that there is somthing creating massive ammounts of logs for the activity_log and that is actually spamming the database?
My thought is something like, if there is something spamming 101 MB every 9 minutes and if only older than 30 days are cleaned up, only after 30 days it would slowly be cleaned up.

[jayonrails]

Hello @james
thanks for your research.

I did not touch the config file for at least some years, but I cannot guarantee that it has never been touched.

####################################################################################################
## If you want to use web installer **DO NOT** create `.env` file manually. 
## If `.env` file exists in the root of your app, web installer won't run.
##
## Every time you are making changes in .env file, in order changes to take an effect you need to run:
## php artisan freescout:clear-cache
#####################################################################################################

# Application URL
APP_URL=https://wont-tell.de

# Use HTTPS protocol and redirect to HTTPS
#APP_FORCE_HTTPS=true

# If you are using HTTPS, feel free to uncomment this line to improve security
#SESSION_SECURE_COOKIE=true

# Enter your proxy address here if freescout.net is not available from your server
# (access to freescout.net is required to obtain official modules)
#APP_PROXY=

# Custom headers to add to all outgoing emails.
#APP_CUSTOM_MAIL_HEADERS="IsTransactional:True;X-Custom-Header:value"

# Timezones: https://github.com/freescout-helpdesk/freescout/wiki/PHP-Timezones
# Comment it to use default timezone from php.ini
APP_TIMEZONE=Europe/Berlin

DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=censored
DB_USERNAME=censored
# Maximum password length is 50 characters
DB_PASSWORD=censored

# Run the following console command to generate the key: php artisan key:generate
# Otherwise application will show the following error: "Whoops, looks like something went wrong"
APP_KEY=base64:km/censored
APP_FORCE_HTTPS = true
APP_DISABLE_UPDATING = true

# Uncomment to see errors in your browser, don't forget to comment it back when debugging finished
#APP_DEBUG=true

APP_CUSTOM_NUMBER=false

APP_USER_PERMISSIONS=

APP_EMAIL_CONV_HISTORY=none

APP_EMAIL_USER_HISTORY=full

APP_LOCALE=de

FASTERSEARCH_URL="https://ms-censored.fra.meilisearch.io"

FASTERSEARCH_API_KEY=censored

APP_ALERT_LOGS=

APP_ALERT_LOGS_PERIOD=hour

export FILE_STORAGE=local
export FILE_STORAGE_UPLOAD_MAX_SIZE=26214400

APP_REMOTE_HOST_WHITE_LIST=mail,172.18.0.0/16,fd00::/8,127.0.1.1,"my.censored.de",172.18.0.1"

After adding the changes you suggested, I noticed that it does not match the styling of the other entries. After starting, I got the following error:

Sep 22 00:27:54 In Loader.php line 273:
Sep 22 00:27:54                                                                  
Sep 22 00:27:54                                                                  
Sep 22 00:27:54   Dotenv values containing spaces must be surrounded by quotes.  
Sep 22 00:27:54 => Clear cache folders
Sep 22 00:27:54 => Ensure directories
Sep 22 00:27:54 => Ensure permissions
Sep 22 00:27:54 => Run migrations
Sep 22 00:27:54 => Set configs
Sep 22 00:27:54 PHP Warning:  Trying to access array offset on null in /app/code/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php on line 159
Sep 22 00:27:55 2026-09-21T22:27:55Z 
Sep 22 00:27:55 2026-09-21T22:27:55Z 

I have then changed it:

# https://github.com/freescout-help-desk/freescout/blob/88734eec10a898682a8dc0e846459f477d60600a/config/activitylog.php#L8
ACTIVITY_LOGGER_ENABLED=true

# https://github.com/freescout-help-desk/freescout/blob/88734eec10a898682a8dc0e846459f477d60600a/config/activitylog.php#L14
ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS=30

...and cleared cache.

root@0a2837d5-d303-4d2b-b9f4-c35199425682:/app/code# php artisan freescout:clear-cache
The compiled services & packages files have been removed.
Cache cleared successfully.
Compiled views cleared!
Configuration cache cleared!
Configuration cached successfully!
Cleared: JS and CSS builds
Created: /js/builds/vars.js

@james sagte:

Also, could it be that there is somthing creating massive ammounts of logs for the activity_log and that is actually spamming the database?
My thought is something like, if there is something spamming 101 MB every 9 minutes and if only older than 30 days are cleaned up, only after 30 days it would slowly be cleaned up.

I also thought about that. After some research I have found out that there is mailboxes / support boxes within FreeScout that try to connect to an IMAP / SMTP mailbox which no longer exists.

Also I have noticed that the meiliseach instance is no longer there. It has been disabled but the respective entries in the config file have not been remove - although the module has been disabled. When I try to uninstall the module, the page reloads, but the module is still there. It does show a confirmation that the module has been deleted.

Also I have noticed that after making the changes to the config file and cleaning the cache, the authentication with SMTP did not work. I checked credentials and other data and when saving, I got the error notice that the hostname is not allowed.

Then I took a deeper look into APP_REMOTE_HOST_WHITE_LIST and changed it as follows:

APP_REMOTE_HOST_WHITE_LIST=mail,172.18.0.0/16,fd00::/8,127.0.1.1,172.17.0.1,172.17.0.0/16,172.18.0.1,fe80::/10,fe80::be24:11ff:fe61:d810,my.example.com

my.example.com being the hostname of my Cloduron.

After saving, restarting the application and clearing cache, this worked again.

When I was sure everything was working again, I cleared all the log files in FreeScout that I could find. Then I disabled the app again, because now the most serious question is: how can I remove the data from the mySQL binlogs?

When that is done and 145 GB more disk space is available, I would to start the application again and check for log files and if the rotation works correctly, also I would like to check if it get's this big again.

[jayonrails]

Addition: I would like to know when "ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS" has been implemented. My FreeScout installtion probably didn't have that when I first installed it. There might be other people that could run into this problem; it could have been a bug some years ago?

It might be clever to re-install it and do an import, however I have not researched yet if an import is possible and if I can keep the ticket IDs the samde and transfer all data including attachments.

[james]

Hello @jayonrails

@jayonrails said:

After adding the changes you suggested, I noticed that it does not match the styling of the other entries. After starting, I got the following error:

The code block I have posted with the URLs are source code and not environment variables.
I should have been more clear, sorry.
The environment variables names are ACTIVITY_LOGGER_ENABLED and ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS.

---

In your /app/data/env you have shared

@jayonrails said:

export FILE_STORAGE=local
export FILE_STORAGE_UPLOAD_MAX_SIZE=26214400

there are these two entries which look incorrect.
No other entries in the /app/data/env have the export $SOME_ENVIRONMENT_VARIABLE=$VALUE style.
Also I did not find anything in the upstream Freescout code that uses these two environment variables.
You might want to look into that as well.

---

@jayonrails said:

how can I remove the data from the mySQL binlogs?

Can you please share where exactly you have found these binlogs?
Are they inside the freescout container or on the host system?
When on the host system please connect to your server with ssh and share the output of the following command:

ls -lah /home/yellowtent/platformdata/mysql/
# and
du -hd 0 /home/yellowtent/platformdata/mysql

---

@jayonrails said:

I would like to know when "ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS" has been implemented.

We Cloudron have never implemented ACTIVITY_LOGGER_DELETE_RECORDS_OLDER_THAN_DAYS.
This is a default from Freescout and has been like that as far as I can tell for 8 years.

@jayonrails said:

My FreeScout installtion probably didn't have that when I first installed it.

With my statement above, no, it has always been like that unless configured otherwise.

[jdaviescoates]

@james said:

No other entries in the /app/data/env have the export $SOME_ENVIRONMENT_VARIABLE=$VALUE style.

If I'm not mistaken, that style used to be how things were routinely done on Cloudron(?), so this is probably just left over from it being an old install?

[jayonrails]

@james sagte:

The code block I have posted with the URLs are source code and not environment variables.
I should have been more clear, sorry.

no problem at all :))

@james sagte:

there are these two entries which look incorrect.
No other entries in the /app/data/env have the export $SOME_ENVIRONMENT_VARIABLE=$VALUE style.
Also I did not find anything in the upstream Freescout code that uses these two environment variables.
You might want to look into that as well.

I will check this!

@james sagte:

Can you please share where exactly you have found these binlogs?
Are they inside the freescout container or on the host system?

root@my:~# # Biggest items inside /var/lib/mysql, one level deep
du -xh --max-depth=1 /var/lib/mysql | sort -h | tail -15
36K	/var/lib/mysql/mysql
116K	/var/lib/mysql/sys
804K	/var/lib/mysql/#innodb_temp
1.7M	/var/lib/mysql/performance_schema
101M	/var/lib/mysql/#innodb_redo
213M	/var/lib/mysql/box
173G	/var/lib/mysql
root@my:~# 
root@my:~# # Binlogs vs everything else, side by side
du -ch /var/lib/mysql/binlog.* | tail -1
du -sh --exclude='binlog.*' /var/lib/mysql
173G	total
424M	/var/lib/mysql
root@my:~# # Oldest and newest, with timestamps
ls -lh --time-style=long-iso /var/lib/mysql/binlog.* | head -5
ls -lh --time-style=long-iso /var/lib/mysql/binlog.* | tail -5

# The index MySQL maintains — one line per binlog it knows about
wc -l /var/lib/mysql/binlog.index
head -3 /var/lib/mysql/binlog.index
-rw-r----- 1 mysql mysql  180 2026-09-01 14:36 /var/lib/mysql/binlog.000001
-rw-r----- 1 mysql mysql  180 2026-09-01 14:38 /var/lib/mysql/binlog.000002
-rw-r----- 1 mysql mysql  180 2026-09-01 14:38 /var/lib/mysql/binlog.000003
-rw-r----- 1 mysql mysql  441 2026-09-11 17:40 /var/lib/mysql/binlog.000004
-rw-r----- 1 mysql mysql 101M 2026-09-11 17:49 /var/lib/mysql/binlog.000005
-rw-r----- 1 mysql mysql 101M 2026-09-22 17:23 /var/lib/mysql/binlog.001764
-rw-r----- 1 mysql mysql 101M 2026-09-22 17:32 /var/lib/mysql/binlog.001765
-rw-r----- 1 mysql mysql 101M 2026-09-22 17:41 /var/lib/mysql/binlog.001766
-rw-r----- 1 mysql mysql  79M 2026-09-22 17:48 /var/lib/mysql/binlog.001767
-rw-r----- 1 mysql mysql  28K 2026-09-22 17:41 /var/lib/mysql/binlog.index
1767 /var/lib/mysql/binlog.index
./binlog.000001
./binlog.000002
./binlog.000003
root@my:~# docker exec mysql ls -lh /var/lib/mysql/ | head -5
docker exec mysql df -h /var/lib/mysql
total 295M
-rw-r----- 1 mysql mysql 6.0M Sep 22 17:48 #ib_16384_0.dblwr
-rw-r----- 1 mysql mysql  14M Sep 21 22:47 #ib_16384_1.dblwr
drwxr-x--- 2 mysql mysql 4.0K Sep 22 17:22 #innodb_redo
drwxr-x--- 2 mysql mysql 4.0K Sep 11 17:40 #innodb_temp
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       295G  290G  4.7G  99% /var/lib/mysql

@james sagte:

ls -lah /home/yellowtent/platformdata/mysql/

and

du -hd 0 /home/yellowtent/platformdata/mysql

root@my:~# ls -lah /home/yellowtent/platformdata/mysql/
total 295M
-rw-r-----  1 102        uuidd      6.0M Sep 22 17:45 '#ib_16384_0.dblwr'
-rw-r-----  1 102        uuidd       14M Sep 21 22:47 '#ib_16384_1.dblwr'
drwxr-x---  2 102        uuidd      4.0K Sep 22 17:22  '#innodb_redo'
drwxr-x---  2 102        uuidd      4.0K Sep 11 17:40  '#innodb_temp'
drwxr-xr-x 14 102        uuidd      4.0K Sep 11 17:40  .
drwxr-xr-x 23 yellowtent yellowtent 4.0K Sep  1 21:03  ..
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:14  a627b901205146af
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:33  a78a01e8ce480b13
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:32  a79f97d58ed20a71
-rw-r-----  1 102        uuidd        56 Sep  1 21:00  auto.cnf
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:31  c894b11c3cecf0d3
-rw-------  1 102        uuidd      1.7K Sep  1 21:00  ca-key.pem
-rw-r--r--  1 102        uuidd      1.1K Sep  1 21:00  ca.pem
-rw-r--r--  1 102        uuidd      1.1K Sep  1 21:00  client-cert.pem
-rw-------  1 102        uuidd      1.7K Sep  1 21:00  client-key.pem
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:31  df2322f70b8c510e
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:29  fc0ce74fabc98ba9
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:32  fd74cc63fc19b991
-rw-r-----  1 102        uuidd      3.9K Sep 11 17:39  ib_buffer_pool
-rw-r-----  1 102        uuidd       12M Sep 22 17:45  ibdata1
-rw-r-----  1 102        uuidd       12M Sep 11 17:40  ibtmp1
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:00  mysql
-rw-r-----  1 102        uuidd       26M Sep 22 17:44  mysql.ibd
-rw-r-----  1 102        uuidd       125 Sep  1 21:00  mysql_upgrade_history
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:00  performance_schema
-rw-------  1 102        uuidd      1.7K Sep  1 21:00  private_key.pem
-rw-r--r--  1 102        uuidd       452 Sep  1 21:00  public_key.pem
-rw-r--r--  1 102        uuidd      1.1K Sep  1 21:00  server-cert.pem
-rw-------  1 102        uuidd      1.7K Sep  1 21:00  server-key.pem
drwxr-x---  2 102        uuidd      4.0K Sep  1 21:00  sys
-rw-r-----  1 102        uuidd      208M Sep 22 17:45  undo_001
-rw-r-----  1 102        uuidd       16M Sep 22 17:45  undo_002
root@my:~# du -hd 0 /home/yellowtent/platformdata/mysql
12G	/home/yellowtent/platformdata/mysql

[jayonrails]

@jdaviescoates sagte:

If I'm not mistaken, that style used to be how things were routinely done on Cloudron(?), so this is probably just left over from it being an old install?

Yes, the installation is around 3 years old I would guess.

[james]

Hello @jayonrails
Since you did run the commands I have asked for against /var/lib/mysql and not /home/yellowtent/platformdata/mysql/ I now understand that this is about the Cloudron MySQL server on the root system itself and not the MySQL service.

This changes things and I will have to look into it again.

[jayonrails]

Perfect. Since I am running out of space now, is there a safe way to do a backup (which I would share with you to investigate) and get rid of the bullshit data so I don't have to move to another server tonight?

[james]

Hello @jayonrails

Before we do a clean-up, please run the following command and share the output with me.

mysql -uroot -ppassword -e "SHOW BINARY LOGS;"

[jayonrails]

I cannot attach a textfile nor add the whole lines of output here because this is too big.

https://paste.cloudron.io/p/bat-cobra-emu

[james]

Hello @jayonrails
Thanks for that.

For a quick clean-up you can run the following command:

mysql -uroot -ppassword -e "RESET BINARY LOGS AND GTIDS;"

After that please check again with the following command if they got removed:

mysql -uroot -ppassword -e "SHOW BINARY LOGS;"

After that we need to understand why your MySQL server even has bin logging enabled.
But let's attempt the quick clean-up first.

[jayonrails]

Seems fine for now. Would you like to aggree on a meeting so we could analyze this together?

root@my:~# mysql -uroot -ppassword -e "RESET BINARY LOGS AND GTIDS;"
mysql: [Warning] Using a password on the command line interface can be insecure.
root@my:~# mysql -uroot -ppassword -e "SHOW BINARY LOGS;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+---------------+-----------+-----------+
| Log_name      | File_size | Encrypted |
+---------------+-----------+-----------+
| binlog.000001 |   1895520 | No        |
+---------------+-----------+-----------+

[james]

Hello @jayonrails

I will have to check some things first, please give me 10-15 minutes.

[james]

Hello @jayonrails

Since 26.02.2021 all Cloudron installation ship the MySQL system server with:

skip-log-bin

in the /etc/mysql/my.cnf file.

Is your Cloudron installation older than 26.02.2021?

Please check your /etc/mysql/my.cnf file if skip-log-bin is set or if for example:

expire_logs_days = 10
max_binlog_size = 100M

or

binlog_expire_logs_seconds = 864000
max_binlog_size = 100M

You can also run the following command to get the live configuration:

mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@expire_logs_days, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"

[jayonrails]

hey @james,

thanks for your investigation. Let met get straight to your questions:

@james sagte:

Is your Cloudron installation older than 26.02.2021?

No, never has been.

@james sagte:

/etc/mysql/my.cnf

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

@james sagte:

You can also run the following command to get the live configuration:

root@my:~# mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@expire_logs_days, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1193 (HY000) at line 1: Unknown system variable 'expire_logs_days'

[james]

Hello @jayonrails
Your MySQL config looks empty and that is wrong.
Is also assume there is nothing in the folders listed in the config.

Since that command also failed for you please try again without the expire_logs_days:

mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"

[CRBear]

Hello,

I seem to have the same or a very similar issue as reported here:

https://forum.cloudron.io/topic/15988/mysql-binlogs-growing-unbounded-145-gb-no-expiry-set/18

My system:

Cloudron: 10.0.5
Ubuntu:   Ubuntu 26.04.1 LTS
Kernel:   Linux 7.0.0-31-generic

The root filesystem is currently at 93%:

/dev/sda1   150G   134G   11G   93% /

The unexpected usage is mainly:

/var/lib/mysql   76G

There are hundreds of binlog.xxxxxx files, mostly about 101 MB each.

MySQL version:

8.4.11-0ubuntu0.26.04.1

Current MySQL settings:

log_bin                       = 1
binlog_expire_logs_seconds    = 2592000
max_binlog_size               = 104857600
binlog_expire_logs_auto_purge = 1

SHOW REPLICA STATUS\G returns no replica.

/etc/mysql/my.cnf only contains:

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

There is no skip-log-bin entry.

I have not deleted any binlogs or changed the MySQL configuration yet.

Could you please advise what I should do next?

In particular, is it safe/recommended in this situation to use:

RESET BINARY LOGS AND GTIDS;

and should skip-log-bin then be restored manually, or is there another Cloudron-supported fix?

The server currently has only about 11 GB free, so I would prefer to resolve this before the root filesystem fills up.

Thanks in advance for your help!

[nebulon]

So just to confirm here, this is about the Ubuntu host system mysql, not the mysql addon service running in docker?

If so how old is that Cloudron? The skip-log-bin was added in Cloudron around 2021 to /etc/mysql/mysql.cnf and a Cloudron update should theoretically ensure that this file gets updated accordingly.

[james]

This indeed does look like the same issue as https://forum.cloudron.io/topic/15988/mysql-binlogs-growing-unbounded-145-gb-no-expiry-set
So maybe the upgrade from Ubuntu 24 to Ubuntu 26 resets the /etc/mysql.cnf.
The guide https://docs.cloudron.io/guides/upgrade-ubuntu-26/ explictly states to:

For all packages (mime, nginx, timesyncd, logrotate, journald, etc.), select N or O to keep your currently-installed version (default).

I will try to reproduce this.