After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace

[SansGuidon]

@james said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

Hello @SansGuidon
You mean the disk usage as a historical statistic and not only a singular point when checking?
If this is what you mean, no that is not part of Cloudron 9 at the moment.
But in my opinion, a very welcome feature request after Cloudron 9 is released!

Exactly, the idea is to be able to notice if something weird is happening (like disk usage growing constantly at a rapid rate)
I'll make a proposal in a separate thread -> Follow up in https://forum.cloudron.io/topic/14292/add-historical-disk-usage-in-system-info-graphs-section

[joseph]

@SansGuidon afaik, Cloudron does not log anything to syslog . Did you happen to check what was inside that massive syslog file? In one of our production cloudrons (running for almost a decade):

$ du -sh /var/log/syslog*
5.1M	/var/log/syslog
6.6M	/var/log/syslog.1
800K	/var/log/syslog.2.gz
796K	/var/log/syslog.3.gz
812K	/var/log/syslog.4.gz

[SansGuidon]

Hi @joseph

root@ubuntu-cloudron-16gb-nbg1-3:~# du -sh /var/log/syslog*
8.2G	/var/log/syslog
0	/var/log/syslog.1
0	/var/log/syslog.1.gz-2025083120.backup
52K	/var/log/syslog.2.gz
4.0K	/var/log/syslog.3.gz
4.0K	/var/log/syslog.4.gz

As mentioned earlier in the discussion , it's due to sqlite backup dumps of UptimeKuma which end in the wrong place.

root@ubuntu-cloudron-16gb-nbg1-3:~# grep 'INSERT INTO' /var/log/syslog | wc -l
47237303

And I think this was started being investigated by @nebulon
This generates a few GBs worth of waste per day on my Cloudron instance which causes regular outages (every few weeks)

[SansGuidon]

For now as a workaround I'm applying this patch, please advise if you have any concern with this

diff --git a/box/src/services.js b/box/src/services.js
--- a/box/src/services.js
+++ b/box/src/services.js
@@ -1,7 +1,7 @@
 'use strict';
 
 exports = module.exports = {
     getServiceConfig,
 
     listServices,
     getServiceStatus,
@@ -308,7 +308,7 @@ async function backupSqlite(app, options) {
     // we use .dump instead of .backup because it's more portable across sqlite versions
     for (const p of options.paths) {
         const outputFile =  path.join(paths.APPS_DATA_DIR, app.id, path.basename(p, path.extname(p)) + '.sqlite');
 
         // we could use docker exec but it may not work if app is restarting
         const cmd = `sqlite3 ${p} ".dump"`;
         const runCmd = `docker run --rm --name=sqlite-${app.id} \
             --net cloudron \
             -v ${volumeDataDir}:/app/data \
             --label isCloudronManaged=true \
-            --read-only -v /tmp -v /run ${app.manifest.dockerImage} ${cmd} > ${outputFile}`;
+            --log-driver=none \
+            --read-only -v /tmp -v /run ${app.manifest.dockerImage} ${cmd} > ${outputFile} 2>/dev/null`;
 
         await shell.bash(runCmd, { encoding: 'utf8' });
     }
 }

[joseph]

@SansGuidon I think @nebulon investigated and could not reproduce. We also run uptime kuma. Our logs are fine. Have you enabled backups inside uptime kuma or something else by any chance?

root@my:~# docker ps | grep uptime
cb00714073cb   cloudron/louislam.uptimekuma.app:202508221422060000    "/app/pkg/start.sh"      2 weeks ago    Up 2 weeks                                            ee6e4628-c370-4713-9cb6-f1888c32f8fb
root@my:~# du -sh /var/log/syslog*
352K	/var/log/syslog
904K	/var/log/syslog.1
116K	/var/log/syslog.2.gz
112K	/var/log/syslog.3.gz
112K	/var/log/syslog.4.gz
108K	/var/log/syslog.5.gz
112K	/var/log/syslog.6.gz
108K	/var/log/syslog.7.gz
root@my:~# grep 'INSERT INTO' /var/log/syslog | wc -l
0

[joseph]

FWIW, our db is pretty big too.

@SansGuidon the command is just sqlite3 ${p} ".dump" and it is redirected to a file. Do you have any ideas of why this will log sql commands to syslog? I can't reproduce this by running the command manually.

[SansGuidon]

@joseph I don't see any special setting in UptimeKuma being applied in my instance. Can you try to reproduce with those instructions below? Hope that makes sense

Ensure your default logdriver is journald:

systemctl show docker -p ExecStart

Should show something like

ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2 --experimental --ip6tables --use>

Then try to mimic what backupSqlite() does (no log driver; redirect only outside docker run):

docker run --rm alpine sh -lc 'for i in $(seq 1 3); do echo "INSERT INTO t VALUES($i);"; done' > /tmp/out.sql

Observe duplicates got logged to syslog anyway:

grep 'INSERT INTO t VALUES' /var/log/syslog | wc -l   # > 0
cat /tmp/out.sql | wc -l                              # same 3 lines

Now repeat with logging disabled (what the fix does):

docker run --rm --log-driver=none alpine sh -lc 'for i in $(seq 1 3); do echo "INSERT INTO t VALUES($i);"; done' > /tmp/out2.sql
grep 'INSERT INTO t VALUES' /var/log/syslog | wc -l   # unchanged

[girish]

@SansGuidon thanks for the repro. I have to say I can easily reproduce not only your test but also uptime kuma backup issue on my test Cloudron. At the same time, I have verified that @joseph's observation is also correct - our prod uptime kuma does not produce any spurious logs. Wonder what is going on... I am debugging.

[girish]

@SansGuidon What is your ubuntu and docker version?

That Cloudron is on Ubuntu 20.04 and docker 27.3.1 . Here it's not reproducible.

My test cloudron is on 24.04 and docker 28.1.1 (it's from dev branch). Here it's reproducible.

[SansGuidon]

@girish Docker 27.3.1 and Ubuntu 24.04.2 LTS

[girish]

@SansGuidon thanks, fixed in https://git.cloudron.io/platform/box/-/commit/e45af9b611f4d0c3b77d4329aac24bacf98e4e6c . I could not figure out why it's not reproducible on that old Cloudron but I can reproduce it everywhere else . Maybe some Ubuntu 20.04 quirk .

[SansGuidon]

Nice! thanks @girish

[zohup]

@girish said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

@SansGuidon thanks, fixed in https://git.cloudron.io/platform/box/-/commit/e45af9b611f4d0c3b77d4329aac24bacf98e4e6c . I could not figure out why it's not reproducible on that old Cloudron but I can reproduce it everywhere else . Maybe some Ubuntu 20.04 quirk .

Can you guide me what commands to type? Thank you.

[james]

Hello @zohup
This is fixed in Cloudron Version 9.

[SansGuidon]

@james said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

Hello @zohup
This is fixed in Cloudron Version 9.

I think what @zohup was asking is how to fix this in production environments which are still running Cloudron Version 8.

[zohup]

yes there it is, and it seems like that's the only way to fix it

@SansGuidon said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

@james said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

Hello @zohup
This is fixed in Cloudron Version 9.

I think what @zohup was asking is how to fix this in production environments which are still running Cloudron Version 8.

thanks for the quick fix! I applied it and it worked perfectly.
@BrutalBirdie said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

Quickfix for users who need it NOW:

# get patch file, apply and remove and restart cloudron-syslog.service
cd /home/yellowtent/box
wget https://git.cloudron.io/platform/box/-/commit/063b1024616706971d4a1f9c50b5032727640120.diff
git apply 063b1024616706971d4a1f9c50b5032727640120.diff
rm -v 063b1024616706971d4a1f9c50b5032727640120.diff
systemctl restart cloudron-syslog.service

du -sh /var/log/syslog*
truncate -s 0 /var/log/syslog
truncate -s 0 /var/log/syslog.1

[SansGuidon]

Thanks @zohup !