@SansGuidon said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

@james said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

Hello @zohup
This is fixed in Cloudron Version 9.

I think what @zohup was asking is how to fix this in production environments which are still running Cloudron Version 8.

thanks for the quick fix! I applied it and it worked perfectly.
@BrutalBirdie said in After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace:

Quickfix for users who need it NOW:

# get patch file, apply and remove and restart cloudron-syslog.service
cd /home/yellowtent/box
wget https://git.cloudron.io/platform/box/-/commit/063b1024616706971d4a1f9c50b5032727640120.diff
git apply 063b1024616706971d4a1f9c50b5032727640120.diff
rm -v 063b1024616706971d4a1f9c50b5032727640120.diff
systemctl restart cloudron-syslog.service

du -sh /var/log/syslog*
truncate -s 0 /var/log/syslog
truncate -s 0 /var/log/syslog.1

Hello @zohup
This is fixed in Cloudron Version 9.

I think what @zohup was asking is how to fix this in production environments which are still running Cloudron Version 8.

@SansGuidon thanks, fixed in https://git.cloudron.io/platform/box/-/commit/e45af9b611f4d0c3b77d4329aac24bacf98e4e6c . I could not figure out why it's not reproducible on that old Cloudron but I can reproduce it everywhere else . Maybe some Ubuntu 20.04 quirk .

Can you guide me what commands to type? Thank you.

That Cloudron is on Ubuntu 20.04 and docker 27.3.1 . Here it's not reproducible.

My test cloudron is on 24.04 and docker 28.1.1 (it's from dev branch). Here it's reproducible.

Ensure your default logdriver is journald:

systemctl show docker -p ExecStart

Should show something like

ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2 --experimental --ip6tables --use>

Then try to mimic what backupSqlite() does (no log driver; redirect only outside docker run):

docker run --rm alpine sh -lc 'for i in $(seq 1 3); do echo "INSERT INTO t VALUES($i);"; done' > /tmp/out.sql

Observe duplicates got logged to syslog anyway:

grep 'INSERT INTO t VALUES' /var/log/syslog | wc -l   # > 0
cat /tmp/out.sql | wc -l                              # same 3 lines

Now repeat with logging disabled (what the fix does):

docker run --rm --log-driver=none alpine sh -lc 'for i in $(seq 1 3); do echo "INSERT INTO t VALUES($i);"; done' > /tmp/out2.sql
grep 'INSERT INTO t VALUES' /var/log/syslog | wc -l   # unchanged

@SansGuidon the command is just sqlite3 ${p} ".dump" and it is redirected to a file. Do you have any ideas of why this will log sql commands to syslog? I can't reproduce this by running the command manually.

root@my:~# docker ps | grep uptime
cb00714073cb   cloudron/louislam.uptimekuma.app:202508221422060000    "/app/pkg/start.sh"      2 weeks ago    Up 2 weeks                                            ee6e4628-c370-4713-9cb6-f1888c32f8fb
root@my:~# du -sh /var/log/syslog*
352K	/var/log/syslog
904K	/var/log/syslog.1
116K	/var/log/syslog.2.gz
112K	/var/log/syslog.3.gz
112K	/var/log/syslog.4.gz
108K	/var/log/syslog.5.gz
112K	/var/log/syslog.6.gz
108K	/var/log/syslog.7.gz
root@my:~# grep 'INSERT INTO' /var/log/syslog | wc -l
0

diff --git a/box/src/services.js b/box/src/services.js
--- a/box/src/services.js
+++ b/box/src/services.js
@@ -1,7 +1,7 @@
 'use strict';
 
 exports = module.exports = {
     getServiceConfig,
 
     listServices,
     getServiceStatus,
@@ -308,7 +308,7 @@ async function backupSqlite(app, options) {
     // we use .dump instead of .backup because it's more portable across sqlite versions
     for (const p of options.paths) {
         const outputFile =  path.join(paths.APPS_DATA_DIR, app.id, path.basename(p, path.extname(p)) + '.sqlite');
 
         // we could use docker exec but it may not work if app is restarting
         const cmd = `sqlite3 ${p} ".dump"`;
         const runCmd = `docker run --rm --name=sqlite-${app.id} \
             --net cloudron \
             -v ${volumeDataDir}:/app/data \
             --label isCloudronManaged=true \
-            --read-only -v /tmp -v /run ${app.manifest.dockerImage} ${cmd} > ${outputFile}`;
+            --log-driver=none \
+            --read-only -v /tmp -v /run ${app.manifest.dockerImage} ${cmd} > ${outputFile} 2>/dev/null`;
 
         await shell.bash(runCmd, { encoding: 'utf8' });
     }
 }

root@ubuntu-cloudron-16gb-nbg1-3:~# du -sh /var/log/syslog*
8.2G	/var/log/syslog
0	/var/log/syslog.1
0	/var/log/syslog.1.gz-2025083120.backup
52K	/var/log/syslog.2.gz
4.0K	/var/log/syslog.3.gz
4.0K	/var/log/syslog.4.gz

As mentioned earlier in the discussion , it's due to sqlite backup dumps of UptimeKuma which end in the wrong place.

root@ubuntu-cloudron-16gb-nbg1-3:~# grep 'INSERT INTO' /var/log/syslog | wc -l
47237303

And I think this was started being investigated by @nebulon
This generates a few GBs worth of waste per day on my Cloudron instance which causes regular outages (every few weeks)

$ du -sh /var/log/syslog*
5.1M	/var/log/syslog
6.6M	/var/log/syslog.1
800K	/var/log/syslog.2.gz
796K	/var/log/syslog.3.gz
812K	/var/log/syslog.4.gz

Hello @SansGuidon
You mean the disk usage as a historical statistic and not only a singular point when checking?
If this is what you mean, no that is not part of Cloudron 9 at the moment.
But in my opinion, a very welcome feature request after Cloudron 9 is released!

Exactly, the idea is to be able to notice if something weird is happening (like disk usage growing constantly at a rapid rate)
I'll make a proposal in a separate thread -> Follow up in https://forum.cloudron.io/topic/14292/add-historical-disk-usage-in-system-info-graphs-section

root@ubuntu-cloudron-16gb-nbg1-3:~# du -sh /var/log/syslog*
15G	/var/log/syslog
26G	/var/log/syslog.1
0	/var/log/syslog.1.gz-2025083120.backup
52K	/var/log/syslog.2.gz
4.0K	/var/log/syslog.3.gz
4.0K	/var/log/syslog.4.gz

Disk graph shows

  docker 25.9 GB
  docker-volumes 7.79 GB
  /apps.swap 4.29 GB
  platformdata 3.77 GB
  boxdata 58.34 MB
  maildata 233.47 kB
  Everything else (Ubuntu, etc) 48.67 GB

root@ubuntu-cloudron-16gb-nbg1-3:~# truncate -s 0 /var/log/syslog
root@ubuntu-cloudron-16gb-nbg1-3:~# truncate -s 0 /var/log/syslog.1

After truncating the logs (see above), I reclaim the disk space, but I really need to work on a more effective patch / housekeeping job to prevent

This disk contains:

  docker 25.9 GB
  docker-volumes 8.02 GB
  /apps.swap 4.29 GB
  platformdata 3.8 GB
  boxdata 57.93 MB
  maildata 233.47 kB
  Everything else (Ubuntu, etc) 7.62 GB

I would also love if the Cloudron disk usage view would be a graph like for CPU and Memory. Maybe it's already planned for Cloudron 9, otherwise should I mention that idea in a new thread, @nebulon ?

From what I can see in your post this all looks correct and as intended.

Thus, none of the docker containers should log to journald or rsyslogd. Well at least if they were created by Cloudron itself of course to set those.

Given that this is uptime kuma, which in turn is just using sqlite, this lead me to https://git.cloudron.io/platform/box/-/blob/master/src/services.js?ref_type=heads#L933 which indeed starts a container without specifying the cloudron logdriver configs. So that is probably one thing we should fix.

This however would still mean the Gbs of sql dump logs just end up in another place. So the main issue then to fix is that sqlite3 app.db .dump which is run to create the sqldump also somehow logs to stdout/err despite redirectding stdou to the dump file....and that ends up in the logs somehow. I haven't found a fix yet but just to share the investigation here.

I'm not sure why, but Cloudron created my app containers with Docker’s syslog log driver. Those containers write their stdout/stderr straight into the host’s rsyslog, which in turn writes to /var/log/syslog.
So when an app (Uptime Kuma in my case) runs a huge sqlite3 .dump during a Cloudron task/backup, that dump goes to stdout → syslog → /var/log/syslog, ballooning the file by GBs. This is not journald forwarding (it’s disabled). Cloudron’s own cloudron-syslog also logs per-app to /home/yellowtent/platformdata/logs/…, so right now there’s duplication.

I’m not looking for a local workaround; I’d like Cloudron to confirm the intent here and provide a platform fix.

Below, the findings and some questions/proposals to pursue

Dockerd default vs. container reality

systemctl show docker -p ExecStart
# ... --log-driver=journald ...

docker ps -a -q | xargs -r -I{} docker inspect {} \
  --format '{{.Name}} {{.HostConfig.LogConfig.Type}}' | sort -u
# ~80 containers → all: syslog

The daemon default is journald, but all existing containers are syslog (likely from when they were created).

Not journald → syslog; it’s Docker → rsyslog

grep -n 'ForwardToSyslog' /etc/systemd/journald.conf
# ForwardToSyslog=no

journald isn’t forwarding.

Rsyslog is writing everything to /var/log/syslog

grep -nH . /etc/rsyslog.d/50-default.conf | sed -n '8,12p'
# *.*;auth,authpriv.none   -/var/log/syslog

Cloudron syslog collector is active (so we have duplicate paths)

systemctl status cloudron-syslog
# active (running)
ls /home/yellowtent/platformdata/logs/
# per-app log dirs + syslog.sock present

The big spill: SQL dump text in logs exactly at backup window

root@ubuntu-cloudron-16gb-nbg1-3:~# grep -nE 'BEGIN TRANSACTION|CREATE TABLE \[heartbeat\]|INSERT INTO heartbeat' /var/log/syslog | head -3
1152:2025-08-31T21:00:37.705303+00:00 ubuntu-cloudron-16gb-nbg1-3 d6750120460b[1123]: BEGIN TRANSACTION;
1153:2025-08-31T21:00:37.705386+00:00 ubuntu-cloudron-16gb-nbg1-3 d6750120460b[1123]: CREATE TABLE [heartbeat](#015
1162:2025-08-31T21:00:37.705789+00:00 ubuntu-cloudron-16gb-nbg1-3 d6750120460b[1123]: INSERT INTO heartbeat VALUES(1,1,1,1,'200 - OK','2025-03-27 23:26:53.602',566,0,0);

And Cloudron task timeline around the same minute:

root@ubuntu-cloudron-16gb-nbg1-3:~# grep -n '2025-08-31T21:0' /home/yellowtent/platformdata/logs/box.log | sed -n '1,40p'
9200:2025-08-31T21:00:00.014Z box:janitor Cleaning up expired tokens
9201:2025-08-31T21:00:00.016Z box:eventlog cleanup: pruning events. creationTime: Mon Jun 02 2025 21:00:00 GMT+0000 (Coordinated Universal Time)
9202:2025-08-31T21:00:00.054Z box:locks write: current locks: {"backup_task":null}
9203:2025-08-31T21:00:00.054Z box:locks acquire: backup_task
9204:2025-08-31T21:00:00.054Z box:janitor Cleaned up 0 expired tokens
9205:2025-08-31T21:00:00.166Z box:tasks startTask - starting task 7053 with options {"timeout":86400000,"nice":15,"memoryLimit":1024,"oomScoreAdjust":-999}. logs at /home/yellowtent/platformdata/logs/tasks/7053.log
9206:2025-08-31T21:00:00.168Z box:shell tasks /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 7053 /home/yellowtent/platformdata/logs/tasks/7053.log 15 1024 -999
9207:2025-08-31T21:00:00.249Z box:shell Running as unit: box-task-7053.service; invocation ID: fa4cf334a41b43fc9e06d6612bf5a9c1
9209:2025-08-31T21:00:00.395Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9210:2025-08-31T21:00:10.288Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9211:2025-08-31T21:00:20.321Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9212:2025-08-31T21:00:30.367Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9213:2025-08-31T21:00:40.579Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9214:2025-08-31T21:00:50.457Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9215:2025-08-31T21:01:00.455Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9216:2025-08-31T21:01:10.350Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9217:2025-08-31T21:01:20.413Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9218:2025-08-31T21:01:30.407Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9219:2025-08-31T21:01:40.367Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9220:2025-08-31T21:01:50.352Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9221:2025-08-31T21:02:00.390Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9222:2025-08-31T21:02:10.709Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9223:2025-08-31T21:02:11.024Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
9224:2025-08-31T21:02:20.338Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9225:2025-08-31T21:02:30.311Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9226:2025-08-31T21:02:40.300Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9227:2025-08-31T21:02:50.308Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9228:2025-08-31T21:03:00.406Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9229:2025-08-31T21:03:10.269Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9230:2025-08-31T21:03:20.363Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9231:2025-08-31T21:03:30.265Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9232:2025-08-31T21:03:40.281Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9233:2025-08-31T21:03:50.312Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9234:2025-08-31T21:04:00.321Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9235:2025-08-31T21:04:10.284Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9236:2025-08-31T21:04:20.357Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9237:2025-08-31T21:04:30.242Z box:apphealthmonitor app health: 31 running / 0 stopped / 0 unresponsive
9238:2025-08-31T21:04:30.281Z box:shell Finished with result: success
9245:2025-08-31T21:04:30.288Z box:shell Service box-task-7053 finished with exit code 0
9247:2025-08-31T21:04:30.289Z box:tasks startTask: 7053 completed with code 0

Questions / Suggestions

• Is syslog the intended log driver for app containers?
  Dockerd on my host now runs with --log-driver=journald, but all app containers remain on syslog unless re-created.
• Platform-level fix proposals (any/all):
  • Migrate app containers to journald on updates/repairs so they inherit the daemon default (no /var/log/syslog involvement).
  • Ensure task/backup helpers don’t emit large dumps to stdout (redirect to files/pipes consumed by cloudron-syslog, not rsyslog).
  • Ship an rsyslog drop-in that stops Docker-originated container stdout from landing in /var/log/syslog, since Cloudron already captures per-app logs under /home/yellowtent/platformdata/logs/.

This would prevent another GB-scale blow-up when an app emits a lot to stdout during backups or maintenance.

What do you think, @nebulon ?
Thanks in advance

That one would dump corresponding logs into the correct places in /home/yellowtent/paltformdata/logs/...

So still I am curious how it ends up in /var/log/syslog and then why it would log db dump data there.

I alone with this setup?

Nope. I see to have the same:

root@Ubuntu-2204-jammy-amd64-base ~ # docker info | grep 'Logging Driver'
 Logging Driver: journald

root@ubuntu-cloudron-16gb-nbg1-3:~# docker info | grep 'Logging Driver'
 Logging Driver: journald

am I alone with this setup? I've no memory about configuring this behavior for logging driver.