After Ubuntu 22/24 Upgrade syslog getting spammed and grows way to much clogging up the diskspace

[BrutalBirdie]

Currently investigating the following issue.
I am upgrading multiple servers to Ubuntu 22/24 and it seems that the /var/log/syslog is getting spammed:

2025-02-27T09:05:33.316275+00:00 my syslog.js[829]: 2025-02-27T09:05:33.316Z syslog:server Unable to parse: <30>1 2025-02-27T09:05:33Z my fc10b9f8-33b4-4f22-b442-74120cfaad1d 1023 fc10b9f8-33b4-4f22-b442-74120cfaad1d - #033[2m2025-02-27T09:05:33.315944Z#033[0m #033[32m INFO#033[0m #033[1mHTTP request#033[0m#033[1m{#033[0m#033[3mmethod#033[0m#033[2m=#033[0mPOST #033[3mhost#033[0m#033[2m=#033[0m"REDACTED.SUB.DOMAIN.TLD" #033[3mroute#033[0m#033[2m=#033[0m/indexes/REDACTED/facet-search #033[3mquery_parameters#033[0m#033[2m=#033[0m #033[3muser_agent#033[0m#033[2m=#033[0mMeilisearch PHP (v1.7.0) #033[3mstatus_code#033[0m#033[2m=#033[0m200#033[1m}#033[0m#033[2m:#033[0m #033[2mmeilisearch#033[0m#033[2m:#033[0m close #033[3mtime.busy#033[0m#033[2m=#033[0m88.0µs #033[3mtime.idle#033[0m#033[2m=#033[0m1.01ms

This log is from a custom Meilisearch app. But other logs with "normal" apps also have the parse error

Growing the syslog files to sizes that my monitoring system kicked in

-rw-r-----   1 syslog    adm              20G Feb 27 08:55 syslog
-rw-r-----   1 syslog    adm              14G Feb 23 00:00 syslog.1

Also noticed this entry:

2025-02-27T09:13:45.449795+00:00 my collectd[1044086]: ERROR: ld.so: object '/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
2025-02-27T09:13:45.450546+00:00 my collectd[1044087]: ERROR: ld.so: object '/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
2025-02-27T09:13:46.373778+00:00 my syslog.js[829]: 2025-02-27T09:13:46.373Z syslog:server Unable to parse: <27>1 2025-02-27T09:13:46Z my sftp 1023 sftp - 120.27.30.185 UNKNOWN - [27/Feb/2025:09:13:46 +0000] "USER root" 331 -
2025-02-27T09:13:47.829026+00:00 my syslog.js[829]: 2025-02-27T09:13:47.828Z syslog:server Unable to parse: <27>1 2025-02-27T09:13:47Z my sftp 1023 sftp - 120.27.30.185 UNKNOWN - [27/Feb/2025:09:13:47 +0000] "USER root" 331 -

Is this maybe connected to the post-upgrade step of Ubuntu 24?

Fixup collectd. /etc/default/collectd might have a LD_PRELOAD line from previous releases. If you make any changes, restart collectd using systemctl restart collectd .

Hmmm yes system in question did not have a cleaned up /etc/default/collectd

But the unable to parse persists:

2025-02-27T09:18:35.333933+00:00 my syslog.js[829]: 2025-02-27T09:18:35.333Z syslog:server Unable to parse: <30>1 2025-02-27T09:18:35Z my d1235d4c-4418-4cfe-8b2b-bc6abdd9afa2 1023 d1235d4c-4418-4cfe-8b2b-bc6abdd9afa2 - Task apps.uptime.tasks.dispatch_checks[427f04a8-9cb7-4914-b26d-db5cc31b860c] succeeded in 0.01911751099396497s: None

[BrutalBirdie]

Ahh The syslog.js is part of Cloudron in /home/yellowtent/box/syslog.js.
So now I can do some javascript investigation

[BrutalBirdie]

Just checked a rather freshly setup Ubuntu 24 with Cloudron and there is also the same issue.
So it seems not related to the do-release-upgrade to 22/24:

2025-02-27T09:34:13.375494+00:00 my-DOMAIN-TLD syslog.js[785]: 2025-02-27T09:34:13.375Z syslog:server Unable to parse: <30>1 2025-02-27T09:34:13Z my-DOMAIN-TLD 4ec5603e-7db7-49c2-a442-bf99721ba379 1043 4ec5603e-7db7-49c2-a442-bf99721ba379 - 162.158.86.97 - - [27/Feb/2025:09:34:13 +0000] "GET /api/config HTTP/1.1" 200 503 "-" "Bitwarden_Mobile/2025.1.2 (release/standard) (Android 14; SDK 34; Model SM-F741B)"

[BrutalBirdie]

After some debugging with the help of @girish a fix was found and will be added.

[girish]

Thanks for reporting. Fix is at https://git.cloudron.io/platform/box/-/commit/063b1024616706971d4a1f9c50b5032727640120

[BrutalBirdie]

Quickfix for users who need it NOW:

# get patch file, apply and remove and restart cloudron-syslog.service
cd /home/yellowtent/box
wget https://git.cloudron.io/platform/box/-/commit/063b1024616706971d4a1f9c50b5032727640120.diff
git apply 063b1024616706971d4a1f9c50b5032727640120.diff
rm -v 063b1024616706971d4a1f9c50b5032727640120.diff
systemctl restart cloudron-syslog.service

[BrutalBirdie]

Extra analysis.
Did this issue really just come up after the upgrade to 22/24?

System Upgrade timers:

2025-02-19 14:33:55	
Linux version 6.8.0-53-generic (buildd@lcy02-amd64-046) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #55-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 17 15:37:52 UTC 2025

2025-02-19 12:33:55	
Linux version 5.15.0-131-generic (buildd@lcy02-amd64-057) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025

2025-02-19 00:33:55	
Linux version 5.4.0-205-generic (buildd@lcy02-amd64-055) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #225-Ubuntu SMP Fri Jan 10 22:23:35 UTC 2025

View of the whole month of February 2025.

Zoom to 2025-02-17:

Yep, this looks very conclusive to me. This issue is only apparent in Ubuntu 22/24 with the non-fixed syslog.js

[jdeighton]

Applied this fix tonight and it worked for me to quiet the errors in the syslog file - thanks for finding and sharing the solution.

[sozialinfo]

We're currently seeing this issue on v8.3.1 (Ubuntu 24.04.1 LTS)

@girish thank you for fixing this! When will this fix be rolled out?

@BrutalBirdie thanks for the quick fix! We applied it and it worked perfectly.

[joseph]

@sozialinfo patch will only be part of cloudron 9 (no ETA, but in coming month)

[tobiasb]

I would like to see this bugfix already in 8.3.x. Please <3.

[necrevistonnezr]

The patch described in https://forum.cloudron.io/topic/13361/after-ubuntu-22-24-upgrade-syslog-getting-spammed-and-grows-way-to-much-clogging-up-the-diskspace/11# Is not available anymore (Error 500)

[BrutalBirdie]

@necrevistonnezr That is because https://git.cloudron.io/ is currently throwing err 500.

---

Has been resolved, is now all good again.

[tobiasb]

Fyi: The fix was not part of 8.3.2. Now we need patch it again.

[bscabl]

@BrutalBirdie this is great, solved the issue for me!