Can't start cloudron service after Default Data Directory migration

[ivan-petro]

We followed the official Cloudron tutorial for moving data to an external volume:
https://docs.cloudron.io/storage/

During the migration process, we faced several issues:

Issue #1: Docker Restart Loop

We attempted to migrate /home/yellowtent/platformdata/logs, but the migration failed because Docker was still using this directory even after stopping the Docker service. Docker automatically restarted itself, preventing the move. To resolve this, we ran the following command:

systemctl mask docker

This allowed us to stop Docker and proceed with the migration.

We attached an ext4 volume from Hetzner to the server, and there is sufficient space on the volume for Cloudron data.

Issue #2: Services Fail to Start After Migration

After completing the migration we restarted services:
box, docker (unmask docker), nginx, collectd

But most services failed to start in "Services" tab in Cloudron. For example graphite, postgresql, unbound, etc. And we can't restart them due to error.

The error message from Cloudron (when we want to start service) is as follows:

(HTTP code 500) server error - Cannot restart container mysql: failed to create task for container: failed to initialize logging driver: dial unix /home/yellowtent/platformdata/logs/syslog.sock: connect: connection refused

Troubleshooting Steps

1. We verified that /mnt/tc-cloudron-volume-1/platformdata/logs/syslog.sock is being created with the root:root ownership and permissions:

srw-rw-rw- 1 root root 0 ...

2. We updated /etc/rsyslog.conf to include the following lines:

module( load="imuxsock" SysSock.Name="/mnt/tc-cloudron-volume-1/platformdata/logs/syslog.sock" SysSock.Owner="yellowtent" SysSock.Group="yellowtent" SysSock.Perm="0666" )

Despite restarting rsyslog and Docker multiple times, the issue persists.

3. We also manually removed the syslog.sock file, restarted rsyslog, and confirmed that it recreates the file. However, it still reverts to root:root ownership.

Current State

• The syslog.sock file is being created with root:root ownership, which prevents Docker containers (e.g., MySQL, PostgreSQL) from initializing correctly.
• Even after applying the correct ownership and permissions manually (chown yellowtent:yellowtent and chmod 0666), the services fail to start.
• We suspect this might be due to a deeper integration issue between rsyslog, Docker, and Cloudron's specific logging requirements.

Environment Details

• Cloudron version: v8.2.3
• OS: Ubuntu 22.04.1 LTS
• Rsyslog version: 8.2.3
• Hosting provider: Hetzner
• Attached volume: ext4 formatted with sufficient space

Request for Assistance

We kindly request guidance on how to resolve the syslog.sock ownership issue and ensure proper integration of logging between Docker and Cloudron after migration to an external volume.

Thank you in advance for your help!

[ivan-petro]

Hi Team,

This issue is particularly significant for users running a large number of apps on Hetzner VPS, as scaling storage is only possible by attaching Hetzner volumes. As you know, Hetzner VPS instances have limited internal disk storage that cannot be expanded directly, making volumes the only viable option for increasing capacity.

In Cloudron, as I understand it, PostgreSQL runs in Docker and stores its data centrally on the server for all apps. Unfortunately, this setup doesn’t allow migrating specific PostgreSQL databases for individual apps to the volume. Instead, the entire /home/yellowtent/platformdata directory needs to be migrated to the attached volume, with symlinks created to point to the new storage location.

This makes the migration process challenging, especially given the dependency on syslog.sock and the logging configuration. Resolving this issue would greatly help users like me who need to scale storage while maintaining Cloudron’s functionality.

[joseph]

@ivan-petro can you try systemctl restart cloudron-syslog ?

[ivan-petro]

Hi @joseph,

No, I haven’t tried running
systemctl restart cloudron-syslog

Do you think this could help resolve the issue?

I followed the official Cloudron documentation for migrating the default data directory: https://docs.cloudron.io/storage/. However, the documentation doesn’t mention restarting cloudron-syslog as part of the process.

To try this command, I’ll need to wait until my scheduled maintenance window at night to work on the server. If you have any other suggestions, please let me know so I can try everything during my maintenance period.

Thank you for your assistance!

[joseph]

@ivan-petro that service is added to docs now . that services holds the syslog.sock file

[ivan-petro]

@joseph , got it!
thanks. will do and let you know about result

[ivan-petro]

Hi @joseph ,

Yes, your new instructions at https://docs.cloudron.io/storage/ were very helpful! However, I still had to manually restart the services on the /services page in the Cloudron admin panel.

Regarding the time it takes to reindex disk information, it happened within the same day for me. I’ll continue monitoring the system's stability over the next few days to ensure everything is running smoothly.

Thank you so much for your help! Thanks to your assistance, I was finally able to resolve the issue with running out of disk space

[luckow]

sorry for bringing up this topic again.
I followed the document (https://docs.cloudron.io/storage/) and ended up at

Feb 22 14:49:00 cos syslog.js[6203]: node:internal/process/promises:391
Feb 22 14:49:00 cos syslog.js[6203]:     triggerUncaughtException(err, true /* fromPromise */);
Feb 22 14:49:00 cos syslog.js[6203]:     ^
Feb 22 14:49:00 cos syslog.js[6203]: [Error: EACCES: permission denied, lstat '/home/yellowtent/platformdata/logs/syslog.sock'] {
Feb 22 14:49:00 cos syslog.js[6203]:   errno: -13,
Feb 22 14:49:00 cos syslog.js[6203]:   code: 'EACCES',
Feb 22 14:49:00 cos syslog.js[6203]:   syscall: 'lstat',
Feb 22 14:49:00 cos syslog.js[6203]:   path: '/home/yellowtent/platformdata/logs/syslog.sock'
Feb 22 14:49:00 cos syslog.js[6203]: }
Feb 22 14:49:00 cos syslog.js[6203]: Node.js v20.18.0
Feb 22 14:49:00 cos systemd[1]: cloudron-syslog.service: Main process exited, code=exited, status=1/FAILURE
Feb 22 14:49:00 cos systemd[1]: cloudron-syslog.service: Failed with result 'exit-code'.

[nebulon]

@luckow have you tried to delete that file /home/yellowtent/platformdata/logs/syslog.sock via ssh?

[luckow]

yes. This is what has happened in the meantime

journalctl -f

Feb 22 17:41:06 cos systemd[1]: box.service: Main process exited, code=exited, status=1/FAILURE
Feb 22 17:41:06 cos systemd[1]: box.service: Failed with result 'exit-code'.
Feb 22 17:41:06 cos systemd[1]: box.service: Scheduled restart job, restart counter is at 3937.
Feb 22 17:41:06 cos systemd[1]: Stopped Cloudron Admin.
Feb 22 17:41:06 cos systemd[1]: Started Cloudron Admin.
Feb 22 17:41:07 cos sshd[32331]: Received disconnect from 218.92.0.237 port 48908:11:  [preauth]
Feb 22 17:41:07 cos sshd[32331]: Disconnected from authenticating user root 218.92.0.237 port 48908 [preauth]
Feb 22 17:41:07 cos sshd[32331]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.237  user=root
Feb 22 17:41:09 cos box.js[32364]: node:fs:827
Feb 22 17:41:09 cos box.js[32364]:   fd = getValidatedFd(fd);
Feb 22 17:41:09 cos box.js[32364]:        ^
Feb 22 17:41:09 cos box.js[32364]: TypeError [ERR_INVALID_ARG_TYPE]: The "fd" argument must be of type number. Received undefined
Feb 22 17:41:09 cos box.js[32364]:     at Object.write (node:fs:827:8)
Feb 22 17:41:09 cos box.js[32364]:     at exitSync (/home/yellowtent/box/box.js:39:26)
Feb 22 17:41:09 cos box.js[32364]:     at main (/home/yellowtent/box/box.js:58:23) {
Feb 22 17:41:09 cos box.js[32364]:   code: 'ERR_INVALID_ARG_TYPE'
Feb 22 17:41:09 cos box.js[32364]: }
Feb 22 17:41:09 cos box.js[32364]: Node.js v20.18.0

[luckow]

fyi: As soon as I have moved platformdata back to the “original” storage, everything works as expected the error disappeared.
I also had to move the appsdata & boxdata back to get a fully functional Cloudron instance again.