Restarting - Waiting for Cloudron to start

[hugoo_souza10]

I have the n8n application installed for a long time, and out of nowhere it entered the update and does not leave this screen anymore. I already checked and it has enough space and I even expanded my server. What can it be? Can you help me?![alt text]

[michaelpope]

In the past when I've had these sort of things happen (pretty rare), it's best to try pressing the x next to the restart to stop the restart. Then start the container again.

If that doesn't fix things, you might need to do a repair on the container (see the repair tab), or restore from a recent backup (which will usually get it to a stable state).

[hugoo_souza10]

Good evening, I already tried pressing "x" and it is not working and neither is restoring from a backup. I'll leave an image here of what appears

[girish]

@hugoo_souza10 Can you check /home/yellowtent/platformdata/logs/box.log ? One or more internal services is not startup up properly (like some database). When those don't start, the apps cannot function properly.

[Sjorsdeleede]

i got exactry the same problem, my Vauldwarden and Guacamole won't start via the cloudron webpage:

I already tried to restart the "box" service with sudo systemctl restart box without any result

I also checked if all the services in cloudron are started and they are:

For now i found a work around (because i needed my password mannager ) by starting the docker containers by hand, but after every restart you have to start them again.

First you need to know the conainer ID, you can do this with sudo docker container ls -a that will show all your containers, including the ones that are offline:

After you got the right container ID you can run sudo docker container start "*your container ID*"

Than run sudo docker ps to check if the container is in "running" status.

For now my 2 apps are working but im still looking to find a propper solution

[girish]

@Sjorsdeleede First, I would do sudo systemctl restart box again and check the logs at /home/yellowtent/platformdata/logs/box.log . Do you see any errors ? It seems to me that one or more services are maybe not starting properly.

[hugoo_souza10]

@girish My log is very big and I didn't understand much, but I checked the services and I didn't find anything in the databases

[hugoo_souza10]

@girish About the log, I really didn't understand very well what happened, I can show you some parts!

[girish]

@hugoo_souza10 The mail service is not starting. Are you able to run docker logs -f mail and what output that gives ?

[hugoo_souza10]

@girish when I run the command: "docker logs -f mail" it returns me this: "No such container: mail"
Which indicates that the container does not exist. Here are the containers

[girish]

@hugoo_souza10 The mail container is not starting/getting created. The logs cut off right at the point where the mail container is getting created (the box.log in your screenshot). Are there any errors after that line?

Usually, I have seen this fail when some other mail server is running on port 25. Do you happen to run postfix/exim or some other mail server ?

[hugoo_souza10]

@girish I only have 1 application installed on this vps, the "N8N". I don't think he even has the email container

[nebulon]

The mail addon, like most addons are always present. Even with n8n it is used for the email sending infrastructure.

[hugoo_souza10]

@nebulon I think that in the case of N8N, the email is configured through some environment variables within the application.

I entered cloudron notifications here and there is this

[nebulon]

Yes those env variables are injected from the platform into the app container and then further used to configure the app to use the platform's email logic.

So the email service is still not working as expected from the services view? Have you checked if port 25 is free for it to take? You can check this for example withnetstat -ltnp | grep -w ':25'

[hugoo_souza10]

@nebulon I can't run this command or install net-tools

[girish]

@hugoo_souza10 that's a good point to start investigating. Why are you unable to install packages ? It says line 11 of /etc/apt/sources.list is malformed. Please chat that file.

[Sjorsdeleede]

@girish Sorry for my late response.

Today i restarted the "box" service again and looked at the log as you sad.

Here is the output from my log:

2023-08-06T17:33:27.114Z box:server ==========================================
2023-08-06T17:33:27.115Z box:server Cloudron 7.4.1
2023-08-06T17:33:27.115Z box:server ==========================================
2023-08-06T17:33:27.178Z box:settings initCache: pre-load settings
2023-08-06T17:33:27.283Z box:tasks stopAllTasks: stopping all tasks
2023-08-06T17:33:27.284Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2023-08-06T17:33:27.396Z box:shell stopTask (stdout): All tasks stopped

2023-08-06T17:33:27.398Z box:reverseproxy writeDashboardConfig: writing admin config for REMOVED.eu
Cloudron is up and running. Logs are at /home/yellowtent/platformdata/logs/box.log
2023-08-06T17:33:27.448Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2023-08-06T17:33:28.258Z box:cloudron onActivated: running post activation tasks
2023-08-06T17:33:28.259Z box:platform initializing platform
2023-08-06T17:33:28.261Z box:platform Updating infrastructure from 49.4.0 to 49.4.0
2023-08-06T17:33:28.263Z box:locker Acquired : platform_start
2023-08-06T17:33:28.269Z box:platform markApps: changedAddons: ["mysql","postgresql","mongodb","redis"]
2023-08-06T17:33:28.301Z box:services startServices: existing infra. incremental service create ["startMail","startTurn","startMysql","startPostgresql","startMongodb","startRedis","startGraphite","start"]
2023-08-06T17:33:28.302Z box:mail startMail: starting
2023-08-06T17:33:28.309Z box:mail restartMail: restarting mail container with mailFqdn:my.REMOVED.eu mailDomain:REMOVED.eu
2023-08-06T17:33:28.393Z box:shell stopMail exec: docker stop mail || true
2023-08-06T17:33:34.677Z box:shell stopMail (stdout): mail

2023-08-06T17:33:34.677Z box:shell stopMail (stderr): null
2023-08-06T17:33:34.677Z box:shell removeMail exec: docker rm -f mail || true
2023-08-06T17:33:34.732Z box:shell removeMail (stdout): mail

2023-08-06T17:33:34.732Z box:shell removeMail (stderr): null
2023-08-06T17:33:34.732Z box:mail createMailConfig: generating mail config with my.REMOVED.eu
2023-08-06T17:33:34.736Z box:shell startMail exec: docker run --restart=always -d --name="mail" --net cloudron --net-alias mail --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:2514 --log-opt syslog-format=rfc5424 --log-opt tag=mail -m 214958080 --memory-swap 536870912 --dns 172.18.0.1 --dns-search=. -e CLOUDRON_MAIL_TOKEN="REMOVED" -e CLOUDRON_RELAY_TOKEN=" REMOVED " -e LOGLEVEL=info -v "/home/yellowtent/boxdata/mail:/app/data" -v "/home/yellowtent/platformdata/addons/mail:/etc/mail:ro" -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995 --label isCloudronManaged=true --read-only -v /run -v /tmp cloudron/mail:3.8.3@sha256: REMOVED
2023-08-06T17:33:36.145Z box:shell startMail (stdout): REMOVED

2023-08-06T17:33:36.147Z box:shell startMail (stderr): null
2023-08-06T17:33:36.199Z box:shell stopTurn exec: docker stop turn || true
2023-08-06T17:33:46.753Z box:shell stopTurn (stdout): turn

2023-08-06T17:33:46.753Z box:shell stopTurn (stderr): null
2023-08-06T17:33:46.754Z box:shell removeTurn exec: docker rm -f turn || true
2023-08-06T17:33:46.947Z box:shell removeTurn (stdout): turn

2023-08-06T17:33:46.947Z box:shell removeTurn (stderr): null
2023-08-06T17:33:46.947Z box:shell startTurn exec: docker run --restart=always -d --name="turn" --hostname turn --net host --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:2514 --log-opt syslog-format=rfc5424 --log-opt tag=turn -m 106954752 --memory-swap 268435456 --dns 172.18.0.1 --dns-search=. -e CLOUDRON_TURN_SECRET=" REMOVED " -e CLOUDRON_REALM="my. REMOVED.eu" --label isCloudronManaged=true --read-only -v /tmp -v /run "cloudron/turn:1.5.0@sha256: REMOVED"
2023-08-06T17:33:47.529Z box:shell startTurn (stdout): REMOVED

2023-08-06T17:33:47.530Z box:shell startTurn (stderr): null
2023-08-06T17:33:47.533Z box:shell stopMysql exec: docker stop mysql || true
2023-08-06T17:33:53.003Z box:shell stopMysql (stdout): mysql

2023-08-06T17:33:53.003Z box:shell stopMysql (stderr): null
2023-08-06T17:33:53.003Z box:shell removeMysql exec: docker rm -f mysql || true
2023-08-06T17:33:53.079Z box:shell removeMysql (stdout): mysql

2023-08-06T17:33:53.079Z box:shell removeMysql (stderr): null
2023-08-06T17:33:53.080Z box:shell startMysql exec: docker run --restart=always -d --name="mysql" --hostname mysql --net cloudron --net-alias mysql --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:2514 --log-opt syslog-format=rfc5424 --log-opt tag=mysql --dns 172.18.0.1 --dns-search=. --ip 172.18.30.1 -e CLOUDRON_MYSQL_TOKEN= REMOVED -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1 -e CLOUDRON_MYSQL_ROOT_PASSWORD=REMOVED -v "/home/yellowtent/platformdata/mysql:/var/lib/mysql" --label isCloudronManaged=true --cap-add SYS_NICE --read-only -v /tmp -v /run "cloudron/mysql:3.3.7@sha256: REMOVED"
2023-08-06T17:33:54.236Z box:shell startMysql (stdout): REMOVED

2023-08-06T17:33:54.236Z box:shell startMysql (stderr): null
2023-08-06T17:33:54.237Z box:services Waiting for mysql
2023-08-06T17:33:54.265Z box:services Attempt 1 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2023-08-06T17:34:10.148Z box:services startPostgresql: postgresql will be upgraded
2023-08-06T17:34:10.148Z box:services exportDatabase: Exporting postgresql
2023-08-06T17:34:10.159Z box:shell exportDatabase - removepostgresql exec: docker rm -f postgresql
2023-08-06T17:34:10.740Z box:shell exportDatabase - removepostgresql (stdout): postgresql

2023-08-06T17:34:10.740Z box:shell exportDatabase - removepostgresql (stderr): null
2023-08-06T17:34:10.740Z box:shell exportDatabase - removeAddonDirpostgresql spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/rmaddondir.sh postgresql
2023-08-06T17:34:10.836Z box:shell stopPostgresql exec: docker stop postgresql || true
2023-08-06T17:34:10.929Z box:shell stopPostgresql (stdout): null
2023-08-06T17:34:10.929Z box:shell stopPostgresql (stderr): Error response from daemon: No such container: postgresql

2023-08-06T17:34:10.929Z box:shell removePostgresql exec: docker rm -f postgresql || true
2023-08-06T17:34:10.978Z box:shell removePostgresql (stdout): null
2023-08-06T17:34:10.978Z box:shell removePostgresql (stderr): Error: No such container: postgresql

2023-08-06T17:34:10.978Z box:shell startPostgresql exec: docker run --restart=always -d --name="postgresql" --hostname postgresql --net cloudron --net-alias postgresql --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:2514 --log-opt syslog-format=rfc5424 --log-opt tag=postgresql --dns 172.18.0.1 --dns-search=. --ip 172.18.30.2 --shm-size=128M -e CLOUDRON_POSTGRESQL_ROOT_PASSWORD=" REMOVED " -e CLOUDRON_POSTGRESQL_TOKEN=" REMOVED" -v "/home/yellowtent/platformdata/postgresql:/var/lib/postgresql" --label isCloudronManaged=true --read-only -v /tmp -v /run "cloudron/postgresql:5.0.6@sha256: REMOVED"
2023-08-06T17:34:12.031Z box:shell startPostgresql (stdout): REMOVED

2023-08-06T17:34:12.031Z box:shell startPostgresql (stderr): null
2023-08-06T17:34:12.031Z box:services Waiting for postgresql
2023-08-06T17:34:12.046Z box:services Attempt 1 failed. Will retry: Network error waiting for postgresql: connect ECONNREFUSED 172.18.30.2:3000
2023-08-06T17:34:27.788Z box:services importDatabase: Importing postgresql
2023-08-06T17:34:27.814Z box:shell stopMongodb exec: docker stop mongodb || true
2023-08-06T17:34:32.542Z box:shell stopMongodb (stdout): mongodb

2023-08-06T17:34:32.543Z box:shell stopMongodb (stderr): null
2023-08-06T17:34:32.543Z box:shell removeMongodb exec: docker rm -f mongodb || true
2023-08-06T17:34:32.635Z box:shell removeMongodb (stdout): mongodb

2023-08-06T17:34:32.635Z box:shell removeMongodb (stderr): null
2023-08-06T17:34:32.635Z box:shell startMongodb exec: docker run --restart=always -d --name="mongodb" --hostname mongodb --net cloudron --net-alias mongodb --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:2514 --log-opt syslog-format=rfc5424 --log-opt tag=mongodb --dns 172.18.0.1 --dns-search=. --ip 172.18.30.3 -e CLOUDRON_MONGODB_ROOT_PASSWORD=" REMOVED" -e CLOUDRON_MONGODB_TOKEN=" REMOVED" -v "/home/yellowtent/platformdata/mongodb:/var/lib/mongodb" --label isCloudronManaged=true --read-only -v /tmp -v /run "cloudron/mongodb:4.3.6@sha256: REMOVED"
2023-08-06T17:34:33.798Z box:shell startMongodb (stdout): REMOVED

2023-08-06T17:34:33.798Z box:shell startMongodb (stderr): null
2023-08-06T17:34:33.799Z box:services Waiting for mongodb
2023-08-06T17:34:33.804Z box:services Attempt 1 failed. Will retry: Network error waiting for mongodb: connect ECONNREFUSED 172.18.30.3:3000
2023-08-06T17:34:48.815Z box:services Attempt 2 failed. Will retry: Network error waiting for mongodb: connect ECONNREFUSED 172.18.30.3:3000
2023-08-06T17:35:06.192Z box:shell stopGraphite exec: docker stop graphite || true
2023-08-06T17:35:10.269Z box:shell stopGraphite (stdout): graphite

2023-08-06T17:35:10.269Z box:shell stopGraphite (stderr): null
2023-08-06T17:35:10.269Z box:shell removeGraphite exec: docker rm -f graphite || true
2023-08-06T17:35:10.337Z box:shell removeGraphite (stdout): graphite

2023-08-06T17:35:10.337Z box:shell removeGraphite (stderr): null
2023-08-06T17:35:10.337Z box:shell startGraphite exec: docker run --restart=always -d --name="graphite" --hostname graphite --net cloudron --net-alias graphite --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:2514 --log-opt syslog-format=rfc5424 --log-opt tag=graphite -m 106954752 --memory-swap 268435456 --dns 172.18.0.1 --dns-search=. -p 127.0.0.1:2003:2003 -v "/home/yellowtent/platformdata/graphite:/var/lib/graphite" --label isCloudronManaged=true --read-only -v /tmp -v /run "cloudron/graphite:3.3.0@sha256: REMOVED"
2023-08-06T17:35:11.538Z box:shell startGraphite (stdout): REMOVED

2023-08-06T17:35:11.538Z box:shell startGraphite (stderr): null
2023-08-06T17:35:11.539Z box:sftp start: re-creating container
2023-08-06T17:35:11.584Z box:sftp ensureSecrets: generating new sftp keys of type rsa
2023-08-06T17:35:13.533Z box:platform Failed to start services. retry=false (attempt 0): Could not generate sftp rsa keys: Command failed: ssh-keygen -m PEM -t rsa -f "/home/yellowtent/platformdata/sftp/ssh/ssh_host_rsa_key" -q -N ""
2023-08-06T17:35:13.534Z box:cloudron Startup task at index 2 failed: Could not generate sftp rsa keys: Command failed: ssh-keygen -m PEM -t rsa -f "/home/yellowtent/platformdata/sftp/ssh/ssh_host_rsa_key" -q -N "" BoxError: Could not generate sftp rsa keys: Command failed: ssh-keygen -m PEM -t rsa -f "/home/yellowtent/platformdata/sftp/ssh/ssh_host_rsa_key" -q -N ""
at ensureKeys (/home/yellowtent/box/src/sftp.js:36:149)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async start (/home/yellowtent/box/src/sftp.js:60:5)
at async Object.startServices (/home/yellowtent/box/src/services.js:779:9)
at async Object.start (/home/yellowtent/box/src/platform.js:68:13)
at async onActivated (/home/yellowtent/box/src/cloudron.js:82:5)
at async Array.<anonymous> (/home/yellowtent/box/src/cloudron.js:134:9)
2023-08-06T17:36:11.559Z box:shell restartcollectd spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh collectd
2023-08-06T17:38:28.302Z box:locker Lock unreleased platform_start

i removed all passwords and tokens etc.

[nebulon]

@Sjorsdeleede that looks not too bad, seems like the host key for SFTP couldn't be generated (no further error details in the logs there) but otherwise things came up?

Given the sftp ssh key error seems rather specific, I guess its fastest if you enable remote SSH support for us and send an email to support@cloudron.io so we can take a direct look there.

[Sjorsdeleede]

@nebulon Thanks for the quick response, I just enabled remote SSH.
I also took a snapshot of the server so don't worry to try thinks out
Will also send a email with my domain i use for cloudron

Everything is running except my 2 apps they are still in the "Restarting - Waiting for Cloudron to initialize" stat