Cloudron Offline

[Setsuneh]

Hi,

Unable to Connect on my portal Cloudron, all seems down. I've checked the issue, but all seem ok unfortunatly :

systemctl status box           
* box.service - Cloudron Admin
     Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-01-25 11:36:59 CET; 169ms ago
   Main PID: 62459 (node)
      Tasks: 7 (limit: 77014)
     Memory: 8.3M (max: 400.0M available: 391.6M)
        CPU: 167ms
     CGroup: /system.slice/box.service
             `-62459 node /home/yellowtent/box/box.js

Jan 25 11:36:59 Webhost systemd[1]: Started Cloudron Admin.

Nginx too :

systemctl status nginx
* nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             `-cloudron.conf
     Active: active (running) since Thu 2024-01-25 11:27:31 CET; 10min ago
       Docs: man:nginx(8)
   Main PID: 853 (nginx)
      Tasks: 21 (limit: 77014)
     Memory: 52.8M
        CPU: 1.515s
     CGroup: /system.slice/nginx.service
             |-853 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             |-854 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-855 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-856 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-857 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-858 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-859 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-860 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-861 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-862 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-863 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-864 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-865 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-866 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-867 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-868 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-869 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-870 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-871 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             |-872 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             `-873 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

Jan 25 11:27:30 Webhost systemd[1]: Starting A high performance web server and a reverse proxy server...
Jan 25 11:27:31 Webhost systemd[1]: Started A high performance web server and a reverse proxy server.

Mysql too :

systemctl status mysql
* mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-01-25 11:27:15 CET; 11min ago
   Main PID: 727 (mysqld)
     Status: "Server is operational"
      Tasks: 38 (limit: 77014)
     Memory: 217.6M
        CPU: 3.545s
     CGroup: /system.slice/mysql.service
             `-727 /usr/sbin/mysqld

Jan 25 11:27:14 Webhost systemd[1]: Starting MySQL Community Server...
Jan 25 11:27:15 Webhost systemd[1]: Started MySQL Community Server.

I've launched start.sh too, the issue is the same.

Last lines ont box.log :

* box-task-1777.service loaded failed failed /home/yellowtent/box/src/scripts/../taskworker.js 1777 /home/yellowtent/platformdata/logs/tasks/1777.log
* box-task-1778.service loaded failed failed /home/yellowtent/box/src/scripts/../taskworker.js 1778 /home/yellowtent/platformdata/logs/7088a5c3-3699-42ba-bfe2-b39c28b77892/apptask.log

2024-01-25T10:08:37.000Z box:shell stopTask (stdout): All tasks stopped

Can you help me please ? Thank you !

[girish]

@Setsuneh what error are you seeing the browser?

Can you try to run cloudron-support --troubleshoot ?

[Setsuneh]

Hi,

I've got "Cloudron is offline, Reconnection ...".

Here is the result

cloudron-support --troubleshoot
[OK]	nginx is running
[OK]	docker is running
[OK]	MySQL is running
[OK]	box is running
[OK]	unbound is running
[OK]	Hairpin NAT is good
[OK]	Domain XXX.fr is valid and has not expired

[girish]

@Setsuneh everything seems fine. Hmm. Do you see anything in /home/yellowtent/platformdata/logs/box.log ?

[Setsuneh]

@girish I've put on the first message :

* box-task-1777.service loaded failed failed /home/yellowtent/box/src/scripts/../taskworker.js 1777 /home/yellowtent/platformdata/logs/tasks/1777.log
* box-task-1778.service loaded failed failed /home/yellowtent/box/src/scripts/../taskworker.js 1778 /home/yellowtent/platformdata/logs/7088a5c3-3699-42ba-bfe2-b39c28b77892/apptask.log

2024-01-25T10:08:37.000Z box:shell stopTask (stdout): All tasks stopped

1777 & 1778 errors provide from Backup Error & Gitlab Update error

[girish]

@Setsuneh mm, in the box logs usually one can see that it is starting up. Looks like it is not for some reason. Can you try systemctl restart box ? You should see some logs like this:

2024-01-25T11:18:22.046Z box:server ==========================================
2024-01-25T11:18:22.047Z box:server            Cloudron 7.6.4  
2024-01-25T11:18:22.047Z box:server ==========================================
2024-01-25T11:18:22.047Z box:platform initializing platform
2024-01-25T11:18:22.099Z box:tasks stopAllTasks: stopping all tasks
2024-01-25T11:18:22.100Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2024-01-25T11:18:22.151Z box:shell stopTask (stdout): All tasks stopped

2024-01-25T11:18:22.170Z box:reverseproxy writeDashboardConfig: writing dashboard config for smartserver.io
2024-01-25T11:18:22.204Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2024-01-25T11:18:22.461Z box:platform onActivated: starting post activation services
2024-01-25T11:18:22.462Z box:platform startInfra: checking infrastructure
2024-01-25T11:18:22.463Z box:platform startInfra: infra is uptodate at version 49.5.0
2024-01-25T11:18:22.463Z box:platform onInfraReady: platform is ready. infra changed: false
2024-01-25T11:18:22.463Z box:apps schedulePendingTasks: scheduling app tasks
2024-01-25T11:18:22.491Z box:cron startJobs: starting cron jobs with hour 8 and minute 13
...

[girish]

As a final resort, if there is nothing helpful there, please write to support@cloudron.io and we have to debug further.

[Setsuneh]

No i don't have theses logs, but theses ones :

2024-01-25T10:01:47.569Z box:shell startTask code: 1, signal: null
2024-01-25T10:01:47.569Z box:tasks startTask: 1775 completed with code 1
2024-01-25T10:01:47.571Z box:tasks setCompleted - 1775: {"error":{"message":"Task 1775 crashed with code 1","code":"crashed"}}
2024-01-25T10:01:47.571Z box:tasks update 1775: {"percent":100,"error":{"message":"Task 1775 crashed with code 1","code":"crashed"}}
2024-01-25T10:01:47.574Z box:apps scheduleTask: task 1775 of 7088a5c3-3699-42ba-bfe2-b39c28b77892 completed
2024-01-25T10:01:47.574Z box:apps Apptask crashed/stopped: Task 1775 crashed with code 1
2024-01-25T10:01:47.575Z box:locker Released : apptask
2024-01-25T10:01:47.575Z box:scheduler resumeJobs: 7088a5c3-3699-42ba-bfe2-b39c28b77892
2024-01-25T10:01:47.575Z box:tasks startTask: 1775 done. error: { message: 'Task 1775 crashed with code 1', code: 'crashed' }
2024-01-25T10:01:50.204Z box:apphealthmonitor app health: 10 running / 0 stopped / 2 unresponsive
2024-01-25T10:01:56.935Z box:updatechecker checkBoxUpdates: checking for updates
2024-01-25T10:01:57.411Z box:updatechecker checkBoxUpdates: Skipping notification of box update 7.6.4 as user was already notified
2024-01-25T10:01:57.411Z box:updatechecker checkAppUpdates: checking for updates
2024-01-25T10:02:00.251Z box:apphealthmonitor app health: 10 running / 0 stopped / 2 unresponsive
2024-01-25T10:02:02.842Z box:locker Acquired : apptask
2024-01-25T10:02:02.842Z box:scheduler suspendJobs: 7088a5c3-3699-42ba-bfe2-b39c28b77892
2024-01-25T10:02:02.842Z box:tasks startTask - starting task 1776 with options {"timeout":72000000,"nice":15,"memoryLimit":400,"logFile":"/home/yellowtent/platformdata/logs/7088a5c3-3699-42ba-bfe2-b39c28b77892/apptask.log"}. logs at /home/yellowtent/platformdata/logs/7088a5c3-3699-42ba-bfe2-b39c28b77892/apptask.log
2024-01-25T10:02:02.842Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 1776 /home/yellowtent/platformdata/logs/7088a5c3-3699-42ba-bfe2-b39c28b77892/apptask.log 15 400
2024-01-25T10:02:02.962Z box:shell startTask (stderr): Running as unit: box-task-1776.service

2024-01-25T10:02:03.142Z box:shell startTask (stderr): /home/yellowtent/box/src/apps.js:359
    let schedulerConfig = app.manifest.addons?.scheduler || null;
                                              ^

SyntaxError: Unexpected token '.'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/home/yellowtent/box/src/apptask.js:14:14)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

2024-01-25T10:02:03.149Z box:shell startTask (stderr): Finished with result: exit-code
Main processes terminated with: code=exited/status=1
Service runtime: 179ms
CPU time consumed: 179ms

Ticket is already open but my mail is down with this problem

[girish]

@Setsuneh got the ticket, can you also run cloudron-support --enable-ssh ? This will let us SSH.

[girish]

@Setsuneh said in Cloudron Offline:

SyntaxError: Unexpected token '.'
at wrapSafe (internal/modules/cjs/loader.js:915:16)
at Module._compile (internal/modules/cjs/loader.js:963:27)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

aha! most likely nodejs in the server has changed. Did you install additional package/things on the server?

[Setsuneh]

Accès done, thank you !
Hum no, only Crowdsev, but seems it does not need nodejs

[girish]

@Setsuneh yeah, node got downgraded

# node --version
v12.22.9

[girish]

@Setsuneh I have switched back node now to 18.16.0 which Cloudron requires.

(generally, this is why we don't recommend installing additional packages. they have side effects.)

I did this to bring it back up (in case it broke other things):

node_version=18.16.0
ln -sf /usr/local/node-${node_version}/bin/node /usr/bin/node
ln -sf /usr/local/node-${node_version}/bin/npm /usr/bin/npm
systemctl restart box

[Setsuneh]

You rocks thank you very much ! If I understand, it's due to Crowdsec ?

[girish]

@Setsuneh that I don't know. Never used crowdsec.