Dashboard unavailable, apps are currently fine

[joshrb91]

My Apps all work and are accessible (although certificates are not automatically renewing) but I am unable to access my dashboard and all troubleshooting attempts have failed. Here is an output of the /home/yellowtent/box/setup/start.sh script

2022-01-04T16:24:13 ==> start: Cloudron Start
media:x:500:
2022-01-04T16:24:13 ==> start: Configuring docker
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable apparmor
Error response from daemon: network with name cloudron already exists
2022-01-04T16:24:17 ==> start: Ensuring directories
2022-01-04T16:24:17 ==> start: Configuring journald
2022-01-04T16:24:18 ==> start: Setting up unbound
2022-01-04T16:24:18 ==> start: Adding systemd services
Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable cron
2022-01-04T16:24:30 ==> start: Configuring sudoers
2022-01-04T16:24:30 ==> start: Configuring collectd
2022-01-04T16:24:30 ==> start: Configuring logrotate
2022-01-04T16:24:30 ==> start: Adding motd message for admins
2022-01-04T16:24:30 ==> start: Configuring nginx
2022-01-04T16:24:31 ==> start: Starting mysql
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-01-04T16:24:31 ==> start: Migrating data
[ERROR] AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Unexpected token '.'
    at /home/yellowtent/box/node_modules/db-migrate/lib/commands/on-complete.js:15:14
    at tryCatcher (/home/yellowtent/box/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.successAdapter [as _fulfillmentHandler0] (/home/yellowtent/box/node_modules/bluebird/js/release/nodeify.js:22:30)
    at Promise._settlePromise (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:601:21)
    at Promise._settlePromise0 (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:456:21)
    let schedulerConfig = app.manifest.addons?.scheduler || null;
                                              ^

SyntaxError: Unexpected token '.'
    at wrapSafe (internal/modules/cjs/loader.js:1053:16)
    at Module._compile (internal/modules/cjs/loader.js:1101:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
    at Module.load (internal/modules/cjs/loader.js:985:32)
    at Function.Module._load (internal/modules/cjs/loader.js:878:14)
    at Module.require (internal/modules/cjs/loader.js:1025:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/home/yellowtent/box/src/dns.js:30:14)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
    at Module.load (internal/modules/cjs/loader.js:985:32)
    at Function.Module._load (internal/modules/cjs/loader.js:878:14)
    at Module.require (internal/modules/cjs/loader.js:1025:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/home/yellowtent/box/src/acme2.js:15:11)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)
2022-01-04T16:24:32 ==> start: DB migration failed

[girish]

@joshrb91 it looks like maybe the node version is not correct . What is the output of node --version ? Also, can you tell me why you are trying to run start.sh manually? Is that because the dashboard went down in the first place?

[joshrb91]

@girish Correct the Dashboard went down first and I used the start.sh script in an attempt to repair it. The version for node is v12.18.4

[girish]

@joshrb91 Ah ok, I think the node version is one part of the issue atleast. The node version should be 14.17.6 (This node version understands the ?. syntax ). Do you have any more context as to how it ended up in this state? Did you run out of disk space or anything?

Can you also tell me what is the version in the file /home/yellowtent/box/VERSION ? This should be 7.0.4

[joshrb91]

@girish Yes the server ran out of space a few weeks back and that is when these issues started. The VERSION result is 7.0.3

[girish]

@joshrb91 I see. Generally, it's not possible that node was not updated but the code itself got updated. I don't know how that can happen even if server ran out of disk space.

My guess is that maybe apt install node was run by mistake. Can you do apt remove nodejs for good measure?

After the above, check if /usr/local/node-14.17.6 exists. If it does, run the below two commands:

    ln -sf /usr/local/node-14.17.6/bin/node /usr/bin/node
    ln -sf /usr/local/node-14.17.6/bin/npm /usr/bin/npm

Now, node --version should be the right version. Then, systemctl restart box should get the box code working.

[joshrb91]

@girish That was able to get box up and running, but the dashboard is still unavailable (I've tried clearing browser cache and checking on a different device/browser with the same result). Could this be caused by the /home/yellowtent/box/VERSION being 7.0.3 instead of 7.0.4?

Also I tried running the start.sh script again and it shows the same error but ultimately still results in box showing active when checking systemctl status box

[girish]

@joshrb91 said in Dashboard unavailable, apps are currently fine:

but the dashboard is still unavailable

What error are you seeing in the browser? Can you check if my.domain.com points to the IP of the server?

[joshrb91]

@girish When loading the page I have the read banner at the top that says "Cloudron is offline. Reconnecting..." I checked the IP address and everything is pointing to the right place. I still have access to all my apps by navigating to their URL.

I also should mention that when logging into the server I see the original post install message saying to navigate to https://publicIP to accept self signed certificate to finish setup. This hadn't appeared before. But when navigating to the IP alone I just get the message "You are seeing this page because this domain points to Cloudron server IP but no app is installed on that domain."

[girish]

@joshrb91 said in Dashboard unavailable, apps are currently fine:

I also should mention that when logging into the server I see the original post install message saying to navigate to https://publicIP to accept self signed certificate to finish setup.

Oh, that's not good. It thinks that Cloudron is not installed/something is corrupt. I think we have to take a look into the server before I suggest something destructive. Can you run cloudron-support --enable-ssh and drop a mail to support@cloudron.io with the server IP? I can take a look.

[joshrb91]

@girish Thanks, I sent you an email with that information

[girish]

@joshrb91 looks like the server has nvm installed. Which then hijacked the system node.

# which node
/root/.nvm/versions/node/v14.17.6/bin/node

The error is not very obvious because systemctl status box says running but it's not really running. journalctl -u box -fa shows that it's crashing constantly. The reason for that is the box code runs as yellowtent user..

# su - yellowtent
$ node --version
v12.18.4
$ which node
/usr/local/bin/node

It seems there is another node installed in /usr/local/bin also. I renamed that directory to bin-old for now and it brings things up.

Generally, we don't support installing external software on Cloudron. Was there any reason you installed nvm on the server? (was it to install cloudron CLI? if so, that's meant to be installed on your PC/laptop and not on the server).