SOLVED Dashboard unavailable, apps are currently fine
My Apps all work and are accessible (although certificates are not automatically renewing) but I am unable to access my dashboard and all troubleshooting attempts have failed. Here is an output of the /home/yellowtent/box/setup/start.sh script
2022-01-04T16:24:13 ==> start: Cloudron Start media:x:500: 2022-01-04T16:24:13 ==> start: Configuring docker Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable apparmor Error response from daemon: network with name cloudron already exists 2022-01-04T16:24:17 ==> start: Ensuring directories 2022-01-04T16:24:17 ==> start: Configuring journald 2022-01-04T16:24:18 ==> start: Setting up unbound 2022-01-04T16:24:18 ==> start: Adding systemd services Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable unbound Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable cron 2022-01-04T16:24:30 ==> start: Configuring sudoers 2022-01-04T16:24:30 ==> start: Configuring collectd 2022-01-04T16:24:30 ==> start: Configuring logrotate 2022-01-04T16:24:30 ==> start: Adding motd message for admins 2022-01-04T16:24:30 ==> start: Configuring nginx 2022-01-04T16:24:31 ==> start: Starting mysql mysqladmin: [Warning] Using a password on the command line interface can be insecure. Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety. mysql: [Warning] Using a password on the command line interface can be insecure. mysql: [Warning] Using a password on the command line interface can be insecure. 2022-01-04T16:24:31 ==> start: Migrating data [ERROR] AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Unexpected token '.' at /home/yellowtent/box/node_modules/db-migrate/lib/commands/on-complete.js:15:14 at tryCatcher (/home/yellowtent/box/node_modules/bluebird/js/release/util.js:16:23) at Promise.successAdapter [as _fulfillmentHandler0] (/home/yellowtent/box/node_modules/bluebird/js/release/nodeify.js:22:30) at Promise._settlePromise (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:601:21) at Promise._settlePromise0 (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:649:10) at Promise._settlePromises (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:729:18) at _drainQueueStep (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:93:12) at _drainQueue (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:86:9) at Async._drainQueues (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:102:5) at Immediate.Async.drainQueues [as _onImmediate] (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:15:14) at processImmediate (internal/timers.js:456:21) let schedulerConfig = app.manifest.addons?.scheduler || null; ^ SyntaxError: Unexpected token '.' at wrapSafe (internal/modules/cjs/loader.js:1053:16) at Module._compile (internal/modules/cjs/loader.js:1101:27) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10) at Module.load (internal/modules/cjs/loader.js:985:32) at Function.Module._load (internal/modules/cjs/loader.js:878:14) at Module.require (internal/modules/cjs/loader.js:1025:19) at require (internal/modules/cjs/helpers.js:72:18) at Object.<anonymous> (/home/yellowtent/box/src/dns.js:30:14) at Module._compile (internal/modules/cjs/loader.js:1137:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10) at Module.load (internal/modules/cjs/loader.js:985:32) at Function.Module._load (internal/modules/cjs/loader.js:878:14) at Module.require (internal/modules/cjs/loader.js:1025:19) at require (internal/modules/cjs/helpers.js:72:18) at Object.<anonymous> (/home/yellowtent/box/src/acme2.js:15:11) at Module._compile (internal/modules/cjs/loader.js:1137:30) 2022-01-04T16:24:32 ==> start: DB migration failed
@joshrb91 Ah ok, I think the node version is one part of the issue atleast. The node version should be
14.17.6(This node version understands the
?.syntax ). Do you have any more context as to how it ended up in this state? Did you run out of disk space or anything?
Can you also tell me what is the version in the file
/home/yellowtent/box/VERSION? This should be 7.0.4
@girish Yes the server ran out of space a few weeks back and that is when these issues started. The VERSION result is 7.0.3
@joshrb91 I see. Generally, it's not possible that node was not updated but the code itself got updated. I don't know how that can happen even if server ran out of disk space.
My guess is that maybe
apt install nodewas run by mistake. Can you do
apt remove nodejsfor good measure?
After the above, check if
/usr/local/node-14.17.6exists. If it does, run the below two commands:
ln -sf /usr/local/node-14.17.6/bin/node /usr/bin/node ln -sf /usr/local/node-14.17.6/bin/npm /usr/bin/npm
node --versionshould be the right version. Then,
systemctl restart boxshould get the box code working.
@girish That was able to get box up and running, but the dashboard is still unavailable (I've tried clearing browser cache and checking on a different device/browser with the same result). Could this be caused by the /home/yellowtent/box/VERSION being 7.0.3 instead of 7.0.4?
Also I tried running the start.sh script again and it shows the same error but ultimately still results in box showing active when checking systemctl status box
@girish When loading the page I have the read banner at the top that says "Cloudron is offline. Reconnecting..." I checked the IP address and everything is pointing to the right place. I still have access to all my apps by navigating to their URL.
I also should mention that when logging into the server I see the original post install message saying to navigate to https://publicIP to accept self signed certificate to finish setup. This hadn't appeared before. But when navigating to the IP alone I just get the message "You are seeing this page because this domain points to Cloudron server IP but no app is installed on that domain."
I also should mention that when logging into the server I see the original post install message saying to navigate to https://publicIP to accept self signed certificate to finish setup.
Oh, that's not good. It thinks that Cloudron is not installed/something is corrupt. I think we have to take a look into the server before I suggest something destructive. Can you run
cloudron-support --enable-sshand drop a mail to email@example.com with the server IP? I can take a look.
@girish Thanks, I sent you an email with that information
@joshrb91 looks like the server has nvm installed. Which then hijacked the system node.
# which node /root/.nvm/versions/node/v14.17.6/bin/node
The error is not very obvious because
systemctl status boxsays running but it's not really running.
journalctl -u box -fashows that it's crashing constantly. The reason for that is the box code runs as
# su - yellowtent $ node --version v12.18.4 $ which node /usr/local/bin/node
It seems there is another node installed in /usr/local/bin also. I renamed that directory to bin-old for now and it brings things up.
Generally, we don't support installing external software on Cloudron. Was there any reason you installed nvm on the server? (was it to install cloudron CLI? if so, that's meant to be installed on your PC/laptop and not on the server).