Unsolved "Cloudron Error" in dashboard UI

[Aizat]

@girish
Thanks!
Not sure if this is related, I started to see this too sometimes, does Cloudron behave like this when the internet connection is not strong? I've never seen this before, prior to the migration. I checked Cloudron logs, it seems related to cronjob? Possibly it may be a potential cause?

I did check:

root@vmi815992:~# systemctl status box
● box.service - Cloudron Admin
     Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-04-01 10:56:46 UTC; 23s ago
   Main PID: 1136454 (node)
      Tasks: 11 (limit: 36046)
     Memory: 65.4M (max: 400.0M)
     CGroup: /system.slice/box.service
             └─1136454 node /home/yellowtent/box/box.js

Apr 01 10:56:46 vmi815992.contaboserver.net systemd[1]: Started Cloudron Admin.
Apr 01 10:56:53 vmi815992.contaboserver.net sudo[1136528]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 01 10:56:53 vmi815992.contaboserver.net sudo[1136528]: pam_unix(sudo:session): session closed for user root
Apr 01 10:56:53 vmi815992.contaboserver.net sudo[1136534]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 01 10:56:53 vmi815992.contaboserver.net sudo[1136534]: pam_unix(sudo:session): session closed for user root
Apr 01 10:56:53 vmi815992.contaboserver.net sudo[1136554]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 01 10:56:54 vmi815992.contaboserver.net sudo[1136554]: pam_unix(sudo:session): session closed for user root

[nebulon]

@Aizat can you look into your browser inspector and see what errors are displayed in the console there? Also some errors should be shown then in the server logs at /home/yellowtent/platformdata/logs/box.log

[girish]

@Aizat I moved this to a new topic, it's not related to SFTP stuff (pretty sure).

[Aizat]

@nebulon Hi
ahh it's forked, and once I click it, now I can see your comment
OK, so it does not happen all the time, so I did not check the console just now.

As for

/home/yellowtent/platformdata/logs/box.log

2022-03-25T00:00:05.538Z box:janitor cleanupTmpVolume ["/5f3f0bd3-db59-4dbe-8b8e-b428235e01f7"]
2022-03-25T00:00:10.429Z box:janitor cleanupTmpVolume ["/9879d76b-d3a0-49e9-b490-6ca436b8d7ef"]
2022-03-25T00:00:11.749Z box:janitor cleanupTmpVolume ["/25dd60eb-1dd0-41ba-a7e4-2fef3c9dcd95"]
2022-03-25T00:00:12.341Z box:apphealthmonitor app health: 24 alive / 0 dead.
2022-03-25T00:00:12.845Z box:janitor cleanupTmpVolume ["/7bab6120-f042-469c-8067-3b5de04732fa"]
2022-03-25T00:00:13.533Z box:janitor cleanupTmpVolume ["/3d641394-9fd7-43d3-bd09-3055261af2bd"]
2022-03-25T00:00:14.336Z box:janitor cleanupTmpVolume ["/34d26d34-8a8a-49de-b75d-0fa632c35d6f"]
2022-03-25T00:00:15.114Z box:janitor cleanupTmpVolume ["/a947ca4c-d42e-4bcb-a29a-2ecdf8f3bff5"]
2022-03-25T00:00:16.161Z box:janitor cleanupTmpVolume ["/8bdbc85b-bfe4-45db-82d2-cf2cd6ed2029"]
2022-03-25T00:00:16.600Z box:janitor cleanupTmpVolume ["/e4cccd87-d627-4899-9fa4-b879fb03e83a"]
2022-03-25T00:00:17.429Z box:janitor cleanupTmpVolume ["/fecdff66-d1c4-406c-9263-9b67785b137b"]
2022-03-25T00:00:18.230Z box:janitor cleanupTmpVolume ["/ff8cea66-0fad-4680-bbec-73cf00b8b1f9"]
2022-03-25T00:00:19.079Z box:janitor cleanupTmpVolume ["/790aba40-7062-4c5d-94ca-2af284795ad3"]
2022-03-25T00:00:19.528Z box:janitor cleanupTmpVolume ["/redis-4d83133c-a6a1-44e0-a539-5c46ab304fa8"]
2022-03-25T00:00:20.105Z box:janitor cleanupTmpVolume ["/508068d0-06a7-487e-b2dc-4a9e46c72d8b"]
2022-03-25T00:00:20.855Z box:apphealthmonitor app health: 24 alive / 0 dead.
2022-03-25T00:00:21.294Z box:janitor cleanupTmpVolume ["/ca285d55-1eaa-4d84-865f-e7e3f457452c"]
2022-03-25T00:00:21.817Z box:janitor cleanupTmpVolume ["/a77f0f5b-b3f1-4901-9c8c-a398bd60dfbb"]
2022-03-25T00:00:22.494Z box:janitor cleanupTmpVolume ["/7a5c21a1-f034-44b0-92ec-fb86de145559"]
2022-03-25T00:00:23.108Z box:janitor cleanupTmpVolume ["/700a6dec-688c-463f-bacd-37b714af5517"]
2022-03-25T00:00:23.616Z box:janitor cleanupTmpVolume ["/5486cc4a-fa5f-4165-96a3-c5bfb0862bc5"]
2022-03-25T00:00:24.267Z box:janitor cleanupTmpVolume ["/439465c2-2614-4821-876c-93093503dd89"]
2022-03-25T00:00:25.125Z box:janitor cleanupTmpVolume ["/sftp"]
2022-03-25T00:00:26.050Z box:janitor cleanupTmpVolume ["/graphite"]
2022-03-25T00:00:26.803Z box:janitor cleanupTmpVolume ["/redis-8bdbc85b-bfe4-45db-82d2-cf2cd6ed2029"]
2022-03-25T00:00:27.370Z box:janitor cleanupTmpVolume ["/redis-7a5c21a1-f034-44b0-92ec-fb86de145559"]
2022-03-25T00:00:28.293Z box:janitor cleanupTmpVolume ["/redis-700a6dec-688c-463f-bacd-37b714af5517"]
2022-03-25T00:00:28.941Z box:janitor cleanupTmpVolume ["/redis-5f3f0bd3-db59-4dbe-8b8e-b428235e01f7"]
2022-03-25T00:00:29.750Z box:janitor cleanupTmpVolume ["/redis-34d26d34-8a8a-49de-b75d-0fa632c35d6f"]
2022-03-25T00:00:30.538Z box:janitor cleanupTmpVolume ["/mongodb"]
2022-03-25T00:00:31.535Z box:apphealthmonitor app health: 24 alive / 0 dead.
2022-03-25T00:00:31.835Z box:janitor cleanupTmpVolume ["/postgresql"]
2022-03-25T00:00:32.560Z box:janitor cleanupTmpVolume ["/mysql"]
2022-03-25T00:00:32.911Z box:janitor cleanupTmpVolume ["/turn"]
2022-03-25T00:00:33.161Z box:janitor cleanupTmpVolume ["/mail"]
2022-03-25T00:00:40.568Z box:apphealthmonitor app health: 24 alive / 0 dead.

[nebulon]

Since the errors shown in the notifications, it does not look like the general connectivity to your server is having an issue, but the server or some services on it fail to respond timely. This is not straightforwards to figure out, especially if this happens only sometimes.

One common issue is lagging disk I/O which can be measured with for example sudo hdparm -Tt /dev/<your main disk>

Also when this happens, most likely you should see some timeout errors in the box.log.

Further checkout if some daemons like the box service possibly crashing or getting restarted (sometimes due to memory shortage) Those cases should be visible in the dmesg or journalctl --system logs. Also here it is best looked at when it happens again, since otherwise they might be harder to find within the vast amount of other logs.

[Aizat]

@nebulon said in "Cloudron Error" in dashboard UI:

journalctl --system

So it happened again just now, I checked the console and immediately tried journalctl --system as you suggested and it showed:

Apr 05 11:29:14 vmi815992.contaboserver.net sshd[3559058]: Received disconnect from 43.154.101.95 port 43174:11: Bye Bye [preauth]
Apr 05 11:29:14 vmi815992.contaboserver.net sshd[3559058]: Disconnected from authenticating user root 43.154.101.95 port 43174 [preauth]
Apr 05 11:29:22 vmi815992.contaboserver.net kernel: Packet dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:74:83:ef:4e:ae:4b:08:00 SRC=161.97.128.11 DST=255.255.2>
Apr 05 11:29:24 vmi815992.contaboserver.net sshd[3559146]: Received disconnect from 43.154.101.95 port 43604:11: Bye Bye [preauth]
Apr 05 11:29:24 vmi815992.contaboserver.net sshd[3559146]: Disconnected from authenticating user root 43.154.101.95 port 43604 [preauth]
Apr 05 11:29:27 vmi815992.contaboserver.net sshd[3559201]: Received disconnect from 43.154.101.95 port 44036:11: Bye Bye [preauth]
Apr 05 11:29:27 vmi815992.contaboserver.net sshd[3559201]: Disconnected from authenticating user root 43.154.101.95 port 44036 [preauth]
Apr 05 11:29:36 vmi815992.contaboserver.net systemd[1]: box.service: Main process exited, code=exited, status=1/FAILURE
Apr 05 11:29:36 vmi815992.contaboserver.net systemd[1]: box.service: Failed with result 'exit-code'.
Apr 05 11:29:36 vmi815992.contaboserver.net systemd[1]: box.service: Scheduled restart job, restart counter is at 4446.
Apr 05 11:29:36 vmi815992.contaboserver.net systemd[1]: Stopped Cloudron Admin.

[girish]

@Aizat Might be better if you can write to us at support@cloudron.io , so we can see what's up. Looks like the box code is crashing on the server side.

[Aizat]

@girish OK!
Also saw a backup failed notification too for the first time ever, yesterday. I'm really not sure what's happening.
I've written to support and let's get on from there. Thanks