my.cloudron.tld is not accessible

JOduMonT

I'm able to ssh my server, and I rebooted it, after few months of non reboot

bitwarden, commento, kutt and others apps works
but since yesterday my nextcloud and now my.cloudron.tld don't work.

I have this error in /home/yellowtent/platformdata/logs/box.log

Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_BAD_FIELD_ERROR: Unknown column 'sourceJson' in 'field list'","nestedError":{"code":"ER_BAD_FIELD_ERROR","errno":1054,"sqlMessage":"Unknown column 'sourceJson' in 'field list'","sqlState":"42S22","index":0,"sql":"INSERT INTO eventlog (id, action, sourceJson, dataJson) VALUES ('679d72ab-7523-45a3-a4b5-47337869f8b7', 'cloudron.update.finish', '{\\\"username\\\":\\\"cron\\\",\\\"userId\\\":null,\\\"ip\\\":null}', '{\\\"errorMessage\\\":\\\"\\\",\\\"oldVersion\\\":\\\"7.0.4\\\",\\\"newVersion\\\":\\\"7.1.3\\\"}')"},"code":"ER_BAD_FIELD_ERROR","sqlMessage":"Unknown column 'sourceJson' in 'field list'"}
at main (/home/yellowtent/box/box.js:47:41)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

nebulon

@JOduMonT maybe some of the database migrations didn't run completely. Can you run /home/yellowtent/box/setup/start.sh manually via SSH?

JOduMonT

@nebulon said in my.cloudron.tld is not accessible:

@JOduMonT maybe some of the database migrations didn't run completely. Can you run /home/yellowtent/box/setup/start.sh manually via SSH?

i ran /home/yellowtent/box/setup/start.sh as root several times

root@cloudron:~# /home/yellowtent/box/setup/start.sh
2022-03-13T02:35:43 ==> start: Cloudron Start
media:x:500:
2022-03-13T02:35:43 ==> start: Configuring docker
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable apparmor
2022-03-13T02:35:44 ==> start: Ensuring directories
2022-03-13T02:35:44 ==> start: Configuring journald
2022-03-13T02:35:45 ==> start: Setting up unbound
2022-03-13T02:35:45 ==> start: Adding systemd services
Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable cron
2022-03-13T02:35:52 ==> start: Configuring sudoers
2022-03-13T02:35:52 ==> start: Configuring collectd
A dependency job for collectd.service failed. See 'journalctl -xe' for details.

the first time journalctl -xe

2022-03-13T02:34:43 ==> start: Configuring journald
Mar 13 02:34:56 cloudron ananicy[959925]: ioclass: node[1403969/1403979] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: sched: node[1403969/1403979] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: Cgroup: node[1403979] added to cpu80
Mar 13 02:34:56 cloudron ananicy[959925]: renice: node[1403969/1403980] -> 19
Mar 13 02:34:56 cloudron ananicy[959925]: ioclass: node[1403969/1403980] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: sched: node[1403969/1403980] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: renice: node[1403969/1403981] -> 19
Mar 13 02:34:56 cloudron ananicy[959925]: ioclass: node[1403969/1403981] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: sched: node[1403969/1403981] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: Cgroup: node[1403981] added to cpu80
Mar 13 02:34:56 cloudron ananicy[959925]: renice: node[1403969/1404009] -> 19
Mar 13 02:34:56 cloudron ananicy[959925]: sched: node[1403969/1404009] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: renice: node[1404202/1404202] -> 19
Mar 13 02:34:56 cloudron ananicy[959925]: ioclass: node[1404202/1404202] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: sched: node[1404202/1404202] -> idle
Mar 13 02:34:56 cloudron ananicy[959925]: Cgroup: node[1404202] added to cpu80
Mar 13 02:34:56 cloudron ananicy[959925]: renice: node[1404202/1404203] -> 19

the second time

media:x:500:
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit box.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Mar 13 02:36:27 cloudron sudo[1408003]: pam_unix(sudo:session): session closed for user root
Mar 13 02:36:27 cloudron systemd[1]: box.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit box.service has entered the 'failed' state with result 'exit-code'.
Mar 13 02:36:28 cloudron systemd[1]: box.service: Scheduled restart job, restart counter is at 67146.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit box.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Mar 13 02:36:28 cloudron systemd[1]: Stopped Cloudron Admin.
-- Subject: A stop job for unit box.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit box.service has finished.
--
-- The job identifier is 5038917 and the job result is done.
Mar 13 02:36:28 cloudron systemd[1]: Started Cloudron Admin.
-- Subject: A start job for unit box.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit box.service has finished successfully.
--
-- The job identifier is 5038917.

systemctl status collectd

● collectd.service - Statistics collection and monitoring daemon
Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:collectd(1)
man:collectd.conf(5)
https://collectd.org

Mar 13 02:34:50 cloudron systemd[1]: Dependency failed for Statistics collection and monitoring daemon.
Mar 13 02:34:50 cloudron systemd[1]: collectd.service: Job collectd.service/start failed with result 'dependency'.
Mar 13 02:35:52 cloudron systemd[1]: Dependency failed for Statistics collection and monitoring daemon.
Mar 13 02:35:52 cloudron systemd[1]: collectd.service: Job collectd.service/start failed with result 'dependency'.
Mar 13 02:38:48 cloudron systemd[1]: Dependency failed for Statistics collection and monitoring daemon.
Mar 13 02:38:48 cloudron systemd[1]: collectd.service: Job collectd.service/start failed with result 'dependency'.

no my drive is not full

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        75G   27G   45G  38% /

joesitsolutions

Hi, sorry to add onto an existing ticket.

I'm also experiencing the same/similar error.

2022-03-13T03:42:22.037Z box:settings initCache: pre-load settings
2022-03-13T03:42:22.070Z box:tasks stopAllTasks: stopping all tasks
2022-03-13T03:42:22.070Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_BAD_FIELD_ERROR: Unknown column 'sourceJson' in 'field list'","nestedError":{"code":"ER_BAD_FIELD_ERROR","errno":1054,"sqlMessage":"Unknown column 'sourceJson' in 'field list'","sqlState":"42S22","index":0,"sql":"INSERT INTO eventlog (id, action, sourceJson, dataJson) VALUES ('7ce937d3-c09e-4b93-9399-1f36d8e3b92b', 'cloudron.update.finish', '{\\\"username\\\":\\\"cron\\\",\\\"userId\\\":null,\\\"ip\\\":null}', '{\\\"errorMessage\\\":\\\"\\\",\\\"oldVersion\\\":\\\"7.0.4\\\",\\\"newVersion\\\":\\\"7.1.3\\\"}')"},"code":"ER_BAD_FIELD_ERROR","sqlMessage":"Unknown column 'sourceJson' in 'field list'"}
    at main (/home/yellowtent/box/box.js:47:41)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-03-13T03:42:23.188Z box:server ==========================================
2022-03-13T03:42:23.189Z box:server            Cloudron 7.1.3
2022-03-13T03:42:23.190Z box:server ==========================================
2022-03-13T03:42:23.290Z box:settings initCache: pre-load settings
2022-03-13T03:42:23.322Z box:tasks stopAllTasks: stopping all tasks
2022-03-13T03:42:23.323Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_BAD_FIELD_ERROR: Unknown column 'sourceJson' in 'field list'","nestedError":{"code":"ER_BAD_FIELD_ERROR","errno":1054,"sqlMessage":"Unknown column 'sourceJson' in 'field list'","sqlState":"42S22","index":0,"sql":"INSERT INTO eventlog (id, action, sourceJson, dataJson) VALUES ('9ca2798c-b843-4562-abe1-54a6db9fa69f', 'cloudron.update.finish', '{\\\"username\\\":\\\"cron\\\",\\\"userId\\\":null,\\\"ip\\\":null}', '{\\\"errorMessage\\\":\\\"\\\",\\\"oldVersion\\\":\\\"7.0.4\\\",\\\"newVersion\\\":\\\"7.1.3\\\"}')"},"code":"ER_BAD_FIELD_ERROR","sqlMessage":"Unknown column 'sourceJson' in 'field list'"}
    at main (/home/yellowtent/box/box.js:47:41)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

This was after a recent Ubuntu security update and reboot. There is plenty of storage on the system.

root@cloudron:~# systemctl status collectd
● collectd.service - Statistics collection and monitoring daemon
     Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:collectd(1)
             man:collectd.conf(5)
             https://collectd.org

Mar 13 03:40:23 cloudron systemd[1]: Dependency failed for Statistics collection and monitoring daemon.
Mar 13 03:40:23 cloudron systemd[1]: collectd.service: Job collectd.service/start failed with result 'dependency'.
Mar 13 03:44:21 cloudron systemd[1]: Dependency failed for Statistics collection and monitoring daemon.
Mar 13 03:44:21 cloudron systemd[1]: collectd.service: Job collectd.service/start failed with result 'dependency'.
Mar 13 03:45:23 cloudron systemd[1]: Dependency failed for Statistics collection and monitoring daemon.
Mar 13 03:45:23 cloudron systemd[1]: collectd.service: Job collectd.service/start failed with result 'dependency'.
Mar 13 03:45:40 cloudron systemd[1]: Dependency failed for Statistics collection and monitoring daemon.
Mar 13 03:45:40 cloudron systemd[1]: collectd.service: Job collectd.service/start failed with result 'dependency'.

I have done /home/yellowtent/box/setup/start.sh multiple times with no luck and have rebooted a few times with no luck.
I was attempting to setup NFS as a backup and was deploying this on a separate server and left the backup page open over night.
Unsure what's caused this but I'm getting the exact same error to @JOduMonT at almost the exact same time it seems.
Feel free to correct me if I'm wrong.

Dumping more logs and info.

-- The unit box.service has entered the 'failed' state with result 'exit-code'.
Mar 13 03:59:20 cloudron unbound[5447]: [5447:0] info: generate keytag query _ta-4f66. NULL IN
Mar 13 03:59:20 cloudron multipathd[691]: sda: add missing path
Mar 13 03:59:20 cloudron multipathd[691]: sda: failed to get udev uid: Invalid argument
Mar 13 03:59:20 cloudron multipathd[691]: sda: failed to get sysfs uid: Invalid argument
Mar 13 03:59:20 cloudron multipathd[691]: sda: failed to get sgio uid: No such file or directory
Mar 13 03:59:20 cloudron systemd[1]: box.service: Scheduled restart job, restart counter is at 50.
-- Subject: Automatic restarting of a unit has been scheduled

Attempted to restart box

root@cloudron:~# systemctl status box
● box.service - Cloudron Admin
     Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Sun 2022-03-13 04:01:24 UTC; 94ms ago
    Process: 7362 ExecStart=/home/yellowtent/box/box.js (code=exited, status=1/FAILURE)
   Main PID: 7362 (code=exited, status=1/FAILURE)

It did come up
But my.WEBSITENAME.com.au is still offine/reconnecting
My apps are running fine and as expected
Going through tshoot steps off cloudron page

root@cloudron:~# systemctl status unbound
● unbound.service - Unbound DNS Resolver
     Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-03-13 03:59:03 UTC; 4min 16s ago
   Main PID: 5447 (unbound)
      Tasks: 1 (limit: 19104)
     Memory: 6.3M
     CGroup: /system.slice/unbound.service
             └─5447 /usr/sbin/unbound -d

Mar 13 03:59:02 cloudron systemd[1]: Starting Unbound DNS Resolver...
Mar 13 03:59:02 cloudron unbound[5447]: [5447:0] notice: init module 0: subnet
Mar 13 03:59:02 cloudron unbound[5447]: [5447:0] notice: init module 1: validator
Mar 13 03:59:02 cloudron unbound[5447]: [5447:0] notice: init module 2: iterator
Mar 13 03:59:03 cloudron unbound[5447]: [5447:0] info: start of service (unbound 1.9.4).
Mar 13 03:59:03 cloudron systemd[1]: Started Unbound DNS Resolver.
Mar 13 03:59:20 cloudron unbound[5447]: [5447:0] info: generate keytag query _ta-4f66. NULL IN
root@cloudron:~# systemctl status nginx
● nginx.service - nginx - high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─cloudron.conf
     Active: active (running) since Sun 2022-03-13 03:58:10 UTC; 5min ago
       Docs: http://nginx.org/en/docs/
   Main PID: 1342 (nginx)
      Tasks: 9 (limit: 19104)
     Memory: 23.2M
     CGroup: /system.slice/nginx.service
             ├─1342 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
             ├─1343 nginx: worker process
             ├─1344 nginx: worker process
             ├─1345 nginx: worker process
             ├─1346 nginx: worker process
             ├─1347 nginx: worker process
             ├─1348 nginx: worker process
             ├─1349 nginx: worker process
             └─1350 nginx: worker process

Mar 13 03:58:08 cloudron systemd[1]: Starting nginx - high performance web server...
Mar 13 03:58:10 cloudron systemd[1]: Started nginx - high performance web server.
root@cloudron:~# systemctl status mysql
● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-03-13 03:58:09 UTC; 5min ago
   Main PID: 992 (mysqld)
     Status: "Server is operational"
      Tasks: 38 (limit: 19104)
     Memory: 206.6M
     CGroup: /system.slice/mysql.service
             └─992 /usr/sbin/mysqld

Mar 13 03:58:08 cloudron systemd[1]: Starting MySQL Community Server...
Mar 13 03:58:09 cloudron systemd[1]: Started MySQL Community Server.

May show updates?? I'm not 100% sure

root@cloudron:~# root@cloudron:~# ls /var/log/apt/history.log
/var/log/apt/history.log
root@cloudron:~# cat /var/log/apt/history.log

Start-Date: 2022-03-02  06:08:51
Commandline: /usr/bin/unattended-upgrade
Upgrade: libpolkit-gobject-1-0:amd64 (0.105-26ubuntu1.2, 0.105-26ubuntu1.3), libpolkit-agent-1-0:amd64 (0.105-26ubuntu1.2, 0.105-26ubuntu1.3), policykit-1:amd64 (0.105-26ubuntu1.2, 0.105-26ubuntu1.3)
End-Date: 2022-03-02  06:08:53

Start-Date: 2022-03-03  06:08:00
Commandline: /usr/bin/unattended-upgrade
Upgrade: libc6-dev:amd64 (2.31-0ubuntu9.2, 2.31-0ubuntu9.7), libc6:amd64 (2.31-0ubuntu9.2, 2.31-0ubuntu9.7), libc-dev-bin:amd64 (2.31-0ubuntu9.2, 2.31-0ubuntu9.7)
End-Date: 2022-03-03  06:08:02

Start-Date: 2022-03-03  06:08:08
Commandline: /usr/bin/unattended-upgrade
Upgrade: libc-bin:amd64 (2.31-0ubuntu9.2, 2.31-0ubuntu9.7)
End-Date: 2022-03-03  06:08:09

Start-Date: 2022-03-03  06:08:15
Commandline: /usr/bin/unattended-upgrade
Upgrade: locales:amd64 (2.31-0ubuntu9.2, 2.31-0ubuntu9.7)
End-Date: 2022-03-03  06:08:20

Start-Date: 2022-03-09  06:40:47
Commandline: /usr/bin/unattended-upgrade
Upgrade: gcc-9-base:amd64 (9.3.0-17ubuntu1~20.04, 9.4.0-1ubuntu1~20.04), cpp-9:amd64 (9.3.0-17ubuntu1~20.04, 9.4.0-1ubuntu1~20.04), libasan5:amd64 (9.3.0-17ubuntu1~20.04, 9.4.0-1ubuntu1~20.04), libstdc++-9-dev:amd64 (9.3.0-17ubuntu1~20.04, 9.4.0-1ubuntu1~20.04), g++-9:amd64 (9.3.0-17ubuntu1~20.04, 9.4.0-1ubuntu1~20.04), gcc-9:amd64 (9.3.0-17ubuntu1~20.04, 9.4.0-1ubuntu1~20.04), libgcc-9-dev:amd64 (9.3.0-17ubuntu1~20.04, 9.4.0-1ubuntu1~20.04)
End-Date: 2022-03-09  06:40:53

Start-Date: 2022-03-10  06:22:00
Commandline: /usr/bin/unattended-upgrade
Install: linux-headers-5.4.0-104-generic:amd64 (5.4.0-104.118, automatic), linux-modules-extra-5.4.0-104-generic:amd64 (5.4.0-104.118, automatic), linux-headers-5.4.0-104:amd64 (5.4.0-104.118, automatic), linux-modules-5.4.0-104-generic:amd64 (5.4.0-104.118, automatic), linux-image-5.4.0-104-generic:amd64 (5.4.0-104.118, automatic)
Upgrade: linux-headers-generic:amd64 (5.4.0.100.104, 5.4.0.104.108), linux-image-generic:amd64 (5.4.0.100.104, 5.4.0.104.108), linux-generic:amd64 (5.4.0.100.104, 5.4.0.104.108)
End-Date: 2022-03-10  06:22:57

Start-Date: 2022-03-10  06:23:03
Commandline: /usr/bin/unattended-upgrade
Upgrade: linux-libc-dev:amd64 (5.4.0-100.113, 5.4.0-104.118)
End-Date: 2022-03-10  06:23:03

Start-Date: 2022-03-11  06:04:58
Commandline: /usr/bin/unattended-upgrade
Remove: linux-headers-5.4.0-99:amd64 (5.4.0-99.112), linux-headers-5.4.0-99-generic:amd64 (5.4.0-99.112)
End-Date: 2022-03-11  06:05:00

Start-Date: 2022-03-11  06:05:07
Commandline: /usr/bin/unattended-upgrade
Remove: linux-image-5.4.0-99-generic:amd64 (5.4.0-99.112), linux-modules-extra-5.4.0-99-generic:amd64 (5.4.0-99.112)
End-Date: 2022-03-11  06:05:13

Start-Date: 2022-03-11  06:05:19
Commandline: /usr/bin/unattended-upgrade
Remove: linux-modules-5.4.0-99-generic:amd64 (5.4.0-99.112)
End-Date: 2022-03-11  06:05:20

Start-Date: 2022-03-11  06:05:27
Commandline: /usr/bin/unattended-upgrade
Upgrade: libdrm2:amd64 (2.4.105-3~20.04.2, 2.4.107-8ubuntu1~20.04.2), libdrm-common:amd64 (2.4.105-3~20.04.2, 2.4.107-8ubuntu1~20.04.2)
End-Date: 2022-03-11  06:05:28

Start-Date: 2022-03-11  06:05:35
Commandline: /usr/bin/unattended-upgrade
Upgrade: libexpat1:amd64 (2.2.9-1ubuntu0.2, 2.2.9-1ubuntu0.4)
End-Date: 2022-03-11  06:05:35

Start-Date: 2022-03-12  19:01:24
Commandline: apt install -y /tmp/containerd.deb /tmp/docker-ce-cli.deb /tmp/docker.deb
Upgrade: containerd.io:amd64 (1.4.6-1, 1.4.9-1), docker-ce:amd64 (5:20.10.7~3-0~ubuntu-focal, 5:20.10.12~3-0~ubuntu-focal), docker-ce-cli:amd64 (5:20.10.7~3-0~ubuntu-focal, 5:20.10.12~3-0~ubuntu-focal)
End-Date: 2022-03-12  19:02:11

Pulled the log and had a read.
This looks to be when the error started for me.

2022-03-12T18:59:50.058Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:00:00.006Z box:cron Starting box autoupdate to {"version":"7.1.3","changelog":["Add mail manager role","mailbox: app can be set as owner when recvmail addon enabled","domains: add well known config UI (for jitsi configuration)","Prefix email addon variables with CLOUDRON_EMAIL instead of CLOUDRON_MAIL","remove support for manifest version 1","Add option to enable/disable mailbox sharing","base image 3.2.0","Update node to 16.13.1","mongodb: update to 4.4","Add `upstreamVersion` to manifest","Add `logPaths` to manifest","Add cifs seal support for backup and volume mounts","add a way for admins to set username when profiles are locked","Add support for secondary domains","postgresql: enable postgis","remove nginx config of stopped apps","mail: use port25check.cloudron.io to check outbound port 25 connectivity","Add import/export of mailboxes and users","LDAP server can now be exposed","Update monaco-editor to 0.32.1","Update xterm.js to 4.17.0","Update docker to 20.10.12","IPv6 support"],"sourceTarballUrl":"https://prod-cloudron-releases.s3.amazonaws.com/box-193dff8c30-be40cea600-7.1.3.tar.gz","sourceTarballSigUrl":"https://prod-cloudron-releases.s3.amazonaws.com/box-193dff8c30-be40cea600-7.1.3.tar.gz.sig","boxVersionsUrl":"https://s3.amazonaws.com/prod-cloudron-releases/versions.json","boxVersionsSigUrl":"https://s3.amazonaws.com/prod-cloudron-releases/versions.json.sig","unstable":false}
2022-03-12T19:00:00.011Z box:janitor Cleaning up expired tokens
2022-03-12T19:00:00.015Z box:janitor Cleaned up 0 expired tokens
2022-03-12T19:00:00.018Z box:locker Acquired : box_update
2022-03-12T19:00:00.038Z box:tasks startTask - starting task 664 with options {"timeout":72000000,"nice":15,"memoryLimit":400}. logs at /home/yellowtent/platformdata/logs/tasks/664.log
2022-03-12T19:00:00.038Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 664 /home/yellowtent/platformdata/logs/tasks/664.log 15 400
2022-03-12T19:00:00.074Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:00:00.127Z box:shell startTask (stdout): Running as unit: box-task-664.service

2022-03-12T19:00:10.058Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:00:20.076Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:00:30.056Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:00:40.063Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:00:50.065Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:01:00.059Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:01:10.057Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:01:20.061Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:01:30.056Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:02:06.915Z box:apphealthmonitor setHealth: 119a7c1a-f70a-4933-bd82-4fec92fe981a (support.joesitsolutions.com.au) waiting for 1163.085 to update health
2022-03-12T19:02:06.925Z box:apphealthmonitor setHealth: 119a7c1a-f70a-4933-bd82-4fec92fe981a (support.joesitsolutions.com.au) waiting for 1163.075 to update health
2022-03-12T19:02:06.926Z box:apphealthmonitor setHealth: 119a7c1a-f70a-4933-bd82-4fec92fe981a (support.joesitsolutions.com.au) waiting for 1163.074 to update health
2022-03-12T19:02:06.991Z box:apphealthmonitor setHealth: 0708633b-f7ec-4a01-9d38-20690f924ade (grafana.joesitsolutions.com.au) waiting for 1163.009 to update health
2022-03-12T19:02:06.992Z box:apphealthmonitor setHealth: 0708633b-f7ec-4a01-9d38-20690f924ade (grafana.joesitsolutions.com.au) waiting for 1163.008 to update health
2022-03-12T19:02:06.992Z box:apphealthmonitor setHealth: 0708633b-f7ec-4a01-9d38-20690f924ade (grafana.joesitsolutions.com.au) waiting for 1163.008 to update health
2022-03-12T19:02:09.085Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:02:09.086Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:02:09.086Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:02:10.018Z box:apphealthmonitor setHealth: 119a7c1a-f70a-4933-bd82-4fec92fe981a (support.joesitsolutions.com.au) waiting for 1159.982 to update health
2022-03-12T19:02:10.161Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:02:20.270Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:02:30.065Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:02:40.062Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:02:50.100Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:03:00.059Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:03:10.077Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:03:20.241Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:03:30.057Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:03:40.070Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:03:50.069Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:04:00.164Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:04:10.055Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:04:20.182Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:04:30.055Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:04:40.062Z box:apphealthmonitor app health: 4 alive / 0 dead.
2022-03-12T19:04:41.098Z box:box Received SIGTERM. Shutting down.
2022-03-12T19:04:41.100Z box:tasks stopAllTasks: stopping all tasks
2022-03-12T19:04:41.100Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2022-03-12T19:04:41.106Z box:shell startTask code: null, signal: SIGTERM
2022-03-13T03:36:48.328Z box:server ==========================================
2022-03-13T03:36:48.329Z box:server            Cloudron 7.1.3  
2022-03-13T03:36:48.329Z box:server ==========================================
2022-03-13T03:36:48.447Z box:settings initCache: pre-load settings
2022-03-13T03:36:48.479Z box:tasks stopAllTasks: stopping all tasks
2022-03-13T03:36:48.480Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_BAD_FIELD_ERROR: Unknown column 'sourceJson' in 'field list'","nestedError":{"code":"ER_BAD_FIELD_ERROR","errno":1054,"sqlMessage":"Unknown column 'sourceJson' in 'field list'","sqlState":"42S22","index":0,"sql":"INSERT INTO eventlog (id, action, sourceJson, dataJson) VALUES ('9344e8c2-c479-42c9-ab49-de7dc4989423', 'cloudron.update.finish', '{\\\"username\\\":\\\"cron\\\",\\\"userId\\\":null,\\\"ip\\\":null}', '{\\\"errorMessage\\\":\\\"\\\",\\\"oldVersion\\\":\\\"7.0.4\\\",\\\"newVersion\\\":\\\"7.1.3\\\"}')"},"code":"ER_BAD_FIELD_ERROR","sqlMessage":"Unknown column 'sourceJson' in 'field list'"}
    at main (/home/yellowtent/box/box.js:47:41)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-03-13T03:36:49.583Z box:server ==========================================
2022-03-13T03:36:49.584Z box:server            Cloudron 7.1.3  
2022-03-13T03:36:49.584Z box:server ==========================================
2022-03-13T03:36:49.683Z box:settings initCache: pre-load settings
2022-03-13T03:36:49.717Z box:tasks stopAllTasks: stopping all tasks
2022-03-13T03:36:49.717Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_BAD_FIELD_ERROR: Unknown column 'sourceJson' in 'field list'","nestedError":{"code":"ER_BAD_FIELD_ERROR","errno":1054,"sqlMessage":"Unknown column 'sourceJson' in 'field list'","sqlState":"42S22","index":0,"sql":"INSERT INTO eventlog (id, action, sourceJson, dataJson) VALUES ('e3aad5b3-eb7d-4d13-95d8-c4b9c3aa1216', 'cloudron.update.finish', '{\\\"username\\\":\\\"cron\\\",\\\"userId\\\":null,\\\"ip\\\":null}', '{\\\"errorMessage\\\":\\\"\\\",\\\"oldVersion\\\":\\\"7.0.4\\\",\\\"newVersion\\\":\\\"7.1.3\\\"}')"},"code":"ER_BAD_FIELD_ERROR","sqlMessage":"Unknown column 'sourceJson' in 'field list'"}

Attempted to do this

root@cloudron:~# /home/yellowtent/box/src/scripts/restartservice.sh all
/home/yellowtent/box/src/scripts/restartservice.sh: line 20: BOX_ENV: unbound variable
root@cloudron:~#

nebulon

First regarding

/home/yellowtent/box/src/scripts/restartservice.sh: line 20: BOX_ENV: unbound variable`

please to not run shell scripts like that directly, the start.sh is a bit special but otherwise it may be risky to run them directly as mostly they are only tested to be run from other code or scripts. Maybe we can add safeguard checks early on to avoid this.

Regarding the main issue, looks like fro both cases collectd does not start, this may be caused by outdated or broken config files.

For a start, can you check journalctl -u collectd and the main log file at /home/yellowtent/platformdata/logs/collectd/collectd.log?

joesitsolutions

First regarding
/home/yellowtent/box/src/scripts/restartservice.sh: line 20: BOX_ENV: unbound variable`
please to not run shell scripts like that directly, the start.sh is a bit special but otherwise it may be risky to run them directly as mostly they are only tested to be run from other code or scripts. Maybe we can add safeguard checks early on to avoid this.

Regarding the main issue, looks like fro both cases collectd does not start, this may be caused by outdated or broken config files.

For a start, can you check journalctl -u collectd and the main log file at /home/yellowtent/platformdata/logs/collectd/collectd.log?

Sorry, wasn't aware of not running those shell scripts but definitely something to take a note of.
journalctl -u collectd
Output:

root@cloudron:~# journalctl -u collectd
-- Logs begin at Sun 2022-03-13 06:27:27 UTC, end at Sun 2022-03-13 09:27:33 UTC. --
-- No entries --
root@cloudron:~# systemctl status collectd
● collectd.service - Statistics collection and monitoring daemon
     Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:collectd(1)
             man:collectd.conf(5)
             https://collectd.org
root@cloudron:~#

Contents of "/home/yellowtent/platformdata/logs/collectd/collectd.log"
(might be very long)

[done in pastebin due to size]
Pastebin link

Also I attempted to start it manually as a test.

root@cloudron:~# journalctl -u collectd
-- Logs begin at Sun 2022-03-13 06:27:27 UTC, end at Sun 2022-03-13 09:30:47 UTC. --
Mar 13 09:30:42 cloudron systemd[1]: Dependency failed for Statistics collection and monitoring daemon.
Mar 13 09:30:42 cloudron systemd[1]: collectd.service: Job collectd.service/start failed with result 'dependency'.
root@cloudron:~#

Please let me know if you'd like the information in a different format or in a file form. Thanks for the assistance.

nebulon

@joesitsolutions do you happen to have stopped apps? If so and if possible, can you start them and run the start.sh script again then? I suspect there is a bug with stopped apps during collectd startup.

joesitsolutions

@nebulon uncertain on how to stop / start apps as I cant access the my.cloudron.com.au side of things.

Docker shows this:

root@cloudron:~# docker ps -a
CONTAINER ID   IMAGE                                                          COMMAND                  CREATED        STATUS                    PORTS                                                          NAMES
2b22cebcd8dd   cloudron/org.getgrav.cloudronapp:20220208-133214-844e92676     "/bin/sh -c 'echo '=…"   29 hours ago   Exited (0) 15 hours ago                                                                  f17bb001-deeb-4120-ba11-55ef90f0e3b4-grav_scheduler
443c1145e43f   cloudron/net.freescout.cloudronapp:20220223-121510-86737ae89   "/bin/sh -c /app/pkg…"   29 hours ago   Exited (0) 15 hours ago                                                                  119a7c1a-f70a-4933-bd82-4fec92fe981a-housekeeping
b811af4c3a3e   cloudron/com.grafana.cloudronapp:20220224-180629-5687a838f     "/app/pkg/start.sh"      2 days ago     Up 6 hours                                                                               0708633b-f7ec-4a01-9d38-20690f924ade
85baeb1e8340   cloudron/redis:3.0.4                                           "/app/code/start.sh"     2 days ago     Up 6 hours                3000/tcp, 6379/tcp                                             redis-0708633b-f7ec-4a01-9d38-20690f924ade
75cadb64d6b8   cloudron/louislam.uptimekuma.app:20220209-152854-0147f4790     "/app/pkg/start.sh"      2 days ago     Up 6 hours                                                                               69dfed39-c1a3-441c-90b7-8b8eb3ccad8b
a75d5ca8221c   cloudron/net.freescout.cloudronapp:20220223-121510-86737ae89   "/app/pkg/start.sh"      2 weeks ago    Up 6 hours                                                                               119a7c1a-f70a-4933-bd82-4fec92fe981a
6620dc0b8126   cloudron/mail:3.5.0                                            "/app/code/start.sh"     3 weeks ago    Up 6 hours                                                                               mail
e596de9e8743   cloudron/org.getgrav.cloudronapp:20220208-133214-844e92676     "/app/pkg/start.sh"      4 weeks ago    Up 6 hours                                                                               f17bb001-deeb-4120-ba11-55ef90f0e3b4
39f11158be76   cloudron/sftp:3.4.2                                            "/app/code/start.sh"     2 months ago   Up 6 hours                0.0.0.0:222->22/tcp, :::222->22/tcp                            sftp
3b1a13866b42   cloudron/graphite:3.0.1                                        "/app/code/start.sh"     2 months ago   Up 6 hours                127.0.0.1:2003-2004->2003-2004/tcp, 127.0.0.1:8417->8000/tcp   graphite
f75c940bc6be   cloudron/mongodb:4.0.2                                         "/app/code/start.sh"     2 months ago   Up 6 hours                                                                               mongodb
55cda1c3bf9f   cloudron/postgresql:4.1.1                                      "/app/code/start.sh"     2 months ago   Up 6 hours                3000/tcp, 5432/tcp                                             postgresql
8a5049411810   cloudron/mysql:3.0.7                                           "/app/code/start.sh"     2 months ago   Up 6 hours                                                                               mysql
d0dac59c07e4   cloudron/turn:1.3.1                                            "/app/code/start.sh"     2 months ago   Up 6 hours                                                                               turn
root@cloudron:~#

These appear to all be running. Or at least appear to have all my apps running as expected. I can go to all of my apps/sites aside from my.cloudron.com.au side of things.

I do see two stopped containers at the top. Is it possible that these are stuck?

nebulon

@joesitsolutions ah of course, I forgot about the dashboard not being accessible. However those two containers are cron job container, so this is fine.

To take one example of the pasted log file table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/f1fff4243dd29c177ec2167b33b1f25c9e98158f6a84aa7706daf4bb89df188d/cpuacct.stat": No such file or directory.

This may indicate that there are collectd configs for some apps referencing container, which do not exist anymore. Can you run the following in /home/yellowtent/platformdata/collectd/collectd.conf.d/:

grep f1fff4243dd29c177ec2167b33b1f25c9e98158f6a84aa7706daf4bb89df188d *

This hopefully reveals which collectd config is referencing this and then you can delete this file and restart collectd via systemctl restart collectd If it still does not start, we have to do the same thing as above for the next errored file.

You can also enable remote ssh for us and send a mail to support@cloudron.io with your dashboard domain.

joesitsolutions

@nebulon doing as requested

root@cloudron:~# cd /home/yellowtent/platformdata/collectd/collectd.conf.d/
root@cloudron:/home/yellowtent/platformdata/collectd/collectd.conf.d# grep f1fff4243dd29c177ec2167b33b1f25c9e98158f6a84aa7706daf4bb89df188d *
root@cloudron:/home/yellowtent/platformdata/collectd/collectd.conf.d# ls -la
total 28
drwxr-xr-x 2 yellowtent yellowtent 4096 Mar 10 14:38 .
drwxr-xr-x 3 yellowtent yellowtent 4096 Dec 18 01:10 ..
-rw-r--r-- 1 yellowtent yellowtent 1175 Mar 10 14:38 0708633b-f7ec-4a01-9d38-20690f924ade.conf
-rw-r--r-- 1 yellowtent yellowtent 1175 Feb 23 15:05 119a7c1a-f70a-4933-bd82-4fec92fe981a.conf
-rw-r--r-- 1 yellowtent yellowtent 1175 Mar 10 13:50 69dfed39-c1a3-441c-90b7-8b8eb3ccad8b.conf
-rw-r--r-- 1 yellowtent yellowtent  138 Dec 18 01:11 cloudron-backup.conf
-rw-r--r-- 1 yellowtent yellowtent 1175 Feb  8 15:07 f17bb001-deeb-4120-ba11-55ef90f0e3b4.conf
root@cloudron:/home/yellowtent/platformdata/collectd/collectd.conf.d#

That file appears to not exist. I'll enable ssh and send through an email now.
Feel free to message or reach out if you need more input from me.

nebulon

Just to update this thread. @joesitsolutions cloudron had a broken line in /etc/fstab which made further local-fs.target dependencies fail, one of them is collectd. Disabling/removing that line following systemctl daemon-reload and running start.sh fixes the issue.

Not sure if this is now related to @JOduMonT 's issue though.

joesitsolutions

@nebulon huge thanks for the assistance!
My poor attempt at adding a cloud backup caused the error as stated. ty for the prompt and friendly assistance

JOduMonT

@nebulon said in my.cloudron.tld is not accessible:

Just to update this thread. @joesitsolutions cloudron had a broken line in /etc/fstab which made further local-fs.target dependencies fail, one of them is collectd. Disabling/removing that line following systemctl daemon-reload and running start.sh fixes the issue.

Not sure if this is now related to @JOduMonT 's issue though.

I had 2 drives, now only one
the drive I removed was attached to moodle, which I removed before removing the drive which I remove before rebooting.

Yes for a long time, statistic (probably populated by collected) was not working in the dashboard

[2021-11-26 02:28:04] Initialization complete, entering read-loop.
d to connect to remote host: Connection refused
d to connect to remote host: Connection refused
[2021-11-26 07:25:15] du plugin: monitoring /home/yellowtent/platformdata
[2021-11-27 02:49:25] du plugin: monitoring /home/yellowtent/appsdata/b72cc559-faeb-4d2f-bf51-2a2da800c5d4/data
feb-bb90-e4be79c69fcb/data"
[2021-11-27 02:49:30] du plugin: monitoring /home/yellowtent/appsdata/6cef9864-730b-4d01-be6c-327e08c4f5d8/data
[2021-11-27 02:49:30] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/aa40b16d-62fd-4[2021-11-27 02:49:47] du plugin: monitoring /home/yellowtent/appsdata/bb96e514-a988-4f90-8d4e-e58726a601c7/data
[2021-11-27 02:49:47] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/bb96e514-a988-4f51-2a2da8
e: 1637981409)
[2021-11-27 02:50:18]   size of /home/yellowtent/appsdata/6deb1be7-d241-4d47-afb9-e63f4a2c00cf/data is 8192 (time: [2021-11-27 02:50:35] du plugin: monitoring /home/yellowtent/platformdata
1637981435)
[2021-11-27 02:50:45] du plugin: monitoring /home/yellowtent/appsdata/bb96e514-a988-4f90-8d4e-e58726a601c7/data
[2021-11-27 02:50:45]   size of /home/yellowtent/appsdata/bb96e514-a988-4f90-8d4e-e58726a601c7/data is 45056 (time:{'name': '6cef9864-730b-4d01-be6c-327e08c4f5d8', 'dir': '/home/yellowtent/appsdata/6cef9864-730b-4d01-be6c-327e08c[2021-11-27 02:50:55]   size of /home/yellowtent/appsdata/bb96e514-a988-4f90-8d4e-e58726a601c7/data is 45056 (time:{'name': '6cef9864-730b-4d01-be6c-327e08c4f5d8', 'dir': '/home/yellowtent/appsdata/6cef9864-730b-4d01-be6c-327e08c1637981480)
a', 'dir': '/home/yellowtent/platformdata', 'exclude': ''}, {'name': '0d4c4ffa-e715-465f-9d8f-2d911df4f0ed', 'dir':[2021-11-27 02:51:37]   size of /var/backups is 1941504 (time: 1637981496)
{'name': '6cef9864-730b-4d01-be6c-327e08c4f5d8', 'dir': '/home/yellowtent/appsdata/6cef9864-730b-4d01-be6c-327e08cb8f-9b4d-e9d09f426230/data"
{'name': '6cef9864-730b-4d01-be6c-327e08c4f5d8', 'dir': '/home/yellowtent/appsdata/6cef9864-730b-4d01-be6c-327e08cf90-8d4e-e58726a601c7/data"
[2021-11-27 02:53:09] Systemd detected, trying to signal readiness.
1637981588)
[2021-11-27 02:53:14] du plugin: monitoring /home/yellowtent/appsdata/e485da60-4fbd-4b8f-9b4d-e9d09f426230/data
: 1637981593)
[2021-11-27 02:53:19] du plugin: monitoring /home/yellowtent/appsdata/4cbc3621-1b02-422d-90fe-3d483d251522/data
1637981599)
[2021-11-27 02:58:40] plugin_load: plugin "interface" successfully loaded.
d01-be6c-327e08c4f5d8/data"
[2021-11-27 02:59:01] plugin_load: plugin "aggregation" successfully loaded.
65f-9d8f-2d911df4f0ed/data"
[2021-11-27 02:59:01]   size of /home/yellowtent/appsdata/f58e115c-abc1-4f3b-a40a-21b91f92c160/data is 1916928 (tim[2021-11-27 03:00:11] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/boxdata" --exclude "mailf3b-a40a-21b91f92c160/data"
b90-e4be79
f90-8d4e-e58726a601c7/data"
[2021-11-27 03:00:28] du plugin: monitoring /home/yellowtent/appsdata/b72cc559-faeb-4d2f-bf51-2a2da800c5d4/data
[2021-11-27 03:00:29] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/ae7fce0c-a5b4-4[2021-11-27 03:00:39] plugin_load: plugin "table" successfully loaded.
1637982039)
[2021-11-27 03:00:39]   size of /home/yellowtent/appsdata/fdb80dbd-191c-4d37-90c2-04c89050f543/data is 630784 (timea96bbe6ebc03bee32f0e85fab2bfe5a3c036/cpuacct.stat": No such file or directory.
: 1637982052)
'/home/yellowtent/appsdata/0d4c4ffa-e715-465f-9d8f-2d911df4f0ed/data', 'exclude': ''}, {'name': '10d75fb4-2ba7-43a[2021-11-27 03:00:57] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/aa40b16d-62fd-4[2021-11-27 03:01:26] plugin_load: plugin "python" successfully loaded.
[2021-11-27 03:01:26] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/6deb1be7-d241-4[2021-11-27 03:01:26] plugin_load: plugin "load" successfully loaded.
[2021-11-27 03:01:27] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/10d75fb4-2ba7-4f3b-a40a-21b91f92c160/data"
377747474e5961b371e8a7e95a0b3632725/memory.stat": No such file or directory.
b90-e4be79
[2021-11-27 03:04:16] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/e485da60-4fbd-4', 'exclude': ''}, {'name': 'boxdata', 'dir': '/home/yellowtent/boxdata', 'exclude': 'mail'}, {'name': 'platformdat: 1637982266)
[2021-11-27 03:04:40] du plugin: monitoring /home/yellowtent/appsdata/6cef9864-730b-4d01-be6c-327e08c4f5d8/data
feb-bb90-e4be79c69fcb/data"
[2021-11-27 03:04:45] plugin_load: plugin "write_graphite" successfully loaded.
e: 1637982285)
[2021-11-27 03:04:53] plugin_load: plugin "memory" successfully loaded.
22d-90fe-3d483d251522/data"
[2021-11-27 03:04:59] plugin_load: plugin "memory" successfully loaded.
e: 1637982299)
[2021-11-27 03:05:08] du plugin: monitoring /home/yellowtent/boxdata
[2021-11-27 03:05:08]   size of /home/yellowtent/appsdata/a27789a4-f7e2-4617-8d11-687b7e24e501/data is 40960 (time:[2021-11-27 03:05:16] Systemd detected, trying to signal readiness.
[2021-11-27 03:05:16]   size of /home/yellowtent/appsdata/bb96e514-a988-4f90-8d4e-e58726a601c7/data is 45056 (time:a32ceef599ebc7cf60c4b7bdacab69cd39f/memory.max_usage_in_bytes": No such file or directory.
[2021-11-27 03:05:34] plugin_load: plugin "memory" successfully loaded.
03c-a654-2e30bad1fba3/data"
[2021-11-27 03:05:52] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/boxdata/mail"
[2021-11-27 03:06:32] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/76ae0d1f17bcc22e21cf39a873c91637982398)
home/yellowtent/appsdata/bb96e514-a988-4f90-8d4e-e58726a601c7/data', 'exclude': ''}, {'name': 'cloudron-backup', 'd[2021-11-27 03:07:07] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/76ae0d1f17bcc22e21cf39a873c90c-a654-2e30bad1fba3', 'dir': '/home/yellowtent/appsdata/aa40b16d-62fd-403c-a654-2e30bad1fba3/data', 'exclude': ''},[2021-11-27 03:08:32] plugin_load: plugin "load" successfully loaded.
22d-90fe-3d483d251522/data"
[2021-11-27 03:09:59] collectd: Stopping 5 read threads.
f90-8d4e-e58726a601c7/data"
[2021-11-27 03:10:07]   size of /home/yellowtent/appsdata/aa40b16d-62fd-403c-a654-2e30bad1fba3/data is 835584 (time[2021-11-27 03:10:09]   size of /home/yellowtent/boxdata is 778240 (time: 1637982608)
[2021-11-27 03:10:16] Systemd detected, trying to signal readiness.
1637982616)
[2021-11-27 03:10:34]   size of /home/yellowtent/appsdata/6deb1be7-d241-4d47-afb9-e63f4a2c00cf/data is 8192 (time: 1fba3/data', 'exclude': ''}, {'name': 'ae7fce0c-a5b4-4618-9add-59384eef3d2a', 'dir': '/home/yellowtent/appsdata/ae7[2021-11-27 03:10:54] collectd: Stopping 5 read threads.
[2021-11-27 03:10:54] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/bb96e514-a988-4[2021-11-27 03:11:05]   size of /home/yellowtent/boxdata/mail is 1208320 (time: 1637982664)
[2021-11-27 03:12:05] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/57d5fc7d52ba29628e46fa72340e[2021-11-27 03:12:29]   size of /home/yellowtent/boxdata is 778240 (time: 1637982748)
[2021-11-27 03:12:49] du plugin: monitoring /home/yellowtent/appsdata/6deb1be7-d241-4d47-afb9-e63f4a2c00cf/data
e: 1637982769)
fce0c-a5b4-4618-9add-59384eef3d2a/data', 'exclude': ''}, {'name': 'cloudron-backup', 'dir': '/var/backups', 'exclud071317a7ec01ac986baeb7ca39997b4f2cd/memory.stat": No such file or directory.
322e2ecc68038ca7839e1085c9a5670641ff/cpuacct.stat": No such file or directory.
[2021-11-27 03:14:43] Systemd detected, trying to signal readiness.
[2021-11-27 03:15:04] plugin_load: plugin "memory" successfully loaded.
03c-a654-2e30bad1fba3/data"
[2021-11-27 03:16:04] du plugin: monitoring /home/yellowtent/appsdata/4cbc3621-1b02-422d-90fe-3d483d251522/data
[2021-11-27 03:16:11] collectd: Stopping 5 write threads.
[2021-11-27 03:16:11]   size of /var/backups is 1941504 (time: 1637982970)
e: 1637982988)
/dev/sdb', '/var/backups', '10255636', '38764', '9676200']]
[2021-11-27 03:17:00] collectd: Stopping 5 write threads.
1637983019)
fce0c-a5b4-4618-9add-59384eef3d2a/data', 'exclude': ''}, {'name': 'bb96e514-a988-4f90-8d4e-e58726a601c7', 'dir': '/[2021-11-27 03:17:15]   size of /var/backups is 1941504 (time: 1637983035)
[2021-11-27 03:18:29] plugin_load: plugin "cpu" successfully loaded.
[2021-11-27 03:18:29] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/platformdata"
[2021-11-27 03:19:01] plugin_load: plugin "swap" successfully loaded.
[2021-11-27 03:19:01]   size of /home/yellowtent/appsdata/4cbc3621-1b02-422d-90fe-3d483d251522/data is 2134016 (tim[2021-11-27 03:19:05] du plugin: monitoring /home/yellowtent/appsdata/ae7fce0c-a5b4-4618-9add-59384eef3d2a/data
[2021-11-27 03:19:05] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/bb96e514-a988-4[2021-11-27 03:19:32] du plugin: monitoring /home/yellowtent/appsdata/ae7fce0c-a5b4-4618-9add-59384eef3d2a/data
[2021-11-27 03:19:32] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/aa40b16d-62fd-4'/home/yellowtent/appsdata/4cbc3621-1b02-422d-90fe-3d483d251522/data', 'exclude': ''}, {'name': '6deb1be7-d241-4d4[2021-11-27 03:19:45] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/f58e115c-abc1-4', 'exclude': ''}, {'name': 'boxdata', 'dir': '/home/yellowtent/boxdata', 'exclude': 'mail'}, {'name': 'platformdat[2021-11-27 03:20:05]   size of /home/yellowtent/appsdata/aa40b16d-62fd-403c-a654-2e30bad1fba3/data is 835584 (time45801737aa558ab816690ec0d033215cc63/memory.max_usage_in_bytes": No such file or directory.
[2021-11-27 03:21:08] du plugin: monitoring /home/yellowtent/appsdata/a27789a4-f7e2-4617-8d11-687b7e24e501/data
617-8d11-687b7e24e501/data"
[2021-11-27 03:21:35] plugin_load: plugin "interface" successfully loaded.
a45801737aa558ab816690ec0d033215cc63/cpuacct.stat": No such file or directory.
f98421229124a116f5e6925429780ff8589/memory.max_usage_in_bytes": No such file or directory.
{'name': 'aa40b16d-62fd-403c-a654-2e30bad1fba3', 'dir': '/home/yellowtent/appsdata/aa40b16d-62fd-403c-a654-2e30bad[2021-11-27 03:22:07] computing size with command: timeout 1800 du -DsB1 "/var/backups"
c00cf/data', 'exclude': ''}, {'name': '93172ad5-f9c8-4feb-bb90-e4be79c69fcb', 'dir': '/home/yellowtent/appsdata/931[2021-12-01 07:44:47] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/4cbc3621-1b02-4[2021-12-11 07:29:56] plugin_load: plugin "aggregation" successfully loaded.
[2021-12-11 07:29:56] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/platformdata"
[2021-12-11 07:29:59] Exiting normally.
1639207799)
[2021-12-11 07:30:01] plugin_load: plugin "aggregation" successfully loaded.
[2021-12-11 07:30:02]   size of /home/yellowtent/boxdata is 1830912 (time: 1639207801)
[2021-12-11 07:30:02]   size of /home/yellowtent/appsdata/fdb80dbd-191c-4d37-90c2-04c89050f543/data is 1191936 (tim[2021-12-11 07:30:23] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/boxdata/mail"
[2021-12-11 07:30:23] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/appsdata/bb96e514-a988-4[2021-12-11 07:33:08] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/boxdata" --exclude "maild37-90c2-04c89050f543/data"
1639208157)
[2021-12-11 07:38:41] collectd: Stopping 5 write threads.
[2022-01-18 22:02:09]   size of /home/yellowtent/boxdata is 2842624 (time: 1642543329)
[2022-01-18 22:02:16] plugin_load: plugin "load" successfully loaded.
[2022-01-18 22:02:16] computing size with command: timeout 1800 du -DsB1 "/home/yellowtent/boxdata" --exclude "mail[2022-01-18 22:02:21] collectd: Stopping 5 read threads.
fe9-9c61-9b0129c7fac7/data"
[2022-01-18 22:02:23] plugin_load: plugin "python" successfully loaded.
ime: 1642543343)
[2022-01-18 22:02:28] du plugin: monitoring /home/yellowtent/appsdata/021b33dd-57a9-4e25-8187-9e2dfac0294c/data
65f-9d8f-2d911df4f0ed/data"
[2022-01-18 22:02:48] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/aec230922c7fe8d26900e718ca281[2022-01-18 22:02:54] custom df plugin initialized with [['/dev/sda1', '/', '78452316', '25895872', '49323180']]
1642543374)
82ffd2a5fd782f44c1556e2bbea96fba3e8/memory.stat": No such file or directory.
82ffd2a5fd782f44c1556e2bbea96fba3e8/memory.stat": No such file or directory.
[2022-01-18 22:06:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721ccf2036f4da0de06fa5f3863e5eee5cac05b8/cpuacct.stat": No such file or directory.
182ffd2a5fd782f44c1556e2bbea96fba3e8/cpuacct.stat": No such file or directory.
[2022-01-18 22:09:34] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/aec230922c7fe8d26900e718ca28f2036f4da0de06fa5f3863e5eee5cac05b8/memory.max_usage_in_bytes": No such file or directory.
82ffd2a5fd782f44c1556e2bbea96fba3e8/memory.max_usage_in_bytes": No such file or directory.
[2022-01-18 22:13:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721cf2036f4da0de06fa5f3863e5eee5cac05b8/memory.stat": No such file or directory.
82ffd2a5fd782f44c1556e2bbea96fba3e8/memory.stat": No such file or directory.
[2022-01-18 22:16:54] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721c182ffd2a5fd782f44c1556e2bbea96fba3e8/cpuacct.stat": No such file or directory.
[2022-01-18 22:19:14] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/f79696562bd1ddb444a34b837721[2022-01-18 22:20:14] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/aec230922c7fe8d26900e718ca28f2036f4da0de06fa5f3863e5eee5cac05b8/memory.max_usage_in_bytes": No such file or directory.
[2022-01-18 22:22:54] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721c[2022-01-18 22:23:54] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/aec230922c7fe8d26900e718ca281f2036f4da0de06fa5f3863e5eee5cac05b8/memory.stat": No such file or directory.
[2022-01-18 22:26:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/aec230922c7fe8d26900e718ca281cf2036f4da0de06fa5f3863e5eee5cac05b8/cpuacct.stat": No such file or directory.
[2022-01-18 22:27:34] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/aec230922c7fe8d26900e718ca281[2022-01-18 22:28:54] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721ccf2036f4da0de06fa5f3863e5eee5cac05b8/cpuacct.stat": No such file or directory.
[2022-01-18 22:30:54] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/aec230922c7fe8d26900e718ca28[2022-01-18 22:32:14] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/aec230922c7fe8d26900e718ca28[2022-01-18 22:33:34] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/f79696562bd1ddb444a34b83772182ffd2a5fd782f44c1556e2bbea96fba3e8/memory.max_usage_in_bytes": No such file or directory.
[2022-01-18 22:35:54] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721c[2022-01-18 22:37:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721c[2022-01-18 22:38:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/aec230922c7fe8d26900e718ca281182ffd2a5fd782f44c1556e2bbea96fba3e8/cpuacct.stat": No such file or directory.
[2022-01-18 22:40:34] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/f79696562bd1ddb444a34b837721[2022-01-18 22:41:34] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/aec230922c7fe8d26900e718ca28[2022-01-18 22:42:54] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/f79696562bd1ddb444a34b83772182ffd2a5fd782f44c1556e2bbea96fba3e8/memory.max_usage_in_bytes": No such file or directory.
[2022-01-18 22:45:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721c[2022-01-18 22:46:34] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721c[2022-01-18 22:47:34] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/aec230922c7fe8d26900e718ca281f2036f4da0de06fa5f3863e5eee5cac05b8/memory.stat": No such file or directory.
[2022-01-18 22:50:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721ccf2036f4da0de06fa5f3863e5eee5cac05b8/cpuacct.stat": No such file or directory.
182ffd2a5fd782f44c1556e2bbea96fba3e8/cpuacct.stat": No such file or directory.
[2022-01-18 22:53:34] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/f79696562bd1ddb444a34b837721f2036f4da0de06fa5f3863e5eee5cac05b8/memory.max_usage_in_bytes": No such file or directory.
82ffd2a5fd782f44c1556e2bbea96fba3e8/memory.max_usage_in_bytes": No such file or directory.
[2022-01-18 22:57:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721c82ffd2a5fd782f44c1556e2bbea96fba3e8/memory.stat": No such file or directory.
f2036f4da0de06fa5f3863e5eee5cac05b8/memory.stat": No such file or directory.
[2022-01-18 23:00:34] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/aec230922c7fe8d26900e718ca281182ffd2a5fd782f44c1556e2bbea96fba3e8/cpuacct.stat": No such file or directory.
cf2036f4da0de06fa5f3863e5eee5cac05b8/cpuacct.stat": No such file or directory.
[2022-01-18 23:04:14] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/f79696562bd1ddb444a34b83772182ffd2a5fd782f44c1556e2bbea96fba3e8/memory.max_usage_in_bytes": No such file or directory.
[2022-01-18 23:06:34] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721c[2022-01-18 23:07:34] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/aec230922c7fe8d26900e718ca28182ffd2a5fd782f44c1556e2bbea96fba3e8/memory.stat": No such file or directory.
[2022-01-18 23:10:14] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/f79696562bd1ddb444a34b837721ccf2036f4da0de06fa5f3863e5eee5cac05b8/cpuacct.stat": No such file or directory.
182ffd2a5fd782f44c1556e2bbea96fba3e8/cpuacct.stat": No such file or directory.
[2022-02-15 03:23:48] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/ee45dfb44e5014ea1e9cce7c93f7efc2587c3759e68c5811d37ff0deef087a6b/cpuacct.stat": No such file or directory.
[2022-02-15 03:24:08] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/cfeb423a79e3e47a8c490064bfe84e5bd42074b497d27be5dd5fe2c1f93ec061/memory.max_usage_in_bytes": No such file or directory.
[2022-02-15 03:24:08] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/520e5e35123f72ad7579b33ffcdafe5bd42074b497d27be5dd5fe2c1f93ec061/cpuacct.stat": No such file or directory.
[2022-02-15 03:24:08] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/ee45dfb44e5014ea1e9cce7c93f7efc2587c3759e68c5811d37ff0deef087a6b/memory.stat": No such file or directory.
[2022-02-15 03:24:08] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/ee45dfb44e5014ea1e9cce7c93f7efc2587c3759e68c5811d37ff0deef087a6b/memory.max_usage_in_bytes": No such file or directory.
[2022-02-15 03:24:08] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/ee45dfb44e5014ea1e9cce7c93f7efc2587c3759e68c5811d37ff0deef087a6b/cpuacct.stat": No such file or directory.
[2022-02-15 03:24:28] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/cfeb423a79e3e47a8c490064bfe84cd53a267f08d1388e1113659d56602136bc/memory.stat": No such file or directory.
[2022-02-15 03:24:28] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/cfeb423a79e3e47a8c490064bfe84cd53a267f08d1388e1113659d56602136bc/memory.max_usage_in_bytes": No such file or directory.
[2022-02-15 03:24:28] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/cfeb423a79e3e47a8c490064bfe84cd53a267f08d1388e1113659d56602136bc/cpuacct.stat": No such file or directory.
[2022-02-15 03:24:28] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/ee45dfb44e5014ea1e9cce7c93f7efc2587c3759e68c5811d37ff0deef087a6b/memory.stat": No such file or directory.
[2022-02-15 03:24:28] table plugin: Failed to open file "/sys/fs/cgroup/memory/docker/ee45dfb44e5014ea1e9cce7c93f7efc2587c3759e68c5811d37ff0deef087a6b/memory.max_usage_in_bytes": No such file or directory.
[2022-02-15 03:24:28] table plugin: Failed to open file "/sys/fs/cgroup/cpuacct/docker/ee45dfb44e5014ea1e9cce7c93f7efc2587c3759e68c5811d37ff0deef087a6b/cpuacct.stat": No such file or directory.
[2022-02-15 03:24:46] Exiting normally.
[2022-02-15 03:24:46] collectd: Stopping 5 read threads.
[2022-02-15 03:24:46] collectd: Stopping 5 write threads.

how could we accelerate the process ?????

nebulon

@JOduMonT to get your server going again faster, I guess it would help if you could enable remote SSH support for us and send a mail with your dashboard domain / raw IP to support@cloudron.io for investigation.

JOduMonT

@nebulon sorry, a lot is happening also on my side
thanks for your help,

as explained here
your ssh id has been added and the ssh support has been activated

if it is not possible to debug, or too long
I would like the database of each services since I already backuped the files (data).

nebulon

So to update this thread, also for @JOduMonT the issue was an error in the /etc/fstab file. This triggered the same issue about systemd's local-fs.target, which is quite crucial to the whole system as a dependency for example for collectd but also many more daemons.

In this case is was a nosudo option for a fileystem. That option does not exist however for this type of filesystem. The solution was to fixup /etc/fstab then run systemctl daemon-reload and systemctl restart local-fs.target following the command to initialize Cloudron again /home/yellowtent/box/setup.start.sh

JOduMonT

@nebulon said in my.cloudron.tld is not accessible:

nosudo

thank you for eyes and help
this is so weird and like you I don't know that option.

also; etckeeper as not being initiated on that server, so I'm unable to retrace when that change has been made

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

my.cloudron.tld is not accessible

the first time journalctl -xe

the second time

systemctl status collectd

no my drive is not full

Yes for a long time, statistic (probably populated by collected) was not working in the dashboard

how could we accelerate the process ?????